feat: openAI explicit value for maxToken and temperature

[panpan0000]

Because when k8sgpt talks with vLLM, the default MaxToken is 16, which is so small.
Given the most model supports 2048 token(like Llama1 ..etc), so put here for a safe value.

Closes # NA

📑 Description

✅ Checks

• My pull request adheres to the code style of this project
• My code requires changes to the documentation
• I have updated the documentation as required
• All the tests have passed

ℹ Additional Information

When k8sgpt talks with local(on-premise) LLM model, for example, model serving by vLLM(refer to blog https://k8sgpt.ai/blog/post-6/)
Problem was met when request from k8sgpt has NO default MaxToken value.

From the log of vLLM

you will see temperature=0.7 and max_tokens=16

such a small max_tokens will cause below issue: the answers from LLM was truncated.

So this fix try to make the value explicit , just like what was done in https://github.com/k8sgpt-ai/k8sgpt/blob/main/pkg/ai/cohere.go#L66

With this fix , we will see the new log from vLLM become better :

Regression Test

when swich back to online openAI, the result is still good

[AlexsJones]

• Please can you add those as constants at the top of file?
• PR name needs to be semantic e.g. "feat: OpenAI: explicit value for MaxToken and Temp "

Nice suggestion!

[panpan0000]

• Please can you add those as constants at the top of file?
• PR name needs to be semantic e.g. "feat: OpenAI: explicit value for MaxToken and Temp "

Nice suggestion!

done . Thanks !

[arbreezy]

Why do we need to specify Temperature value as a constant ?

Is this worth doing across all AI backends ?

[panpan0000]

Why do we need to specify Temperature value as a constant ?

Is this worth doing across all AI backends ?

Hi, @arbreezy :
(1) I learned from previous code for cohere backend : https://github.com/k8sgpt-ai/k8sgpt/blob/main/pkg/ai/cohere.go#L66
(2) no default temperature value from go-openAI library
(3) so what the values will be depends on openAI API server. openAI make it 0.7 by default. but if we build a on-premise LLM server , it may vary.
(4) to be honest, I think for k8sgpt is more serious case that normal chat case, so the 0.7 temperature should not be too high to lead hallucination, but I don't want to change the value for now..

[arbreezy]

@panpan0000 makes sense.
Two additional things to consider,
a) is it worth making temperature's value configurable to users?
b) i think it makes sense have temperature explicitly defined in all our ai backends e.g azure openai

[panpan0000]

@panpan0000 makes sense. Two additional things to consider, a) is it worth making temperature's value configurable to users? b) i think it makes sense have temperature explicitly defined in all our ai backends e.g azure openai

sure. @arbreezy . already updated with second commit : (1) add it for all backend (2) add it as a cmd flag.

can you please review again? Thanks a lot

[arbreezy]

@panpan0000 @AlexsJones ah I think we need to also set it in the server mode right?

[AlexsJones]

@panpan0000 @AlexsJones ah I think we need to also set it in the server mode right?

Feels like an oversight that needs fixing yes