An expensive bug GPT4. no matter what I do #125

twobob · 2023-10-06T03:40:02Z

I set this up on a new machine today.
I realised that in 2 games I had suddenly spent 7 dollars instantly. Pretty steep considering it was supposed to be using 3.5-turbo-16k

Checked over all the setting. Then parsed the entire source code for ANY reference to default models. anything. Changed every single reference. No matter what I set it is using the GPT4 API endpoint.

I have tried; the usual inline declarations; using the built in preferred template; just jamming it in as the hardcoded value to the method that checks it. So.. unless anyone has any other ideas. I'm all out.

7 bucks is no big shakes but I run systems in parallel and no doubt others batch too. Wouldn't want anyone to get an AMAZON AWS level surprise and not have reported. so. Enjoy. Reported.

EDIT: It's a linux box but I did install it under windows on there as well and test it (that test is probably the top 50c of the green bar ) It was identical so "not OS". and "yes, I also tried that."

twobob · 2023-10-06T03:47:31Z

Green 2 games. The blue bit, about 10 others....

afourney · 2023-10-06T04:15:10Z

Thanks for reporting and sorry about the spend. It sounds like you did due diligence on debugging, so I’m not sure how it was charged to GPT-4. We’ve not seen it call the wrong models before, so I am very curious to try and replicate this.

victordibia · 2023-10-06T04:46:12Z

To help reproduce, would be great to see

Your code
Emphasis on how you specify your OAI_CONFIG_LIST (the data structure used to define which LLM is used).

LittleLittleCloud · 2023-10-06T08:45:10Z

Maybe here? Looks like the default model is set to GPT-4

autogen/autogen/code_utils.py

Line 18 in 20d77a1

DEFAULT_MODEL = "gpt-4"

afourney · 2023-10-06T17:22:49Z

@LittleLittleCloud No I don't think that's it. That default is only accessed from the improve_code, improve_function, and implement calls -- and they aren't part of the main control path.

I am eyeing this:

autogen/autogen/oai/openai_utils.py

Line 331 in 20d77a1

# Ensure default models are always considered

   # Ensure default models are always considered
    default_models = ["gpt-4", "gpt-3.5-turbo"]

    for model in default_models:
        # Only assign default API key if the model is not present in the map.
        # If model is present but set to invalid/empty, do not overwrite.
        if model not in model_api_key_map:
            model_api_key_map[model] = "OPENAI_API_KEY"

@twobob How are you loading the config list? Are you using .env?

twobob · 2023-10-07T14:54:20Z

[
{
"model": "gpt-3.5-turbo-16k-0613",
"api_key": "ssdsdsdsdsdxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxA"
},
{
"model": "gpt-3.5-turbo-16k",
"api_key": "ssdsdsdsdsdxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxA"
},
{
"model": "gpt-3.5-turbo-0613",
"api_key": "ssdsdsdsdsdxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxA"
},
{
"model": "gpt-3.5-turbo",
"api_key": "ssdsdsdsdsdxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxA"
},
{
"model": "gpt-4", <--- -the single remaining reference - and its for azure so ¯\(ツ)/¯
"api_key": "",
"api_base": "",
"api_type": "azure",
"api_version": "2023-07-01-preview"
},
{
"model": "gpt-3.5-turbo",
"api_key": "",
"api_base": "",
"api_type": "azure",
"api_version": "2023-07-01-preview"
}
]

twobob · 2023-10-07T14:55:47Z

thanks for all the feedback. Yes there is zero reference to the word gpt-4 in my code now so
DEFAULT_MODEL = "gpt-4"
would have already been changed. I literally gutted it

twobob · 2023-10-07T15:00:24Z

@LittleLittleCloud No I don't think that's it. That default is only accessed from the improve_code, improve_function, and implement calls -- and they aren't part of the main control path.

I am eyeing this:

autogen/autogen/oai/openai_utils.py

Line 331 in 20d77a1

# Ensure default models are always considered
   # Ensure default models are always considered
    default_models = ["gpt-4", "gpt-3.5-turbo"]

    for model in default_models:
        # Only assign default API key if the model is not present in the map.
        # If model is present but set to invalid/empty, do not overwrite.
        if model not in model_api_key_map:
            model_api_key_map[model] = "OPENAI_API_KEY"
@twobob How are you loading the config list? Are you using .env?

me@mebuntu:~/repo/autogen$ echo $OPENAI_API_KEY
ssdsdsdsdsdxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxA

my env is indeed setup as well as the config file

twobob · 2023-10-07T15:01:16Z

To help reproduce, would be great to see

* Your code

* Emphasis on how you specify your OAI_CONFIG_LIST (the data structure used to define which LLM is used).

hi , which code. its the default code gutted to not contain any references to gpt_4 pretty much?

sonichi · 2023-10-07T15:02:52Z

Could you show the code that contains llm_config? If it contains "config_list" in it, could you also show the code which sets config_list?

twobob · 2023-10-07T15:07:16Z

I forgot to mention but I DID do this first on a windows box and wa using bash as my interpreter because "inline commands working for windows would be cool to do for everyone" I thought. That was when I noticed the bug.

Thought it was /maaaaybe/ the funky bash hack that was causing the error
Hence it being a vanilla ubuntu - like fresh for this job - and a very unhacked autogen other than the gutting.

Like this is the dream support box. literally nothing has happened on it other than this. since I have installed chatdev on the same box but meh. these tests were before that and nothing has changed in terms of the autogen anyway.

So there. the full sordid details. Honestly not sure where to go from there

It's not account since chatdev can use 3.5 just fine from the ubunutu install (why I installed chatdev after)

twobob · 2023-10-07T15:16:05Z

Could you show the code that contains llm_config? If it contains "config_list" in it, could you also show the code which sets config_list?

hi. it is the defaults straight from the readme or from a notebook, does it write logs anywhere? Ive mounted onto the windows drive from ubuntu so I can access anything...

twobob · 2023-10-07T15:34:36Z

i ran agentchat_groupchat_research.ipynb from the notebook folder in jupyter notebook

in the beginning I was running it

{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import autogen\n",
"\n",
"config_list_gpt4 = autogen.config_list_from_json(\n",
" "OAI_CONFIG_LIST",\n",
" filter_dict={\n",
" "model": ["gpt-4-32k", "gpt-4-32k-0314", "gpt-4-32k-v0314"],\n",
" },\n",
")\n",
"\n",
"config_list_gpt3 = autogen.config_list_from_json(\n",
" "OAI_CONFIG_LIST",\n",
" filter_dict={\n",
" "model": ["gpt-3.5-turbo-16k-0613", "gpt-3.5-turbo-16k", "gpt-3.5-turbo-0613"],\n",
" },\n",
")"
]
},

and only the engineer was actually given gpt 4 access

"gpt4_config = {\n",
" "seed": 42, # change the seed for different trials\n",
" "temperature": 0,\n",
" "config_list": config_list_gpt4,\n",
" "request_timeout": 120,\n",
"}\n",
"gpt3_config = {\n",
" "seed": 42, # change the seed for different trials\n",
" "temperature": 0,\n",
" "config_list": config_list_gpt3,\n",
" "request_timeout": 90,\n",
"}\n",
"user_proxy = autogen.UserProxyAgent(\n",
" name="Admin",\n",
" system_message="A human admin. Interact with the planner to discuss the plan. Plan execution needs to be approved by this admin.",\n",
" code_execution_config=False,\n",
")\n",
"engineer = autogen.AssistantAgent(\n",
" name="Engineer",\n",
" llm_config=gpt3_config,\n",

but as shown there by the end all actors had been reduced to gpt_3config

twobob · 2023-10-07T19:24:47Z

@LittleLittleCloud No I don't think that's it. That default is only accessed from the improve_code, improve_function, and implement calls -- and they aren't part of the main control path.

I am eyeing this:

autogen/autogen/oai/openai_utils.py

Line 331 in 20d77a1

# Ensure default models are always considered
   # Ensure default models are always considered
    default_models = ["gpt-4", "gpt-3.5-turbo"]

    for model in default_models:
        # Only assign default API key if the model is not present in the map.
        # If model is present but set to invalid/empty, do not overwrite.
        if model not in model_api_key_map:
            model_api_key_map[model] = "OPENAI_API_KEY"
@twobob How are you loading the config list? Are you using .env?

Hmmm. you know I am certain I did NOT remove that one (I errantly excluded that from the replace)...
Hmm.. Time for new test then

twobob · 2023-10-07T19:54:35Z

So. i reran it just to make sure it wasnt anything really stupid. BEFORE I made the above change

Blue bar is me making 30 games yesterday with chatDev, green bar is just setting up the requirements (basically 4 round of chat) in autogen on "supposed 3.5" before I stopped it again and waited for the OpenAi usage to catch up.

so. I wasn't going mad it really was doing that.

I made the above change. and only added 3.5 models in the list and ran it again/

Still no change.

I attached my actual notebook.
agentchat_groupchat_research_That_WAS_ACTUTALLY_USED.zip

twobob · 2023-10-07T19:55:34Z

maybe I should remove the API key from env? Any other ideas?

It is fair to say I am very sad this does not work hahahah

twobob · 2023-10-07T19:57:40Z

@LittleLittleCloud

twobob · 2023-10-07T20:02:43Z

To help reproduce, would be great to see

* Your code

* Emphasis on how you specify your OAI_CONFIG_LIST (the data structure used to define which LLM is used).

attached the zip

twobob · 2023-10-07T20:04:21Z

In the notebook I attached I had actually increased the engineer to 4 again. Hardly mattered I was already timing out on the gpt4 token/min before it popped

sonichi · 2023-10-07T21:00:39Z

The code looks good to me. Please make sure:

OAI_CONFIG_LIST is either an environment variable, or a file under the same path as the notebook. Please print the config_list_gpt3 dict to make sure it contains the gpt-3.5 models.
make sure config_list_gpt3 is contained in all the llm_config.
set max_consecutive_reply to 0 for UserProxyAgent to make a single-turn conversation. Check your bill.

twobob · 2023-10-07T22:04:33Z

it is a file in the root of the path. I didnt realise I could do it as an env... Did I miss a doc? maybe, I shall insert the print
Will do. In terms of the llm_configs as you can see in the code I believe that twas the case
Ah okay. If its all the same I will probably run up like 50c or so - just so I can FOR SURE see it. (I've set aside like $15 of left over credit to get this fixed..)

twobob · 2023-10-07T22:06:03Z

Ill do it after midnight. THEN I will for sure see it. it will be the only cost. lightbulb

twobob · 2023-10-07T22:13:36Z

Ooooookay. It seems like maybe we have a suspect.

config_list_from_json not actually populating ... hmm.. I DID NOT see that coming. good call.

So... I guess time to make that env... Better find those docs I guess

EDIT: https://microsoft.github.io/autogen/docs/FAQ/#set-your-api-endpoints I see

sonichi · 2023-10-07T22:20:08Z

The file is supposed be located at:

twobob · 2023-10-07T22:27:24Z

I dont run this on colab sir.
in addition I have setup the env version. in .bashrc and then started up a new terminal - run jupyter notebook FROM the new terminal . I also made a copy of the OAI_CONFIG_LIST and placed it in the notbooks folder. in addition to the root of the drive.

sonichi · 2023-10-07T22:29:42Z

ok. then it should be located next to the .ipynb file. otherwise you need to specify file_location. https://microsoft.github.io/autogen/docs/FAQ#set-your-api-endpoints

twobob · 2023-10-07T22:30:50Z

Yup ^^^ This ^^^ I was NOT in the notebook. folder. It was one level below. Ill restest now this is all a bit more "like it should be"

now when I print the config the keys are present.

running test. There are 0 gpt3 cost for the day and I noted the gpt4 usage. Here's hoping!

twobob · 2023-10-07T22:51:20Z

That sliver on blue IS A WELCOME SIGHT!

Might I suggest a tweak to the notebooks? something that actually PRINTS or even CHECKS the content of that array and just refuses to continue otherwise with a warning?

Thanks so much again

Engineer (to chat_manager):

As an AI language model, I don't have the capability to actively participate in the development process. However, I can provide guidance and suggestions based on the information provided. If you have any specific questions or need assistance with any particular step of the development process, please let me know and I'll be happy to help!

How unfortunate... Ah well

twobob · 2023-10-07T23:17:10Z

Engineer (to chat_manager):

As an AI language model, I don't have the capability to actively participate in the development process. However, I can provide guidance and suggestions based on the information provided. If you have any specific questions or need assistance with any particular step of the development process, please let me know and I'll be happy to help!

How unfortunate... Ah well

Yup they spent $1.70 and never wrote a single line of code. Just kept planning the project over and over despite intervention guiding them not to several times....

23:00
Local time: 8 Oct 2023, 00:00
gpt-3.5-turbo-16k-0613, 38 requests
169,170 prompt + 2,509 completion = 171,679 tokens
23:05
Local time: 8 Oct 2023, 00:05
gpt-3.5-turbo-16k-0613, 23 requests
159,774 prompt + 2,554 completion = 162,328 tokens

I guess Ill wait until this allows local LLMS

sonichi · 2023-10-09T00:45:41Z

or try some non-coding tasks, or try including a code example in the initial message.

afourney · 2023-10-09T22:39:47Z

Ok, here is my understanding of what's happening:

If the OAI_CONFIG_LIST is not found, then the config that gets returned is an empty list, (see:

autogen/autogen/oai/openai_utils.py

Line 246 in 4522900

return []

). Then

autogen/autogen/agentchat/conversable_agent.py

Line 111 in 4522900

self.llm_config = self.DEFAULT_CONFIG.copy()

populates the config with a model="gpt-4" default, and an empty config_list.

Then

autogen/autogen/oai/completion.py

Line 771 in 4522900

if config_list:

defaults to the classic OpeanAI chat completion behavior because config_list evaluates to false. The default is to read from the "model" attribute.

Finally OAI's openai library reads the missing key from environment variable.

twobob closed this as completed Oct 7, 2023

afourney mentioned this issue Oct 9, 2023

Output a warning if the OAI_CONFIG_LIST file is not found. #174

Merged

3 tasks

This was referenced Oct 9, 2023

GPT4 being used when not called in code #176

Closed

Warn if oai.Completion is provided with an empty config_list #178

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

An expensive bug GPT4. no matter what I do #125

An expensive bug GPT4. no matter what I do #125

twobob commented Oct 6, 2023 •

edited

Loading

twobob commented Oct 6, 2023

afourney commented Oct 6, 2023

victordibia commented Oct 6, 2023

LittleLittleCloud commented Oct 6, 2023

afourney commented Oct 6, 2023 •

edited

Loading

twobob commented Oct 7, 2023 •

edited

Loading

twobob commented Oct 7, 2023

twobob commented Oct 7, 2023

twobob commented Oct 7, 2023

sonichi commented Oct 7, 2023

twobob commented Oct 7, 2023

twobob commented Oct 7, 2023 •

edited

Loading

twobob commented Oct 7, 2023 •

edited

Loading

twobob commented Oct 7, 2023

twobob commented Oct 7, 2023

twobob commented Oct 7, 2023 •

edited

Loading

twobob commented Oct 7, 2023

twobob commented Oct 7, 2023

twobob commented Oct 7, 2023 •

edited

Loading

sonichi commented Oct 7, 2023

twobob commented Oct 7, 2023 •

edited

Loading

twobob commented Oct 7, 2023

twobob commented Oct 7, 2023 •

edited

Loading

sonichi commented Oct 7, 2023

twobob commented Oct 7, 2023 •

edited

Loading

sonichi commented Oct 7, 2023

twobob commented Oct 7, 2023 •

edited

Loading

twobob commented Oct 7, 2023 •

edited

Loading

twobob commented Oct 7, 2023 •

edited

Loading

sonichi commented Oct 9, 2023

afourney commented Oct 9, 2023 •

edited

Loading

An expensive bug GPT4. no matter what I do #125

An expensive bug GPT4. no matter what I do #125

Comments

twobob commented Oct 6, 2023 • edited Loading

twobob commented Oct 6, 2023

afourney commented Oct 6, 2023

victordibia commented Oct 6, 2023

LittleLittleCloud commented Oct 6, 2023

afourney commented Oct 6, 2023 • edited Loading

twobob commented Oct 7, 2023 • edited Loading

twobob commented Oct 7, 2023

twobob commented Oct 7, 2023

twobob commented Oct 7, 2023

sonichi commented Oct 7, 2023

twobob commented Oct 7, 2023

twobob commented Oct 7, 2023 • edited Loading

twobob commented Oct 7, 2023 • edited Loading

twobob commented Oct 7, 2023

twobob commented Oct 7, 2023

twobob commented Oct 7, 2023 • edited Loading

twobob commented Oct 7, 2023

twobob commented Oct 7, 2023

twobob commented Oct 7, 2023 • edited Loading

sonichi commented Oct 7, 2023

twobob commented Oct 7, 2023 • edited Loading

twobob commented Oct 7, 2023

twobob commented Oct 7, 2023 • edited Loading

sonichi commented Oct 7, 2023

twobob commented Oct 7, 2023 • edited Loading

sonichi commented Oct 7, 2023

twobob commented Oct 7, 2023 • edited Loading

twobob commented Oct 7, 2023 • edited Loading

twobob commented Oct 7, 2023 • edited Loading

sonichi commented Oct 9, 2023

afourney commented Oct 9, 2023 • edited Loading

twobob commented Oct 6, 2023 •

edited

Loading

afourney commented Oct 6, 2023 •

edited

Loading

twobob commented Oct 7, 2023 •

edited

Loading

twobob commented Oct 7, 2023 •

edited

Loading

twobob commented Oct 7, 2023 •

edited

Loading

twobob commented Oct 7, 2023 •

edited

Loading

twobob commented Oct 7, 2023 •

edited

Loading

twobob commented Oct 7, 2023 •

edited

Loading

twobob commented Oct 7, 2023 •

edited

Loading

twobob commented Oct 7, 2023 •

edited

Loading

twobob commented Oct 7, 2023 •

edited

Loading

twobob commented Oct 7, 2023 •

edited

Loading

twobob commented Oct 7, 2023 •

edited

Loading

afourney commented Oct 9, 2023 •

edited

Loading