Add GPT-4

[yifanmai]

Addresses #1456

[percyliang]

Description should mention that the context length is 32K tokens.

[yifanmai]

Added to description.

[percyliang]

Great! Yes, agreed that we should make the window size less clunky...

[pramitchoudhary]

Hey Guys, this is a great initiative. Thanks for starting it.
I was curious to try it out on the GPT4 models, so I was playing with the changes on this branch.

1. From my limited understanding, with these changes, one would need to specify an open API key using by adding it to prod_env/credentials.conf
2. echo 'entries: [{description: "mmlu:subject=philosophy,model=openai/gpt-4-32k-031", priority: 1}]' > run_specs.conf
3. On attempting helm-run --conf-paths run_specs.conf --local --suite v1 --max-eval-instances 10 throws error,

File "./helm/helm-venv/lib/python3.8/site-packages/helm/proxy/models.py", line 684, in get_model
    raise ValueError(f"No model with name: {model_name}")
ValueError: No model with name: openai/gpt-4-32k-031

Am i missing an obvious step?

[yifanmai]

You are probably misspelling the model name: it is openai/gpt-4-32k-0314, not openai/gpt-4-32k-031.

The other steps are correct; I will add them to the documentation soon.

[pramitchoudhary]

openai/gpt-4-32k-031

Sorry, above was a typo. Same error with openai/gpt-4-32k-0314 --> ValueError: No model with name: openai/gpt-4-32k-0314

Also, on, trying with other previous versions of GPT, getting ChatGPT error: local variable 'driver' referenced before assignment, probably from the ChatGPT client used? Current version: revChatGPT==0.1.1

I will add them to the documentation soon.

No problem, 🙏

[yifanmai]

@pramitchoudhary, we will be releasing v0.2.4 of the package next week; you should try installing the updated package (or running from main today) and see if that resolves your issues. You should open a new GitHub issue if this still doesn't work for you.