Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Ollama, Palm, Claude-2, Cohere, Replicate, Llama2 CodeLlama (100+LLMs) [LiteLLM] #160

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Add support for Ollama, Palm, Claude-2, Cohere, Replicate, Llama2 CodeLlama (100+LLMs) [LiteLLM] #160

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
5c05c2a
v0 litellm
ishaan-jaff Nov 9, 2023
85ced61
litellm add v
ishaan-jaff Nov 9, 2023
2d74323
Update bigcode_eval/tasks/humanevalpack_openai.py
ishaan-jaff Nov 15, 2023
5188189
Update bigcode_eval/tasks/humanevalpack_openai.py
ishaan-jaff Nov 15, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions bigcode_eval/tasks/humanevalpack_openai.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ def get_prompt_base(doc, language="python"):

import os
import openai
import litellm
import jsonlines
import termcolor

Expand Down Expand Up @@ -170,7 +171,8 @@ def __call__(self, prompt: str, n: int) -> str:
]
while True:
try:
response = openai.ChatCompletion.create(
# Equivalent to response = openai.ChatCompletion.create(
response = litellm.completion(
ishaan-jaff marked this conversation as resolved.
Show resolved Hide resolved
model=self._model,
messages=messages,
temperature=0.2,
Expand Down Expand Up @@ -199,8 +201,11 @@ def __call__(self, prompt: str, n: int) -> str:
with jsonlines.open(f"completions_{LANGUAGE}_humanevalexplaindescribe.jsonl", "r") as f:
descriptions = [line["raw_generation"][0] for line in f]

openai.organization = os.getenv("OPENAI_ORGANIZATION")
openai.api_key = os.getenv("OPENAI_API_KEY")
# Equivalent to:
# openai.organization = os.getenv("OPENAI_ORGANIZATION")
# openai.api_key = os.getenv("OPENAI_API_KEY")
litellm.organization = os.getenv("OPENAI_ORGANIZATION")
litellm.api_key = os.getenv("OPENAI_API_KEY")
ishaan-jaff marked this conversation as resolved.
Show resolved Hide resolved

samples = [s for s in load_dataset("bigcode/humanevalpack", LANGUAGE)["test"]]

Expand Down
1 change: 1 addition & 0 deletions setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
"matplotlib==3.5.2",
"numpy==1.21.6",
"openai==0.23.0",
"litellm==0.13.2",
Copy link
Collaborator

@loubnabnl loubnabnl Nov 16, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's not the right place for the requirement as this is for DS1000 task that doesn't support external APIs, it seems we didn't have openai package properly added.

I would leave this list as it is and create a new variable
external_api_requirements=["litellm==0.13.2"] then add it to the extras_require dict here so people that want this evaluation install the package separately

"pandas==1.3.5",
"pandas-datareader==0.10.0",
"pathlib==1.0.1",
Expand Down