Add support for Bedrock, Ollama, Palm, Claude-2, Cohere, Replicate Llama2, CodeLlama, Hugging Face (100+LLMs) - using LiteLLM

[ishaan-jaff]

This PR adds support for the above mentioned LLMs using LiteLLM https://github.com/BerriAI/litellm/
LiteLLM is a lightweight package to simplify LLM API calls - use any llm as a drop in replacement for gpt-3.5-turbo.

Example

from litellm import completion

## set ENV variables
os.environ["OPENAI_API_KEY"] = "openai key"
os.environ["COHERE_API_KEY"] = "cohere key"

messages = [{ "content": "Hello, how are you?","role": "user"}]

# openai call
response = completion(model="gpt-3.5-turbo", messages=messages)

# cohere call
response = completion(model="command-nightly", messages)

# anthropic call
response = completion(model="claude-instant-1", messages=messages)

[ishaan-jaff]

addressing:
#18
#12
#44 also allow you to track cost for llm requests)

[ishaan-jaff]

@cpacker can you take a look at this PR when possible ?

[angar4ik]

when I was trying to work this with LM Studio, it was complaining about missing function role

"[2023-10-21 23:34:06.027] [ERROR] Error: 'messages' array must only contain objects with a 'role' field that is either 'user', 'assistant', or 'system'."

Model: Mistral 7b

Mistral doesn't know function role

[theCuratorCM]

Voting this as one of the highest leverage PRs of all time.

[vivi]

Thank you for taking the time to propose the integration of the litellm package into our project. We recognize the effort and dedication that has gone into developing litellm, and we genuinely appreciate your interest in contributing to our platform.

We've decided that we won't be integrating litellm directly into our codebase for the following reasons:

1. Applicability to MemGPT: We are committed to preserving the flexibility of our current codebase and ensuring that our users are not locked into a specific solution. It is also misleading to users currently because the “100+LLMs” support does not directly apply to MemGPT. Almost all of these models—including the last two in your example—do not support function calling for the purposes of MemGPT.
2. Complexity: While litellm offers various features, we believe that our specific needs can be addressed in a more streamlined manner without the need for adding another package dependency. See [Feature Request] Support for local LLMs like Ollama #18 (comment). We are also rolling our own support for function calling on local LLMs (Add local LLM support (with function calling) #97).

We hope you understand our decision and wish you all the best with the continued development of litellm.

[krrishdholakia]

hey @vivi , thanks for the feedback:

• Re: function calling for unsupported models via litellm - For unsupported models, you can choose to add it to the prompt as part of the system message via litellm.add_function_to_prompt = True.

• Re: streamlined manner - we work extremely hard to make sure litellm is as light as possible. Is there anything specific bloat we can help address here?

[cpacker]

We believe our users are better off using OpenAI's officially supported openai python API with our function calling parsers if they want to use non-OpenAI LLMs (which have to be written regardless to get good MemGPT performance with non-OpenAI models). This is not even touching on the privacy implications of potentially unnecessarily routing every MemGPT user's personal traffic through a startup's servers. If they really want to use litellm for some reason, they can install it on their own and swap the imports.

Please stop spamming github repositories with pull requests, thank you.

[krrishdholakia]

wait @cpacker litellm isn't a proxy server. we let users spin up an openai-compatible server if they'd like.

It's just a python package for translating llm api calls. I agree with you, unnecessarily routing things through a proxy would be a bit weird.

Is there nothing we can do to help you translate llm api calls across providers?