feat(llm): Add openailike llm mode #1447
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This mode behaves the same as the openai mode, except that it allows setting custom models not supported by OpenAI. It can be used with any tool that serves models from an OpenAI compatible API. Implements zylon-ai#1424
|
Published docs preview URL: https://privategpt-preview-339c6258-82f1-423a-a8ad-229fcd11ffdb.docs.buildwithfern.com |
imartinez
previously approved these changes
Dec 22, 2023
Add the "required" fields. The fields aren't really required, but mypy throws an error for fields that don't have default values set.
|
Published docs preview URL: https://privategpt-preview-98fba892-baa6-45d4-b54c-888eaed655af.docs.buildwithfern.com |
|
Pushed a fix for the linting errors. I ran |
imartinez
approved these changes
Dec 26, 2023
simonbermudez
pushed a commit
to simonbermudez/saimon
that referenced
this pull request
Feb 24, 2024
This mode behaves the same as the openai mode, except that it allows setting custom models not supported by OpenAI. It can be used with any tool that serves models from an OpenAI compatible API. Implements zylon-ai#1424
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This mode behaves the same as the openai mode, except that it allows setting custom models not supported by OpenAI. It can be used with any tool that serves models from an OpenAI compatible API.
Note that the mentioned libraries, LocalAI and vLLM, also have their own providers in LlamaIndex. This is a more generic solution that should support those and many more libraries.
One fun implication that I'm testing is the possibility of running PrivateGPT on top of another instance of PrivateGPT. The top-level instance manages embeddings, and the bottom-level instance runs an LLM. With a little additional logic, it could be possible to route requests from the parent instance to multiple child instances based on model id in the request, effectively letting a single PrivateGPT instance serve multiple models while benefiting from a single document store.
Implements #1424