Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🔥 add LLaVA support and GPT vision API, Multiple requests for llama.cpp, return JSON types #1254

Merged
merged 13 commits into from
Nov 11, 2023

Conversation

mudler
Copy link
Owner

@mudler mudler commented Nov 6, 2023

Description

This PR fixes #1054 and updates the llama.cpp engine

Example: https://github.com/mudler/LocalAI/tree/master/examples/configurations/llava

Notes for Reviewers*

Notable changes:

  • The field for the Message content now is not implied to be a string anymore but is an interface dynamically inferenced. This is in order to support GPT vision with message content of image types
  • There is a breaking change: now the roles when being encoded do not imply a space between the role and the message. The space has to be passed by as part of the role if needed. E.g. before "USER:" as a role was automatically templated as "USER: " (note the space). This is not the case anymore. You need to adapt your configurations if necessary

Signed commits

  • Yes, I signed my commits.

Video:

llava

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler mudler changed the title wip: update llama.cpp, add LaVA support update llama.cpp, add LaVA support Nov 10, 2023
@mudler mudler changed the title update llama.cpp, add LaVA support 🔥 add LaVA support and GPT vision API, Multiple requests for llama.cpp, return JSON types Nov 10, 2023
@mudler mudler mentioned this pull request Nov 10, 2023
4 tasks
mudler and others added 5 commits November 10, 2023 19:18
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
This reverts commit f1a4731.
@mudler mudler force-pushed the llava_updates branch 2 times, most recently from dd4fbd2 to 982fa9a Compare November 11, 2023 09:53
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler mudler merged commit 0eae727 into master Nov 11, 2023
14 checks passed
@mudler mudler deleted the llava_updates branch November 11, 2023 12:15
@mudler mudler changed the title 🔥 add LaVA support and GPT vision API, Multiple requests for llama.cpp, return JSON types 🔥 add LLaVA support and GPT vision API, Multiple requests for llama.cpp, return JSON types Nov 13, 2023
@mudler mudler added enhancement New feature or request breaking-change labels Nov 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
breaking-change enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

feat: Add support for LLaVA (text+images) and miniGPT4
1 participant