Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(multimodal): Video understanding #2318

Closed
mudler opened this issue May 13, 2024 · 0 comments · Fixed by #3729
Closed

feat(multimodal): Video understanding #2318

mudler opened this issue May 13, 2024 · 0 comments · Fixed by #3729
Labels
enhancement New feature or request roadmap up for grabs Tickets that no-one is currently working on

Comments

@mudler
Copy link
Owner

mudler commented May 13, 2024 •
edited
Loading

It should be possible now to expand the vision support to understand videos, there are projects like
https://github.com/Efficient-Large-Model/VILA
https://github.com/LLaVA-VL/LLaVA-NeXT
https://huggingface.co/Qwen/Qwen2-VL-2B-Instruct?s=09

which make this possible nowadays. Since OpenAI has announced GPT4o, makes sense start looking into open solutions that we can plug into the API with specific backends.

llama.cpp: ggerganov/llama.cpp#9165
vLLM: #3670
@mudler mudler added the enhancement New feature or request label May 13, 2024
@mudler mudler mentioned this issue May 13, 2024
Open
@mudler mudler added roadmap up for grabs Tickets that no-one is currently working on labels May 13, 2024
@mudler mudler mentioned this issue Sep 19, 2024
Merged
@mudler mudler mentioned this issue Sep 26, 2024
Closed
@mudler mudler mentioned this issue Oct 4, 2024
Merged
1 task
mudler added a commit that referenced this issue Oct 4, 2024
@mudler
feat(vllm): add support for video-to-text
f3f9d1d 
Closes: #2318

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler mudler mentioned this issue Oct 4, 2024
Merged
1 task
mudler added a commit that referenced this issue Oct 4, 2024
@mudler
feat(vllm): add support for image-to-text and video-to-text (#3729)
2553de0 
* feat(vllm): add support for image-to-text

Related to #3670

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(vllm): add support for video-to-text

Closes: #2318

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(vllm): support CPU installations

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(vllm): add bnb

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* chore: add docs reference

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Apply suggestions from code review

Signed-off-by: Ettore Di Giacinto <mudler@users.noreply.github.com>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Signed-off-by: Ettore Di Giacinto <mudler@users.noreply.github.com>
siddimore pushed a commit to siddimore/LocalAI that referenced this issue Oct 6, 2024
@mudler @siddimore
feat(vllm): add support for image-to-text and video-to-text (mudler#3729
5b19cee 
)

* feat(vllm): add support for image-to-text

Related to mudler#3670

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(vllm): add support for video-to-text

Closes: mudler#2318

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(vllm): support CPU installations

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(vllm): add bnb

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* chore: add docs reference

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Apply suggestions from code review

Signed-off-by: Ettore Di Giacinto <mudler@users.noreply.github.com>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Signed-off-by: Ettore Di Giacinto <mudler@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request roadmap up for grabs Tickets that no-one is currently working on
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat(vllm): add support for image-to-text and video-to-text mudler/LocalAI
1 participant
@mudler