Llava 34b model is working with llamacpp?

[mirek190]

Llava 34b model is working with llamacpp?

[cmp-nct]

Overall if you use the draft PR #5267 you can convert and inference llava-1.6

But at lower quality input images, likely breaking the finetune a bit.
I tested the 13B already and it works very well, despite all those shortcommings

Here are 13B and Mistral 7B - as a demo
https://huggingface.co/cmp-nct/llava-1.6-gguf/tree/main

[cjpais]

ive got some basic quants for 34b generated from a hacked up script. seems to work pretty good.

surely the pr mentioned above will improve it further

hf link

not sure if this kosher to share here or not, but give a go

[svenstaro]

This issue should be closed as LLaVA v1.6 support is now merged.

[BarfingLemurs]

@mirek190 working!

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.