lmm.cpp

Inference of Large Multimodal Models in C/C++

Warning

This is still work in progress and not ready for anything.

Description

This repo implements LLaVA inference in C/C++ on top of clip.cpp and llama.cpp. Eventually, it will support inference of other large multimodal models, but LLaVA is chosen as a starting point.

Roadmap

• Get rid of text model and other unnecessary artifacts in clip.cpp
• Write the conversion script for LLaVA. Initially, it should be two-file format **one for the visual encoder and the other for LLaMA.
• Come up with a way to support single-file format the includes the CLIP backbone, the multimodal projector and LLaMA weights together.
• Support other models such as instructblip.