Vision Foundation Model deployment in the edge cases - How adapters will be useful to deploy VFM in the edge device?

[solomonmanuelraj]

Hi team,
I want to fine tune a vision foundation model using custom dataset and deploy it in an edge device which has resource constraints ( around 150 MB memory) .
( VFM - OWL-ViT Size - 611 MB ( https://huggingface.co/google/owlvit-base-patch16)). when i do peft finetuning additional tiny checkpoint is getting created and, in the inference time final model size ( base model checkpoint + tiny checkpoint) is greater than the base model size. In that case final model is not possible to deploy in the edge device because of limited edge device. Is there any way only tiny model checkpoint alone can be deployed in the edge device for the prediction? any other way is available so that custom fine-tuned foundation model can be deployed in the edge device?

with thanks

[BenjaminBossan]

Yes, when you call merge_and_unload after loading the adapter into the base model, the adapter weights are merged into the base weights, resulting in a model of the same size as the original model (small numerical differences are expected though). Note that not all adapter types support this, but if you use LoRA, it works.

[solomonmanuelraj]

ok thanks.
Is it any way we can optimize the merged model so that we can reduce the model size and increase the inference speed in the edge device?

your comments will be useful . thanks

[younesbelkada]

hi @solomonmanuelraj
to further optimize the merged model one could use torch.scaled_dot_product_attention that we recently added in transformers. You can open an issue in https://github.com/huggingface/transformers/ and ask for SDPA support for that model

[solomonmanuelraj]

Hi @younesbelkada & @BenjaminBossan for owl-vit model (google/owlvit-base-patch32) is existing adapters are available? let me know the link?

thanks