Add BLIP

[younesbelkada]

What does this PR do?

BLIP is a model from salesforce, capable of performing Visual question answering, image captioning and image-text retrieval. This model has been also used in several Stable-diffusion finetuned variants, such as Pokemon stable diffusion or Naruto Stable diffusion to generate text descriptions from images in order to create text-image paired dataset.

Original repo: https://github.com/salesforce/BLIP

•  add integration tests
• Push weights
• document everything

Users would be able to use Blip for three main usecases:

1- Conditional Generation (Image captioning):

from PIL import Image
import requests
from transformers import BlipForConditionalGeneration, BlipProcessor

img_url = 'https://storage.googleapis.com/sfr-vision-language-research/BLIP/demo.jpg' 
image = Image.open(requests.get(img_url, stream=True).raw).convert('RGB')   

model = BlipForConditionalGeneration.from_pretrained("Salesfoce/blip-image-captioning-base")
processor = BlipProcessor.from_pretrained("Salesforce/blip-image-captioning-base")
text = "a picture of" # the prefix is optional

inputs = processor(image, text, return_tensors="pt")
output = model.generate(**inputs)

print(processor.decode(output[0], skip_special_tokens=True))
>>> a picture of a woman and a dog sitting in a beach

1- bis Conditional Generation (Image captioning with no prefix!):

from PIL import Image
import requests
from transformers import BlipForConditionalGeneration, BlipProcessor

img_url = 'https://storage.googleapis.com/sfr-vision-language-research/BLIP/demo.jpg' 
image = Image.open(requests.get(img_url, stream=True).raw).convert('RGB')   

model = BlipForConditionalGeneration.from_pretrained("Salesfoce/blip-image-captioning-base")
processor = BlipProcessor.from_pretrained("Salesforce/blip-image-captioning-base")

inputs = processor(image, return_tensors="pt")
output = model.generate(**inputs)

print(processor.decode(output[0], skip_special_tokens=True))
>>> an image of a woman and a dog sitting in a beach

2- Visual question answering

from PIL import Image
import requests
from transformers import BlipForQuestionAnswering, BlipProcessor

img_url = 'https://storage.googleapis.com/sfr-vision-language-research/BLIP/demo.jpg' 
image = Image.open(requests.get(img_url, stream=True).raw).convert('RGB')   

model = BlipForVisualQuestionAnswering.from_pretrained("Salesforce/blip-vqa-base")
processor = BlipProcessor.from_pretrained("Salesforce/blip-vqa-base")

question = ["How many dogs are in this image?"]
inputs = processor(image, text, return_tensors="pt")

output = model.generate(**inputs)
print(processor.decode(output[0], skip_special_tokens=True))
>>> 1 

3- Image text retrieval (score matching)

import torch
from PIL import Image
import requests
from transformers import BlipForQuestionAnswering, BlipProcessor

img_url = 'https://storage.googleapis.com/sfr-vision-language-research/BLIP/demo.jpg' 
image = Image.open(requests.get(img_url, stream=True).raw).convert('RGB')   

model = BlipForImageTextRetrieval.from_pretrained("Salesforce/blip-vqa-base")
processor = BlipProcessor.from_pretrained("Salesforce/blip-vqa-base")

question = ["A picture of a woman with a dog sitting in a beach"]
inputs = processor(image, question, return_tensors="pt")

out_itm = model(**inputs, use_itm_head=True)
out = model(**inputs, use_itm_head=False)

print(out) # cosine similarity score
>>> 0.21
print(torch.nn.functional.softmax(out_itm[0], dim=1)[:, 1])
>>> 0.46

cc @NielsRogge

Fixes salesforce/LAVIS#64

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[younesbelkada]

The failing CI test seems to be related to #20790

[ivelin]

Awesome contribution! Thank you @younesbelkada.

Just noticed Salesforce released BLIP2. Not sure how much work it would be to port to huggingface.
https://github.com/salesforce/LAVIS/tree/main/projects/blip2