Chen Li1, Yixiao Ge1, Dian Li2, and Ying Shan1.
1ARC Lab, Tencent PCG
2Foundation Technology Center, Tencent PCG
This paper is a review of all the works related to vision-language instruction tuning (VLIT). We will periodically update the recent public VLIT dataset and the VLIT data constructed by the pipeline in this paper.
- Release New Vision-Language Instruction Data (periodically) ...
- Update Public VLIT Datasets and Related Work (periodically) ...
- Release Construction Tools
- [2023.11.16] Release Instruction Data
- [2023.11.15] Paper Released (ArXiv)
Currently, the existing VLIT generation schemes can be divided into two categories, among which Annotation Adaption mainly relies on directly adjusting and rewriting the existing annotation data to adapt to the VLIT data template. Self-Instruct relies on the Large Language Model (LLM) to synthesize annotation data from more sources and reorganize it to generate VLIT data with more diversity and complexity (of course, it also brings more noise and hallucination).
VLIT Data
├─ General Instruction
│ ├─ Annotation Adaption
│ └─ Self-Instruct
├─ Specific Instruction
│ ├─ Object/Task-Specific
│ │ ├─ Region
│ │ ├─ Video
│ │ └─ Text
│ └─ Domain-Specific
│ ├─ Medicine
│ ├─ Document
│ └─ PointCloud
├─ Construction Tools
└─ Data Mixing
If there is any missing, please notify us by email(palchenli@tencent.com) and we will update as soon as possible.
In this paper, we propose a vision-language instruction construction pipeline and generate a corresponding dataset. Specifically, the generated instruction data is a multi-round question answering about a given image. Here are some examples of the generated instruction data:
| Image | Conversation |
|---|---|
 |
User: Can you describe the scene in the image? AI: Sure, the image captures a large grizzly bear wading through a moving river. The bear has a brown coat and appears to be quite large and furry. It's currently not holding anything in its mouth and it doesn't seem to have caught any fish from the river. User: Are grizzly bears typically found in such environments? AI: Yes, indeed. Grizzly bears are often found near rivers or streams. They are excellent swimmers and often go into the water in search of food. During salmon spawning seasons, they are known to frequent river areas to catch the fish. User: Is the bear possibly looking for food? AI: That's a strong possibility. While there are no fish present in the river in the image, grizzly bears are known to forage for various food sources in rivers, including fish and other aquatic organisms. However, without further context, it's hard to say definitively. User: What other characteristics does a grizzly bear have? AI: Grizzly bears, which are a subspecies of brown bear, are known for their size, strength, and excellent swimming ability. They have a distinct hump over their shoulders which is a muscle mass used for digging and clawing. Their fur color can vary from very light tan to dark brown, with the fur on their back often being darker than the rest of their body. Also, despite their large size, grizzlies are quite fast and can run at speeds of up to 30 miles per hour. |
 |
User: What is the main animal in the picture? AI: The main animal in the picture is a giraffe. User: What is the giraffe doing? AI: The giraffe is standing next to a tree and some rocks. It's not doing much, just hanging around. User: Is the giraffe standing on the rocks? AI: No, the giraffe is not standing on the rocks. It is standing next to them. User: Can you describe the setting in more detail? AI: The picture mainly focuses on the giraffe, a tree, and some rocks. The giraffe is standing right next to the tree. There are rocks scattered around them. User: Are there any elephants in the picture? AI: No, there are no elephants in this picture. Only a single giraffe is present. User: Are there any unique features of the giraffe? AI: As per the details provided in the image, the giraffe is standing and has a spotted pattern on its skin, which is a characteristic feature of giraffes. User: Is the giraffe sitting on the rocks? AI: No, the giraffe is not sitting on the rocks. It's standing by them. User: Is the giraffe eating the tree leaves? AI: There's no indication in the image that the giraffe is eating the leaves of the tree. It's simply standing beside it. User: Where do giraffes usually live? AI: Giraffes are native to Africa and they usually inhabit savannas, grasslands, and open woodlands. However, in this particular image, the exact location isn't specified. |
 |
User: Is there a car in the picture? AI: Yes, there is a car in the picture and it appears to be parked. User: What can you tell me about the car? AI: Based on the image, the car is parked, which suggests that it is not currently in use. However, I can't provide specifics about its make, model, or color without more visual information. User: Do you think the car is related to the boy on the skateboard? AI: It's hard to definitively state a connection between the car and the boy on the skateboard based on the image alone. The car may belong to someone who is watching the boy, or it may just be parked in the vicinity where the boy is skateboarding. User: Is the car moving? AI: No, the car isn't moving. It's parked. User: Could the car potentially pose a danger to the skateboarder? AI: Since the car is parked, it's unlikely to pose a direct threat to the skateboarder. However, any vehicle, parked or moving, in the vicinity of skateboarders could potentially become a hazard, especially if the skateboarder loses control. This emphasizes the importance of skateboarding in safe, designated areas. User: Is the car possibly there for a skateboarding event? AI: Based on the information available, I can't definitely say if the car is there for a skateboarding event. It could belong to a spectator or participant, or it could be unrelated to the skateboarding activity. More context would be needed to make a clear conclusion. |
There are three different types of instruction data. The data statistics and download links are as follows.
| Data Type | Baidu CLoud | Google Drive | Huggingface |
|---|---|---|---|
| COCO_2014_Images | url | url | url |
| Global | url | url | url |
| Negative | url | url | url |
| Region | url | url | url |
| Region_Images | url | url | url |
{
"image_source": "",
"construction_time": "",
"annotations": [
{
"img_ids": "",
"instruction_type": "",
"conversations": []
},
{
"img_ids": "",
"instruction_type": "",
"conversations": []
}
]
}If you found this repository useful, please consider citing:
@article{li2023visionlanguage,
title={Vision-Language Instruction Tuning: A Review and Analysis},
author={Chen Li and Yixiao Ge and Dian Li and Ying Shan},
year={2023},
eprint={2311.08172},
archivePrefix={arXiv},
primaryClass={cs.MM}
}
We would like to thank LLaVA, LAVIS and OpenFlamingo for their well-architcated multi-modal LLMs. Thanks to SEED-Bench for being an open source and convenient benchmark for evaluating MLLMs.