Adaptive Image Quality Assessment via Teaching Large Multimodal Model to Compare

Hanwei Zhu^1*, Haoning Wu^2*, Yixuan Li¹, Zicheng Zhang³, Baoliang Chen¹, Lingyu Zhu¹,

Yuming Fang⁴, Guangtao Zhai³, Weisi Lin², Shiqi Wang^1#

¹City University of Hong Kong ²Nanyang Technological University, ³Shanghai Jiao Tong University, ⁴Jiangxi University of Finance and Economics

^*Equal contribution. ^#Corresponding author.

Compare2Score (HF Model) | Arxiv | Homepage

Motivation

Training & Inference

Structure

Quicker Start with Hugging Face AutoModel

No need to install this GitHub repo.

import requests
import torch
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("q-future/Compare2Score", trust_remote_code=True, torch_dtype=torch.float16, device_map="auto")

from PIL import Image
image_path_url = "https://raw.githubusercontent.com/Q-Future/Q-Align/main/fig/singapore_flyer.jpg"
print("The quality score of this image is {}".format(model.score(image_path_url)) 

Installation

Evaluation:

git clone https://github.com/Q-Future/Compare2Score.git
cd Compare2Score
pip install -e .

Training:

pip install -e ".[train]"
pip install flash_attn --no-build-isolation

Visual Quality Scorer

from q_align import Compare2Scorer
from PIL import Image

scorer = Compare2Scorer()
image_path = "figs/i04_03_4.bmp"
print("The quality score of this image is {}.".format(scorer(image_path)))

Training & Evaluation

Get Datasets

Download all IQA datasets and training JSONs

import os, glob
from huggingface_hub import snapshot_download


snapshot_download("VQA-CityU/IQA_data", repo_type="dataset", local_dir="./playground/data", local_dir_use_symlinks=False)

gz_files = glob.glob("playground/data/*.zip")

for gz_file in gz_files:
    print(gz_file)
    os.system("unzip ./playground/data/".format(gz_file))

Evaluation

After preparing the datasets, you can evaluate pre-trained Compare2Score as follows:

python q_align/evaluate/IQA_dataset_eval.py --model-path q-future/Compare2Score --device cuda:0

Training from Scratch

sh scripts/train_bid_csiq_clive_kadid_koniq_live_compare.sh

Citation

@article{zhu2024adaptive,
  title={Adaptive Image Quality Assessment via Teaching Large Multimodal Model to Compare},
  author={Zhu, Hanwei and Wu, Haoning and Li, Yixuan and Zhang, Zicheng and Chen, Baoliang and Zhu, Lingyu and Fang, Yuming and Zhai, Guangtao and Lin, Weisi and Wang, Shiqi},
  journal={arXiv preprint arXiv:2405.19298},
  year={2024},
}