| 🏠Home Page | 🤗HuggingFace | 🏆Leaderboard | 📖Paper | 🖥️ Code |
- ✅ NaturalBench-Retrieval Dataset: the download link from huggingface homepage.
- ✅ We have integrated NaturalBench into lmms-eval and VLMEvalKit.
- 🎉 Sep. 26, 2024. NaturalBench was accepted by NeurIPS!
You can learn how to use and evaluate NaturalBench by reviewing the simple examples in example.py.
@inproceedings{naturalbench,
title={NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples},
author={Li, Baiqi and Lin, Zhiqiu and Peng, Wenxuan and Nyandwi, Jean de Dieu and Jiang, Daniel and Ma, Zixian and Khanuja, Simran and Krishna, Ranjay and Neubig, Graham and Ramanan, Deva},
booktitle={The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
year={2024},
url={https://openreview.net/forum?id=Dx88A9Zgnv}
}