Cross_Modality_Relevance

The source code of ACL 2020 paper: "Cross-Modality Relevance for Reasoning on Language and Vision"

Author: Chen Zheng, Quan Guo, Parisa Kordjamshidi

ArXiv pre-print version link: https://arxiv.org/abs/2005.06035

data download link:

The data link is the same as the nlvr data page. We only need to download nlvr2. Please open the below page link and download it.

https://github.com/lil-lab/nlvr

image bounding box feature download link:

Since the training image bounding box feature file is too large, we only provide the valid and test image feature files in this time. After we find a larger storage space, we will consider to upload the training image bounding box feature file.

https://drive.google.com/file/d/1Ywpe-Vq5FKHCIPMMEToIDytlG_BMbxpt/view?usp=sharing

model checkpoint download link:

The checkpoint file of nlvr2 parameter weights:

https://drive.google.com/file/d/10SBGpAXQ-tV0qpEjlxatyWYbwND5u0Hd/view?usp=sharing

Tips

• The image bounding box feature files are very large, espeically the training bounding box file has around 40GB.
• Make sure both CPU and GPU memory are enough to load the data and model.

experiment environment:

• Machine: Lambda GPU machine.
• GPU: TITAN RTX.

Load conda environment:

• conda env create -f cmr.yaml
• source activate cmr

How to run the code?

• before runing the code, please make sure your config file is correct: configs/global_config.py

python run_cmr_nlvr2_test.py