Multi-Grained Multimodal Interaction Network for Entity Linking

This repository is the official implementation for the paper titled "Multi-Grained Multimodal Interaction Network for Entity Linking".

News

• [2023.09.17] Updating the method of accessing low-resource data and considerations for reproduce the results.
• [2023.09.16] The detailed values of the experimental results have been updated. Please refer here.

Usage

Step 1: Set up the environment

We recommend using Conda to manage virtual environments, and we use Python version 3.8.12.

conda create -n mimic python==3.8.12
conda activate mimic

Please install the specified versions of Python libraries according to the requirements.txt file.

Note that the versions of PyTorch, Transformers, and PyTorch Lightning may have a slight impact on the results. To fully reproduce the results of the paper, we recommend installing the specified versions.

Step 2: Download the data

You may download WikiMEL and RichpediaMEL from https://github.com/seukgcode/MELBench and WikiDiverse from https://github.com/wangxw5/wikiDiverse.

Or download our cleaned data WikiMEL, RichpediaMEL, WikiDiverse (Password: kdd2023).

Step 3: Modify the data path

Please modify the configuration files under the "config" directory (including the YAML files for all 3 datasets) and replace YOUR_PATH in the data field of each configuration file with the path to your corresponding dataset.

NOTE: Due to the uploaded training files of RichpediaMEL, mention images are stored in the folder mention_images. You need to modify the mention_img_folder in the richpediamel.yaml config file or rename the mention_images folder to mention_image. (Thank Zhiwei Hu for bringing up this issue)

Step 4: Start the training

Now you can execute bash run.sh <gpu_id> <dataset_name> to begin the training.

bash run.sh 0 wikimel       # for WikiMEL
bash run.sh 0 richpediamel  # for RichpediaMEL
bash run.sh 0 wikidiverse   # for WikiDiverse

Code Structure

The code is organized as follows:

├── codes
│   ├── main.py
│   ├── model
│   │   ├── lightning_mimic.py
│   │   └── modeling_mimic.py
│   └── utils
│       ├── dataset.py
│       └── functions.py
├── config
│   ├── richpediamel.yaml
│   ├── wikidiverse.yaml
│   └── wikimel.yaml
├── readme.md
├── requirements.txt
└── run.sh

Results

Main Result

Model	WikiMEL					RichpediaMEL	WikiDiverse
	H@1↑	H@3↑	H@5↑	MRR↑	MR↓	H@1↑	H@3↑	H@5↑	MRR↑	MR↓	H@1↑	H@3↑	H@5↑	MRR↑	MR↓
BLINK	74.66	86.63	90.57	81.72	51.48	58.47	81.51	88.09	71.39	178.57	57.14	78.04	85.32	69.15	332.03
BERT	74.82	86.79	90.47	81.78	51.23	59.55	81.12	87.16	71.67	278.08	55.77	75.73	83.11	67.38	373.96
RoBERTa	73.75	85.85	89.80	80.86	31.02	61.34	81.56	87.15	72.80	218.16	59.46	78.54	85.08	70.52	405.22
DZMNED	78.82	90.02	92.62	84.97	152.58	68.16	82.94	87.33	76.63	313.85	56.90	75.34	81.41	67.59	563.26
JMEL	64.65	79.99	84.34	73.39	285.14	48.82	66.77	73.99	60.06	470.90	37.38	54.23	61.00	48.19	996.63
VELML	76.62	88.75	91.96	83.42	102.72	67.71	84.57	89.17	77.19	332.85	54.56	74.43	81.15	66.13	463.25
GHMFC	76.55	88.40	92.01	83.36	54.75	72.92	86.85	90.60	80.76	214.64	60.27	79.40	84.74	70.99	628.87
CLIP	83.23	92.10	94.51	88.23	17.60	67.78	85.22	90.04	77.57	107.16	61.21	79.63	85.18	71.69	313.35
ViLT	72.64	84.51	87.86	79.46	220.76	45.85	62.96	69.80	56.63	675.93	34.39	51.07	57.83	45.22	2421.49
ALBEF	78.64	88.93	91.75	84.56	47.95	65.17	82.84	88.28	75.29	122.30	60.59	75.59	81.30	69.93	291.17
METER	72.46	84.41	88.17	79.49	111.90	63.96	82.24	87.08	74.15	376.42	53.14	70.93	77.59	63.71	944.48
MIMIC	87.98	95.07	96.37	91.82	11.02	81.02	91.77	94.38	86.95	55.11	63.51	81.04	86.43	73.44	227.08

Low-resource Setting Result

To access low-resource training data, please refer here.

10% RichpediaMEL

Model	H@1	H@3	H@5	H@10	MRR
DZMNED	22.57	34.95	41.33	50.48	31.79
JMEL	16.70	27.68	33.63	41.55	25.01
VELML	27.15	38.60	43.99	51.99	35.52
GHMFC	68.00	83.38	87.73	91.97	76.69
ViLT	11.73	18.59	22.07	27.32	17.05
METER	60.89	79.23	84.78	89.42	71.40
CLIP	62.66	79.14	85.06	90.68	72.51
ALBEF	63.19	79.31	84.25	89.42	72.51
MIMIC	64.49	82.03	87.59	92.45	74.62

---

20% RichpediaMEL

Model	H@1	H@3	H@5	H@10	MRR
DZMNED	36.38	52.25	58.28	67.46	47.01
JMEL	28.92	43.35	50.59	61.54	39.38
VELML	48.85	64.91	71.76	79.42	59.24
GHMFC	72.57	86.69	90.15	93.77	80.42
ViLT	30.24	42.39	48.40	55.73	38.81
METER	61.51	79.56	84.48	89.50	71.82
CLIP	64.32	79.59	85.54	90.96	73.72
ALBEF	64.21	79.47	85.32	89.92	73.02
MIMIC	75.60	88.63	91.72	94.67	82.73

---

10% WikiDiverse

Model	H@1	H@3	H@5	H@10	MRR
DZMNED	11.45	22.52	29.50	37.15	19.99
JMEL	19.97	32.19	37.58	44.37	28.26
VELML	30.51	46.20	52.36	59.62	40.70
ViLT	13.19	21.27	26.37	32.68	19.57
METER	40.42	61.31	70.26	78.78	53.53
CLIP	59.87	76.52	81.57	85.95	69.49
ALBEF	51.83	69.20	74.64	81.57	62.26
GHMFC	48.08	66.31	74.25	81.91	59.56
MIMIC	60.54	76.18	81.33	86.14	69.70

---

20% WikiDiverse

Model	H@1	H@3	H@5	H@10	MRR
DZMNED	28.73	47.35	56.69	63.96	40.97
JMEL	29.26	44.23	49.90	57.22	39.05
VELML	43.65	61.36	67.66	74.88	54.76
ViLT	20.93	32.92	38.93	47.26	29.48
METER	40.23	61.16	70.45	80.56	53.46
CLIP	59.96	77.05	82.24	86.86	69.95
ALBEF	56.40	73.87	78.97	85.08	66.56
GHMFC	51.73	71.85	78.54	84.50	63.46
MIMIC	61.01	77.67	83.35	88.88	70.52

---

Citation

If you find this project useful in your research, please cite the following paper:

@inproceedings{luo2023multi,
    author = {Luo, Pengfei and Xu, Tong and Wu, Shiwei and Zhu, Chen and Xu, Linli and Chen, Enhong},
    title = {Multi-Grained Multimodal Interaction Network for Entity Linking},
    year = {2023},
    publisher = {Association for Computing Machinery},
    booktitle = {Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining},
    pages = {1583–1594},
}

Contact Information

If you have any questions, please contact pfluo@mail.ustc.edu.cn.