KTNMT

Introduction

Source code for the ACL 2023 main conference long paper "Knowledge Transfer in Incremental Learning for Multilingual Neural Machine Translation" (Outstanding Paper Award)

In this work, we propose a knowledge transfer method that can efficiently adapt original MNMT models to diverse incremental language pairs.

---

Get Started

(Core) Data Preprocessing.

Standard BPE Procedure: following https://github.com/google/sentencepiece with 64k merged BPE tokens.

After obtaining the original vocabulary and the incremental vocabulary, you must get an incremental vocabulary of external models.

Model Training:

This system has been tested in the following environment.

Python version == 3.7

Pytorch version == 1.8.0

Fairseq version == 0.12.0 (pip install fairseq)

Note that it only influences the training procedure of the original and incremental model. You can choose your favorite deep learning library for model training.

---

Incremental Learning

We build the incremental learning procedure for Multilingual Neural Machine Translation as follows:

1. Get original multilingual translation models (or train a multilingual translation model by yourself).

2. Preprocessing incremental data

• Data Clean (optional, if needed)
• Get Vocabulary (optional, if needed, follow standard BPE procedure)

3. Incremental Training. We provide all runing scripts in the folder ''src''. Here is an example:

   echo "Task format: "
   export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
   export SEED=222
   export CHECKPOINT_PATH='' # save new model
   export CL_CHECKPOINT_PATH='' # old model
   export DATA_PATH='' # incremental data bin
   export USER_PATH='mkt/' # user dir
   export ARCH=transformer_adapter
   export TASK=translation_multi_adapter
   export ADAPTER_MODULE=serial # serial / parallel / parallel_gate / parallel_norm / parallel_dp
   export KRAIL_MODULE='' # krail / krail_ffn / krail_map / krail_map_last / krail_map_avg / krail_adaptive / krail_adaptive_final
   export LANG_DICT='misc/cl_adapter/lang_dicts_uk.txt' # new lang dict
   export KRAIL_MODEL='mPLM/418M_last_checkpoint.pt' # external model path
   export NEW_DICT_PATH='misc/cl_adapter/model_dict.m2m100.vex.txt' # combination model dict, we can get this vocab by scripts
   export KRAIL_DICT_PATH='misc/m2m100/model_dict.128k.txt' # external model dict
   export lang_pairs=en-uk
   export CLIP_NORM=0.0
   export OPTIMIZER=adam
   export ADAM_EPS=1e-9
   export LR=5e-4
   export LR_SCHEDULER=inverse_sqrt
   export WARMUP=4000
   export DROPOUT=0.2
   export ATT_DROPOUT=0.2
   export WEIGHT_DECAY=0.0001
   export CRITERION=label_smoothed_cross_entropy
   export LABEL_SMOOTHING=0.1
   export MAX_TOKENS=1024
   export UPDATE_FREQ=8
   export SAVE_INTERVAL_UPDATES=2500
   export KEEP_INTERVAL_UPDATES=1
   export MAX_UPDATE=500000
   
   fairseq-train $DATA_PATH \
   --user-dir $USER_PATH \
   --share-all-embeddings \
   --encoder-normalize-before --decoder-normalize-before \
   --encoder-embed-dim 1024 --encoder-ffn-embed-dim 8192 --encoder-attention-heads 16 \
   --decoder-embed-dim 1024 --decoder-ffn-embed-dim 8192 --decoder-attention-heads 16 \
   --encoder-layers 24 --decoder-layers 24 \
   --left-pad-source False --left-pad-target False \
   --arch $ARCH \
   --task $TASK \
   --sampling-method temperature \
   --sampling-temperature 5 \
   --lang-tok-style multilingual \
   --lang-dict $LANG_DICT \
   --lang-pairs $lang_pairs \
   --encoder-langtok src \
   --decoder-langtok \
   --clip-norm $CLIP_NORM \
   --optimizer $OPTIMIZER \
   --adam-betas '(0.9, 0.98)' \
   --adam-eps $ADAM_EPS \
   --lr $LR \
   --lr-scheduler $LR_SCHEDULER \
   --warmup-updates $WARMUP \
   --dropout $DROPOUT \
   --attention-dropout $ATT_DROPOUT \
   --weight-decay $WEIGHT_DECAY \
   --criterion $CRITERION \
   --label-smoothing $LABEL_SMOOTHING \
   --max-tokens $MAX_TOKENS \
   --save-dir $CHECKPOINT_PATH/checkpoints/ \
   --update-freq $UPDATE_FREQ \
   --save-interval-updates $SAVE_INTERVAL_UPDATES \
   --keep-interval-updates $KEEP_INTERVAL_UPDATES \
   --max-update $MAX_UPDATE \
   --no-epoch-checkpoints \
   --seed $SEED --log-format simple --log-interval 300 \
   --patience 10 \
   --tensorboard-logdir $CHECKPOINT_PATH/logs/ \
   --no-progress-bar \
   --ddp-backend no_c10d \
   --finetune-from-model $CL_CHECKPOINT_PATH \
   --freeze-all \
   --adapter-inner-dims 4096 \
   --adapter-module $ADAPTER_MODULE \
   --krail-module $KRAIL_MODULE \
   --vocab-adapter \
   --base-embed-dim 32211 \
   --krail-model $KRAIL_MODEL \
   --new-dict-path $NEW_DICT_PATH \
   --krail-dict-path $KRAIL_DICT_PATH

---

Inference & Evaluation

Please refer to src/inference.sh & run_sh/evaluate.sh

---

Citation

@inproceedings{huang-etal-2023-knowledge,
    title = "Knowledge Transfer in Incremental Learning for Multilingual Neural Machine Translation",
    author = "Huang, Kaiyu  and
      Li, Peng  and
      Ma, Jin  and
      Yao, Ting  and
      Liu, Yang",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-long.852",
    doi = "10.18653/v1/2023.acl-long.852",
    pages = "15286--15304",
}