-
Notifications
You must be signed in to change notification settings - Fork 495
The BLEU decreased when train on Unsupervised NMT #39
Comments
I'm surprised by the perplexity of your language model:
482 is really high, I'm afraid the model does not encode Chinese sentences properly here. Can you pretrain for a longer time? Did you stop at epoch 11 because the model had stopped converging? Another thing to check is whether the word embeddings in the pretrained LM lookup table are somehow aligned. You can for instance print the nearest neighbors of Chinese words and see if they are close to their Mongolian translations. |
How can I get the nearest neighbors of Chinese words?@glample |
This notebook will show you how to reload a model with the associated dictionary: https://github.com/facebookresearch/XLM/blob/master/generate-embeddings.ipynb Once you have it, you can simply extract the embeddings with |
If the word embeddings in the pretrained LM lookup table are aligned not good,How can I improve it.@glample |
First you would need to fix the language model quality on the Chinese sentences, the perplexity should be much lower than this, so right now it is not surprising that it does not work. Maybe train with more data / check that the Chinese segmentation is correct and that there is a not a bug in your data preprocessing? Or simply train longer / with more GPUs, as 11 epochs is not much. |
I got it. Thanks! |
Hi,@glample
I pre-trained a language model and use it to train on Unsupervised NMT,but the BLEU becomes lower and lower. Is there something wrong?
Details:
The language model:
INFO - 03/14/19 16:57:00 - 23:43:04 - ============ End of epoch 11 ============
INFO - 03/14/19 16:57:06 - 23:43:10 - epoch -> 11.000000
INFO - 03/14/19 16:57:06 - 23:43:10 - valid_mn_mlm_ppl -> 12.698742
INFO - 03/14/19 16:57:06 - 23:43:10 - valid_mn_mlm_acc -> 61.901453
INFO - 03/14/19 16:57:06 - 23:43:10 - valid_zh_mlm_ppl -> 482.045657
INFO - 03/14/19 16:57:06 - 23:43:10 - valid_zh_mlm_acc -> 24.392448
INFO - 03/14/19 16:57:06 - 23:43:10 - valid_mlm_ppl -> 247.372200
INFO - 03/14/19 16:57:06 - 23:43:10 - valid_mlm_acc -> 43.146951
INFO - 03/14/19 16:57:06 - 23:43:10 - test_mn_mlm_ppl -> 34.794975
INFO - 03/14/19 16:57:06 - 23:43:10 - test_mn_mlm_acc -> 52.602524
INFO - 03/14/19 16:57:06 - 23:43:10 - test_zh_mlm_ppl -> 124.785448
INFO - 03/14/19 16:57:06 - 23:43:10 - test_zh_mlm_acc -> 34.501062
INFO - 03/14/19 16:57:06 - 23:43:10 - test_mlm_ppl -> 79.790211
INFO - 03/14/19 16:57:06 - 23:43:10 - test_mlm_acc -> 43.551793
Unsupervised NMT:
python3.6.2 train.py --exp_name unsupMT_mnzh --dump_path ./dumped/ --exp_id '190315' --reload_model './dumped/my_mnzh_mlm/190313/best-valid_mlm_ppl.pth,./dumped/my_mnzh_mlm/190313/best-valid_mlm_ppl.pth' --data_path ./data/processed/mn-zh/ --lgs 'mn-zh' --ae_steps 'mn,zh' --bt_steps 'mn-zh-mn,zh-mn-zh' --word_shuffle 3 --word_dropout 0.1 --word_blank 0.1 --lambda_ae '0:1,100000:0.1,300000:0' --encoder_only false --emb_dim 768 --n_layers 6 --n_heads 8 --dropout 0.1 --attention_dropout 0.1 --gelu_activation true --tokens_per_batch 1000 --batch_size 16 --max_batch_size 64 --bptt 256 --optimizer adam_inverse_sqrt,beta1=0.9,beta2=0.98,lr=0.0001,weight_decay=0 --epoch_size 300000 --eval_bleu true --stopping_criterion 'valid_mn-zh_mt_bleu,10' --validation_metrics 'valid_mn-zh_mt_bleu'
INFO - 03/15/19 12:54:23 - 3:17:34 - ============ End of epoch 0 ============
INFO - 03/15/19 12:56:06 - 3:19:16 - BLEU ./dumped/unsupMT_mnzh/190315/hypotheses/hyp0.mn-zh.valid.txt ./dumped/unsupMT_mnzh/190315/hypotheses/ref.mn-zh.valid.txt : 0.180000
INFO - 03/15/19 12:58:15 - 3:21:25 - BLEU ./dumped/unsupMT_mnzh/190315/hypotheses/hyp0.zh-mn.valid.txt ./dumped/unsupMT_mnzh/190315/hypotheses/ref.zh-mn.valid.txt : 2.740000
INFO - 03/15/19 12:58:36 - 3:21:47 - BLEU ./dumped/unsupMT_mnzh/190315/hypotheses/hyp0.mn-zh.test.txt ./dumped/unsupMT_mnzh/190315/hypotheses/ref.mn-zh.test.txt : 0.000000
INFO - 03/15/19 12:59:01 - 3:22:12 - BLEU ./dumped/unsupMT_mnzh/190315/hypotheses/hyp0.zh-mn.test.txt ./dumped/unsupMT_mnzh/190315/hypotheses/ref.zh-mn.test.txt : 2.160000
INFO - 03/15/19 12:59:01 - 3:22:12 - epoch -> 0.000000
INFO - 03/15/19 12:59:01 - 3:22:12 - valid_mn-zh_mt_ppl -> 6020.106288
INFO - 03/15/19 12:59:01 - 3:22:12 - valid_mn-zh_mt_acc -> 9.684522
INFO - 03/15/19 12:59:01 - 3:22:12 - valid_mn-zh_mt_bleu -> 0.180000
INFO - 03/15/19 12:59:01 - 3:22:12 - valid_zh-mn_mt_ppl -> 146.305114
INFO - 03/15/19 12:59:01 - 3:22:12 - valid_zh-mn_mt_acc -> 40.263721
INFO - 03/15/19 12:59:01 - 3:22:12 - valid_zh-mn_mt_bleu -> 2.740000
INFO - 03/15/19 12:59:01 - 3:22:12 - test_mn-zh_mt_ppl -> 6059.479785
INFO - 03/15/19 12:59:01 - 3:22:12 - test_mn-zh_mt_acc -> 12.168889
INFO - 03/15/19 12:59:01 - 3:22:12 - test_mn-zh_mt_bleu -> 0.000000
INFO - 03/15/19 12:59:01 - 3:22:12 - test_zh-mn_mt_ppl -> 488.040713
INFO - 03/15/19 12:59:01 - 3:22:12 - test_zh-mn_mt_acc -> 34.044409
INFO - 03/15/19 12:59:01 - 3:22:12 - test_zh-mn_mt_bleu -> 2.160000
INFO - 03/16/19 06:23:05 - 20:46:16 - ============ End of epoch 5 ============
INFO - 03/16/19 06:25:41 - 20:48:51 - BLEU ./dumped/unsupMT_mnzh/190315/hypotheses/hyp5.mn-zh.valid.txt ./dumped/unsupMT_mnzh/190315/hypotheses/ref.mn-zh.valid.txt : 0.000000
INFO - 03/16/19 06:27:31 - 20:50:41 - BLEU ./dumped/unsupMT_mnzh/190315/hypotheses/hyp5.zh-mn.valid.txt ./dumped/unsupMT_mnzh/190315/hypotheses/ref.zh-mn.valid.txt : 0.280000
INFO - 03/16/19 06:27:58 - 20:51:09 - BLEU ./dumped/unsupMT_mnzh/190315/hypotheses/hyp5.mn-zh.test.txt ./dumped/unsupMT_mnzh/190315/hypotheses/ref.mn-zh.test.txt : 0.000000
INFO - 03/16/19 06:28:22 - 20:51:33 - BLEU ./dumped/unsupMT_mnzh/190315/hypotheses/hyp5.zh-mn.test.txt ./dumped/unsupMT_mnzh/190315/hypotheses/ref.zh-mn.test.txt : 0.920000
INFO - 03/16/19 06:28:22 - 20:51:33 - epoch -> 5.000000
INFO - 03/16/19 06:28:22 - 20:51:33 - valid_mn-zh_mt_ppl -> 9263.390210
INFO - 03/16/19 06:28:22 - 20:51:33 - valid_mn-zh_mt_acc -> 7.963293
INFO - 03/16/19 06:28:22 - 20:51:33 - valid_mn-zh_mt_bleu -> 0.000000
INFO - 03/16/19 06:28:22 - 20:51:33 - valid_zh-mn_mt_ppl -> 195.211674
INFO - 03/16/19 06:28:22 - 20:51:33 - valid_zh-mn_mt_acc -> 36.910448
INFO - 03/16/19 06:28:22 - 20:51:33 - valid_zh-mn_mt_bleu -> 0.280000
INFO - 03/16/19 06:28:22 - 20:51:33 - test_mn-zh_mt_ppl -> 9938.071239
INFO - 03/16/19 06:28:22 - 20:51:33 - test_mn-zh_mt_acc -> 6.666667
INFO - 03/16/19 06:28:22 - 20:51:33 - test_mn-zh_mt_bleu -> 0.000000
INFO - 03/16/19 06:28:22 - 20:51:33 - test_zh-mn_mt_ppl -> 619.158340
INFO - 03/16/19 06:28:22 - 20:51:33 - test_zh-mn_mt_acc -> 32.541759
INFO - 03/16/19 06:28:22 - 20:51:33 - test_zh-mn_mt_bleu -> 0.920000
The text was updated successfully, but these errors were encountered: