-
Notifications
You must be signed in to change notification settings - Fork 495
Issue with MT model training from pretrained model (Vector size mismatch) #144
Comments
Hi, It seems that the binarized dataset you are using was created without the BPE codes. Did you download the provided BPE en-fr codes and preprocessed the dataset with them? Be careful: if you initially preprocessed the dataset without them, and downloaded them afterwards, the script will not do anything as it will already see a |
@glample Thanks! That helped. However, I am still facing GPU memory issues while starting training.
Do you reckon it will be possible to train the MT model using a couple of GTX 1060s(6 GB VRAM each)? |
I'm afraid 6 GB is not enough.. you can try, but I'm not sure that even with batch size 1 it will fit. Or you would need to use less layers / a smaller dimension than the pretrained models with 6 layers / dimension 1024. |
oh, I see. Then it looks like I will have to do the training on a cloud platform. May I ask what is the recommended amount of GPU RAM for training an MT model from the provided pre-trained model? |
16 GB is good. 32 GB is even better if you can (larger batches work better, and you can fit larger batches with 32 GB). |
@glample On a related note, is it possible to run I checked out the code but couldn't find any way to start training with the CPU. |
Hi, I never tried, but I expect that the model would be too slow to run on CPU. I don't think you can train anything on CPU in a reasonable amount of time. |
I'm stuck with this mistake, help me get through it. I am running the Unsupervised machine translation from a pretrained cross lingual langauge model in google colab. The language model is succesfully trained, while the MT training breaks with the following error: Illegal division by zero at /content/XLM/src/evaluation/multi-bleu.perl line 154, line 10. By looking at the hypothesis files (hyp0.?-?.valid.txt, hyp0.?-?.test.txt ... in dumped/unsupMT?-?/???????/hypotheses) which are supposed to contain the translations produced by the mt at that time, I noticed that they are all empty, while the reference files (ref.?-?.valid.txt, ref.?-?.test.txt) which are supposed to contain the target translations, are not empty. This is consistent with the error because lines 153-154-155 of the multi-bleu.perl file contain these : if ($length_translation<$length_reference) { So the question is why the hypothesis files are empty. |
I tried to train an unsupervised MT model using the script below. The script was obtained from the repo readme.
I created the binarised en-fr dataset also by following the documentation.
However, I am getting an error as follows:-
Any help with the same would be much appreciated. Thanks
The text was updated successfully, but these errors were encountered: