Divergence in pretraining BERT large #244
Comments
|
I found the same issue of divergence in #140. I trained this BERT large model using the training configuration recommended in RoBERTa paper, which is I wonder if my actual effective batch size is much smaller than 8192 since each model replica may update the gradients on its own and the gradients of the checkpoint is gathered in the end by the master GPU by averaging or in some other way. So my effective batch size could be just 16 and this causes the instability issue due to the model is very large (24 layers). Could you please help me explain how the gradients is gathered and how to get the actual batch size if the model is trained on multiple nodes and multiple GPUs? Thank you so much! |
|
Hi, The gradients are accumulated and averaged at the end. So if you train with 32 GPUs instead of 8, you multiply the effective batch size by 4. An equivalent way to do this if you don't have 4 nodes is to do: |
|
@glample In the case above, the effective batch size is 8192 yet training is not stable right at the second epoch, which is very surprising to me... |
|
Do you have the same issue with 12 or 16 layers? We also observed diverging issues with 24 layers, but usually they tend to appear at a later stage of training. One thing that helped was to use a L2 regularization on the logits of the self-attention (if you print these you may observe that they converge to huge values). Another was to use a larger epsilon in the layer norm (something around 1e-6, for instance). These things should help, but there is no guarantee that the model will never diverge. There is this paper that came out recently: https://openreview.net/pdf?id=SylO2yStDr that maybe you can try? Idea is very simple to implement: at training time, each transformer layer is dropped with (for instance) 10% probably. This apparently makes the training much more stable, and only takes a few lines to implement, but we have not tried it with this codebase yet. |
|
Sorry for the late reply, I couldn't find the time to advance on this earlier. @glample @aconneau Thanks again for your active support (I thanked you in our recent Flaubert submission, c.f. Acknowledgements section).
I did not train the 16 layer model, but I had no convergence issue at all with the 12 layer (i.e. base) model.
Thanks. I didn't notice that the default value was very small (1e-12). The one of fairseq (also used in RoBERTa) is 1e-5. Since I followed RoBERTa's configurations to train my models, I think using 1e-5 or 1e-6 would be better.
Thanks a lot for the reference! I have implemented LayerDrop, it looks like this in It seems to work. However, training speed decreased significantly:
This is very strange because theoretically adding LayerDrop should be faster than no LayerDrop. Could you please tell me if you have an idea on this? Thank you again for your help! |
formiel
changed the title
|
It seems that the problem lies in the backward passes or in the gradient gathering steps, because I try timing the forward passes and there was indeed a speedup. I added the following code to count the number of layers and the running time of each forward pass: # transformer layers
num_skips = 0
start = time.time()
for i in range(self.n_layers):
# LayerDrop
dropout_probability = random.uniform(0, 1)
if self.training and (dropout_probability < self.layerdrop):
num_skips += 1
continue
# self attention
...
tensor *= mask.unsqueeze(-1).to(tensor.dtype)
logger.info('{} layers took {}s'.format(self.n_layers - num_skips, time.time() - start))Training BERT base on one node with 2 GPUs, I obtained: Without LayerDrop:
With LayerDrop = 0.5:
The forward passes are clearly faster with LayerDrop, while the final speed is slower (36 sent/s vs 48 sent/s). And this is even worse in multi-node training. I also tried LayerDrop in another implementation (following this PyTorch official tutorial), and did not have this issue (LayerDrop is faster overall). I wonder if this is related to the use of Apex for distributed training... |
I encountered a very weird problem in which the validation perplexity and MLM accuracy were extremely bad after the first epoch of training. Specifically:
An iteration over all of my training data takes 2.5 epochs and one epoch would take 10 hours to finish. I trained this model on multiple nodes using Slurm. Due to the time limit of my server, I only submit jobs with the requested time around 10 hours so that 1 epoch is run and saved in each job. The next job will resume the checkpoint from its previous job and continue training. However, after the first job, I observed a very bad performance in validation perplexity and MLM accuracy as above.
I think it's very weird and I suspect there maybe some kind of changes in configuration between the nodes in my server that caused the nodes not synchronised and collected gradients properly like this...
However, I just want to ask you to understand how the model on the master GPU collects gradients signal from the model replicas on the remaining devices. Is this process handled by Slurm or PyTorch distributed package or both? Besides, is this normal if the master GPU save checkpoint while some of the remaining devices are still in evaluation mode and hasn't finished the scoring part on validation and test set yet?
Thank you so much in advance for your help!
The text was updated successfully, but these errors were encountered: