TransformerEncoder is not causal

[bigheary]

Add Link

https://pytorch.org/tutorials/beginner/transformer_tutorial.html

for language modeling， src_mask should be mask future words

Describe the bug

is there anything wrong?

Describe your environment

colab

cc @pytorch/team-text-core @Nayef211 @sekyondaMeta @svekars @carljparker @NicolasHug @kit1980 @subramen

[jorendorff]

Yes, this has to be a bug. I tried using the resulting model to do some basic predictions and it was never able to give a prediction for the next word that made any sense, despite reaching a perplexity of 2.9 on the validation and test data. Without a mask, every word except the last (the one that isn't present in the input) should be easy for the model to predict flawlessly.

[jorendorff]

You can play with it here (a copy of the notebook with a cell added at the end for making predictions)

[xanderex-sid]

/assigntome

[nvs-abhilash]

/assigntome

[xanderex-sid]

/assigntome

[xanderex-sid]

@bigheary Can you tell me more specifically, why you think src_mask should be mask future words (tgt_mask) ??

[jorendorff]

The text of the tutorial says:

Along with the input sequence, a square attention mask is required because the self-attention layers in nn.TransformerDecoder are only allowed to attend the earlier positions in the sequence. For the language modeling task, any tokens on the future positions should be masked.

[jorendorff]

The purpose of the mask is to prevent the model from seeing the token it's supposed to be predicting. Without a mask, the task is trivial and the model doesn't learn.

Other tutorials also include this mask. Andrej Karpathy's video, for example, lingers for 15 minutes on the subject, starting here, and then later comes back and discusses it again, saying "you have to mask ... so that nodes from the future never talk with the past, because they would give away the answer". (He points out that non-predictive applications don't need the mask, but the tutorial at issue here begins "This is a tutorial on training a model to predict the next word[...].")

[ahoblitz]

/assigntome

[ahoblitz]

Added a generate_square_subsequent_mask with 30 epochs to get the following based on some of the discussion here

Let me know if this follows the spirit of the conversation or not