[BUG] - Accelerating PyTorch Transformers by replacing nn.Transformer with Nested Tensors and torch.compile()

[AdrianOrenstein]

Add Link

https://pytorch.org/tutorials/intermediate/transformer_building_blocks.html

Describe the bug

Unfinished sentence in the tutorial:

"Thanks to this PR this is no longer the case. Instead, fully masked rows in scaled_dot_product_attention [missing text]. For cases where nn.MHA does not employ the “fast-path”, this will also apply."

Describe your environment

Brave Browser.

[mikaylagawarecki]

Thanks for catching this