0.5.1

What's Changed

• Bug fix: decay mask for bf16, bf32 by @fkodom in #25

Full Changelog: 0.5.0...0.5.1

0.5.0

Significant efficiency improvements to the chunkwise formulation, thanks to @leor-c 🎉

What's Changed

• a more efficient computation of the state in the chunkwise formulation by @leor-c in #22

Full Changelog: 0.4.2...0.5.0

0.4.2

What's Changed

• Slightly more efficient / cleaner implementation of the chunkwise relative pos. enc. by @leor-c in #21

New Contributors

• @leor-c made their first contribution in #21

Full Changelog: 0.4.1...0.4.2

0.4.1

What's Changed

• Performance Tuning by @fkodom in #20
• float32 -> 32-true by @fkodom in #16
• CPU benchmark support by @fkodom in #19

Full Changelog: 0.4.0...0.4.1

0.4.0

What's Changed

• Fixed issue where the dimensions of the Group norm seems to be incorrect by @draguve in #11

New Contributors

• @draguve made their first contribution in #11

Full Changelog: 0.3.1...0.4.0

0.3.1

What's Changed

• bug fix: change F.relu to F.silu by @Dongyeongkim in #7

New Contributors

• @Dongyeongkim made their first contribution in #7

Full Changelog: 0.3.0...0.3.1

0.3.0

More streamlined support for training

• example training script
• RetNet.forward is no longer just a wrapper for RetNet.forward_parallel. It accepts inputs, labels Tensors, and returns a loss value.

  class RetNet:
      ...
      def forward(self, inputs: Tensor, labels: Tensor) -> Tensor:
          pred = self.forward_parallel(inputs)
          criterion = nn.CrossEntropyLoss()
          return criterion(rearrange(pred, "b n c -> (b n) c"), labels.flatten())

• include example TorchData datapipe -- top 100 project gutenberg books
• example streaming text generation with trained RetNet

0.2.0

What's Changed

• Chunkwise Formulation by @fkodom in #4

Full Changelog: 0.1.3...0.2.0

0.1.3

Set default layer_norm_eps=1e-6, as updated in the official implementation:
microsoft/torchscale@2c29de0

0.1.2

Remove extra complex conjugation from the relative position embedding.
Reference: microsoft/torchscale#49