Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[feat] Rotary embeddings #32

Closed
blefaudeux opened this issue Oct 24, 2021 · 1 comment
Closed

[feat] Rotary embeddings #32

blefaudeux opened this issue Oct 24, 2021 · 1 comment
Labels
enhancement New feature or request ongoing

Comments

@blefaudeux
Copy link
Contributor

🚀 Feature

Implement rotary embeddings support

Motivation

Clear improvement over SOTA, improves some of the variants (Favor at least), possible good candidate for Triton to get some speed beyond pytorch

Alternatives

Not doing it

Additional context

Reference paper https://arxiv.org/abs/2104.09864
Related blog post https://blog.eleuther.ai/rotary-embeddings/
@blefaudeux blefaudeux added the enhancement New feature or request label Oct 24, 2021
@blefaudeux blefaudeux mentioned this issue Oct 25, 2021
Merged
10 tasks
@blefaudeux blefaudeux changed the title Rotary embeddings [feat] Rotary embeddings Oct 25, 2021
@blefaudeux blefaudeux mentioned this issue Oct 25, 2021
Merged
10 tasks
@blefaudeux
Copy link
Contributor Author

done

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request ongoing
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@blefaudeux