Rebase attention off triton v2

[colehawkins]

What does this PR do?

Addresses bf16 support in #231 by basing off Triton v2

Before submitting

• [ X] Did you have fun?
  • Make sure you had fun coding 🙃
• Did you read the contributor guideline?
• [ X] Was this discussed/approved via a Github issue? (no need for typos, doc improvements)
  • N/A
• Did you make sure to update the docs?
  • N/A
• [X ] Did you write any new necessary tests?
  • N/A
• Did you update the changelog? (if needed)
  • N/A

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

[facebook-github-bot]

Hi @colehawkins!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@fb.com. Thanks!

[colehawkins]

I'm aware there's now a triton 2 dev branch, so I can just work off that branch.

One now-resolved delay was adding me (AWS employee) to the CCLA. Am I supposed to sign the CLA as well?

[blefaudeux]

I'm aware there's now a triton 2 dev branch, so I can just work off that branch.

One now-resolved delay was adding me (AWS employee) to the CCLA. Am I supposed to sign the CLA as well?

I'm not sure about how the CLA works, I would just follow the guide (never done that myself, as a Meta employee..). Sorry about that ! Slowly getting there with the triton2 branch, I can only work on that from time to time unfortunately, but it seems that layernorm is the only remaining piece of the puzzle

[blefaudeux]

looking at the errors @colehawkins, there's a lint issue which is easy to fix, but the rest is a little more fundamental, basically /experimental does not know about the rest of the repo (like, it does not know that xformers exist), and the unit tests are done accordingly (this was for instance because at the time it depended on triton2 while the rest was on triton stable, so they could not coexist anyway). I think it may be best to finish the move to triton2 and you would get bf16 for "free"

[blefaudeux]

you should be good with #272, let me know if that works for you @colehawkins ?

[colehawkins]

@blefaudeux thanks!