[fix] Deepnorm self-attention value proj init scaling missing

[blefaudeux]

What does this PR do?

Hotfix an init of the projection matrix for the value being skipped when using deepnorm and self attention. See bottom of #219 for context.

TODO:

• Init seperately Q, K, V weights when using a common buffer
• Add a unit test to catch faulty weight inits with deepnorm

Before submitting

• Did you have fun?
  • Make sure you had fun coding 🙃
• Did you read the contributor guideline?
• Was this discussed/approved via a Github issue? (no need for typos, doc improvements)
  • N/A
• Did you make sure to update the docs?
  • N/A
• Did you write any new necessary tests?
  • N/A
• Did you update the changelog? (if needed)
  • N/A

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

[blefaudeux]

cc @jramapuram, may not be enough to fix but sorry for this bug already, I should have added a test when penning this option

[blefaudeux]

@SeanNaren we discussed this at some point I think, but this flag completely destroys the speed, on a single GPU (so it's not because of communication). May be worth a warning/explanation ?

[blefaudeux]

Nothing to write home about on vit/cifar, but this is only 6 layers. (blue is pre-norm, orange is deepnorm)

[codecov-commenter]

Codecov Report

Merging #284 (c69b98e) into main (ac94252) will increase coverage by 0.02%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main     #284      +/-   ##
==========================================
+ Coverage   92.72%   92.74%   +0.02%     
==========================================
  Files          61       61              
  Lines        3407     3417      +10     
==========================================
+ Hits         3159     3169      +10     
  Misses        248      248              

Flag	Coverage Δ
Python	92.74% <100.00%> (+0.02%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
xformers/components/residual.py	97.10% <ø> (ø)
xformers/factory/model_factory.py	96.12% <100.00%> (+0.32%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ac94252...c69b98e. Read the comment docs.