[feat] Adding a conv MLP, following VAN

[blefaudeux]

What does this PR do?

One step towards #319, adding the MLP/Conv2d hybrid proposed by the VAN paper. Interestingly, testing this with a "Metaformer" (in true xformers fashion you can mix and match) on a tiny example does bring a measurable benefit.

Small (6M) Metaformer on Cifar10

Orange is the default (scaled dot product attention, not poolformer) + MLP White is the same but with the ConvMLP that this PR introduces

Before submitting

• Did you have fun?
  • Make sure you had fun coding 🙃
• Did you read the contributor guideline?
• Was this discussed/approved via a Github issue? (no need for typos, doc improvements)
  • N/A
• Did you make sure to update the docs?
  • N/A
• Did you write any new necessary tests?
  • N/A
• Did you update the changelog? (if needed)
  • N/A

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

[blefaudeux]

cc @fmassa, it's an interesting take I think

[codecov-commenter]

Codecov Report

Merging #321 (1b13a83) into main (5ccbcd9) will decrease coverage by 0.04%.
The diff coverage is 89.79%.

@@            Coverage Diff             @@
##             main     #321      +/-   ##
==========================================
- Coverage   93.75%   93.70%   -0.05%     
==========================================
  Files          68       69       +1     
  Lines        3840     3889      +49     
==========================================
+ Hits         3600     3644      +44     
- Misses        240      245       +5     

Flag	Coverage Δ
Python	93.70% <89.79%> (-0.05%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
xformers/factory/weight_init.py	90.82% <ø> (ø)
xformers/helpers/hierarchical_configs.py	100.00% <ø> (ø)
xformers/components/feedforward/conv_mlp.py	89.36% <89.36%> (ø)
xformers/components/feedforward/base.py	100.00% <100.00%> (ø)
xformers/factory/block_factory.py	97.03% <100.00%> (+0.02%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5ccbcd9...1b13a83. Read the comment docs.

[blefaudeux]

this does 2D convolutions, meaning that the layer needs to be able to go from [Batch x Context x Embedding] to [Batch x H x W x Embedding]. A solution which is not too intrusive is to force the use of sequences being squared numbers, meaning essentially that we only work with square pictures. It's pretty common in vision codebases, I think that another solution would be to keep track of the original H and W prior to flattening this dimension.

[blefaudeux]

Codecov Report

Merging #321 (1b13a83) into main (5ccbcd9) will decrease coverage by 0.04%.
The diff coverage is 89.79%.

@@            Coverage Diff             @@
##             main     #321      +/-   ##
==========================================
- Coverage   93.75%   93.70%   -0.05%     
==========================================
  Files          68       69       +1     
  Lines        3840     3889      +49     
==========================================
+ Hits         3600     3644      +44     
- Misses        240      245       +5     

Flag Coverage Δ
Python 93.70% <89.79%> (-0.05%) arrow_down

Flags with carried forward coverage won't be shown. Click here to find out more.
Impacted Files Coverage Δ
xformers/factory/weight_init.py 90.82% <ø> (ø)
xformers/helpers/hierarchical_configs.py 100.00% <ø> (ø)
xformers/components/feedforward/conv_mlp.py 89.36% <89.36%> (ø)
xformers/components/feedforward/base.py 100.00% <100.00%> (ø)
xformers/factory/block_factory.py 97.03% <100.00%> (+0.02%) arrow_up

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5ccbcd9...1b13a83. Read the comment docs.

should be fixed with the last update

[blefaudeux]

[dianaml0]

LGTM! Nice!