Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Minor] Expose bias options for both MLP and FusedMLP, use same defaults #220

Merged
merged 2 commits into from
Mar 2, 2022

Conversation

blefaudeux
Copy link
Contributor

What does this PR do?

Looking into #219 exposed that MLP and FusedMLP were not using the same defaults (bias and no bias respectively), and that Timm defaulted to using a bias. This PR

  • exposes the bias flag
  • uses the same default for both MLP and FusedMLP
  • use default bias=True so that it's the same as Timm

Before submitting

  • Did you have fun?
    • Make sure you had fun coding 🙃
  • Did you read the contributor guideline?
  • Was this discussed/approved via a Github issue? (no need for typos, doc improvements)
    • N/A
  • Did you make sure to update the docs?
    • N/A
  • Did you write any new necessary tests?
    • N/A
  • Did you update the changelog? (if needed)
    • N/A

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Mar 1, 2022
@blefaudeux
Copy link
Contributor Author

cc @jramapuram, but this is not the reason for your issue, since you were seeing the same number of parameters

@blefaudeux blefaudeux linked an issue Mar 1, 2022 that may be closed by this pull request
@blefaudeux blefaudeux changed the title [Minor] expose the bias options for both MLP and FusedMLP, use the same defaults [Minor] Expose bias options for both MLP and FusedMLP, use same defaults Mar 1, 2022
@codecov-commenter
Copy link

codecov-commenter commented Mar 1, 2022

Codecov Report

Merging #220 (2926d46) into main (a65c243) will not change coverage.
The diff coverage is 100.00%.

Impacted file tree graph

@@           Coverage Diff           @@
##             main     #220   +/-   ##
=======================================
  Coverage   91.72%   91.72%           
=======================================
  Files          60       60           
  Lines        3214     3214           
=======================================
  Hits         2948     2948           
  Misses        266      266           
Flag Coverage Δ
Python 91.72% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
xformers/components/feedforward/fused_mlp.py 91.30% <ø> (ø)
xformers/components/feedforward/mlp.py 100.00% <ø> (ø)
xformers/triton/layer_norm.py 76.92% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a65c243...2926d46. Read the comment docs.

@blefaudeux blefaudeux requested a review from jieru-hu March 1, 2022 22:48
@blefaudeux
Copy link
Contributor Author

Guessing that this is small enough to land, not changing anything really deep, exposing existing params and aligning defaults

@blefaudeux blefaudeux merged commit d4c28fb into main Mar 2, 2022
@blefaudeux blefaudeux deleted the align_mlp_fused_mlp branch March 2, 2022 04:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

xformers ViT-B ImageNet MAE + Deepnorm training instability
3 participants