Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature Request] Addition of Kaiming He Normalization Option for Layer Initialization #51

Open
1 task done
Zach-Attach opened this issue Oct 29, 2024 · 0 comments · May be fixed by #53
Open
1 task done

[Feature Request] Addition of Kaiming He Normalization Option for Layer Initialization #51

Zach-Attach opened this issue Oct 29, 2024 · 0 comments · May be fixed by #53
Labels
enhancement New feature or request

Comments

@Zach-Attach
Copy link

🚀 Feature

Add a third option for weight_init as 'kaiming he' (see https://pytorch.org/docs/stable/nn.init.html and https://arxiv.org/pdf/1803.08494).

Motivation

I have been using this method of initialization for a while as it performs better than default or orthogonal. I thought it would make sens to add this to the public repo.sense

Pitch

Already wrote the code changes here.
5b08294

Alternatives

No response

Additional context

Tests do not currently exist for layer initialization, so I haven't added any for such a simple addition.

Checklist

  • I have checked that there is no similar issue in the repo
@Zach-Attach Zach-Attach added the enhancement New feature or request label Oct 29, 2024
@Zach-Attach Zach-Attach linked a pull request Nov 8, 2024 that will close this issue
15 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant