Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add tensor_ops::prelu and nn::PReLU #389

Closed
leofidus opened this issue Jan 23, 2023 · 3 comments
Closed

Add tensor_ops::prelu and nn::PReLU #389

leofidus opened this issue Jan 23, 2023 · 3 comments

Comments

@leofidus
Copy link

Lately I've tried porting various computer vision networks to dfdx, and one function that seems quite popular in that field but isn't supported yet is PReLU. For now I just stay away from networks that use it, but it would be nice to have it available in the future.

https://pytorch.org/docs/stable/generated/torch.nn.PReLU.html#torch.nn.PReLU

ParametricReLU is bascially a LeakyReLU where the negative slope is a learned parameter, so there is going to be some overlap with the LeakyReLU implemenation (#287).

@TimerErTim
Copy link
Contributor

TimerErTim commented Feb 26, 2023

Can this really be implemented on a tensor level? LeakyReLU should be no problem but PReLU, which effectively has learnable parameters, is not really possible on purely tensors, is it? nn::PReLU should be easy to do on the other hand.

Edit: Have been thinking about this for some time. Maybe we could make tensor_ops::LeakyReLU use a simple f32 for the slope a whereas tensor_ops::PReLU uses a scalar or (to stay aligned with PyTorch) a vector. This way, a can be backpropagated through in the PReLU operation whereas in LeakyReLU it can't (which is basically it's only major difference).

@opfromthestart
Copy link
Contributor

This should be closed

@coreylowman
Copy link
Owner

Yep, thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants