Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding dilation and dropout #4

Open
wants to merge 19 commits into
base: main
Choose a base branch
from
Open

Adding dilation and dropout #4

wants to merge 19 commits into from

Conversation

jmoso13
Copy link

@jmoso13 jmoso13 commented Jun 12, 2023

This PR adapts the residual units inside the UNet encoder and decoder blocks to follow more closely the encoder/decoder architecture described in both:

Specifically it adapts the internal residual units to follow an exponentially increasing dilation schedule by way of a dilation factor base (usually 3):

  • dilation=3^0 -> dilation=3^1 -> dilation=3^2

As well as adapting the second convolutional layer in the residual unit to use dilation=1 and kernel_size=1. This follows the residual units described in the papers above.

Screen Shot 2023-06-12 at 3 20 16 PM

To optionally remove dilation from the blocks, use dilation factor of 1:

  • dilation=1^0 -> dilation=1^1 -> dilation=1^2

Additionally I have added dropout in the residual units for experimentation after the activation. I tried to adhere as closely as possible to the code style you have laid out, let me know if there are stylistic or other tweaks you'd like to see. Hopefully this is a useful contribution! I have already been experimenting in adding dilation to https://github.com/archinetai/audio-diffusion-pytorch using this update and will push a PR in that repo as well for controlling dilation factor and dropout for UNet use in diffusion.

🙏

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant