Animatediff Proposal

[DN6]

What does this PR do?

Adds the AnimateDiff model to diffusers.

This PR:

1. Introduces a dedicated UNetMotionModel and motion blocks that allow using existing 2D UNet models with the Motion Modules introduced in AnimateDiff.

The approach taken by AnimateDiff is slightly different to a 3D UNet. No temporal convs are applied to the inputs and the motion module weights are saved separately from the 2D UNet weights. This is slightly similar to a ControlNet/Adapter approach so IMO it would be better to create a dedicated model and blocks to support this type of functionality, rather than adapt the existing 3D UNets.

2. Adds a MotionAdapter module that acts as a dedicated container for saving and loading motion module weights from the hub.

Proposed API

# Loading motion module into Pipeline
from diffusers import MotionAdapter, AnimateDiffPipeline

motion_adapter = MotionAdapter.from_pretrained("<path to saved motion modules>")
pipe = AnimateDiffPipeline.from_pretrained("runwayml/stable-diffusion-v1-5", motion_adapter=motion_adapter)

# Calling pipe.unet should return an UNetMotionModel object

# Create brand new UNetMotionModel with all random weights
unet = UNetMotionModel()

# Load from an existing 2D UNet and MotionAdapter
unet2D = UNet2DConditionModel.from_pretrained("...")
motion_adapter = MotionAdapter.from_pretrained("...")

unet_motion = UNetMotionModel.from_unet2d(unet2D, motion_adapter: Optional = None)

# Or load motion module after init
unet_motion.load_motion_modules(motion_adapter)

# Save only motion modules
unet_motion.save_motion_module(<path to save model>, push_to_hub=True)

# Save all weights to a single model repo (Including UNet weights) 
unet_motion.save_pretrained()

# Load fused models (Where the motion weights are saved along with the UNet weights in a single repo)
unet_motion = UNetMotionModel.from_pretrained("<path to model>") 

TODO:

1. Test forward pass of UNetMotionModel to verify outputs match the original model
2. Test pipeline outputs for the same prompts defined in the original model repo
3. Support loading Motion LoRAs
4. Add documentation
5. Add tests
6. Add support for training/fine tuning AnimateDiff models.

Fixes # (issue)

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline?
• Did you read our philosophy doc (important for complex PRs)?
• Was this discussed/approved via a Github issue or the forum? Please add a link to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[patrickvonplaten]

I've added some final comments. It would be great if you could go over them and then I think we can merge tomorrow!

[patrickvonplaten]

I still don't understand why we need this. If cross_attention_dim is None then why do we have to manually set encodre_hidden_states to None. This looks more like a hacky bug correction. Why do we pass encoder_hidden_states in the first place if we don't have cross attention?

[patrickvonplaten]

Great job!