Add Latent Consistency Models Pipeline #5448

dg845 · 2023-10-19T05:26:50Z

What does this PR do?

This PR adds a pipeline and scheduler for Latent Consistency Models (LCM; paper, project page) based on the LCM community pipeline added in #5438, originally authored by @luosiallen.

Latent consistency models are an extension of consistency models which operate in the latent space of a VAE. Analogous to how diffusion models can be distilled to a consistency model for fast onestep or few-step generation, latent diffusion models such as Stable Diffusion can be distilled to a latent consistency model for fast onestep or few-step generation.

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline?
Did you read our philosophy doc (important for complex PRs)?
Was this discussed/approved via a Github issue or the forum? Please add a link to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@patrickvonplaten
@williamberman
@luosiallen

…sed on the community pipeline

src/diffusers/pipelines/latent_consistency_models/pipeline_latent_consistency_models.py

patrickvonplaten · 2023-10-19T09:39:55Z

src/diffusers/pipelines/latent_consistency_models/pipeline_latent_consistency_models.py

+    def __call__(
+        self,
+        prompt: Union[str, List[str]] = None,
+        height: Optional[int] = 768,


Is the default 512 or 768?

In the SimianLuo/LCM_Dreamshaper_v7 checkpoint the VAE sample_size is 768 and the community pipeline also uses 768:

diffusers/examples/community/latent_consistency_txt2img.py

Lines 202 to 203 in e516858

height: Optional[int] = 768,

width: Optional[int] = 768,

@luosiallen to confirm what the best default resolution is.

I think we can set the default as 768 has the best generation quality, most of the SD model like SD-V2.1 or Dreamshaper is trained on 768 resolution.

Actually, I think it might be more convenient to set the default height and width to None and let __call__ infer them.

patrickvonplaten · 2023-10-19T09:40:42Z

Very clean! Think we can merge this soon :-)

src/diffusers/pipelines/__init__.py

src/diffusers/pipelines/latent_consistency_models/pipeline_latent_consistency_models.py

luosiallen · 2023-10-19T12:09:26Z

src/diffusers/pipelines/latent_consistency_models/pipeline_latent_consistency_models.py

+            prompt,
+            device,
+            num_images_per_prompt,
+            False,  # Don't need to get negative prompts due to LCM guided distillation


To clarify: LCM currently supports only the unconditional prompt "" and does not support any other types of negative prompts.

I see. So if I understand correctly, the pipeline __call__ method should not take in negative_prompt or negative_prompt_embeds arguments, but we should still allow encode_prompt to prepare the default unconditional prompt "" if guidance_scale > 0.0, and perform CFG as normal in the rest of the pipeline?

[My understanding is that the diffusers convention is to use the Imagen CFG formulation $\tilde{\epsilon}_\theta(z_t, c) = \omega'\epsilon_\theta(z_t, c) + (1 - \omega')\epsilon_\theta(z_t, \varnothing)$ where $\omega' = \omega + 1$ and $\omega$ is the guidance scale used in the LCM paper, so I will probably change the guidance scale to follow the above definition.]

Well, i think currently setting the negative prompt as False is correct. All we need to do is to change the comment for better clarification: # LCM guided distillation currently only support unconditional prompt "", don't support other types of negative prompt.

The unconditional prompt is not used in LCM actually. Because it has already been distilled in LCM. So we only take input of prompt_embeds. Also LCM does not support negative prompts currently, maybe try to fix it in the next version of LCM?

I have updated the comment to provide a (hopefully) better description about negative prompt support.

[My understanding is that the diffusers convention is to use the Imagen CFG formulation
where
and
is the guidance scale used in the LCM paper, so I will probably change the guidance scale to follow the above definition.]

Indeed.

…gen) CFG formulation.

luosiallen

Nice clarification!

luosiallen · 2023-10-19T14:02:41Z

src/diffusers/pipelines/latent_consistency_models/pipeline_latent_consistency_models.py

+            prompt,
+            device,
+            num_images_per_prompt,
+            False,  # Don't need to get negative prompts due to LCM guided distillation


The unconditional prompt is not used in LCM actually. Because it has already been distilled in LCM. So we only take input of prompt_embeds. Also LCM does not support negative prompts currently, maybe try to fix it in the next version of LCM?

luosiallen · 2023-10-19T14:12:10Z

src/diffusers/pipelines/latent_consistency_models/pipeline_latent_consistency_models.py

+    def __call__(
+        self,
+        prompt: Union[str, List[str]] = None,
+        height: Optional[int] = 768,


I think we can set the default as 768 has the best generation quality, most of the SD model like SD-V2.1 or Dreamshaper is trained on 768 resolution.

luosiallen

Good!

…/config (as origin_steps).

src/diffusers/pipelines/latent_consistency_models/pipeline_latent_consistency_models.py

a-r-r-o-w

Looks great overall! It was fun reading through the paper, although I don't fully comprehend it, and quite awesome to see fast inference in such few steps yielding great results.

Apologies about the aggressive type hinting suggestions 😅 I'm trying to help make everything typed for better static type analysis and docs support for those digging deeper into the intrinsics of diffusers.

src/diffusers/schedulers/scheduling_lcm.py

src/diffusers/pipelines/latent_consistency_models/pipeline_latent_consistency_models.py

…mesteps doesn't work under default settings.

…y doesn't support CFG.

…int.

HuggingFaceDocBuilderDev · 2023-10-24T01:02:31Z

The documentation is not available anymore as the PR was closed or merged.

Co-authored-by: Aryan V S <avs050602@gmail.com>

src/diffusers/schedulers/scheduling_lcm.py

Co-authored-by: Aryan V S <avs050602@gmail.com>

* initial commit for LatentConsistencyModelPipeline and LCMScheduler based on the community pipeline * Add callback and freeu support. * apply suggestions from review * Clean up LCMScheduler * Remove timeindex argument to LCMScheduler.step. * Add support for clipping or thresholding the predicted original sample. * Remove unused methods and arguments in LCMScheduler. * Improve comment about (lack of) negative prompt support. * Change input guidance_scale to match the StableDiffusionPipeline (Imagen) CFG formulation. * Move lcm_origin_steps from pipeline __call__ to LCMScheduler.__init__/config (as origin_steps). * Fix typo when clipping/thresholding in LCMScheduler. * Add some initial LCMScheduler tests. * add type annotations from review * Fix type annotation bug. * Override test_add_noise_device in LCMSchedulerTest since hardcoded timesteps doesn't work under default settings. * Add generator argument pipeline prepare_latents call. * Cast LCMScheduler.timesteps to long in set_timesteps. * Add onestep and multistep full loop scheduler tests. * Set default height/width to None and don't hardcode guidance scale embedding dim. * Add initial LatentConsistencyPipeline fast and slow tests. * Add initial documentation for LatentConsistencyModelPipeline and LCMScheduler. * Make remaining failing fast tests pass. * make style * Make original_inference_steps configurable from pipeline __call__ again. * make style * Remove guidance_rescale arg from pipeline __call__ since LCM currently doesn't support CFG. * Make LCMScheduler defaults match config of LCM_Dreamshaper_v7 checkpoint. * Fix LatentConsistencyPipeline slow tests and add dummy expected slices. * Add checks for original_steps in LCMScheduler.set_timesteps. * make fix-copies * Improve LatentConsistencyModelPipeline docs. * Apply suggestions from code review Co-authored-by: Aryan V S <avs050602@gmail.com> * Apply suggestions from code review Co-authored-by: Aryan V S <avs050602@gmail.com> * Apply suggestions from code review Co-authored-by: Aryan V S <avs050602@gmail.com> * Update src/diffusers/schedulers/scheduling_lcm.py * Apply suggestions from code review Co-authored-by: Aryan V S <avs050602@gmail.com> * finish --------- Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> Co-authored-by: Aryan V S <avs050602@gmail.com>

dg845 added 2 commits October 18, 2023 22:08

initial commit for LatentConsistencyModelPipeline and LCMScheduler ba…

904b7f5

…sed on the community pipeline

Add callback and freeu support.

4e25452

patrickvonplaten reviewed Oct 19, 2023

View reviewed changes

src/diffusers/pipelines/latent_consistency_models/pipeline_latent_consistency_models.py Outdated Show resolved Hide resolved

patrickvonplaten reviewed Oct 19, 2023

View reviewed changes

src/diffusers/pipelines/latent_consistency_models/pipeline_latent_consistency_models.py Outdated Show resolved Hide resolved

patrickvonplaten reviewed Oct 19, 2023

View reviewed changes

echarlaix reviewed Oct 19, 2023

View reviewed changes

src/diffusers/pipelines/__init__.py Outdated Show resolved Hide resolved

patrickvonplaten reviewed Oct 19, 2023

View reviewed changes

src/diffusers/pipelines/latent_consistency_models/pipeline_latent_consistency_models.py Outdated Show resolved Hide resolved

patrickvonplaten reviewed Oct 19, 2023

View reviewed changes

src/diffusers/pipelines/latent_consistency_models/pipeline_latent_consistency_models.py Outdated Show resolved Hide resolved

apply suggestions from review

3eec980

luosiallen reviewed Oct 19, 2023

View reviewed changes

dg845 added 6 commits October 19, 2023 05:19

Clean up LCMScheduler

cadb86b

Remove timeindex argument to LCMScheduler.step.

0740f36

Add support for clipping or thresholding the predicted original sample.

b960037

Remove unused methods and arguments in LCMScheduler.

468511a

Improve comment about (lack of) negative prompt support.

6b00756

Change input guidance_scale to match the StableDiffusionPipeline (Ima…

cbc4d55

…gen) CFG formulation.

luosiallen reviewed Oct 20, 2023

View reviewed changes

Move lcm_origin_steps from pipeline __call__ to LCMScheduler.__init__…

e8f65bf

…/config (as origin_steps).

echarlaix reviewed Oct 20, 2023

View reviewed changes

src/diffusers/pipelines/latent_consistency_models/pipeline_latent_consistency_models.py Show resolved Hide resolved

a-r-r-o-w approved these changes Oct 20, 2023

View reviewed changes

dg845 added 5 commits October 20, 2023 04:25

Fix typo when clipping/thresholding in LCMScheduler.

f800547

Add some initial LCMScheduler tests.

21cb605

add type annotations from review

ae358ac

Fix type annotation bug.

00f4b48

Override test_add_noise_device in LCMSchedulerTest since hardcoded ti…

959e7f8

…mesteps doesn't work under default settings.

echarlaix mentioned this pull request Oct 20, 2023

Enable Latent Consistency models ONNX export huggingface/optimum#1469

Merged

dg845 added 2 commits October 20, 2023 06:11

Add generator argument pipeline prepare_latents call.

696d987

Cast LCMScheduler.timesteps to long in set_timesteps.

43361b7

dg845 added 9 commits October 22, 2023 22:38

Make remaining failing fast tests pass.

9df89ea

make style

0dd2a82

Make original_inference_steps configurable from pipeline __call__ again.

cbbd187

make style

8868d34

Remove guidance_rescale arg from pipeline __call__ since LCM currentl…

92b68b2

…y doesn't support CFG.

Make LCMScheduler defaults match config of LCM_Dreamshaper_v7 checkpo…

24c8cad

…int.

Fix LatentConsistencyPipeline slow tests and add dummy expected slices.

e191555

Add checks for original_steps in LCMScheduler.set_timesteps.

e439118

make fix-copies

516d8fd

Improve LatentConsistencyModelPipeline docs.

e57913d

dg845 changed the title ~~[WIP] Add Latent Consistency Models Pipeline~~ Add Latent Consistency Models Pipeline Oct 24, 2023

patrickvonplaten and others added 3 commits October 24, 2023 20:34

Apply suggestions from code review

6faa29f

Co-authored-by: Aryan V S <avs050602@gmail.com>

Apply suggestions from code review

b08df59

Co-authored-by: Aryan V S <avs050602@gmail.com>

Apply suggestions from code review

264bd77

Co-authored-by: Aryan V S <avs050602@gmail.com>

patrickvonplaten reviewed Oct 24, 2023

View reviewed changes

src/diffusers/schedulers/scheduling_lcm.py Outdated Show resolved Hide resolved

patrickvonplaten and others added 4 commits October 24, 2023 20:36

Update src/diffusers/schedulers/scheduling_lcm.py

888b96d

Apply suggestions from code review

686c0c0

Co-authored-by: Aryan V S <avs050602@gmail.com>

Merge branch 'main' into latent-consistency-models-pipeline

2cd1019

finish

2f2d14c

patrickvonplaten mentioned this pull request Oct 24, 2023

[Community] Make lcm backward comp #5514

Closed

patrickvonplaten approved these changes Oct 24, 2023

View reviewed changes

patrickvonplaten merged commit 958e17d into huggingface:main Oct 24, 2023
11 checks passed

a-r-r-o-w mentioned this pull request Oct 24, 2023

LCM pipeline fails when initialized without scheduler #5515

Closed

dg845 mentioned this pull request Oct 25, 2023

[WIP] Add Latent Consistency Distillation Script #5517

Closed

6 tasks

dg845 deleted the latent-consistency-models-pipeline branch October 25, 2023 13:15

echarlaix mentioned this pull request Oct 26, 2023

Enable Latent Consistency models OpenVINO export and inference huggingface/optimum-intel#463

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Latent Consistency Models Pipeline #5448

Add Latent Consistency Models Pipeline #5448

dg845 commented Oct 19, 2023

patrickvonplaten Oct 19, 2023

dg845 Oct 19, 2023

luosiallen Oct 19, 2023

dg845 Oct 20, 2023

luosiallen Oct 20, 2023

patrickvonplaten commented Oct 19, 2023

luosiallen Oct 19, 2023

dg845 Oct 19, 2023

luosiallen Oct 19, 2023

luosiallen Oct 19, 2023

dg845 Oct 20, 2023

luosiallen Oct 20, 2023

sayakpaul Oct 20, 2023

luosiallen left a comment

luosiallen Oct 19, 2023

luosiallen Oct 19, 2023

luosiallen left a comment

a-r-r-o-w left a comment •

edited

Loading

HuggingFaceDocBuilderDev commented Oct 24, 2023 •

edited

Loading

Add Latent Consistency Models Pipeline #5448

Add Latent Consistency Models Pipeline #5448

Conversation

dg845 commented Oct 19, 2023

What does this PR do?

Before submitting

Who can review?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

patrickvonplaten commented Oct 19, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

luosiallen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

luosiallen left a comment

Choose a reason for hiding this comment

a-r-r-o-w left a comment • edited Loading

Choose a reason for hiding this comment

HuggingFaceDocBuilderDev commented Oct 24, 2023 • edited Loading

a-r-r-o-w left a comment •

edited

Loading

HuggingFaceDocBuilderDev commented Oct 24, 2023 •

edited

Loading