-
Notifications
You must be signed in to change notification settings - Fork 27k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Feature Request]: new sampler "lcm" for lcm-loras #13952
Comments
I'm awaiting the integration of the LCM sampler into AUTOMATIC1111, While AUTOMATIC1111 is an excellent program, the implementation of new features, such as the LCM sampler and consistency VAE, appears to be sluggish. The consistency VAE is currently only accessible in beta, while comfyui already offers both consistency VAE and LCM sampler support at no time :( |
Let's not forget that TensorRT for SDXL is sitting on the Dev branch, too. I desperately need all three of these moved up into the Main branch. |
let's hope this will be soon 🤞🤞🤞🤞 |
Someone reported nearly 5 gens per second on a 3090 with both TensorRT and LCM Lora. We're living in a pretty good timeline over here. Can't wait to see all this stuff get in so we can use it all together. Never ever thought I'd see a factor of 15+ speed up this quick. |
Makes me feel happy with my 3090 Ti! |
I could make tensorrt works with lcm only after merging lcm lora into model (and training difference in supermerger) |
I’m looking forward to this feature too. |
I just installed TensorRT and yeah its amazing (3090Ti) |
@iChristGit I assume you ran this on the Dev branch? |
Yeah git checkout dev |
Someone from Reddit implemented it 4 day ago You can add the LCM sampler to A1111 with a little trick Tutorial | Guide |
But also there was found a bug in LCM Scheduler, and the algorithm will be updated |
I've modified patch from reddit to not edit external repository: diff --git a/modules/sd_samplers_extra.py b/modules/sd_samplers_extra.py
index 1b981ca8..d154a2b6 100644
--- a/modules/sd_samplers_extra.py
+++ b/modules/sd_samplers_extra.py
@@ -72,3 +72,20 @@ def restart_sampler(model, x, sigmas, extra_args=None, callback=None, disable=No
last_sigma = new_sigma
return x
+
+
+@torch.no_grad()
+def sample_lcm(model, x, sigmas, extra_args=None, callback=None, disable=None, noise_sampler=None):
+ extra_args = {} if extra_args is None else extra_args
+ noise_sampler = k_diffusion.sampling.default_noise_sampler(x) if noise_sampler is None else noise_sampler
+ s_in = x.new_ones([x.shape[0]])
+ for i in tqdm.auto.trange(len(sigmas) - 1, disable=disable):
+ denoised = model(x, sigmas[i] * s_in, **extra_args)
+ if callback is not None:
+ callback({'x': x, 'i': i, 'sigma': sigmas[i], 'sigma_hat': sigmas[i], 'denoised': denoised})
+
+ x = denoised
+ if sigmas[i + 1] > 0:
+ x += sigmas[i + 1] * noise_sampler(sigmas[i], sigmas[i + 1])
+ return x
+
diff --git a/modules/sd_samplers_kdiffusion.py b/modules/sd_samplers_kdiffusion.py
index 8a8c87e0..b6c3dc44 100644
--- a/modules/sd_samplers_kdiffusion.py
+++ b/modules/sd_samplers_kdiffusion.py
@@ -36,6 +36,7 @@ samplers_k_diffusion = [
('DPM2 a Karras', 'sample_dpm_2_ancestral', ['k_dpm_2_a_ka'], {'scheduler': 'karras', 'discard_next_to_last_sigma': True, "uses_ensd": True, "second_order": True}),
('DPM++ 2S a Karras', 'sample_dpmpp_2s_ancestral', ['k_dpmpp_2s_a_ka'], {'scheduler': 'karras', "uses_ensd": True, "second_order": True}),
('Restart', sd_samplers_extra.restart_sampler, ['restart'], {'scheduler': 'karras', "second_order": True}),
+ ('LCM Test', sd_samplers_extra.sample_lcm, ['lcm'], {}),
]
|
I have wrapped this patch in extention! 🎉 |
We also need Skipping-Step from paper. Without it we still have incomplete implementation. |
We hope @AUTOMATIC1111 will add this sampler natively and correctly, because I know nothing about defussion theory |
There's this: https://github.com/0xbitches/sd-webui-lcm |
Nooo. It is converted gradio demo of lcm model before the loras were released It is separated tab. It's no longer relevant |
Thanks. So is the light-and-ray thing an actual solution? I'm hoping for a version that will work in Deform. |
Yes, you're right |
use my extension to use LCM sampler in exaly the way you describe. https://github.com/continue-revolution/sd-webui-animatediff#lcm |
Thanks for your research! Any ideas why in sdxl other samplers not work? |
Do @AUTOMATIC1111 have time to support the LCM sampler? |
This referenced an issue discussion of the stable-diffusion-webui at AUTOMATIC1111/stable-diffusion-webui#13952, which may not be too perfect.
If you install the latest update of the Animatediff extension, that installs the LCM sampler for you. No need for this workaround, now. |
Dude, what a BLESSING! Installing Animatediff just added the LCM sampler as well. Also you can totally increase the step count to 10-15 and get even better quality it seems. Thank you! |
I am getting an image per second on 3090Ti, with TensorRT (no LCM) it is around 3.5 second per image. |
I'm using an RTX 3060 12GB and I'm getting surprisingly good results. My settings are lora weight of 0.75, LCM sampler, 6-8 steps and CFG of 2.0. Try these settings and see if things improve at all for you. |
And with LCM you want TinyVAE. It turns 70ms 512x512 4 step generations into 45ms on my 4090. |
You need to understand several things
|
Do you feel like Automatic1111 is a dead-end in the long run? Should we all be putting more effort into learning ComfyUI? |
I've never had a second having any thoughts like this, and I will almost certainly stick to A1111. I believe that creating a great user experience is our (programmer's) mission, and you, users, should focus more on how to use, and teach us how to actually use our software. A lot of people are much better at using my extension than myself. Designing a user-friendly software is never easy in any UI, but I enjoy that. A1111 has done his best. A1111 WebUI is easy-to-use, fast and memory efficient. We should not critisize A1111 before having any clear evidence. Dispite the fact that it is tricky to hook some functions, some other functions are designed so good that it can easily fit my need. That said, we programmers do need money. Working for love is never sustainable, unless we are as rich as Mark Zuckerberg. Mark has already been extremely rich through Facebook, so he open-sourced almost everything about ML in Meta. Sam is not as rich as Mark, so OpenAI becomes CloseAI. |
They have updated the algorithm of sampler |
@light-and-ray How do I use this update? I tested lcm and there are some problems in controlnet inpaint, it doesn't recognize red anymore. |
please can you elaborate how ti install TENSOR RT? I have an RTX 2060, do i need to install a driver from NVIDIA, or just the webui tensor rt extension?????? |
@continue-revolution cold you apply this update in your extension?
|
post a feature request in my repo and AT the original author of LCM. He will make the decision. |
@LIQUIDMIND111 Why do you say NOT for photos? The images aren't perfect, but after an Ultimate Upscale application, the results look really, really nice. 🤷🏼♂️ |
sorry i meant that for photos you dont need as much steps as for
animations, but i have done photos with LCM at 12 steps and look good also,
but i meant for photos is enough with 6 steps, but for animatediff i do
animations with LCM at 14 steps and look WAY BETTER than 8 steps.... so it
gives GOOD result to do MORE steps than what the authors recommend.
…On Fri, Nov 24, 2023 at 1:23 AM DarthBuckeye ***@***.***> wrote:
@LIQUIDMIND111 <https://github.com/LIQUIDMIND111> Why do you say NOT for
photos? The images aren't perfect, but after an Ultimate Upscale
application, the results look really, really nice. 🤷🏼♂️
—
Reply to this email directly, view it on GitHub
<#13952 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/A22MJBASEMLYAXFKM3SDZTLYGAVM5AVCNFSM6AAAAAA7HPKZ5WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMRVGE3DEOJTG4>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
--
"It was from the Mind of GOD, the Universe was created."
- - -- --- -)( ((---IMAGINATION IS CREATION---)) )(- --- -- - -
- -- -- --- -)( ((---liquid[)(]mind---)) )(- --- -- -- -
|
@continue-revolution |
StreamMultiDiffusion solved the issue for regional prompting with LCM - https://github.com/ironjr/StreamMultiDiffusion. |
Care to share a link to TinyVAE? |
Google'ing TinyVAE doesn't come up with a direct result, but I wonder if they intended to refer to this one:
|
Yeah this is what I came across also, though it doesn't use the same name oddly. :) |
Is there an existing issue for this?
What would your feature do ?
This feature requires not so much: no a new model class support, only a new sampler
LCM had major update and now we can use it like a regular lora:
https://huggingface.co/latent-consistency/lcm-lora-sdv1-5
https://huggingface.co/latent-consistency/lcm-lora-ssd-1b
https://huggingface.co/latent-consistency/lcm-lora-sdxl
We only need to rename them and put into lora models directory. Set sampling steps to 4, CFG Scale to 1.0. Sampling method to "DPM2" or "Euler a". It gives decent results, but for better work it requires a special sampler, with is similar to others but with little change. You can look how it works in ComfyUI: comfyanonymous/ComfyUI@002aefa
Proposed workflow
Additional information
How it works now
CompyUI's recent update:
The text was updated successfully, but these errors were encountered: