Fix Karras scheduler doesn't start from the actual max to min

[liuliu]

I am looking into various of the new samplers and try it in my https://github.com/liuliu/swift-diffusion

There is this magical sigma_min / sigma_max for Karras scheduler seems a bit abrupt. Looking deeper into it, I believe that it is best to match the actual sigma max / sigma min used when training the model. That means for Stable Diffusion, it is the sigma at step 999 and at step 1.

I switched to that value, and both DPM++ 2S Karras and DPM++ 2M Karras looks much more consistent between step 10, step 20, step 30 and step 40. I believe this is a good improvement and don't want to keep it secret only in my repo.

I renounce all the copyright of this PR. Just merge it and fix it for everyone if people verified this is indeed a good fix.

[aliencaocao]

Any info on the values of these for NovelAI leaked model?

[AUTOMATIC1111]

considering this will change existing seeds there should be some more demonstration/feedback from users over whether this really is better.

[kjerk]

I happened to be doing some image studies already when I saw this so I dove in to work out some prompts to try to stratify the difference a bit for inspection, keeping Euler and DDIM as a control.

Subjectively it looks like you can still get fine results without the change but after the change the step differences could bit more linear as stated, which makes sense (if it aligns with the vanilla settings, taking OP's word for it). Dreambooth models seem to be exacerbated a bit more (last links below).

I think even if you roll with this change, it seems like the stability/volatility these changes introduce probably make them a candidate for tweakable settings down the line, as per the house example, leaving the current settings in place can make the ancestral+Karras samplers extra wild, which is a feature to many.

Before Change

After Change

Cropped to Side-By-Side Karras Only

Settings

CLI Options: --xformers --opt-channelslast
Prompt: detailed illustration of Thor God of Thunder , guilty gear, by conrad roset, by yoshitaka amano, by ruan jia, cgsociety, artstation, portfolio quality, deep background
Negative prompt: boring, amateur, 3d
CFG scale: 8.5, Seed: 3847627383, Size: 512x512
Model: sd-v15-pruned-emaonly
VAE: vae-ft-ema-560000-ema-pruned.pt

More Examples

House

House Painting 3847627383 NO CHANGE
House Painting 3847627383 WITH CHANGE

CLI Options: --xformers --opt-channelslast
Prompt: small house at night. Country road, country landscape, trees. wide view, desolate. digital illustration, very vibrant colors, soft lighting, adventurous, atmospheric lighting, 8K, octane render. By Makoto Shinkai, Stanley Artgerm Lau, WLOP, Rossdraws, James Jean, Andrei Riabovitchev, Marc Simonetti, krenz
Negative prompt: boring, amateur, 3d
CFG scale: 8.5, Seed: 3847627383, Size: 512x512
Model: sd-v15-pruned-emaonly
VAE: vae-ft-ema-560000-ema-pruned.pt

Dreambooth

Arcane v3 34854 NO CHANGE
Arcane v3 34854 WITH CHANGE

CLI Options: --xformers --opt-channelslast
Prompt: character portrait in arcane style of a warrior with an eye patch and a huge mohawk, wide angle
Negative prompt: blurry, boring, amateur
CFG scale: 12.5, Seed: 34854, Size: 512x512
Model: arcane-diffusion-v3.ckpt
VAE: vae-ft-ema-560000-ema-pruned.pt

[C43H66N12O12S2]

@AUTOMATIC1111 I had posted this in my original PR that added Karras scheduler variants - its 'improvement' is a matter of preference TBH.

Enabling quantization should alleviate the non-compliant sigma range (aka anything other than 0.03-14.6).

If we insist on changing this, the range should be set as sigma_min, sigma_max = self.model_wrap.sigmas[0].item(), self.model_wrap.sigmas[-1].item() anyways.

IMO minute tweaks like this are unnecessary (especially since it would break seeds) considering dfaker already has a script for this sort of experimenting.

[GhostDragon69]

this is more a matter of taste then anything else, i prefer as it is right now. some of the other samplers already give the type of image, lets leave this one as is, diversity is good.

[liuliu]

Thanks for the feedback. My original motivation is to arrive at a stable diffusion result with lower step counts, and this sigma choice seems to do that (With this choice, I am consistently arriving at same results for DPM++ 2M Karras with 30 steps). But agreed with other concerns.

[kjerk]

Per the Thor test images I posted, DPM++ 2s a Karras is converged to something more salient and correct per the prompt at 14, 24, and 50 steps after the sigmas changes, on prior settings it's less resolved and he's missing an arm. It shouldn't be dismissed merely as taste if the effect is that distribution sampling + prompt embed is winding up poorer for it.

I'd be interested to see examples of convergence being worse, I'd echo the OP's point that anecdotally results have been equivalent or better.

@AUTOMATIC1111 I had posted this in my original PR that added Karras scheduler variants - its 'improvement' is a matter of preference TBH.

Enabling quantization should alleviate the non-compliant sigma range (aka anything other than 0.03-14.6).

If we insist on changing this, the range should be set as sigma_min, sigma_max = self.model_wrap.sigmas[0].item(), self.model_wrap.sigmas[-1].item() anyways.

IMO minute tweaks like this are unnecessary (especially since it would break seeds) considering dfaker already has a script for this sort of experimenting.

When I tested this quantization didn't have any effect (assuming you're talking about the toggleable option), other than the micro effects it has always had (a tree leaf goes appears or disappears etc, nothing structural or about the convergence)

[KohakuBlueleaf]

Is it possible to add some settings for sigma_min or sigma_max?
It seems like it actually affect something.
And since the difference is more like "taste" I think it could and should be a setting(like a slide bar or somewhat)

[C43H66N12O12S2]

@kjerk Here's a example: #1560 (comment)

Now, I don't know if it's worse, but it is certainly very different from the SD scheduler version.

[f8upd8]

Is it possible to add some settings for sigma_min or sigma_max? It seems like it actually affect something. And since the difference is more like "taste" I think it could and should be a setting(like a slide bar or somewhat)

I think this is a very solid point, more significant settings, the merrier

[Birch-san]

b34b25b#commitcomment-86212295