[Feature Request]: Double model load for multi process(hires fix)

[FlowDownTheRiver]

Is there an existing issue for this?

• I have searched the existing issues and checked the recent builds/commits

What would your feature do ?

Hi there Automatic and other devs.

1. I have done an experiment pushing up the cfg scale to 20 by mimicing the value of 7 with the extension https://github.com/mcmonkeyprojects/sd-dynamic-thresholding . This gave me the ability to lower the sampling steps and push up the details really fast. So sampling steps 6-8-10 is used.

2. I have tested so many inpainting models for inference in the txt2img tab. So I am not running a regular model. Keep this in mind. With denoising strength 1, I enabled hires fix. Since I am running an inpainting model I have the ability to control "Inpainting conditioning mask strength" value to how much of an influence it affects the image especially on the second pass of hires fix. I put "Inpainting conditioning mask strength" to my quick settings to have easy access and manipulate this value. for the Hires fix I used dpm++ samplers with 6-10 steps. The reason it works these samplers does a great job at low sampling steps while I push the cfg scale to 20 but mimicing the value of 7. The results were satisfying with really fast inference times. Why inpainting model you may ask,cos it can reconstruct the image really good like a healing effect with the cfg scale high and with the guidance of the prompt.

3. I could even push this further with token merging but I didn't do it since it was fast enough.

This experiment gave me the idea to load 2 models at the same time. The first model shall be a regular model for the first pass of hires fix and for the second pass we can use inpainting models or another regular model(maybe generate the base info before hires is applied with a model and style change on the second pass,so much flexibility there). For your information I have done this experiment with the old hires fix. I didn't use latent upscalers but used "Lanczos" upscaling method for the sake of speed and latent upscalers doesn't play good with low denosing values.

Optionally this can be used with https://github.com/klimaleksus/stable-diffusion-webui-conditioning-highres-fix by tieing up the values of mask conditionating strength to denoising strenght dynamically changing,with some presets there for ease of use for the end user.

What I request is simple :

I want to be be able to load 2 models.Since we can prune the models to 2 gb,memory shall not be an issue when we load them and keep them in the memory.
Since we have the option to set different prompting for hires fix,and sampling method as well,why not to have a model loading option for the hires pass?

There is one thing there I didn't understand why.The aspect ratio has to be 1:1 for this to work. I have no idea about this and why it is forced to have.

For the next level if this gets implemented, styles(prompt saving) shall be updated to include the hires prompting.

Note that, this was not a regular testing but pushing the limits of what's available on the table.

I hope this gets implemented and the idea behind spreads to masses...

Note : To people who see this one,the numbers here are situational,so these are not magical numbers.

I have made this photo down below completely with inpainting model in txt2img,no controlnet,upscaled in img2img tab with an inpainting model 2x with ultimate sr upscale extension,then in extras tab,first model : 2x lolipop(esrgan model) ,second model: 1x focus(esrgan model), low value of gfpgan and codeformer in combination.Just to give an example of how it works.

[w-e-w]

#12181

[FlowDownTheRiver]

Thank you very much devs. You are great!