integrate abehonest's ControlNet_AnimalPose as a preporcessor/model

[Lefkowitz-s]

So it looks like abehonest has made a branch of this Extension that works with quadrupeds instead of humanoids. However, it doesn't look like there's any way to directly install it to your personal Stable Diffusion program. Is there any to download the code and install it, or better yet, use it as a preprocessor/model for the existing Controlnet UI? I'm not exactly an expert with Github.

[geroldmeisinger]

you can find the model at the provided gdrive link https://drive.google.com/file/d/1Q0lvCVFx2hGlgI7Q-npJKDFqZRwZltYY/view?usp=share_link . just copy it to your controlnet folder.
a preprocessor is also provided, but you have to call it on your image manually. the preprocessor can be integrated directly in any Stable diffusion UI of course, but this for convinience. you can open a feature request in the corresponding project (a1111 extension, comfy nodes etc.) or ask the author animalpose. to increase your chance of realization, you should provide examples of art created with it, as to support "people are using it, it is useful, we need this!"

[abehonest]

Agreed! I poked around a bit to try to figure out how to prune my model to run on <8gb of VRAM (that's the main reason why I haven't ported it to WebUI yet) and never got too far on that. The models for WebUI seem to be significantly smaller than the base ControlNet models and I want to replicate that. If anyone knows more about this or has seen any good resources I'm glad to re-start efforts on that. I'm currently training a new version of the quadruped ControlNet at the moment but once that is finished I'll have more time to look into this issue.

[geroldmeisinger]

huh, why is it 8GB? did you fine-tune a SD model on fp32? :D
I wrote an article on controlnet training. the diffuser version creates a 1.4GB file on default, if you convert it to fp16 it has 700MB. There are other tricks like Loras for controlnet (see SDXL), T2I-Adapter, Controlnet-XS which are even smaller but I don't how to make them yet. DM me if you need help.

[abehonest]

I just read the article. It's the best I have ever read on training a ControlNet! I will use the script you linked in your article, https://github.com/Akegarasu/sd-model-converter, to shrink my model and then I can work on porting it to WebUI!