ControlNet support?

[katopz]

I think it would be nice to have ControlNet support.
Not sure how hard it gonna take for this task. 🤔

[zolrath]

Would absolutely love to use ControlNet scribble here!

[LaurentMazare]

I just merged some very basic support for ControlNet via #68 . The weight file data/controlnet.safetensors can be obtained from the HuggingFace hub here.

Note that this is for the "canny" version of ControlNet and the edge detection part has to be done before submitting the input image (I plan on fixing this soonish). Here is how to run it:

cargo run --example controlnet --features clap -- --cpu all --prompt "disco dancer with colorful lights" --input-image vermeer_canny_edged.png

The input image vermeer_canny_edged.png looked like:

Generated image:

[katopz]

Cool! I learn a lot from this feature PR. Thanks!
I will find some free time to add other ControlNet capabilities.

[LaurentMazare]

Great that you find it helpful, I've also included the edge detection bit now (through an external crate) so there is no need to pre-process the original image anymore. There are also some specific instructions in the readme file.

[otdavies]

If I wanted to stack multiple controlnets together (depth, canny, normal) for example, how difficult would you estimate this to be to implement @LaurentMazare?