Add controlnet-1.5-face #13
Conversation
|
|
||
| ControlNetModel.from_pretrained( | ||
| settings.CONTROLNET_MODEL, | ||
| subfolder="diffusion_sd15", |
There was a problem hiding this comment.
We need the sd15 variant instead of the default 21 variant
| controlnet = ControlNetModel.from_pretrained( | ||
| settings.CONTROLNET_MODEL, | ||
| subfolder="diffusion_sd15", | ||
| torch_dtype=torch.float16, | ||
| cache_dir=settings.MODEL_CACHE, | ||
| local_files_only=True, | ||
| ) |
There was a problem hiding this comment.
Load the sd15 variant for ControlNetMediaPipeFace
| def process_control(self, control_image): | ||
| if control_image is None: | ||
| return None | ||
|
|
||
| # TODO do some actual pre-processing instead of returning the already pre-processed control image | ||
| return control_image |
There was a problem hiding this comment.
This currently just returns the already preprocessed control image. We could add some actual preprocessing here, eg. by implementing https://huggingface.co/CrucibleAI/ControlNetMediaPipeFace/blob/main/tool_generate_face_poses.py
| @@ -0,0 +1,4 @@ | |||
| MODEL_CACHE = "diffusers-cache" | |||
| SAFETY_MODEL_ID = "CompVis/stable-diffusion-safety-checker" | |||
| CONTROLNET_MODEL = "CrucibleAI/ControlNetMediaPipeFace" | |||
There was a problem hiding this comment.
|
Confirmed working on replicate, see this template
|
|
@SabatinoMasala what do you think about the idea of having a separate I can see it being useful for other controlnets as well |
|
I wrote this up the other day: #12 |
|
@anotherjesse, a separate |
|
In the following commit I introduced |
|
With the introduction of ControlNet 1.1, Openpose face is included. We can probably consider this PR irrelevant. |
I added a template for CrucibleAI/ControlNetMediaPipeFace.
To test, please add the following file to the
imagesfolder and call itface_annotation.png:https://huggingface.co/CrucibleAI/ControlNetMediaPipeFace/resolve/main/samples_laion_face_dataset/happy_annotation.png
Go into the controlnet-1.5-face folder, add some weights and run:
This will output the controlled image like so:
Control image for reference:
Known limitations:
The template currently does not preprocess images and expects a preprocessed image as an input. We could implement https://huggingface.co/CrucibleAI/ControlNetMediaPipeFace/blob/main/tool_generate_face_poses.py to preprocess the images.