Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a new community pipeline #5477

Conversation

nagolinc
Copy link
Contributor

@nagolinc nagolinc commented Oct 22, 2023

This PR adds an img2img pipeline for Latent Consistency Models: luosiallen/latent-consistency-model#8 (comment)

examples/community/latent_consistency_img2img.py

which can be called like this

import torch
from diffusers import DiffusionPipeline

pipe = DiffusionPipeline.from_pretrained(
"SimianLuo/LCM_Dreamshaper_v7", custom_pipeline="latent_consistency_txt2img", custom_revision="main")

        # To save GPU memory, torch.float16 can be used, but it may compromise image quality.

pipe.to(torch_device="cuda", torch_dtype=torch.float32)

img2img=LatentConsistencyModelPipeline_img2img(
vae=pipe.vae,
text_encoder=pipe.text_encoder,
tokenizer=pipe.tokenizer,
unet=pipe.unet,
#scheduler=pipe.scheduler,
scheduler=None,
safety_checker=None,
feature_extractor=pipe.feature_extractor,
requires_safety_checker=False,
)

img = Image.open("thisismyimage.png")

result = img2img(prompt,img,strength,num_inference_steps=4)

What does this PR do?

Fixes # (issue)

Before submitting

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

examples/community/latent_consistency_img2img.py

which can be called like this

import torch
from diffusers import DiffusionPipeline

pipe = DiffusionPipeline.from_pretrained(
                "SimianLuo/LCM_Dreamshaper_v7", custom_pipeline="latent_consistency_txt2img", custom_revision="main")

            # To save GPU memory, torch.float16 can be used, but it may compromise image quality.
pipe.to(torch_device="cuda", torch_dtype=torch.float32)

img2img=LatentConsistencyModelPipeline_img2img(
    vae=pipe.vae,
    text_encoder=pipe.text_encoder,
    tokenizer=pipe.tokenizer,
    unet=pipe.unet,
    #scheduler=pipe.scheduler,
    scheduler=None,
    safety_checker=None,
    feature_extractor=pipe.feature_extractor,
    requires_safety_checker=False,
)

img = Image.open("thisismyimage.png")

result = img2img(prompt,img,strength,num_inference_steps=4)
Copy link
Contributor

@patrickvonplaten patrickvonplaten left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool addition! Could we maybe also add a small entry to the README here: https://github.com/huggingface/diffusers/tree/main/examples/community#latent-consistency-pipeline

@HuggingFaceDocBuilderDev
Copy link

HuggingFaceDocBuilderDev commented Oct 23, 2023

The documentation is not available anymore as the PR was closed or merged.

Fix name formatting for scheduler

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
@nagolinc nagolinc force-pushed the LatentConsistencyModelPipeline_img2img branch from 7e76ac4 to 76f9f1e Compare October 23, 2023 20:51
@nagolinc nagolinc force-pushed the LatentConsistencyModelPipeline_img2img branch from 76f9f1e to 072e682 Compare October 23, 2023 20:55
@nagolinc
Copy link
Contributor Author

would reformat /home/runner/work/diffusers/diffusers/examples/community/latent_consistency_img2img.py

which formatter do I need to run?

@patrickvonplaten
Copy link
Contributor

make style => I can run it on "main"

@patrickvonplaten patrickvonplaten merged commit a1fad82 into huggingface:main Oct 25, 2023
10 of 11 checks passed
@nagolinc
Copy link
Contributor Author

thanks!

kashif pushed a commit to kashif/diffusers that referenced this pull request Nov 11, 2023
* Add a new community pipeline

examples/community/latent_consistency_img2img.py

which can be called like this

import torch
from diffusers import DiffusionPipeline

pipe = DiffusionPipeline.from_pretrained(
                "SimianLuo/LCM_Dreamshaper_v7", custom_pipeline="latent_consistency_txt2img", custom_revision="main")

            # To save GPU memory, torch.float16 can be used, but it may compromise image quality.
pipe.to(torch_device="cuda", torch_dtype=torch.float32)

img2img=LatentConsistencyModelPipeline_img2img(
    vae=pipe.vae,
    text_encoder=pipe.text_encoder,
    tokenizer=pipe.tokenizer,
    unet=pipe.unet,
    #scheduler=pipe.scheduler,
    scheduler=None,
    safety_checker=None,
    feature_extractor=pipe.feature_extractor,
    requires_safety_checker=False,
)

img = Image.open("thisismyimage.png")

result = img2img(prompt,img,strength,num_inference_steps=4)

* Apply suggestions from code review

Fix name formatting for scheduler

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

* update readme (and run formatter on latent_consistency_img2img.py)

---------

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
@Zaniyar
Copy link

Zaniyar commented Nov 20, 2023

Cool addition! Could we maybe also add a small entry to the README here: https://github.com/huggingface/diffusers/tree/main/examples/community#latent-consistency-pipeline

 from diffusers import Diffusionpipeline
ImportError: cannot import name 'Diffusionpipeline' from 'diffusers' (/home/user/anaconda3/lib/python3.11/site-packages/diffusers/__init__.py)

I tried:
$ conda install -c conda-forge diffusers
$ conda install diffusers
$ pip install diffusers
$ pip3 install diffusers

it is installed, but can't be imported in python - any ideas?

user@Desktop:/mnt/c/Users/user/projects/code/realtimeDiffusion$ pip install diffuser
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: diffuser in /home/user/.local/lib/python3.10/site-packages (0.0.1)

user@Desktop:/mnt/c/Users/user/projects/code/realtimeDiffusion$ python3 bild2bild.py 
/home/user/.local/lib/python3.10/site-packages/huggingface_hub/file_download.py:672: FutureWarning: 'cached_download' is the legacy way to download files from the HF hub, please consider upgrading to 'hf_hub_download'
  warnings.warn(
Traceback (most recent call last):
  File "/mnt/c/Users/user/projects/code/realtimeDiffusion/bild2bild.py", line 12, in <module>
    pipe = LDMTextToImagePipeline.from_pretrained(model_id_or_path, torch_dtype=torch.float16, custom_pipeline="pipeline_fabric").to("cuda")
  File "/home/user/.local/lib/python3.10/site-packages/diffusers/pipelines/pipeline_utils.py", line 1078, in from_pretrained
    cached_folder = cls.download(
  File "/home/user/.local/lib/python3.10/site-packages/diffusers/pipelines/pipeline_utils.py", line 1754, in download
    pipeline_class = _get_pipeline_class(
  File "/home/user/.local/lib/python3.10/site-packages/diffusers/pipelines/pipeline_utils.py", line 360, in _get_pipeline_class
    return get_class_from_dynamic_module(
  File "/home/user/.local/lib/python3.10/site-packages/diffusers/utils/dynamic_modules_utils.py", line 445, in get_class_from_dynamic_module
    final_module = get_cached_module_file(
  File "/home/user/.local/lib/python3.10/site-packages/diffusers/utils/dynamic_modules_utils.py", line 318, in get_cached_module_file
    modules_needed = check_imports(resolved_module_file)
  File "/home/user/.local/lib/python3.10/site-packages/diffusers/utils/dynamic_modules_utils.py", line 150, in check_imports
    raise ImportError(
ImportError: This modeling file requires the following packages that were not found in your environment: diffuser. Run `pip install diffuser`

AmericanPresidentJimmyCarter pushed a commit to AmericanPresidentJimmyCarter/diffusers that referenced this pull request Apr 26, 2024
* Add a new community pipeline

examples/community/latent_consistency_img2img.py

which can be called like this

import torch
from diffusers import DiffusionPipeline

pipe = DiffusionPipeline.from_pretrained(
                "SimianLuo/LCM_Dreamshaper_v7", custom_pipeline="latent_consistency_txt2img", custom_revision="main")

            # To save GPU memory, torch.float16 can be used, but it may compromise image quality.
pipe.to(torch_device="cuda", torch_dtype=torch.float32)

img2img=LatentConsistencyModelPipeline_img2img(
    vae=pipe.vae,
    text_encoder=pipe.text_encoder,
    tokenizer=pipe.tokenizer,
    unet=pipe.unet,
    #scheduler=pipe.scheduler,
    scheduler=None,
    safety_checker=None,
    feature_extractor=pipe.feature_extractor,
    requires_safety_checker=False,
)

img = Image.open("thisismyimage.png")

result = img2img(prompt,img,strength,num_inference_steps=4)

* Apply suggestions from code review

Fix name formatting for scheduler

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

* update readme (and run formatter on latent_consistency_img2img.py)

---------

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants