[Core] enable lora for sdxl adapters too and add slow tests.

[ilisparrow]

Fixes #5516
Additionally, adds SLOW tests for SDXL T2i adapters.

Hello,
as requested here is a PR to fix the #5516, WIth it's corresponding tests.

I have a few questions :

• The test seem to fail when this is removed from the pipeline initialization : torch_dtype=torch.float16, But your PR that inspired mine didn't have it(https://github.com/huggingface/diffusers/pull/4666/files).
  This is the error that I get with it :
  RuntimeError: Input type (float) and bias type (c10::Half) should be the same
• I get this warning when I add .to("CPU") to the pipeline and the adapter :
  Pipelines loaded with 'torch_dtype=torch.float16' cannot run with 'cpu' device. It is not recommended to move them to 'cpu' as running them will fail. Please make sure to use an accelerator to run the pipeline in inference, due to the lack of support for'float16' operations on this device in PyTorch. Please, remove the 'torch_dtype=torch.float16' argument, or use another device for inference.
• Last, In the tests, we load this image : https://huggingface.co/datasets/hf-internal-testing/diffusers-images/resolve/main/t2i_adapter/toy_canny.png, but if we are testing for canny of functions that process images with canny, shouldn't we input a real image and expect this as an output ? Because otherwise we are applying a canny to an already canny image.

I hope this helps.

@sayakpaul

[sayakpaul]

LoRA related additions look good to me! Are there any cool results that you would like so share with us here?

For the tests, I will defer to @DN6.

[DN6]

LGTM 👍🏽

[ilisparrow]

Thanks, great. Do you know when it will be merged?
I have some feedback or comments in or comment, that you might find interesting.

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[DN6]

@ilisparrow I think you would need to run make style && make quality as well as make fix-copies on the PR again.

[patrickvonplaten]

Great job @ilisparrow !

[patrickvonplaten]

Actually this PR was very much incorrect, SDXL Adapter already had the LoRA load mixin and the tests had errors. Reverting this PR here: #5555

[ilisparrow]

Hello,
Thank you for taking the time to have a look.
This PR solves this problem : #5516
That sayakpaul was able to reproduce. Where I tried to add alora to an adapter constrained SDXL model.
It was inspired by this one : #4664
Which was solving the same for controlnet and SDXL .

I might be missing something, but it fixed a real problem, and was reproduced by Sayakpaul.

[sayakpaul]

That might have been the case because I reproduced it with the stable release of diffusers. Installing from main might have actually solved it:

diffusers/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_xl_adapter.py

Line 127 in 75ea54a

DiffusionPipeline, FromSingleFileMixin, StableDiffusionXLLoraLoaderMixin, TextualInversionLoaderMixin

I apologize for the oversight caused on my part.