[LoRA] support LyCORIS

[sayakpaul]

What does this PR do?

Fixes #5079.

Usage example

from diffusers import StableDiffusionPipeline, DPMSolverMultistepScheduler
from diffusers.utils import make_image_grid
import torch

pipe = StableDiffusionPipeline.from_single_file(
    "Amixx.safetensors",
    safety_checker=None,
    torch_dtype=torch.float16,
)
pipe.scheduler = DPMSolverMultistepScheduler.from_config(
    pipe.scheduler.config, use_karras_sigmas=True
)
pipe.load_lora_weights(".",  weight_name="edgLycorisMugler-light.safetensors")
pipe.to("cuda")

# Weighting in prompts isn't natively supported in diffusers (be aware). It's just copy-pasted from CivitAI.
prompt = "((Masterpiece, best quality,edgQuality)),smirk,smug,white hair, Haute_Couture, blue gown,golden embroidery, woman wearing a Haute_Couture dress, (((edgTM, wearing edgTM_style fashion, eccentric clothing))),feathers, <lora:edgLycorisMugler-light:1>"
neg_prompt = "(low quality, worst quality:1.4),(holding,multiple limbs, mutated limbs, mutated human, poorly drawn human body:1.5)"

images = pipe(
    prompt, negative_prompt=neg_prompt,
    num_inference_steps=25, guidance_scale=6.0,
    generator=torch.manual_seed(2014727055),
    num_images_per_prompt=4
).images
make_image_grid(images, 1, len(images)).save("results.png")

Ensure the following models are downloaded before running the pipeline:

get https://civitai.com/api/download/models/141190 -O Amixx.safetensors
wget https://civitai.com/api/download/models/159507 -O edgLycorisMugler-light.safetensors

Cc: @George0726.

[patrickvonplaten]

Super, maybe also one slow test?

[sayakpaul]

@patrickvonplaten done!

Okay for you to merge once the CI is partially green (barring the documentation workflow)?

[sayakpaul]

Failing tests are flaky. Merging.