enhancement - translucent background

[johndpope]

First, confirm

• I have read the instruction carefully
• I have searched the existing issues
• I have updated the extension to the latest version

What happened?

I'm using with https://github.com/layerdiffusion/sd-forge-layerdiffuse + sd forge.
when I use this plugin - it puts in a grid background instead of a transparent pixel.

Steps to reproduce the problem

install sd forge
https://github.com/lllyasviel/stable-diffusion-webui-forge

add extension
https://github.com/layerdiffusion/sd-forge-layerdiffuse

Sysinfo

ubuntu

Relevant console log

a

Additional information

.

[Dutch77]

+1

I'm doing currently very painful procedure because of this. Be able to faceswap on transparent png would be great feature.

[altoiddealer]

ReActor face swap actually works... technically... if you take the transparent output from layerdiffuse and send it to Extras tab, then use ReActor.

However, it totally borks the image and makes it fully opaque

Btw I did make an issue over there, but it seems like ultimately ReActor would have to do some kind of magic to be compatible with images with alpha. lllyasviel/sd-forge-layerdiffuse#50

[altoiddealer]

Sorry to double post, but it seems like this could be a solution... I'll be trying it personally tomorrow when time permits

ChatGPT says this works supposedly

[Gourieff]

There're several issues we have in SD WebUI with all these things

1. The main image that is generated by "sd-forge-layerdiffuse" - the checkerboarded image but not the one with alpha-channel (and it still doesn't save the image with alpha)
2. It seems that SD WebUI cuts Alpha channel when even we process the image via Extras tab (correct me if I'm wrong)
3. Insightface doesn't work with Alpha-channel as well :( It needs BGR as the input...

Sorry to double post, but it seems like this could be a solution... I'll be trying it personally tomorrow when time permits
ChatGPT says this works supposedly

But it's rather interesting solution if we could get alpha-png from the SD WebUI pipeline to postprocess it without the alpha-channel being cut
Or we even could simply blend it by the face mask

[Woisek]

Why using a transparent PNG before the face swap?
Why not make a transparent PNG opaque, face swap and then remove the bg or use the transparency of the original to re-cut out the face swapped copy ... ?
🤷‍♂️

[Dutch77]

@Gourieff Or we even could simply blend it by the face mask this would be great! If I understand it correctly, you would

1. get the target pic with transparent bg
2. faceswap as is now so it has black bg
3. crop faces via faces masks
4. cropped faces apply to original target img to keep it with transparent bg

This would be awesome and solve a lot of pain I'm currently doing in my pipeline to be even able to swap face to cropped characters :D

Also really good feature would be if you can return new faces as mask. I bet that would also help a lot of people.

Why using a transparent PNG before the face swap? Why not make a transparent PNG opaque, face swap and then remove the bg or use the transparency of the original to re-cut out the face swapped copy ... ? 🤷‍♂️

@Woisek One of the reason is because after you bring back bg, several other people are gonna be present and face indexes would be different. It cannot be automated without human intervention.

[johndpope]

a hack from chatgpt -
I asked it to remove this grid

RGB
if item[0] == item[1] == item[2]: // eg. 3333333

problem is gray is a little off.

from PIL import Image

# Open the image file with the grid background
with Image.open("path_to_your_image.png") as img:
    # Convert the image to RGBA if it's not already in that mode
    if img.mode != 'RGBA':
        img = img.convert('RGBA')
    
    # Get the data of the image
    datas = img.getdata()
    
    newData = []
    for item in datas:
        # Check if the pixel is part of the grid (assumed to be shades of gray)
        if item[0] == item[1] == item[2]:
            # Make the grid transparent
            newData.append((255, 255, 255, 0))
        else:
            newData.append(item)

    # Update image data with new data
    img.putdata(newData)
    
    # Save the modified image with a transparent grid
    img.save("path_to_your_new_image.png", "PNG")

[Woisek]

@Woisek One of the reason is because after you bring back bg, several other people are gonna be present and face indexes would be different. It cannot be automated without human intervention.

Huh? What 'several other people are gonna be present' when you make the transparent image opaque with any color ... ??

[Dutch77]

Huh? What 'several other people are gonna be present' when you make the transparent image opaque with any color ... ??

Sry. Should have better describe it. My use case currently is stable-diffusio with inpainting person to custom backround. I have it automated via api but problem here is when the new background has multiple other characters and I don't know which id is the face I want to swap. I have the mask of person but when I feed masked image to reactor, it removes alpha channel. Because of this I have to use another layer that detect head a feeds only the head to reactor and that head is positioned back to composition.

tldr: if reactor wouldn't strip alpha channels life would be easier. I can imagine case where you have some people assets without background and you want them face swapped. Now it's impossible.

[altoiddealer]

So what I said earlier that ChatGPT suggested, is working for my discord bot.

I just had to play around with it a bit.

        layerdiffuse = img_payload['alwayson_scripts'].get('layerdiffuse', {})
        if len(images) > 1 and layerdiffuse and layerdiffuse['args'][0]:
             # Extract alpha channel from layerdiffuse output
            img1 = images[1]
            _, _, _, alpha = img1.split()
            
            # Open ReActor output, convert to RGBA, apply alpha
            img0 = Image.open(f'temp_img_0.png')
            img0 = img0.convert('RGBA')
            img0.putalpha(alpha)
            
            # Save the image, update discord image payload
            img0.save('temp_img_0.png')
            images[0] = img0

Downsides:

• if the layerdiffusion output has a 'mild cloudiness' to the background transparency layer, the result of this method here will have a faint checker pattern in those semitranslucent areas.

• If the overall shape changes (head turned sideways), then the alpha won't align and it will look wonky

[Gourieff]

Ok guys
With the quick commit 99b0f19 ReActor can work with RGBA via the Extras Tab (Single Image) - but only if SD WebUI doesn't cut Alpha channel when run postprocessing
There's a quick fix to make SD WebUI allow RGBA for Extras: Gourieff/stable-diffusion-webui-forge@main...Gourieff:stable-diffusion-webui-forge:extras--allow-png-rgba
Fill free to test

[Dutch77]

@Gourieff Idk what im doing wrong but it still thinks it's RGB

ai-stable-diffusion-webui-1  | 10:22:13 - ReActor - DEBUG - We're here: process()
ai-stable-diffusion-webui-1  | 10:22:13 - ReActor - DEBUG - We're here: process()
ai-stable-diffusion-webui-1  | 10:22:13 - ReActor - STATUS - Working: source face index [0], target face index [0]
ai-stable-diffusion-webui-1  | 10:22:13 - ReActor - STATUS - image = <PIL.Image.Image image mode=RGB size=1080x1440 at 0x7F59341C2110>
ai-stable-diffusion-webui-1  | 10:22:13 - ReActor - STATUS - alpha = None

Using extra tab in webui. Tried even transparent png in target and source. Also could you do the fix also for API? Thx

webui version 1.8.0.

[Gourieff]

There's a quick fix to make SD WebUI allow RGBA for Extras: Gourieff/stable-diffusion-webui-forge@main...Gourieff:stable-diffusion-webui-forge:extras--allow-png-rgba

@Dutch77 Did you apply this fix?

Also could you do the fix also for API?

Yes, this is in the plan as well

[Dutch77]

I'm gonna create seperate ticket. My problem is no longer on topic.