Corrupted images on Vukan backend

[wbruna]

I'm getting either corrupted or inconsistent images on Vulkan backend, for any resolution other than 512x512.

My system is a Linux PC with a Ryzen 3400G, almost-vanilla Debian 12 (using the distro graphic stack). All following tests with:
--type f16 --lora-model-dir ./LoRA --model ./SD/dreamshaper_8.safetensors --prompt 'a fantasy character, detailed background, colorful<lora:lcm-lora-sdv1-5:1>' --cfg-scale 1.0 --sampling-method lcm --steps 4 --rng cuda --seed 42 -b 1 --color
, and a script alternating resolution and compiled binary (Vulkan or CPU backend).

320x512:

Vulkan 1	Vulkan 2	CPU

Vulkan images look ok-ish (for such a small resolution anyway), but the same seed should produce the same image. And the CPU render looks quite different.

Second test, 384x384; similar behavior (changes between Vulkan 1 and 2 may not be apparent on the thumbnail):

Vulkan 1	Vulkan 2	CPU

The third test, 448x448, gets weird:

Vulkan 1	Vulkan 2	CPU

At first, I blamed my PC drivers. But then, the 512x512 test:

Vulkan 1	Vulkan 2	CPU

Looks absolutely fine, and identical between Vulkan and CPU.

In summary:

• 512x512 works fine;
• any other resolution produces inconsistent images between runs;
• some resolutions introduce artifacts

(related: #122 )

[stduhpf]

Good catch. Using the same prompt, i get a smilar behavior as you.
Though I don't get anything as dramatic as your 448x448 results. (I only get variations of your "Vulkan 1" images, no matter how many times I try).

I might try to investigate what's going on, but I'm not confident I'll figure it out.

[stduhpf]

@0cc4m Do you have a clue?

[stduhpf]

This also happens with images bigger than 512x512 if the resolution isn't a multiple of 128...

[0cc4m]

Thank you for the detailed report, I'll look into it when I find some time.

[jeffbolznv]

I used GGML_VULKAN_CHECK_RESULTS to narrow down that group_norm was failing, reproduced it with a backend test, and pushed a fix to ggerganov/llama.cpp#10496.

BTW, GGML_VULKAN_CHECK_RESULTS is really helpful, but it looks like it may get harder to build with the recent backend split.

[0cc4m]

BTW, GGML_VULKAN_CHECK_RESULTS is really helpful, but it looks like it may get harder to build with the recent backend split.

Yeah, I built it to narrow down these kinds of model issues. I hope we can keep it around.

[wbruna]

I manually applied 56d8a95 on a local build, and it seems to fix this issue. Thanks!