Using Zluda for running Fooocus on AMD GPUs on Windows (instead of DirectML)

[Grey3016]

Firstly, this guide is more for current users of ZLuda on SDNext or elsewhere (or new fork of Forge with ZLuda). The main errors in it not working in my observation is the setting of PATHs - it must be correct or it goes tits up pronto.
I would advise using Notepad ++ for editing the python files, as it's far easier to ensure code is aligned (misalignment of code can also mess it up).
Given the extra installs at the start it's not practical to fork this - I lack the python skills.

1.If ZLuda runs ok on SDNext following https://github.com/vladmandic/automatic/wiki/ZLUDA

• you have installed Visual C++
• you have installed the proper HIP for your card (important)
• you have downloaded/unzipped ZLuda to a folder, mine ran with v3.5 but v3.71 is also stable https://github.com/lshqqytiger/ZLUDA/releases/
• you have set the PATHs https://github.com/brknsoul/ROCmLibs/wiki/Adding-folders-to-PATH
• note the section on disabling your iGpu

2.Download Fooocus & unzip https://github.com/lllyasviel/Fooocus

3.Navigate to "Fooocus/ldm_patched/modules/"

4.Open "model_management.py" in preferably notepad ++

5.Find this line (line 216)
FORCE_FP32 = False

6.Change line to FORCE_FP32 = True
(this allows the ChatGPT2 / Fooocus Expansion V2 to work - you can if you want alternately add a startup argument for this)

7.Find these lines (259-262)

try:
    print("Device:", get_torch_device_name(get_torch_device()))
except:
    print("Could not pick default device.")

8.Change the lines above by copy/pasting the following over them (this stop errors for cuda functions that don't work yet, enable bits that do and disables pytorch attention as ZLuda doesn't support it )

try:
    torch_device_name = get_torch_device_name(get_torch_device())

    if "[ZLUDA]" in torch_device_name:
        print("Detected ZLUDA, support for it is experimental and Fooocus may not work properly.")

        if torch.backends.cudnn.enabled:
            torch.backends.cudnn.enabled = False
            print("Disabling cuDNN because ZLUDA does currently not support it.")

        torch.backends.cuda.enable_flash_sdp(False)
        torch.backends.cuda.enable_math_sdp(True)
        torch.backends.cuda.enable_mem_efficient_sdp(False)

        if ENABLE_PYTORCH_ATTENTION:
            print("Disabling pytorch cross attention because ZLUDA does currently not support it.")
            ENABLE_PYTORCH_ATTENTION = False

    print("Device:", torch_device_name)
except:
    print("Could not pick default device.")

9.Install needed torch requirements after uninstalling the ones that don't work - open cmd window from main Fooocus folder with Imbeded folder

.\python_embeded\python.exe -m pip uninstall torch torchvision -y
.\python_embeded\python.exe -m pip install torch==2.2.0 torchvision --index-url https://download.pytorch.org/whl/cu118

10.As per the older ZLuda instructions replace cublas64_11.dll, cusparse64_11.dll, and nvrtc64_112_0.dll in "python_embeded\Lib\site-packages\torch\lib" with the ones from the ZLUDA
folder (replace and rename - checkout out the ZLuda wiki).

11. Start Fooocus by any of the 3 bat files, it should not need --use-zluda to start . It will download the files it wants for each of the 3 bat files as needed.

12. PATHs checks, open a cmd window and type
    zluda
    hipinfo

13. Start by running any of the startup up bat files (edited in as this was missing). No startup arguments needed - add Fooocus ones if you wish

'ZLuda' should return text about its arguments
HIPinfo should return details of the installed HIP files - if it doesn't, you have an 'issue' with your PATH to them

---

This was another proof on concept project for me - it's far slower (by its/s anyway or use Lightning models) than SDNext on my 7900xtx (SDNext 7.5its/s & Fooocus 1.2 - 2.2 its/s depending on speed setting).

Speed Increase
Use Turbo, Lightning etc models
On step 6, leave it at fp16 BUT disable Fooocus Expansion V2 (or it crashes)

ALL credit and kudos to Vosen, lshqqytiger, BrknSoul & LeagueRaNi

[Dog-brother0113]

Your article is simply amazing. I am currently researching how Zluda works in various software I use, and you are like the lighthouse of night navigation. Thank you.

[Grey3016]

You are welcome.

[wtnwin]

@Grey3016 OSError: [WinError 126] The specified module could not be found. Error loading "E:\Fooocus\python_embeded\lib\site-packages\torch\lib\caffe2_nvrtc.dll" or one of its dependencies.
on 6700XT GPU while sd.next does work as instrunction followed.

plus zluda folder only has cublas.dll, cusparse.dll, and nvrtc64.dll, when these replace cublas64_11.dll, cusparse64_11.dll, and nvrtc64_112_0.dll in "python_embeded\Lib\site-packages\torch\lib" , do I need to rename change ublas.dll, cusparse.dll, and nvrtc64.dll to cublas64_11.dll, cusparse64_11.dll, and nvrtc64_112_0.dll in "python_embeded\Lib\site-packages\torch\lib"?

[Grey3016]

It looks like you have the wrong ZLuda zip downloaded, after unzipping the folder should have about 8-10 files as I recall. Try downloading v3.5 which definitely works with Fooocus (I'm away from my pc for a while).
And yes to your final question - same files as per SDnext.

[wtnwin]

Thx for help, I did rename change cublas.dll, cusparse.dll, and nvrtc64.dll to cublas64_11.dll, cusparse64_11.dll, and nvrtc64_112_0.dll in "python_embeded\Lib\site-packages\torch\lib" to resolve problem.

After success in running run.bat, should I use --use-zluda as arg to run run.bat to allow zluda to work?

How to share the model with sd.next? so both sd UI can share the same model.

[Grey3016]

The --use-zluda argyment is not needed for this, the original method did but this version allows the usage of other Fooocus startup arguments - as a side note, I couldn't get it to work quicker with the ones I tried. I tried as many of the tools (advanced menus) as I could and it all worked (slow but it works).

Sharing folders appears to be a bit of saga, you can symlink or someone says how to do it here #236

[wtnwin]

loading in lowvram mode 6935.72490978241
[Fooocus Model Management] Moving model(s) has taken 25.50 seconds
0%|
...
return torch.nn.functional.linear(input, weight, bias)
RuntimeError: CUDA error: CUBLAS_STATUS_NOT_SUPPORTED when calling cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)
Total time: 229.42 seconds

Once it takes full ram and varam, it switches to lowvram mode, then CUDA error above happens.

[Grey3016]

Ahh, I have a 7900, so it wouldn't have triggered the lowvram for me. I don't know if there are other arguments that could help with this, trying out ZLuda 3.71 might also help - I can't force lower vram to test it sorry.

[f1am3d]

I got this:

RuntimeError: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx

[f1am3d]

Other Rocm projects are working (ex. koboldcpp_rocm), and zluda too (rendering in blender with cuda on AMD gpu).
Only this doesn't work.

[Grey3016]

You've asked several things here and given no gpu and sparse info, I can only say that it definitely works for newer AMD cards, if you are using something old, then you must ensure you have the correct HIP files installed & your AMD drivers are up to date -

1. Freezing fault > Did you do 'FORCE_FP32 = True' (step 6) ?
2. The ROCBlas error > does that file exist (TensileLibrary.dat) ? if it doesn't, reinstall the HIP files, if it does, it implies your Paths aren't set correctly .
3. Try this, at a cmd, enter -

ZLuda 
Hipinfo

Both should return something. Returned info on each would imply that Paths have been set correctly.

[f1am3d]

@Grey3016 I have 7900 XTX and did install the latest ROCm HIP SDK (5.7.1).
I had enabled APU, that's why it showed gfx1036, but now I disabled it, but the problem persists.

[f1am3d]

Update:
My model_management.py file was automatically reverted after step #9 for some reason.

[f1am3d]

@Grey3016 now it's working. Thank you for your assistance.

[f1am3d]

I tested twice: the code from steps 5 - 8 is reverted after performing step 9. It's need to change the order.

[Grey3016]

That's the idea, it automatically installed the latest torch for cuda (when you installed Fooocus) and this guide works by you manually uninstalling it and installing the ZLuda compatible torch (step 9).
That's the reason every step is numbered, this isn't a guide for those that decide they're going to do their own thing. Don't run it between steps, step 9 only changes files inside the embeded folder.
As I said it works if the steps are followed.

[f1am3d]

As I said - I followed these steps in exact order and the code was reverted after the execution of step 9.

[Grey3016]

Whatever you have done is outside of the scope of the steps I have provided. I have asked you to carry out checks on which you provided no answers, I will have to terminate this discussion as I'm banging my head on the wall.

[lgwjames]

i spent close to 3 hours doing this on my windows bootcamp and first the run bat just opened and closed, then i go tsomething to do something but loads of errors and just closed

[chejo]

Followed this instruction and had different results:

On the day performing the changes and starting for the first time I ended up with an error I unfortunaely did not make a screenshot of, but it was something like "GPU(0), falling back on CPU" but then it did nothing anymore.

Second day it was just doing nothing for over 1 1/2 hours after i started and entered a prompt (suprisingly the error from day one did not occur)

Today it started fine, loaded the model but then stopped at "preparing prompt" step and after approx. 5 minutes ended up with "Exception Code: 0xC0000094". Also there are several dozens of lines similar to the following:

0x00007FFF760E45D6, \python310.dll(0x00007FFF76060000) + 0x845D6 byte(s), PyType_IsSubtype() + 0x6FE byte(s) 0x00007FFF7607F33C, \python310.dll(0x00007FFF76060000) + 0x1F33C byte(s), _PyEval_EvalFrameDefault() + 0x3F1C byte(s) 0x00007FFF7607A4B7, \python310.dll(0x00007FFF76060000) + 0x1A4B7 byte(s), _PyFunction_Vectorcall() + 0x87 byte(s) 0x00007FFF760A7763, \python310.dll(0x00007FFF76060000) + 0x47763 byte(s), _PyObject_FastCallDictTstate() + 0x6B byte(s)

Does anyone have an idea what I could to to get it working?

I am using a Radeon RX 6800 and a Ryzen 7 5800X

Thanks in advance!

That happened to me because fooocus has auto updated. You need to change the model_management.py file again

[Selos86]

It indeed reverted to the initial file content. I edited the file again and started it and the same errors still occur despite the model_management.py having the Zluda-conform content.

Does anyone have any further ideas?

[Selos86]

Ok, the above errors obviously occured due to Fooocus V2 being checked in the styles tab.

But now there is another error occuring when it tries to start image generation after all models are loaded:

[Fooocus] Preparing task 1/2 ...
[Sampler] refiner_swap_method = joint
[Sampler] sigma_min = 0.0291671771556139, sigma_max = 14.614643096923828
Traceback (most recent call last):
  File "D:\KI\Fooocus\Fooocus\modules\async_worker.py", line 998, in worker
    handler(task)
  File "D:\KI\Fooocus\python_embeded\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "D:\KI\Fooocus\python_embeded\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "D:\KI\Fooocus\Fooocus\modules\async_worker.py", line 888, in handler
    imgs = pipeline.process_diffusion(
  File "D:\KI\Fooocus\python_embeded\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "D:\KI\Fooocus\python_embeded\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "D:\KI\Fooocus\Fooocus\modules\default_pipeline.py", line 372, in process_diffusion
    modules.patch.BrownianTreeNoiseSamplerPatched.global_init(
  File "D:\KI\Fooocus\Fooocus\modules\patch.py", line 198, in global_init
    BrownianTreeNoiseSamplerPatched.tree = BatchedBrownianTree(x, t0, t1, seed, cpu=cpu)
  File "D:\KI\Fooocus\Fooocus\ldm_patched\k_diffusion\sampling.py", line 85, in __init__
    self.trees = [torchsde.BrownianTree(t0, w0, t1, entropy=s, **kwargs) for s in seed]
  File "D:\KI\Fooocus\Fooocus\ldm_patched\k_diffusion\sampling.py", line 85, in <listcomp>
    self.trees = [torchsde.BrownianTree(t0, w0, t1, entropy=s, **kwargs) for s in seed]
  File "D:\KI\Fooocus\python_embeded\lib\site-packages\torchsde\_brownian\derived.py", line 155, in __init__
    self._interval = brownian_interval.BrownianInterval(t0=t0,
  File "D:\KI\Fooocus\python_embeded\lib\site-packages\torchsde\_brownian\brownian_interval.py", line 540, in __init__
    W = self._randn(initial_W_seed) * math.sqrt(t1 - t0)
  File "D:\KI\Fooocus\python_embeded\lib\site-packages\torchsde\_brownian\brownian_interval.py", line 234, in _randn
    return _randn(size, self._top._dtype, self._top._device, seed)
  File "D:\KI\Fooocus\python_embeded\lib\site-packages\torchsde\_brownian\brownian_interval.py", line 33, in _randn
    return torch.randn(size, dtype=dtype, device=device, generator=generator)
**RuntimeError: Expected a 'cuda' device type for generator but found 'cpu'**
Total time: 368.88 seconds

Running "zluda --help" in the commandline returns the following, so I assume registering Zluda to PATH was successful:

C:\>zluda --help
Usage: zluda <exe> [<args...>] [--nccl <nccl>] [--nvrtc <nvrtc>] [--nvcuda <nvcuda>] [--nvml <nvml>] [--nvapi <nvapi>] [--nvoptix <nvoptix>]

Launch application with custom CUDA libraries

Positional Arguments:
  exe               executable to be injected with custom CUDA libraries
  args              arguments to the executable

Options:
  --nccl            DLL to be injected instead of system nccl.dll. If not
                    provided zluda, will use nccl.dll from its own directory
  --nvrtc           DLL to be injected instead of system nvrtc.dll. If not
                    provided zluda, will use nvrtc.dll from its own directory
  --nvcuda          DLL to be injected instead of system nvcuda.dll. If not
                    provided zluda, will use nvcuda.dll from its own directory
  --nvml            DLL to be injected instead of system nvml.dll. If not
                    provided zluda, will use nvml.dll from its own directory
  --nvapi           DLL to be injected instead of system nvapi64.dll. If not
                    provided, no injection will take place
  --nvoptix         DLL to be injected instead of system nvoptix.dll. If not
                    provided, no injection will take place
  --help            display usage information

Hipinfo also returns several information about my graphics card so I guess this is also fine.

After starting Fooocus with zluda, it states in the dialogue:

Detected ZLUDA, support for it is experimental and Fooocus may not work properly.
Disabling cuDNN because ZLUDA does currently not support it.
Disabling pytorch cross attention because ZLUDA does currently not support it.
Device: cuda:0 AMD Radeon RX 6800 [ZLUDA] : native

[Grey3016]

Ok, the above errors obviously occured due to Fooocus V2 being checked in the styles tab.

But now there is another error occuring when it tries to start image generation after all models are loaded:

[Fooocus] Preparing task 1/2 ...
[Sampler] refiner_swap_method = joint
[Sampler] sigma_min = 0.0291671771556139, sigma_max = 14.614643096923828
Traceback (most recent call last):
  File "D:\KI\Fooocus\Fooocus\modules\async_worker.py", line 998, in worker
    handler(task)
  File "D:\KI\Fooocus\python_embeded\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "D:\KI\Fooocus\python_embeded\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "D:\KI\Fooocus\Fooocus\modules\async_worker.py", line 888, in handler
    imgs = pipeline.process_diffusion(
  File "D:\KI\Fooocus\python_embeded\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "D:\KI\Fooocus\python_embeded\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "D:\KI\Fooocus\Fooocus\modules\default_pipeline.py", line 372, in process_diffusion
    modules.patch.BrownianTreeNoiseSamplerPatched.global_init(
  File "D:\KI\Fooocus\Fooocus\modules\patch.py", line 198, in global_init
    BrownianTreeNoiseSamplerPatched.tree = BatchedBrownianTree(x, t0, t1, seed, cpu=cpu)
  File "D:\KI\Fooocus\Fooocus\ldm_patched\k_diffusion\sampling.py", line 85, in __init__
    self.trees = [torchsde.BrownianTree(t0, w0, t1, entropy=s, **kwargs) for s in seed]
  File "D:\KI\Fooocus\Fooocus\ldm_patched\k_diffusion\sampling.py", line 85, in <listcomp>
    self.trees = [torchsde.BrownianTree(t0, w0, t1, entropy=s, **kwargs) for s in seed]
  File "D:\KI\Fooocus\python_embeded\lib\site-packages\torchsde\_brownian\derived.py", line 155, in __init__
    self._interval = brownian_interval.BrownianInterval(t0=t0,
  File "D:\KI\Fooocus\python_embeded\lib\site-packages\torchsde\_brownian\brownian_interval.py", line 540, in __init__
    W = self._randn(initial_W_seed) * math.sqrt(t1 - t0)
  File "D:\KI\Fooocus\python_embeded\lib\site-packages\torchsde\_brownian\brownian_interval.py", line 234, in _randn
    return _randn(size, self._top._dtype, self._top._device, seed)
  File "D:\KI\Fooocus\python_embeded\lib\site-packages\torchsde\_brownian\brownian_interval.py", line 33, in _randn
    return torch.randn(size, dtype=dtype, device=device, generator=generator)
**RuntimeError: Expected a 'cuda' device type for generator but found 'cpu'**
Total time: 368.88 seconds

Can you confirm you have ZLuda working with SDNext ?
Can you confirm you have done EVERY step, in the order they are in ?

I would guess that either the torch wasn't installed or the the 3 files that need to be changed/renamed were not. SDNext now does most of the original steps automatically with scripts, so the instructions will change.
If you are getting errors due to Expansion V2 being turned on means you haven't carried out steps 5/6. I don't know what to say other than it works if the steps are followed.
Mmmm... check that the copy/pasted code in model_management aligns exactly as my original post - no extra spaces, code aligns with code above and below. This is why I mentioned Notepad++ instead of any old text editor, N++ shows the alignment of code blocks .

[Selos86]

I followed the instructions step by step. I renamed the 3 files as suggested (see screenshot) and also steh code in model_management.py is as suggested (see other screenshot)
SD.Next I wanted to try but it does not seem to come with an embedded python and I don't hava a general python installation on this machine.

[cavalia88]

I tried following the instructions but could not get Fooocus to work with zluda. I have zluda working with Automatic1111 and SD_NEXT, so most of the required stuff should already be there.

OSError: [WinError 126] The specified module could not be found. Error loading "E:\Fooocus\python_embeded\lib\site-packages\torch\lib\caffe2_nvrtc.dll" or one of its dependencies.

[cavalia88]

I double checked and confirm that "FORCE_FP32 = True". Still getting the same error. Not sure what else i can try to get it working...probably have to fall back on automatic1111 and sd_NEXT

[Grey3016]

Turn off the Expansion V2 filter before you start the first generation, in case that is playing up.

[cavalia88]

Yes, I have already unchecked "Fooocus V2" in the style tab before running the first generation. Still getting the same error

[Grey3016]

Recheck the copy & pasted code in model_managment - check the code is aligned ie no extra spaces or code not in line with code above or below (this is why I advocated for Notepad+ not any old text editor, it shows the alignment of code).
Other than that, go back and recheck each step. I know it will work on a 6800.

[cavalia88]

Finally managed to get it to work on my 7900XTX. The issue was that the code I pasted previously had reverted back to its original form. I assume this was due to the upgrading of one of the components. So each time there is an upgrade, we will need to copy/paste the code over one more time. Nonetheless, thank you for your patience and advise along the way.

[urostrickovic]

Did anyone solve this problem:
RuntimeError: CUDA error: CUBLAS_STATUS_NOT_SUPPORTED when calling cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)

[Grey3016]

Did anyone solve this problem:

RuntimeError: CUDA error: CUBLAS_STATUS_NOT_SUPPORTED when calling cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)

I think (seems a long time ago now), this issue is caused by the wrong version of ZLuda being used. At the time of writing this, I recall using v1.5 (my HD is now wiped) as it was a datum point (this ZLuda version worked with Fooocus, SDNext and for that version of Forge - all of these have been updated since)