How to use cpu offload when using LoKr?

[michaelcyshield]

Hello, I have a hard time using cpu offload when loading in the lycoris config, is there anyone who has had the same issue and was able to solve it?
Currently I am getting this error when trying to run inference, I want to try and run flux on bfloat16 with cpu offload instead of int8-quanto but can't seem to get it to work with lycoris
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

[bghira]

it probably won't work that way. you should quantise instead.

[michaelcyshield]

it probably won't work that way. you should quantise instead.

In that case, I see you have a function called quantoise, would it be possible to quantise the transformer with it but not the unet or vice versa?
my main goal here is to try and get my 24gb vram to run on fp16 because fp8 and int8 are too unstable and don't truly offer the quality of flux (at least with my inference notebook) and I am unsure of what to do

[bghira]

fp8 and int8 are equal to bf16, if it's low quality, it's your data and/or settings