Add FLUX nf4 quantization support

[agv-zx82]

Please add FLUX nf4 quantization support

[stduhpf]

I'm not familiar with nf4. From what I managed to understand it's basically just 4 bit fixed point numbers with a range of [-1,1]? Or am I misunderstanding?

[cb88]

I'm not familiar with nf4. From what I managed to understand it's basically just 4 bit fixed point numbers with a range of [-1,1]? Or am I misunderstanding?

Not fixed point apparently. Just a very limited non standard float.

https://huggingface.co/blog/4bit-transformers-bitsandbytes

[stduhpf]

https://www.ai-bites.net/qlora-train-your-llms-on-a-single-gpu/#normalfloat

This looks somewhat similar to GGML's IQ4_NL type in principle? It's not quite the same though.

[Green-Sky]

Pretty sure all was need is the ability to read it and convert to whatever quant the user wants.