feat: IQ quants support

[mr-september]

Problem
GGUF models quantized with IQ quants fail to load.

Success Criteria
Load and play as usual

Additional context
IQ quants: ggerganov/llama.cpp#4773

Example model with both traditional Q and new IQ quants: https://huggingface.co/bartowski/Starling_Monarch_Westlake_Garten-7B-v0.1-GGUF

[Van-QA]

hi @mr-september,

• As I tested, the imported gguf model (Starling_Monarch_Westlake_Garten-7B-v0.1-Q2_K) is working. Can you tell us more details about the issue that you are facing?

• If the issue is about broken UI due to long model name, we will resolve it soon 🙏
• In case you need it, here is the guideline on how to import GGUF model: https://jan.ai/docs/models#import-or-symlink-local-models.

Many thanks

[mr-september]

Thanks for the quick reply. I think the models you downloaded are using traditional quants, could you please check with an IQ quant model? For example:

Also, is there some logs or some other outputs I could share which may help with troubleshooting?

[Van-QA]

thank @mr-september,
We were able to reproduce the issue using Starling_Monarch_Westlake_Garten-7B-v0.1-IQ4_XS.gguf. Dev team will investigate the issue soon.

[Van-QA]

hi @mr-september,

Using Jan v0.4.10-368 ✅, the Starling_Monarch_Westlake_Garten-7B-v0.1-IQ4_XS.gguf is able to generate response, would you like to try it as well?

Thank you

[mr-september]

Beautiful, it's working flawlessly! Very impressive turnaround!

[mr-september]

Hi, I think the latest nightly (-376) broke support again. It was a prompted update at startup. Rolling back to -368 still works.

[Van-QA]

hi @mr-september, sorry for the inconvenience, due to the Nitro that supports IQ quants is currently facing many issues. Which we have to temporally revert it, and currently working on a fix atm.
cc: @CameronNg @vansangpfiev

[Van-QA]

hi @mr-september, the latest nightly build Jan v0.4.11-386 resolved the issue with IQ Quant. Thanks.