New ggml llamacpp file format support

[ParisNeo]

Hi and thanks for this beautiful work.

Are you planning on supporting the version 2 of the llamacpp files. I want to add the OpenAssistant model to my GPT4ALL-ui and can't find a python binding that supports it.

Here is the bloke's version of the model:
https://huggingface.co/TheBloke/OpenAssistant-SFT-7-Llama-30B-GGML/tree/main

Best regards

[marella]

Hi,

Support for the new quantization format was discussed previously here. Since it is a breaking change I haven't updated yet. There are few more features I'm adding to the Python library after which I will update the C backend.

I will look into adding llama.cpp support over the weekend but don't know if I will be to finish it this week. I'm also waiting for the MPT PR to be merged so that I can add that as well.

Thanks for the model link. Do you have a link to a smaller LLaMA-7B model quantized in the latest format that I can use for testing? My machine doesn't have enough RAM to run larger models.

[ParisNeo]

Of course,
The Bloke has you covered. He has all kinds of models in the new format:
https://huggingface.co/TheBloke

His hugging face models are well organized. Just search for GGML models on his space and you can find anything you want.

The best one is WizardLM-7B-uncensored-GGML, that you can find here:

https://huggingface.co/TheBloke/WizardLM-7B-uncensored-GGML

[marella]

This is released in the latest version 0.2.0

It supports LLaMA, MPT models now. It also includes the most recent breaking change ggerganov/llama.cpp#1508