[Usage]: how to use openai compatible api to run GGUF model?

[weiminw]

Your current environment

The output of `python collect_env.py`

How would you like to use vllm

I want to run inference of a [specific model](put link here). I don't know how to integrate it with vllm.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[bhavnicksm]

Hey @weiminw,
If you pass the GGUF Quantized model path directly to the OpenAI compatible API, it should be able to detect the quantization to use with it and run.

Let me know if you are seeing something else.

[IgorBeHolder]

Hey @weiminw, If you pass the GGUF Quantized model path directly to the OpenAI compatible API, it should be able to detect the quantization to use with it and run.

Let me know if you are seeing something else.

Hi,
I tried but

rag-vllm-1  |     raise EnvironmentError(
rag-vllm-1  | OSError: mradermacher/Nemo-12B-Marlin-v7-GGUF does not appear to have a file named config.json. Checkout 'https://huggingface.co/mradermacher/Nemo-12B-Marlin-v7-GGUF/tree/main' for available files.

GGUF format has all metadata, isnt it?
Why did vllm ask for config.json?

[paolovic]

The same holds for me as also described in here
#4416

When trying to load a GGUF model, e.g., https://huggingface.co/bartowski/reader-lm-1.5b-GGUF , vLLM requires a config.json although the new (?) GGUF quantized models come only with an .imatrix

OSError: /reader-lm-1.5b-GGUF/ does not appear to have a file named config.json. Checkout 'https://huggingface.co//u01/app/mlo/models/reader-lm-1.5b-GGUF//tree/Nles.

[bhavnicksm]

Hey @paolovic,

Yes, this error occurs because vLLM is currently not looking for .gguf files inside the folder but instead assumes you pass the model as the .gguf weight path. Additionally, you would also need to pass the tokenizer as the path to the tokenizer.

I have tested this for v0.6.1.post2 and it works properly with the .gguf weights path.

llm =  LLM(model = "/path/to/model.gguf", tokenizer="/path/to/tokenizer")

cc: @IgorBeHolder @weiminw

[paolovic]

Hey @paolovic,

Yes, this error occurs because vLLM is currently not looking for .gguf files inside the folder but instead assumes you pass the model as the .gguf weight path. Additionally, you would also need to pass the tokenizer as the path to the tokenizer.

I have tested this for v0.6.1.post2 and it works properly with the .gguf weights path.

llm =  LLM(model = "/path/to/model.gguf", tokenizer="/path/to/tokenizer")

cc: @IgorBeHolder @weiminw

ahhhh....ok, easy
thank you very much, somehow I overlooked this information! @bhavnicksm