Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

outlines with llamacpp has KeyError over 'n_gpu_layers' #761

Closed
AndreaRiboni opened this issue Mar 20, 2024 · 2 comments
Closed

outlines with llamacpp has KeyError over 'n_gpu_layers' #761

AndreaRiboni opened this issue Mar 20, 2024 · 2 comments
Labels

Comments

@AndreaRiboni
Copy link

AndreaRiboni commented Mar 20, 2024

Describe the issue as clearly as possible:

i have used llamacpp on itself or via langchain with no problems. When loading the same gguf model using outlines' llamacpp wrapper, i get a keyerror

Steps/code to reproduce the bug:

install llamacpp:

`MAKE_ARGS="-DLLAMA_CUBLAS=on" FORCE_CMAKE=1 pip install --upgrade --force-reinstall llama-cpp-python==0.2.43 --no-cache-dir`

install outlines:
`pip install outlines`

load the model:

import outlines
llm = outlines.models.llamacpp(
       model_path,
       device="cuda", 
       model_kwargs={
          'n_ctx': context_size,
          'n_gpu_layers': -1
       }
)

Expected result:

the model should be loaded correctly

Error message:

Traceback (most recent call last):
  File "/usr/src/app/llm-exposer.py", line 56, in <module>
    llm_wrapper = LLMModel(default_model)
  File "/usr/src/app/model.py", line 29, in __init__
    self.load_model()
  File "/usr/src/app/model.py", line 32, in load_model
    self.llm = outlines.models.llamacpp(
  File "/usr/local/lib/python3.10/dist-packages/outlines/models/llamacpp.py", line 29, in llamacpp
    model_kwargs["n_gpu_layers"].setdefault(-1)
KeyError: 'n_gpu_layers'

Outlines/Python version information:

Version information

``` (command output here) ```

Context for the issue:

No response

@isamu-isozaki
Copy link
Contributor

isamu-isozaki commented Mar 26, 2024

I think this is fixed in here. Not sure if the latest version includes it though.
For a quick fix, I recommend just proving n_ctx and n_gpu_layers directly as keyword arguments instead of under model_kwargs

@rlouf
Copy link
Member

rlouf commented Apr 10, 2024

I completely overhauled the llama.cpp integration in #779 and this shouldn't be an issue anymore!

@rlouf rlouf closed this as completed Apr 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants