Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix memory issue with llama.cpp LLM pipeline #824

Closed
davidmezzetti opened this issue Nov 30, 2024 · 0 comments
Closed

Fix memory issue with llama.cpp LLM pipeline #824

davidmezzetti opened this issue Nov 30, 2024 · 0 comments
Assignees
Labels
bug Something isn't working
Milestone

Comments

@davidmezzetti
Copy link
Member

The current behavior of the llama.cpp LLM pipeline is to always set n_ctx=0. When n_ctx=0, the context size defaults to n_ctx_train which can be very large with some models.

This change will fallback to the default n_ctx when n_ctx=0 fails due to being out of memory. It will also allow n_ctx as a input parameter. If a manually set n_ctx is too large, this will fail since it's user-specified.

@davidmezzetti davidmezzetti added the bug Something isn't working label Nov 30, 2024
@davidmezzetti davidmezzetti added this to the v8.1.0 milestone Nov 30, 2024
@davidmezzetti davidmezzetti self-assigned this Nov 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant