You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The current behavior of the llama.cpp LLM pipeline is to always set n_ctx=0. When n_ctx=0, the context size defaults to n_ctx_train which can be very large with some models.
This change will fallback to the default n_ctx when n_ctx=0 fails due to being out of memory. It will also allow n_ctx as a input parameter. If a manually set n_ctx is too large, this will fail since it's user-specified.
The text was updated successfully, but these errors were encountered:
The current behavior of the llama.cpp LLM pipeline is to always set
n_ctx=0
. Whenn_ctx=0
, the context size defaults ton_ctx_train
which can be very large with some models.This change will fallback to the default
n_ctx
whenn_ctx=0
fails due to being out of memory. It will also allown_ctx
as a input parameter. If a manually setn_ctx
is too large, this will fail since it's user-specified.The text was updated successfully, but these errors were encountered: