Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Specifying chat history using tokens #77

Closed
minipasila opened this issue Feb 15, 2023 · 2 comments
Closed

Specifying chat history using tokens #77

minipasila opened this issue Feb 15, 2023 · 2 comments
Labels
enhancement New feature or request

Comments

@minipasila
Copy link
Contributor

Since each individual messages can vary in size, setting how many messages it remembers can vary somewhat if you are running low on memory. If you could set a hard limit using tokens, then I could pretty much stop worrying about cuda memory errors.
@oobabooga
Copy link
Owner

That's a very good idea and has been implemented, replacing the old "history size" parameter.

7be3728

You can still OOM if you use a too long context string (character description/world scenario), an that is by construction never truncated.

@oobabooga oobabooga added the enhancement New feature or request label Feb 15, 2023
@oobabooga
Copy link
Owner

I will close this issue, but feel free to comment again if you experience any issues.

@mironkraft mironkraft mentioned this issue May 19, 2023
Closed
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@minipasila @oobabooga