Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat: Correct context length for models #2867

Merged
merged 19 commits into from
May 6, 2024
Merged

Conversation

hahuyhoang411
Copy link
Contributor

@hahuyhoang411 hahuyhoang411 commented May 4, 2024

Describe Your Changes

  • Correct upper bound for models on the Catalog

Fixes Issues

  • Closes #
  • Closes #

Self Checklist

  • Codeninja 4k -> 8k
  • CmdR 4k -> 130k
  • Package bump

@hahuyhoang411
Copy link
Contributor Author

hahuyhoang411 commented May 5, 2024

I think we need a dynamic sync between upper bound of max_len and ctx_len.

  • My approach right now is to set max_len = ctx_len in the model.json file to display the maximum but we need to handle this dynamically

  • The upper bound of max_len should also be the same as configured ctx_len

  • E.g. ctx_len = 2048, max_len should be 2048

  • Otherwise, it would confuse the user like this:

image

@hahuyhoang411
Copy link
Contributor Author

Thanks @urmauur for the fast work

#2870

@hahuyhoang411 hahuyhoang411 marked this pull request as ready for review May 6, 2024 08:05
@hahuyhoang411 hahuyhoang411 added the type: feature request A new feature label May 6, 2024
@hahuyhoang411 hahuyhoang411 added this to the v0.5.0 Broken Rice milestone May 6, 2024
@Van-QA Van-QA force-pushed the chore/local-model-adjust branch from 981dfd4 to 5bcd8e6 Compare May 6, 2024 10:37
Copy link
Contributor

@Van-QA Van-QA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Van-QA Van-QA merged commit 2008aae into dev May 6, 2024
6 of 7 checks passed
@Van-QA Van-QA deleted the chore/local-model-adjust branch May 6, 2024 11:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

2 participants