Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix off-by-one error in tokenizer::normalizer::Range::len #1638

Merged
merged 1 commit into from
Oct 14, 2024

Conversation

rlanday
Copy link
Contributor

@rlanday rlanday commented Sep 29, 2024

I was reading through this code and observed an apparent bug. This pull request fixes the bug and adds test cases. Note that the case where the start bound is exclusive is not tested, as range expressions always create ranges with inclusive start bounds.

@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Copy link
Collaborator

@ArthurZucker ArthurZucker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey! Could you rebase on main for the failing rust build.
Otherwise, I am not entirely sure where this is used but logically this makes sense yes!

@rlanday
Copy link
Contributor Author

rlanday commented Oct 1, 2024

Hey! Could you rebase on main for the failing rust build. Otherwise, I am not entirely sure where this is used but logically this makes sense yes!

Done

@rlanday
Copy link
Contributor Author

rlanday commented Oct 12, 2024

Hi @ArthurZucker, do I need to do anything else?

Copy link
Collaborator

@ArthurZucker ArthurZucker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nope sorry!

@ArthurZucker ArthurZucker merged commit 9b77c05 into huggingface:main Oct 14, 2024
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants