Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add an AutoTokenizer #1994

Merged
merged 122 commits into from
Nov 20, 2023
Merged

Add an AutoTokenizer #1994

merged 122 commits into from
Nov 20, 2023

Conversation

JosselinSomervilleRoberts
Copy link
Contributor

@JosselinSomervilleRoberts JosselinSomervilleRoberts commented Nov 9, 2023 •
edited
Loading

As described in the title, AutoClient.tokenize, AutoClient.decode and AutoClient._get_tokenizer are now raising errors and should be replaced with calls to AutoTokenizer.

This shows a lot of commits but builds on top of #1903 that's why

@JosselinSomervilleRoberts
Add the Tokenizer object logic
e6b9729
@JosselinSomervilleRoberts
Changed most clients to use a Tokenizer object
38784f7
@JosselinSomervilleRoberts
Changed remaining clients to use a Tokenizer object
daf431b
@JosselinSomervilleRoberts
Removed calls to CachableClient.tokenize()
cfd0c15
@JosselinSomervilleRoberts
Add TODOs
4bf3c46
@JosselinSomervilleRoberts
Make client methods abstract
d6341ef
@JosselinSomervilleRoberts
Resolve merge conflicts
d2ac135
@JosselinSomervilleRoberts
Fix ICE Tokenizer test
4e83fd2
@JosselinSomervilleRoberts
Fix Critique breaking change
0445a78
@JosselinSomervilleRoberts
Revert fix
e1cbe32
@JosselinSomervilleRoberts
Fix all window service test issues except for Cohere
40337f7
@JosselinSomervilleRoberts
Resolve merge conflicts with HuggingFace refactorization
037a869
@JosselinSomervilleRoberts
Refactor CachableClient -> CachingClient
1077699
@JosselinSomervilleRoberts
Refactor yalm_tokenizer_src -> yalm_tokenizer_data
b0fefef
@JosselinSomervilleRoberts
Merge #1891
0e47750
@JosselinSomervilleRoberts
First draft of the model deployment/metadata refactorization
c1b06e3
@JosselinSomervilleRoberts
Fix one of the TODO
802b2ec
@JosselinSomervilleRoberts
Merge branch 'main' into joss-refactor-1-tokenizer
a907774
@JosselinSomervilleRoberts
CachableTokenizer -> CachingTokenizer
c9aa4fd
@JosselinSomervilleRoberts
Port VLM model Idefics to use new Tokenizer logic
b1badb7
@JosselinSomervilleRoberts
Add TODOs to remove tokenize and decode methods from Client
57fd565
@JosselinSomervilleRoberts
Change methods of CachingTokenizer to preserve existing Cache
50bb454
@JosselinSomervilleRoberts
Change raw_request to request in _tokenization_raw_response_to_tokens…
9876162 
… and _decode_raw_response_to_text
@JosselinSomervilleRoberts
Merge branch 'main' into joss-refactor-1-tokenizer
359994c
@JosselinSomervilleRoberts
First draft of the model deployment/metadata refactorization
4d8c080
@JosselinSomervilleRoberts
Fix one of the TODO
ec97f8d
@JosselinSomervilleRoberts
Merge branch 'joss-refactor-4-deployments' of https://github.com/stan…
068b199 
…ford-crfm/helm into joss-refactor-4-deployments
@JosselinSomervilleRoberts
Fixing black and mypy
427e723
@JosselinSomervilleRoberts
Merge main
15d3cc4
@JosselinSomervilleRoberts
Support model in conf
06d5707
yifanmai
yifanmai requested changes Nov 15, 2023
View reviewed changes
Copy link
Collaborator

@yifanmai yifanmai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can review this now if you change the merge target from main to joss-refactor-4-deployments. You can do this by clicking the "Edit" button in the top right of the PR. Otherwise I can't tell what the intended changes are.

@JosselinSomervilleRoberts JosselinSomervilleRoberts changed the base branch from main to joss-refactor-4-deployments November 17, 2023 23:46
Base automatically changed from joss-refactor-4-deployments to main November 18, 2023 00:37
yifanmai
yifanmai approved these changes Nov 18, 2023
View reviewed changes
src/helm/common/auto_utils.py Outdated Show resolved Hide resolved
@JosselinSomervilleRoberts JosselinSomervilleRoberts merged commit 32342d9 into main Nov 20, 2023
6 checks passed
@JosselinSomervilleRoberts JosselinSomervilleRoberts deleted the joss-refactor-6-auto-tokenizer branch November 20, 2023 19:37
This was referenced Nov 22, 2023
Merged
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yifanmai yifanmai yifanmai approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@JosselinSomervilleRoberts @yifanmai