Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add private configs #1996

Merged
merged 121 commits into from
Dec 11, 2023
Merged

Add private configs #1996

merged 121 commits into from
Dec 11, 2023

Conversation

JosselinSomervilleRoberts
Copy link
Contributor

@JosselinSomervilleRoberts JosselinSomervilleRoberts commented Nov 10, 2023 •
edited
Loading

Follow up to #1903 to add private configs.

@JosselinSomervilleRoberts
Add the Tokenizer object logic
e6b9729
@JosselinSomervilleRoberts
Changed most clients to use a Tokenizer object
38784f7
@JosselinSomervilleRoberts
Changed remaining clients to use a Tokenizer object
daf431b
@JosselinSomervilleRoberts
Removed calls to CachableClient.tokenize()
cfd0c15
@JosselinSomervilleRoberts
Add TODOs
4bf3c46
@JosselinSomervilleRoberts
Make client methods abstract
d6341ef
@JosselinSomervilleRoberts
Resolve merge conflicts
d2ac135
@JosselinSomervilleRoberts
Fix ICE Tokenizer test
4e83fd2
@JosselinSomervilleRoberts
Fix Critique breaking change
0445a78
@JosselinSomervilleRoberts
Revert fix
e1cbe32
@JosselinSomervilleRoberts
Fix all window service test issues except for Cohere
40337f7
@JosselinSomervilleRoberts
Resolve merge conflicts with HuggingFace refactorization
037a869
@JosselinSomervilleRoberts
Refactor CachableClient -> CachingClient
1077699
@JosselinSomervilleRoberts
Refactor yalm_tokenizer_src -> yalm_tokenizer_data
b0fefef
@JosselinSomervilleRoberts
Merge #1891
0e47750
@JosselinSomervilleRoberts
First draft of the model deployment/metadata refactorization
c1b06e3
@JosselinSomervilleRoberts
Fix one of the TODO
802b2ec
@JosselinSomervilleRoberts
Merge branch 'main' into joss-refactor-1-tokenizer
a907774
@JosselinSomervilleRoberts
CachableTokenizer -> CachingTokenizer
c9aa4fd
@JosselinSomervilleRoberts
Port VLM model Idefics to use new Tokenizer logic
b1badb7
@JosselinSomervilleRoberts
Add TODOs to remove tokenize and decode methods from Client
57fd565
@JosselinSomervilleRoberts
Change methods of CachingTokenizer to preserve existing Cache
50bb454
@JosselinSomervilleRoberts
Change raw_request to request in _tokenization_raw_response_to_tokens…
9876162 
… and _decode_raw_response_to_text
@JosselinSomervilleRoberts
Merge branch 'main' into joss-refactor-1-tokenizer
359994c
@JosselinSomervilleRoberts
First draft of the model deployment/metadata refactorization
4d8c080
@JosselinSomervilleRoberts
Fix one of the TODO
ec97f8d
@JosselinSomervilleRoberts
Merge branch 'joss-refactor-4-deployments' of https://github.com/stan…
068b199 
…ford-crfm/helm into joss-refactor-4-deployments
@JosselinSomervilleRoberts
Fixing black and mypy
427e723
@JosselinSomervilleRoberts
Merge main
15d3cc4
@JosselinSomervilleRoberts
Support model in conf
06d5707
@JosselinSomervilleRoberts
Use importlib so that local paths work on a pypi install
3f97b6d
@JosselinSomervilleRoberts
Add default model metadata registration for huggingface models
7cecdbd
@JosselinSomervilleRoberts
Changing Request so that model and model_deployment are always both f…
a8d931b 
…illed
@JosselinSomervilleRoberts
Fix test server service
57431f1
@JosselinSomervilleRoberts
Update tutorial
67252bc
@yifanmai @JosselinSomervilleRoberts
Alternative model deployment proposal (#2002)
abf9d0a 
Co-authored-by: JosselinSomervilleRoberts <josselin.somerville@gmail.com>
@JosselinSomervilleRoberts
Merge main
6ba0199
@JosselinSomervilleRoberts
Fix helm-run and a few tests
4b849a7
@JosselinSomervilleRoberts
Merge branch 'main' into joss-refactor-4-deployments
5749a8f
@JosselinSomervilleRoberts
Fix broken test
6f8a13c
@JosselinSomervilleRoberts
Small fixes to the configs
e7eb250
@JosselinSomervilleRoberts
Fix Mistral #1998
f177503
@JosselinSomervilleRoberts
Fix files that were still not speciying both model and deployment in …
a3323bd 
…request
@JosselinSomervilleRoberts
Fix some mypy issues
c2d3e03
@JosselinSomervilleRoberts
Merge branch 'main' into joss-refactor-4-deployments
5ce5648
@JosselinSomervilleRoberts
Merge branch 'joss-refactor-4-deployments' into joss-refactor-8-private
0959f1d
@JosselinSomervilleRoberts JosselinSomervilleRoberts changed the base branch from main to joss-refactor-4-deployments November 18, 2023 00:00
Base automatically changed from joss-refactor-4-deployments to main November 18, 2023 00:37
@yifanmai yifanmai mentioned this pull request Nov 30, 2023
Merged
yifanmai
yifanmai requested changes Dec 4, 2023
View reviewed changes
Copy link
Collaborator

@yifanmai yifanmai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks mostly good. Just a few changes needed. Thanks!

src/helm/benchmark/model_metadata_registry.py Show resolved Hide resolved
src/helm/benchmark/model_deployment_registry.py Show resolved Hide resolved
src/helm/benchmark/model_metadata_registry.py Show resolved Hide resolved
src/helm/benchmark/config_registry.py Show resolved Hide resolved
@yifanmai yifanmai mentioned this pull request Dec 7, 2023
Closed
yifanmai
yifanmai approved these changes Dec 7, 2023
View reviewed changes
Copy link
Collaborator

@yifanmai yifanmai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One change suggested, looks good otherwise!

src/helm/benchmark/run.py
for model_metadata_path in args.model_metadata_paths:
register_model_metadata_from_path(model_metadata_path)
for model_deployment_paths in args.model_deployment_paths:
register_model_deployments_from_path(model_deployment_paths)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you keep these flags for now because it looks like some people are using it e.g. #2110

@RawthiL
Copy link

RawthiL commented Dec 7, 2023

Hi, I'm here from PR #2110 , I tried to install this branch (on a docker image)
pip install git+https://github.com/stanford-crfm/helm.git@joss-refactor-8-private

But I cannot execute the helm-run, I get an error saying ModuleNotFoundError: No module named 'helm.config'. Were there any requirements changes?

@yifanmai yifanmai merged commit af55ff5 into main Dec 11, 2023
9 checks passed
@yifanmai yifanmai deleted the joss-refactor-8-private branch December 11, 2023 22:18
@yifanmai
Copy link
Collaborator

But I cannot execute the helm-run, I get an error saying ModuleNotFoundError: No module named 'helm.config'. Were there any requirements changes?

I believe this was broken by #1903 and fixed in #2021

For the record, we're continuing this discussion in #2110.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yifanmai yifanmai yifanmai approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@JosselinSomervilleRoberts @RawthiL @yifanmai