Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use pre-downloaded models with the Transformers backend #3776

Closed
joshbtn opened this issue Oct 9, 2024 · 1 comment · Fixed by #3777
Closed

Use pre-downloaded models with the Transformers backend #3776

joshbtn opened this issue Oct 9, 2024 · 1 comment · Fixed by #3777
Labels
enhancement New feature or request

Comments

@joshbtn
Copy link
Contributor

joshbtn commented Oct 9, 2024

Is your feature request related to a problem? Please describe.
I have a use case where I pre-download the models to be used with the transformers backend and drop them into a my models directory. My server does not have access to the internet. When trying to specify the model for transformers to use it tries to download the model rather than finding the model that already exists in the file system.

Describe the solution you'd like
I'd like to detect if the model has already been downloaded and tell transformers to use that path.

Describe alternatives you've considered
I considered passing the full path as the model name but this fails validation, both in the model yaml config and when trying to pass it in a request to the completions endpoint.

Additional context
I'd like this to work with the transformers backend.

name: some-model
backend: transformers
parameters:
    model: "some-model-from-huggingface"
type: AutoModelForCausalLM
...

This was a work around I tried, however if fails on some validation of the model name in the API. This would have worked had the model name made it's way all the way to the transformers backend. Seeing as there's already a ModelFile provided as part of the request object it seems like we should check that first rather than changing the model name validation.

name: some-model
backend: transformers
parameters:
    model: "/full/path/to/some-model-from-huggingface"
type: AutoModelForCausalLM
...
@joshbtn joshbtn added the enhancement New feature or request label Oct 9, 2024
@joshbtn
Copy link
Contributor Author

joshbtn commented Oct 9, 2024

This may be related to, or even fix this issue #3594

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
1 participant