Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tgi correct clear implementation #609

Merged
merged 3 commits into from
May 27, 2024
Merged

Tgi correct clear implementation #609

merged 3 commits into from
May 27, 2024

Commits on May 27, 2024

  1. fix(tgi): allow wrapper script to catch SIGTERM

    @dacorvo
    dacorvo committed May 27, 2024
    Configuration menu
    Copy the full SHA
    cf838d0 View commit details
    Browse the repository at this point in the history

  2. fix(tgi): bogus max_new_tokens with static batching

    @dacorvo
    dacorvo committed May 27, 2024
    Configuration menu
    Copy the full SHA
    99f1937 View commit details
    Browse the repository at this point in the history

  3. fix(tgi): allow clearing requests from a single batch 

    When all requests from a prefill batch are cancelled, the router will not send
a filter request, but rather a clear cache request with the batch_id.
We previously ignored that value and cleared everything.
    @dacorvo
    dacorvo committed May 27, 2024
    Configuration menu
    Copy the full SHA
    58eafcb View commit details
    Browse the repository at this point in the history