Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement Speculative Decoding #242

Merged
merged 79 commits into from
May 11, 2024
Merged

Implement Speculative Decoding #242

merged 79 commits into from
May 11, 2024

Commits on Apr 28, 2024

  1. Temp

    @EricLBuehler
    EricLBuehler committed Apr 28, 2024
    Configuration menu
    Copy the full SHA
    e9610e4 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into speculative

    @EricLBuehler
    EricLBuehler committed Apr 28, 2024
    Configuration menu
    Copy the full SHA
    097b8e0 View commit details
    Browse the repository at this point in the history

  3. Temp

    @EricLBuehler
    EricLBuehler committed Apr 28, 2024
    Configuration menu
    Copy the full SHA
    63c4c86 View commit details
    Browse the repository at this point in the history

Commits on Apr 30, 2024

  1. Merge branch 'master' into speculative

    @EricLBuehler
    EricLBuehler committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    3871180 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into speculative

    @EricLBuehler
    EricLBuehler committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    fa01657 View commit details
    Browse the repository at this point in the history

  3. Use arc for chat template

    @EricLBuehler
    EricLBuehler committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    945e985 View commit details
    Browse the repository at this point in the history

  4. Update reset_non_granular_state

    @EricLBuehler
    EricLBuehler committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    462dea3 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'master' into speculative

    @EricLBuehler
    EricLBuehler committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    aeb2460 View commit details
    Browse the repository at this point in the history

  6. Begin abstraction of sampling

    @EricLBuehler
    EricLBuehler committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    ec24cf2 View commit details
    Browse the repository at this point in the history

  7. Begin abstraction of sampling

    @EricLBuehler
    EricLBuehler committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    e7f36ca View commit details
    Browse the repository at this point in the history

Commits on May 1, 2024

  1. Abstract sampling process

    @EricLBuehler
    EricLBuehler committed May 1, 2024
    Configuration menu
    Copy the full SHA
    72b5755 View commit details
    Browse the repository at this point in the history

  2. Abstract sampling process

    @EricLBuehler
    EricLBuehler committed May 1, 2024
    Configuration menu
    Copy the full SHA
    4dc5db1 View commit details
    Browse the repository at this point in the history

  3. Almost there!

    @EricLBuehler
    EricLBuehler committed May 1, 2024
    Configuration menu
    Copy the full SHA
    cc79226 View commit details
    Browse the repository at this point in the history

  4. It compiles

    @EricLBuehler
    EricLBuehler committed May 1, 2024
    Configuration menu
    Copy the full SHA
    ae4877a View commit details
    Browse the repository at this point in the history

  5. Implement the rest of the todos

    @EricLBuehler
    EricLBuehler committed May 1, 2024
    Configuration menu
    Copy the full SHA
    07caf41 View commit details
    Browse the repository at this point in the history

  6. Implement the rest of the todos

    @EricLBuehler
    EricLBuehler committed May 1, 2024
    Configuration menu
    Copy the full SHA
    5fb731a View commit details
    Browse the repository at this point in the history

  7. Use cache instructions

    @EricLBuehler
    EricLBuehler committed May 1, 2024
    Configuration menu
    Copy the full SHA
    5adef60 View commit details
    Browse the repository at this point in the history

  8. Clippy

    @EricLBuehler
    EricLBuehler committed May 1, 2024
    Configuration menu
    Copy the full SHA
    d228757 View commit details
    Browse the repository at this point in the history

Commits on May 2, 2024

  1. Add to server api

    @EricLBuehler
    EricLBuehler committed May 2, 2024
    Configuration menu
    Copy the full SHA
    254e405 View commit details
    Browse the repository at this point in the history

  2. Update cache manager

    @EricLBuehler
    EricLBuehler committed May 2, 2024
    Configuration menu
    Copy the full SHA
    6bb0971 View commit details
    Browse the repository at this point in the history

  3. Remove manual rt

    @EricLBuehler
    EricLBuehler committed May 2, 2024
    Configuration menu
    Copy the full SHA
    1bf81b2 View commit details
    Browse the repository at this point in the history

  4. Partially working

    @EricLBuehler
    EricLBuehler committed May 2, 2024
    Configuration menu
    Copy the full SHA
    1a468bf View commit details
    Browse the repository at this point in the history

  5. Rewrite it

    @EricLBuehler
    EricLBuehler committed May 2, 2024
    Configuration menu
    Copy the full SHA
    0da46fa View commit details
    Browse the repository at this point in the history

  6. Set cache to none

    @EricLBuehler
    EricLBuehler committed May 2, 2024
    Configuration menu
    Copy the full SHA
    303eb5d View commit details
    Browse the repository at this point in the history

  7. Disable prefix cache here

    @EricLBuehler
    EricLBuehler committed May 2, 2024
    Configuration menu
    Copy the full SHA
    bf548a8 View commit details
    Browse the repository at this point in the history

  8. Format

    @EricLBuehler
    EricLBuehler committed May 2, 2024
    Configuration menu
    Copy the full SHA
    880b0f1 View commit details
    Browse the repository at this point in the history

  9. Clippy

    @EricLBuehler
    EricLBuehler committed May 2, 2024
    Configuration menu
    Copy the full SHA
    73b0cd8 View commit details
    Browse the repository at this point in the history

  10. Merge branch 'master' into speculative

    @EricLBuehler
    EricLBuehler committed May 2, 2024
    Configuration menu
    Copy the full SHA
    9a8d22e View commit details
    Browse the repository at this point in the history

  11. Narrow target model kv cache

    @EricLBuehler
    EricLBuehler committed May 2, 2024
    Configuration menu
    Copy the full SHA
    dd4fa6a View commit details
    Browse the repository at this point in the history

  12. Only narrow if non prompt

    @EricLBuehler
    EricLBuehler committed May 2, 2024
    Configuration menu
    Copy the full SHA
    1e625cc View commit details
    Browse the repository at this point in the history

  13. Small fixes

    @EricLBuehler
    EricLBuehler committed May 2, 2024
    Configuration menu
    Copy the full SHA
    f2af3ef View commit details
    Browse the repository at this point in the history

  14. Set tmp tok

    @EricLBuehler
    EricLBuehler committed May 2, 2024
    Configuration menu
    Copy the full SHA
    690cec1 View commit details
    Browse the repository at this point in the history

  15. Clamp

    @EricLBuehler
    EricLBuehler committed May 2, 2024
    Configuration menu
    Copy the full SHA
    0d78dfb View commit details
    Browse the repository at this point in the history

  16. Merge

    @EricLBuehler
    EricLBuehler committed May 2, 2024
    Configuration menu
    Copy the full SHA
    311f48c View commit details
    Browse the repository at this point in the history

Commits on May 4, 2024

  1. Fix adding and removing the last token

    @EricLBuehler
    EricLBuehler committed May 4, 2024
    Configuration menu
    Copy the full SHA
    621cf5d View commit details
    Browse the repository at this point in the history

  2. Update

    @EricLBuehler
    EricLBuehler committed May 4, 2024
    Configuration menu
    Copy the full SHA
    72e0cb5 View commit details
    Browse the repository at this point in the history

  3. So close

    @EricLBuehler
    EricLBuehler committed May 4, 2024
    Configuration menu
    Copy the full SHA
    50dc127 View commit details
    Browse the repository at this point in the history

  4. Update with no kv cache for target

    @EricLBuehler
    EricLBuehler committed May 4, 2024
    Configuration menu
    Copy the full SHA
    86e7188 View commit details
    Browse the repository at this point in the history

  5. Got it to work!

    @EricLBuehler
    EricLBuehler committed May 4, 2024
    Configuration menu
    Copy the full SHA
    4d40453 View commit details
    Browse the repository at this point in the history

Commits on May 5, 2024

  1. Clippy and add kv cache

    @EricLBuehler
    EricLBuehler committed May 5, 2024
    Configuration menu
    Copy the full SHA
    7da675b View commit details
    Browse the repository at this point in the history

  2. Rewind kv cache of draft model

    @EricLBuehler
    EricLBuehler committed May 5, 2024
    Configuration menu
    Copy the full SHA
    a2c8684 View commit details
    Browse the repository at this point in the history

  3. Clone in and out draft cache

    @EricLBuehler
    EricLBuehler committed May 5, 2024
    Configuration menu
    Copy the full SHA
    4a0feca View commit details
    Browse the repository at this point in the history

  4. Proper source and dst for cache ops

    @EricLBuehler
    EricLBuehler committed May 5, 2024
    Configuration menu
    Copy the full SHA
    bb41f77 View commit details
    Browse the repository at this point in the history

  5. Fix

    @EricLBuehler
    EricLBuehler committed May 5, 2024
    Configuration menu
    Copy the full SHA
    6f37f47 View commit details
    Browse the repository at this point in the history

  6. Fix

    @EricLBuehler
    EricLBuehler committed May 5, 2024
    Configuration menu
    Copy the full SHA
    fb2b6fd View commit details
    Browse the repository at this point in the history

  7. Fix it

    @EricLBuehler
    EricLBuehler committed May 5, 2024
    Configuration menu
    Copy the full SHA
    94a2fb0 View commit details
    Browse the repository at this point in the history

  8. Fix n not accepted

    @EricLBuehler
    EricLBuehler committed May 5, 2024
    Configuration menu
    Copy the full SHA
    6033230 View commit details
    Browse the repository at this point in the history

  9. Slight refactor and add new entrypt

    @EricLBuehler
    EricLBuehler committed May 5, 2024
    Configuration menu
    Copy the full SHA
    085fe95 View commit details
    Browse the repository at this point in the history

  10. Refactor causal mask generation

    @EricLBuehler
    EricLBuehler committed May 5, 2024
    Configuration menu
    Copy the full SHA
    a5b6180 View commit details
    Browse the repository at this point in the history

  11. Fix dtypes

    @EricLBuehler
    EricLBuehler committed May 5, 2024
    Configuration menu
    Copy the full SHA
    72c43a9 View commit details
    Browse the repository at this point in the history

  12. Fix deadlock

    @EricLBuehler
    EricLBuehler committed May 5, 2024
    Configuration menu
    Copy the full SHA
    d75722c View commit details
    Browse the repository at this point in the history

Commits on May 6, 2024

  1. Sample with gumbel for speculative step

    @EricLBuehler
    EricLBuehler committed May 6, 2024
    Configuration menu
    Copy the full SHA
    6dd27ec View commit details
    Browse the repository at this point in the history

  2. Use argmax

    @EricLBuehler
    EricLBuehler committed May 6, 2024
    Configuration menu
    Copy the full SHA
    5cbbc53 View commit details
    Browse the repository at this point in the history

  3. Fix len and broadcast div

    @EricLBuehler
    EricLBuehler committed May 6, 2024
    Configuration menu
    Copy the full SHA
    7777648 View commit details
    Browse the repository at this point in the history

  4. Fix seqlen when using tmp toks

    @EricLBuehler
    EricLBuehler committed May 6, 2024
    Configuration menu
    Copy the full SHA
    8f311f2 View commit details
    Browse the repository at this point in the history

  5. Narrow caches correctly

    @EricLBuehler
    EricLBuehler committed May 6, 2024
    Configuration menu
    Copy the full SHA
    f67b9e9 View commit details
    Browse the repository at this point in the history

  6. Add toml selecter interface

    @EricLBuehler
    EricLBuehler committed May 6, 2024
    Configuration menu
    Copy the full SHA
    7875fd8 View commit details
    Browse the repository at this point in the history

  7. Add initial config file

    @EricLBuehler
    EricLBuehler committed May 6, 2024
    Configuration menu
    Copy the full SHA
    fdaa712 View commit details
    Browse the repository at this point in the history

  8. Fix deser

    @EricLBuehler
    EricLBuehler committed May 6, 2024
    Configuration menu
    Copy the full SHA
    3bc6405 View commit details
    Browse the repository at this point in the history

  9. Fix filename

    @EricLBuehler
    EricLBuehler committed May 6, 2024
    Configuration menu
    Copy the full SHA
    d350121 View commit details
    Browse the repository at this point in the history

  10. Fixes

    @EricLBuehler
    EricLBuehler committed May 6, 2024
    Configuration menu
    Copy the full SHA
    21e6d3e View commit details
    Browse the repository at this point in the history

  11. Add same gguf toml

    @EricLBuehler
    EricLBuehler committed May 6, 2024
    Configuration menu
    Copy the full SHA
    cc2f60a View commit details
    Browse the repository at this point in the history

Commits on May 8, 2024

  1. Merge

    @EricLBuehler
    EricLBuehler committed May 8, 2024
    Configuration menu
    Copy the full SHA
    f5c9970 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into speculative

    @EricLBuehler
    EricLBuehler committed May 8, 2024
    Configuration menu
    Copy the full SHA
    9fe7591 View commit details
    Browse the repository at this point in the history

Commits on May 9, 2024

  1. Cache last (not working yet

    @EricLBuehler
    EricLBuehler committed May 9, 2024
    Configuration menu
    Copy the full SHA
    b382602 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into speculative

    @EricLBuehler
    EricLBuehler committed May 9, 2024
    Configuration menu
    Copy the full SHA
    19afd58 View commit details
    Browse the repository at this point in the history

  3. Use causal masker

    @EricLBuehler
    EricLBuehler committed May 9, 2024
    Configuration menu
    Copy the full SHA
    482859f View commit details
    Browse the repository at this point in the history

  4. Merge branch 'master' into speculative

    @EricLBuehler
    EricLBuehler committed May 9, 2024
    Configuration menu
    Copy the full SHA
    bf8878a View commit details
    Browse the repository at this point in the history

  5. Merge

    @EricLBuehler
    EricLBuehler committed May 9, 2024
    Configuration menu
    Copy the full SHA
    4adbd6d View commit details
    Browse the repository at this point in the history

  6. Merge branch 'master' into speculative

    @EricLBuehler
    EricLBuehler committed May 9, 2024
    Configuration menu
    Copy the full SHA
    281040a View commit details
    Browse the repository at this point in the history

Commits on May 10, 2024

  1. Merge branch 'master' into speculative

    @EricLBuehler
    EricLBuehler committed May 10, 2024
    Configuration menu
    Copy the full SHA
    2ab9d3b View commit details
    Browse the repository at this point in the history

  2. It works

    @EricLBuehler
    EricLBuehler committed May 10, 2024
    Configuration menu
    Copy the full SHA
    830478e View commit details
    Browse the repository at this point in the history

  3. It works

    @EricLBuehler
    EricLBuehler committed May 10, 2024
    Configuration menu
    Copy the full SHA
    9d78b4e View commit details
    Browse the repository at this point in the history

  4. Add speculative api to runner

    @EricLBuehler
    EricLBuehler committed May 10, 2024
    Configuration menu
    Copy the full SHA
    8fa467b View commit details
    Browse the repository at this point in the history

Commits on May 11, 2024

  1. Docs

    @EricLBuehler
    EricLBuehler committed May 11, 2024
    Configuration menu
    Copy the full SHA
    7b8ac2a View commit details
    Browse the repository at this point in the history

  2. Fix

    @EricLBuehler
    EricLBuehler committed May 11, 2024
    Configuration menu
    Copy the full SHA
    cfc9a0c View commit details
    Browse the repository at this point in the history

  3. Fix

    @EricLBuehler
    EricLBuehler committed May 11, 2024
    Configuration menu
    Copy the full SHA
    74a9d0d View commit details
    Browse the repository at this point in the history

  4. Fix deadlock

    @EricLBuehler
    EricLBuehler committed May 11, 2024
    Configuration menu
    Copy the full SHA
    0b3ba2c View commit details
    Browse the repository at this point in the history

  5. More masking fixes

    @EricLBuehler
    EricLBuehler committed May 11, 2024
    Configuration menu
    Copy the full SHA
    d630c4a View commit details
    Browse the repository at this point in the history