Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Llama.cpp example for cpp backend #2904

Merged
merged 19 commits into from
Jan 26, 2024
Merged

Llama.cpp example for cpp backend #2904

merged 19 commits into from
Jan 26, 2024

Commits on Jan 26, 2024

  1. Version1 of llm inference with cpp backend 

    Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

Updating llm handler - loadmodel, preprocess, inference methods

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

Fixed infinite lock by adding request ids to the preprocess method

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

Adding test script for finding tokens per second llama-7b-chat and ggml version

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

GGUF Compatibility

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

Fixing unit tests

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

Fix typo

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

Using folly to read config path

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

Removing debug couts

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

Processing all the items in the batch

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

Adopted llama.cpp api changes
    @shrinath-suresh @mreso
    shrinath-suresh authored and mreso committed Jan 26, 2024
    Configuration menu
    Copy the full SHA
    26abfd7 View commit details
    Browse the repository at this point in the history

  2. Adapt to removal of TS backend

    @mreso
    mreso committed Jan 26, 2024
    Configuration menu
    Copy the full SHA
    94b9c9c View commit details
    Browse the repository at this point in the history

  3. Re-add test for llama.cpp example

    @mreso
    mreso committed Jan 26, 2024
    Configuration menu
    Copy the full SHA
    49649d5 View commit details
    Browse the repository at this point in the history

  4. Add llama.cpp as a submodule

    @mreso
    mreso committed Jan 26, 2024
    Configuration menu
    Copy the full SHA
    5804216 View commit details
    Browse the repository at this point in the history

  5. Point to correct llama.cpp installation

    @mreso
    mreso committed Jan 26, 2024
    Configuration menu
    Copy the full SHA
    6add8da View commit details
    Browse the repository at this point in the history

  6. Build llama.cpp in build.sh

    @mreso
    mreso committed Jan 26, 2024
    Configuration menu
    Copy the full SHA
    ed4114d View commit details
    Browse the repository at this point in the history

  7. Skip llama.cpp example test if model weights are not available

    @mreso
    mreso committed Jan 26, 2024
    Configuration menu
    Copy the full SHA
    4946825 View commit details
    Browse the repository at this point in the history

  8. renamed torchscript_model folder into examples

    @mreso
    mreso committed Jan 26, 2024
    Configuration menu
    Copy the full SHA
    25992e8 View commit details
    Browse the repository at this point in the history

  9. Adjust to new base_handler interface

    @mreso
    mreso committed Jan 26, 2024
    Configuration menu
    Copy the full SHA
    0f00fed View commit details
    Browse the repository at this point in the history

  10. Remove debug statement

    @mreso
    mreso committed Jan 26, 2024
    Configuration menu
    Copy the full SHA
    d50b19b View commit details
    Browse the repository at this point in the history

  11. Rename llamacpp class + remove dummy.pt file

    @mreso
    mreso committed Jan 26, 2024
    Configuration menu
    Copy the full SHA
    7c5e242 View commit details
    Browse the repository at this point in the history

  12. Move llamacpp config.json

    @mreso
    mreso committed Jan 26, 2024
    Configuration menu
    Copy the full SHA
    ccb9c4d View commit details
    Browse the repository at this point in the history

  13. Moved and created prompt file

    @mreso
    mreso committed Jan 26, 2024
    Configuration menu
    Copy the full SHA
    641386c View commit details
    Browse the repository at this point in the history

  14. Reset context for mutiple batch entries

    @mreso
    mreso committed Jan 26, 2024
    Configuration menu
    Copy the full SHA
    8e33517 View commit details
    Browse the repository at this point in the history

  15. Add doc for llamacpp example

    @mreso
    mreso committed Jan 26, 2024
    Configuration menu
    Copy the full SHA
    9a9b69e View commit details
    Browse the repository at this point in the history

  16. Fix spell check

    @mreso
    mreso committed Jan 26, 2024
    Configuration menu
    Copy the full SHA
    d08e4eb View commit details
    Browse the repository at this point in the history

  17. Replace output example in llamacpp example

    @mreso
    mreso committed Jan 26, 2024
    Configuration menu
    Copy the full SHA
    2d070a2 View commit details
    Browse the repository at this point in the history

  18. Move cpp example src into main examples folder

    @mreso
    mreso committed Jan 26, 2024
    Configuration menu
    Copy the full SHA
    f9dccd3 View commit details
    Browse the repository at this point in the history

  19. Convert cerr/cout into logs

    @mreso
    mreso committed Jan 26, 2024
    Configuration menu
    Copy the full SHA
    2cb7b03 View commit details
    Browse the repository at this point in the history