Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

text embedding cache speed-up for slow backends (eg. S3 or spinning disks) #271

Merged
merged 14 commits into from
Jan 16, 2024
Merged

text embedding cache speed-up for slow backends (eg. S3 or spinning disks) #271

merged 14 commits into from
Jan 16, 2024

Commits on Jan 15, 2024

  1. text embedding cache: reduce file existence checks

    bghira committed Jan 15, 2024
    Configuration menu
    Copy the full SHA
    06d34a2 View commit details
    Browse the repository at this point in the history

  2. text embedding cache: start the thread when re-scanning a new subset … 

    …of a dataset
    bghira committed Jan 15, 2024
    Configuration menu
    Copy the full SHA
    aba313f View commit details
    Browse the repository at this point in the history

  3. text embedding cache: do not log as much

    bghira committed Jan 15, 2024
    Configuration menu
    Copy the full SHA
    7475acc View commit details
    Browse the repository at this point in the history

  4. text embedding cache: write until we exhaust it

    bghira committed Jan 15, 2024
    Configuration menu
    Copy the full SHA
    7b30a21 View commit details
    Browse the repository at this point in the history

  5. text embedding cache: write until we exhaust it, faster

    bghira committed Jan 15, 2024
    Configuration menu
    Copy the full SHA
    dfb4f15 View commit details
    Browse the repository at this point in the history

  6. text embedding cache: write until we exhaust it, faster

    bghira committed Jan 15, 2024
    Configuration menu
    Copy the full SHA
    b5457bc View commit details
    Browse the repository at this point in the history

  7. text embedding cache: do not log the list

    bghira committed Jan 15, 2024
    Configuration menu
    Copy the full SHA
    822619a View commit details
    Browse the repository at this point in the history

  8. text embedding cache: do not log as much, speed up further

    bghira committed Jan 15, 2024
    Configuration menu
    Copy the full SHA
    2e8da41 View commit details
    Browse the repository at this point in the history

  9. Use unified method which checks for the existence of the prompts in t… 

    …he cache
    bghira committed Jan 15, 2024
    Configuration menu
    Copy the full SHA
    5d59aaf View commit details
    Browse the repository at this point in the history

  10. AWS: log when we check for files, since this is an expensive operation

    bghira committed Jan 15, 2024
    Configuration menu
    Copy the full SHA
    bfdf03b View commit details
    Browse the repository at this point in the history

Commits on Jan 16, 2024

  1. text embedding cache: switch log level back to user-provided

    bghira committed Jan 16, 2024
    Configuration menu
    Copy the full SHA
    982b080 View commit details
    Browse the repository at this point in the history

  2. multidatabackend: fix the probability implementation, and add ignore_… 

    …epochs parameter to dataset
    bghira committed Jan 16, 2024
    Configuration menu
    Copy the full SHA
    20fe88b View commit details
    Browse the repository at this point in the history

  3. multidatabackend: fix the probability implementation, and add ignore_… 

    …epochs parameter to dataset (fix)
    bghira committed Jan 16, 2024
    Configuration menu
    Copy the full SHA
    77412a7 View commit details
    Browse the repository at this point in the history

  4. multidatabackend: better docs and implementation for ignore_epochs

    bghira committed Jan 16, 2024
    Configuration menu
    Copy the full SHA
    e0b1bb5 View commit details
    Browse the repository at this point in the history