Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

s2: Add 'best' compression option #310

Merged
merged 4 commits into from
Jan 13, 2021
Merged

Commits on Jan 12, 2021

  1. s2: Add 'best' compression option

    Add compression mode that will compress the data very well, but at a significant performance decrease.
    
    Mainly for offline compression, but where decompression speed should still be high.
    
    Comparing to 'better':
    
    ```
    github-june-2days-2019.json -> (discarded) 6273951764 -> 950079555 [15.14%]; 3541.4MB/s
    github-june-2days-2019.json -> (discarded) 6273951764 -> 846260870 [13.49%]; 661.5MB/s
    
    nyc-taxi-data-10M.csv -> (discarded) 3325605752 -> 960330423 [28.88%]; 3249.5MB/s
    nyc-taxi-data-10M.csv -> (discarded) 3325605752 -> 794873295 [23.90%]; 437.5MB/s
    
    10gb.tar -> (discarded) 10065157632 -> 5650133605 [56.14%]; 3137.9MB/s
    10gb.tar -> (discarded) 10065157632 -> 5250102961 [52.16%]; 390.1MB/s
    
    consensus.db.10gb -> (discarded) 10737418240 -> 4542443833 [42.30%]; 3008.2MB/s
    consensus.db.10gb -> (discarded) 10737418240 -> 4272335541 [39.79%]; 248.8MB/s
    ```
    klauspost committed Jan 12, 2021
    Configuration menu
    Copy the full SHA
    9ce8311 View commit details
    Browse the repository at this point in the history
  2. Update tests and docs.

    klauspost committed Jan 12, 2021
    Configuration menu
    Copy the full SHA
    e0b6170 View commit details
    Browse the repository at this point in the history
  3. Don't retest same offsets.

    klauspost committed Jan 12, 2021
    Configuration menu
    Copy the full SHA
    e590bd2 View commit details
    Browse the repository at this point in the history

Commits on Jan 13, 2021

  1. Update benchmarks

    klauspost committed Jan 13, 2021
    Configuration menu
    Copy the full SHA
    bdb7aab View commit details
    Browse the repository at this point in the history