Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Compress/Encrypt Blocks in the background (#1227)
This commit adds support for compression/encryption in the background. All incoming writes are saved to a common buffer and then worker go routine pick up each block (a block is just a start and end point in the main buffer). The worker goroutines also fix the table start and end points. We fix the interleaving space (which might be caused because the block was compressed) in the builder.Finish() call. All the blocks are copied over to their correct position. Master branch vs This commit (new) Benchmarks. ``` benchstat master.txt new.txt name old time/op new time/op delta (-ve is better. Reduction in time) no_compression-16 177ms ± 1% 177ms ± 2% ~ (p=0.690 n=5+5) encryption-16 313ms ± 4% 245ms ± 3% -21.69% (p=0.008 n=5+5) zstd_compression/level_1-16 330ms ± 1% 227ms ± 2% -31.22% (p=0.008 n=5+5) zstd_compression/level_3-16 345ms ± 2% 227ms ± 3% -34.39% (p=0.008 n=5+5) zstd_compression/level_15-16 10.7s ± 1% 1.2s ± 0% -88.96% (p=0.008 n=5+5) name old speed new speed delta (+ve is better. Speed improvement) no_compression-16 471MB/s ± 1% 471MB/s ± 2% ~ (p=0.690 n=5+5) encryption-16 266MB/s ± 4% 340MB/s ± 2% +27.67% (p=0.008 n=5+5) zstd_compression/level_1-16 252MB/s ± 1% 367MB/s ± 2% +45.40% (p=0.008 n=5+5) zstd_compression/level_3-16 241MB/s ± 2% 367MB/s ± 3% +52.44% (p=0.008 n=5+5) zstd_compression/level_15-16 7.80MB/s ± 1% 70.62MB/s ± 0% +805.33% (p=0.008 n=5+5) ``` Please look at #1227 (comment) for the code used for benchmarking.
- Loading branch information