Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

s2: Add LZ4 block converter #748

Merged
merged 14 commits into from
Feb 17, 2023
Merged

s2: Add LZ4 block converter #748

merged 14 commits into from
Feb 17, 2023

Commits on Feb 7, 2023

  1. s2: Add LZ4 block converter 

    ## Single threaded performance

Speed excluding LZ4 encoding:

```
BenchmarkLZ4Converter_ConvertBlock/html-32         	   22450	     49962 ns/op	2049.58 MB/s	       559.0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/urls-32         	    1933	    608889 ns/op	1153.06 MB/s	     -3943 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/jpg-32          	  428572	      2816 ns/op	43715.74 MB/s	       482.0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/jpg_200b-32     	  413655	      2793 ns/op	44075.51 MB/s	       482.0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/pdf-32          	  163200	      7226 ns/op	14171.69 MB/s	       136.0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/html4-32        	    5451	    214569 ns/op	1908.94 MB/s	      1840 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt1-32         	    5100	    233525 ns/op	 651.27 MB/s	       106.0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt2-32         	    5997	    205362 ns/op	 609.55 MB/s	     -1427 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt3-32         	    1948	    616892 ns/op	 691.78 MB/s	       384.0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt4-32         	    1419	    835555 ns/op	 576.70 MB/s	     -9125 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/pb-32           	   29461	     40528 ns/op	2926.08 MB/s	         1.000 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/gaviota-32      	    5454	    214264 ns/op	 860.25 MB/s	      9303 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt1_128b-32    	    5216	    233042 ns/op	 652.62 MB/s	       106.0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt1_1000b-32   	    5000	    232401 ns/op	 654.43 MB/s	       106.0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt1_10000b-32  	    4999	    232989 ns/op	 652.77 MB/s	       106.0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt1_20000b-32  	    4999	    233068 ns/op	 652.55 MB/s	       106.0 b_saved	       0 B/op	       0 allocs/op
```

Reference comnpression speed:

```
BenchmarkCompressBlockReference/html/default-32 	   14392	     82754 ns/op	1237.41 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/html/better-32  	    6663	    178830 ns/op	 572.61 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/urls/default-32 	    1174	   1002967 ns/op	 700.01 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/urls/better-32  	     607	   1942559 ns/op	 361.42 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/jpg/default-32  	  196694	      6023 ns/op	20436.73 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/jpg/better-32   	   51148	     23613 ns/op	5212.85 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/jpg_200b/default-32         	  197408	      6015 ns/op	20464.42 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/jpg_200b/better-32          	   51267	     23509 ns/op	5236.04 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/pdf/default-32              	   91663	     12891 ns/op	7943.73 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/pdf/better-32               	    8433	    128542 ns/op	 796.62 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/html4/default-32            	   10000	    100385 ns/op	4080.29 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/html4/better-32             	    5712	    204505 ns/op	2002.88 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/txt1/default-32             	    3427	    340552 ns/op	 446.60 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/txt1/better-32              	    2032	    576497 ns/op	 263.82 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/txt2/default-32             	    4359	    266102 ns/op	 470.42 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/txt2/better-32              	    2306	    509822 ns/op	 245.53 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/txt3/default-32             	    1219	    971355 ns/op	 439.34 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/txt3/better-32              	     748	   1598189 ns/op	 267.02 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/txt4/default-32             	    1056	   1129044 ns/op	 426.79 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/txt4/better-32              	     579	   2064813 ns/op	 233.37 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/pb/default-32               	   17960	     65621 ns/op	1807.16 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/pb/better-32                	    7062	    159908 ns/op	 741.60 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/gaviota/default-32          	    3870	    302586 ns/op	 609.15 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/gaviota/better-32           	    2396	    495081 ns/op	 372.30 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/txt1_128b/default-32        	    3426	    342366 ns/op	 444.23 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/txt1_128b/better-32         	    2066	    581844 ns/op	 261.39 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/txt1_1000b/default-32       	    3524	    340442 ns/op	 446.74 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/txt1_1000b/better-32        	    1996	    577981 ns/op	 263.14 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/txt1_10000b/default-32      	    3423	    340195 ns/op	 447.06 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/txt1_10000b/better-32       	    2013	    578202 ns/op	 263.04 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/txt1_20000b/default-32      	    3518	    340101 ns/op	 447.19 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/txt1_20000b/better-32       	    2031	    581532 ns/op	 261.53 MB/s	       0 B/op	       0 allocs/op
```

Size comparisons (using Go lz4 encoder):

```
=== RUN   TestLZ4Converter_ConvertBlock
=== RUN   TestLZ4Converter_ConvertBlock/html
    lz4convert_test.go:34: input size: 102400
    lz4convert_test.go:35: lz4 size: 21195
    lz4convert_test.go:52: lz4->s2 size: 20636
    lz4convert_test.go:64: s2 (better) size: 18969
    lz4convert_test.go:66: lz4 -> s2 bytes saved: 559
    lz4convert_test.go:67: data -> s2 (better) bytes saved: 2226
    lz4convert_test.go:68: direct data -> s2 better compared to converted from lz4: 1667
=== RUN   TestLZ4Converter_ConvertBlock/urls
    lz4convert_test.go:34: input size: 702087
    lz4convert_test.go:35: lz4 size: 292514
    lz4convert_test.go:52: lz4->s2 size: 296457
    lz4convert_test.go:64: s2 (better) size: 248076
    lz4convert_test.go:66: lz4 -> s2 bytes saved: -3943
    lz4convert_test.go:67: data -> s2 (better) bytes saved: 44438
    lz4convert_test.go:68: direct data -> s2 better compared to converted from lz4: 48381
=== RUN   TestLZ4Converter_ConvertBlock/jpg
    lz4convert_test.go:34: input size: 123093
    lz4convert_test.go:35: lz4 size: 123522
    lz4convert_test.go:52: lz4->s2 size: 123040
    lz4convert_test.go:64: s2 (better) size: 123097
    lz4convert_test.go:66: lz4 -> s2 bytes saved: 482
    lz4convert_test.go:67: data -> s2 (better) bytes saved: 425
    lz4convert_test.go:68: direct data -> s2 better compared to converted from lz4: -57
=== RUN   TestLZ4Converter_ConvertBlock/jpg_200b
    lz4convert_test.go:34: input size: 123093
    lz4convert_test.go:35: lz4 size: 123522
    lz4convert_test.go:52: lz4->s2 size: 123040
    lz4convert_test.go:64: s2 (better) size: 123097
    lz4convert_test.go:66: lz4 -> s2 bytes saved: 482
    lz4convert_test.go:67: data -> s2 (better) bytes saved: 425
    lz4convert_test.go:68: direct data -> s2 better compared to converted from lz4: -57
=== RUN   TestLZ4Converter_ConvertBlock/pdf
    lz4convert_test.go:34: input size: 102400
    lz4convert_test.go:35: lz4 size: 83152
    lz4convert_test.go:52: lz4->s2 size: 83016
    lz4convert_test.go:64: s2 (better) size: 82884
    lz4convert_test.go:66: lz4 -> s2 bytes saved: 136
    lz4convert_test.go:67: data -> s2 (better) bytes saved: 268
    lz4convert_test.go:68: direct data -> s2 better compared to converted from lz4: 132
=== RUN   TestLZ4Converter_ConvertBlock/html4
    lz4convert_test.go:34: input size: 409600
    lz4convert_test.go:35: lz4 size: 81908
    lz4convert_test.go:52: lz4->s2 size: 80068
    lz4convert_test.go:64: s2 (better) size: 18979
    lz4convert_test.go:66: lz4 -> s2 bytes saved: 1840
    lz4convert_test.go:67: data -> s2 (better) bytes saved: 62929
    lz4convert_test.go:68: direct data -> s2 better compared to converted from lz4: 61089
=== RUN   TestLZ4Converter_ConvertBlock/txt1
    lz4convert_test.go:34: input size: 152089
    lz4convert_test.go:35: lz4 size: 79672
    lz4convert_test.go:52: lz4->s2 size: 79566
    lz4convert_test.go:64: s2 (better) size: 71608
    lz4convert_test.go:66: lz4 -> s2 bytes saved: 106
    lz4convert_test.go:67: data -> s2 (better) bytes saved: 8064
    lz4convert_test.go:68: direct data -> s2 better compared to converted from lz4: 7958
=== RUN   TestLZ4Converter_ConvertBlock/txt2
    lz4convert_test.go:34: input size: 125179
    lz4convert_test.go:35: lz4 size: 70801
    lz4convert_test.go:52: lz4->s2 size: 72228
    lz4convert_test.go:64: s2 (better) size: 65938
    lz4convert_test.go:66: lz4 -> s2 bytes saved: -1427
    lz4convert_test.go:67: data -> s2 (better) bytes saved: 4863
    lz4convert_test.go:68: direct data -> s2 better compared to converted from lz4: 6290
=== RUN   TestLZ4Converter_ConvertBlock/txt3
    lz4convert_test.go:34: input size: 426754
    lz4convert_test.go:35: lz4 size: 207038
    lz4convert_test.go:52: lz4->s2 size: 206654
    lz4convert_test.go:64: s2 (better) size: 184936
    lz4convert_test.go:66: lz4 -> s2 bytes saved: 384
    lz4convert_test.go:67: data -> s2 (better) bytes saved: 22102
    lz4convert_test.go:68: direct data -> s2 better compared to converted from lz4: 21718
=== RUN   TestLZ4Converter_ConvertBlock/txt4
    lz4convert_test.go:34: input size: 481861
    lz4convert_test.go:35: lz4 size: 277731
    lz4convert_test.go:52: lz4->s2 size: 286856
    lz4convert_test.go:64: s2 (better) size: 264987
    lz4convert_test.go:66: lz4 -> s2 bytes saved: -9125
    lz4convert_test.go:67: data -> s2 (better) bytes saved: 12744
    lz4convert_test.go:68: direct data -> s2 better compared to converted from lz4: 21869
=== RUN   TestLZ4Converter_ConvertBlock/pb
    lz4convert_test.go:34: input size: 118588
    lz4convert_test.go:35: lz4 size: 19003
    lz4convert_test.go:52: lz4->s2 size: 19002
    lz4convert_test.go:64: s2 (better) size: 17686
    lz4convert_test.go:66: lz4 -> s2 bytes saved: 1
    lz4convert_test.go:67: data -> s2 (better) bytes saved: 1317
    lz4convert_test.go:68: direct data -> s2 better compared to converted from lz4: 1316
=== RUN   TestLZ4Converter_ConvertBlock/gaviota
    lz4convert_test.go:34: input size: 184320
    lz4convert_test.go:35: lz4 size: 71749
    lz4convert_test.go:52: lz4->s2 size: 62446
    lz4convert_test.go:64: s2 (better) size: 55395
    lz4convert_test.go:66: lz4 -> s2 bytes saved: 9303
    lz4convert_test.go:67: data -> s2 (better) bytes saved: 16354
    lz4convert_test.go:68: direct data -> s2 better compared to converted from lz4: 7051
=== RUN   TestLZ4Converter_ConvertBlock/txt1_128b
    lz4convert_test.go:34: input size: 152089
    lz4convert_test.go:35: lz4 size: 79672
    lz4convert_test.go:52: lz4->s2 size: 79566
    lz4convert_test.go:64: s2 (better) size: 71608
    lz4convert_test.go:66: lz4 -> s2 bytes saved: 106
    lz4convert_test.go:67: data -> s2 (better) bytes saved: 8064
    lz4convert_test.go:68: direct data -> s2 better compared to converted from lz4: 7958
=== RUN   TestLZ4Converter_ConvertBlock/txt1_1000b
    lz4convert_test.go:34: input size: 152089
    lz4convert_test.go:35: lz4 size: 79672
    lz4convert_test.go:52: lz4->s2 size: 79566
    lz4convert_test.go:64: s2 (better) size: 71608
    lz4convert_test.go:66: lz4 -> s2 bytes saved: 106
    lz4convert_test.go:67: data -> s2 (better) bytes saved: 8064
    lz4convert_test.go:68: direct data -> s2 better compared to converted from lz4: 7958
=== RUN   TestLZ4Converter_ConvertBlock/txt1_10000b
    lz4convert_test.go:34: input size: 152089
    lz4convert_test.go:35: lz4 size: 79672
    lz4convert_test.go:52: lz4->s2 size: 79566
    lz4convert_test.go:64: s2 (better) size: 71608
    lz4convert_test.go:66: lz4 -> s2 bytes saved: 106
    lz4convert_test.go:67: data -> s2 (better) bytes saved: 8064
    lz4convert_test.go:68: direct data -> s2 better compared to converted from lz4: 7958
=== RUN   TestLZ4Converter_ConvertBlock/txt1_20000b
    lz4convert_test.go:34: input size: 152089
    lz4convert_test.go:35: lz4 size: 79672
    lz4convert_test.go:52: lz4->s2 size: 79566
    lz4convert_test.go:64: s2 (better) size: 71608
    lz4convert_test.go:66: lz4 -> s2 bytes saved: 106
    lz4convert_test.go:67: data -> s2 (better) bytes saved: 8064
    lz4convert_test.go:68: direct data -> s2 better compared to converted from lz4: 7958
```
    @klauspost
    klauspost committed Feb 7, 2023
    Configuration menu
    Copy the full SHA
    fe0c088 View commit details
    Browse the repository at this point in the history

  2. Tweak tests.

    @klauspost
    klauspost committed Feb 7, 2023
    Configuration menu
    Copy the full SHA
    519a51c View commit details
    Browse the repository at this point in the history

  3. Return error when truncated.

    @klauspost
    klauspost committed Feb 7, 2023
    Configuration menu
    Copy the full SHA
    b691e88 View commit details
    Browse the repository at this point in the history

Commits on Feb 14, 2023

  1. Improve speed with inlining

    @klauspost
    klauspost committed Feb 14, 2023
    Configuration menu
    Copy the full SHA
    b214753 View commit details
    Browse the repository at this point in the history

  2. Tweak benchmarks

    @klauspost
    klauspost committed Feb 14, 2023
    Configuration menu
    Copy the full SHA
    0751108 View commit details
    Browse the repository at this point in the history

  3. Add assembly converter

    @klauspost
    klauspost committed Feb 14, 2023
    Configuration menu
    Copy the full SHA
    95a0452 View commit details
    Browse the repository at this point in the history

Commits on Feb 15, 2023

  1. Add asm function stub for other platforms.

    @klauspost
    klauspost committed Feb 15, 2023
    Configuration menu
    Copy the full SHA
    f082d16 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into add-lz4-block-converter

    @klauspost
    klauspost authored Feb 15, 2023
    Configuration menu
    Copy the full SHA
    66f7ba9 View commit details
    Browse the repository at this point in the history

Commits on Feb 16, 2023

  1. Regen assembly.

    @klauspost
    klauspost committed Feb 16, 2023
    Configuration menu
    Copy the full SHA
    80de799 View commit details
    Browse the repository at this point in the history

  2. Add fuzz tests.

    @klauspost
    klauspost committed Feb 16, 2023
    Configuration menu
    Copy the full SHA
    ea57f35 View commit details
    Browse the repository at this point in the history

  3. Combine write

    @klauspost
    klauspost committed Feb 16, 2023
    Configuration menu
    Copy the full SHA
    55def3f View commit details
    Browse the repository at this point in the history

Commits on Feb 17, 2023

  1. Add Snappy block converter. Why not I guess...

    @klauspost
    klauspost committed Feb 17, 2023
    Configuration menu
    Copy the full SHA
    7f1ea45 View commit details
    Browse the repository at this point in the history

  2. Fix lz4->snappy assembly.

    @klauspost
    klauspost committed Feb 17, 2023
    Configuration menu
    Copy the full SHA
    928a1c4 View commit details
    Browse the repository at this point in the history

  3. Add fuzz corpus

    @klauspost
    klauspost committed Feb 17, 2023
    Configuration menu
    Copy the full SHA
    25a9ae5 View commit details
    Browse the repository at this point in the history