bug in zstd : compression produces an invalid file with SpeedBestCompression

[LeGEC]

We hit a bug in the zstd package of this library : when compressing the attached file (the ungzipped version of that file) with level SpeedBestCompression, the compression succeeds without error, but the decompression fails with error "CRC check failed". The same error appears when trying to decompress with an external zstd decompressor.

This error did not appear in version 1.13.2 and appears in version 1.13.3.
In version 1.13.3, this error does not appear either when changing the compression level to any of the 3 other values.

Sample code :

func TestCompress(t *testing.T) {
	var b bytes.Buffer

	enc, err := zstd.NewWriter(&b, zstd.WithEncoderLevel(zstd.SpeedBestCompression))
	if err != nil {
		t.Fatal(err)
	}

	in, _ := os.Open("/tmp/failer.dat")
	_, err = io.Copy(enc, in)
	if err != nil {
		t.Fatal(err)
	}

	err = enc.Close()
	if err != nil {
		t.Fatal(err)
	}

	d, err := zstd.NewReader(&b)
	if err != nil {
		t.Fatal(err)
	}

	_, err = io.ReadAll(d)
	if err != nil {
		t.Fatal(err)  // test fails here : CRC check failed
	}
}

$ go version
go version go1.16.5 linux/amd64

failer.dat.gz

[klauspost]

@LeGEC Thanks for the report. I will investigate.

[LeGEC]

updated version range : this bug appeared between 1.13.2 and 1.13.3

[klauspost]

Yeah, likely #404

[klauspost]

Though it could be just because it creates different matches. Checking...

[klauspost]

Fix is in #415 - I will test and release.

[klauspost]

@LeGEC Released as 1.13.4. Thanks for the report!