release v1.5.4 #3487
Merged
release v1.5.4 #3487
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Centralize the logic about whether to print the progress bar or not in the `*_PROGRESS()` macros. * Centralize the logc about whether to print the summary line or not in `FIO_shouldDisplayFileSummary()` and `FIO_shouldDisplayMultipleFileSummary()`. * Make `--progress` work for non-zstd (de)compressors. * Clean up several edge cases in compression and decompression progress printing along the way. E.g. wrong log level, or missing summary line. One thing I don't like about stdout mode, which sets the display level to 1, is that warnings aren't displayed. After this PR, we could change stdout mode from lowering the display level, to defaulting to implied `--no-progress`. But, I think that deserves a separate PR.
Instead of using packed attribute hack, just use aligned attribute. It improves code generation on armv6 and armv7, and slightly improves code generation on aarch64. GCC generates identical code to regular aligned access on ARMv6 for all versions between 4.5 and trunk, except GCC 5 which is buggy and generates the same (bad) code as packed access: https://gcc.godbolt.org/z/hq37rz7sb
Split the logic for parameter adaption from the logic to update the display rate. This decouples the two updates, so changes to display updates don't affect parameter adaption. Also add a test case that checks that parameter adaption actually happens. This fixes Issue #3353, where --adapt is broken when --no-progress is passed.
Fix an instance of `NULL + 0` in `ZSTD_decompressStream()`. Also, improve our `stream_decompress` fuzzer to pass `NULL` in/out buffers to `ZSTD_decompressStream()`, and fix 2 issues that were immediately surfaced. Fixes #3351
Delete unaligned memory access code from the legacy codebase by removing all the non-memcpy functions. We don't care about speed at all for this codebase, only simplicity.
Convert references to https from http
Fix an off-by-one error in the compressor that emits corrupt blocks if: * Zstd is compiled in 32-bit mode * The windowLog == 25 exactly * An offset of 2^25-3, 2^25-2, 2^25-1, or 2^25 is emitted * The bitstream had 7 bits leftover before writing the offset This bug has been present since before v1.0, but wasn't able to easily be triggered, since until somewhat recently zstd wasn't able to find matches that were within 128KB of the window size. Add a test case, and fix 2 bugs in `ZSTD_compressSequences()`: * The `ZSTD_isRLE()` check was incorrect. It wouldn't produce corruption, but it could waste CPU and not emit RLE even if the block was RLE * One windowSize was `1 << windowLog`, not `1u << windowLog` Thanks to @tansy for finding the issue, and giving us a reproducer! Fixes Issue #3350.
It uses non-portable compiler options unconditionally. Elsewhere, we check the compiler ID and only add the right ones, globally. Do the same here. NDEBUG can actually be handled by a core option, so while we are moving things around, do so. Unfortunately, this doesn't fix things entirely. The remaining issue is not Meson's issue though -- MSVC simply does not like this source code and somehow chokes on innocent code with the inscrutable "syntax error" and "illegal token".
In commit 031de3c some code was added that returned a boolean, but was treated as if it returned a dependency object. This wasn't tested and could not work. Moreover, zstd no longer built at all unless the entire programs directory was disabled and not even evaluated. Fix the return type checking.
In commit 031de3c a feature of Meson 0.50.0 was added, but the minimum specified version of Meson is 0.48.0. Meson therefore emitted a warning: WARNING: Project targets '>=0.48.0' but uses feature introduced in '0.50.0': required arg in compiler.has_header. And if anyone actually used Meson 0.48.0 to build with, it would error out with mysterious claims that the build file itself is invalid, rather than telling the user to install a newer version of Meson. Solve this by bumping the minimum version to align with reality. This e.g. drops support for Debian oldstable (buster)'s packaged version of Meson, but still works if backports are enabled, or if the user can `pip install` a newer version.
meson: fix broken commit that broke the build
Update CircleCI to focal and try to re-enable aarch64build. Fixes #2785 if it works.
It's entirely possible some people don't have valgrind installed, but still want to run the tests. If they don't have it installed, then they probably don't intend to run those precise test targets anyway. Also, this solves an error when running the tests in an automated environment. The valgrind tests have a hard dependency on behavior such as `./zstd` erroring out with the message "stdin is a console, aborting" which does not work if the automated environment doesn't have a console. As a rough heuristic, automated environments lacking a console will *probably* also not have valgrind, so avoiding that test definition neatly sidesteps the issue. Also, valgrind is not easily installable on macOS, at least homebrew says it isn't available there. This makes it needlessly hard to enable the testsuite on macOS.
playTests.sh has an option to run really slow tests. This is enabled by default in Meson, but what we really want is to do like the Makefile, and run the fast ones by default, but with an option to run the slow ones instead.
Travis is no longer run, but this wasn't ported to something else.
There are a couple of oddities here. We don't attempt to build e.g. contrib, because that doesn't seem to work at the moment. Also notice that each command is its own step. This happens because github actions runs in powershell, which doesn't seem to let you abort on the first failure.
It seems like with the deletion of Travis CI we didn't successfully transfer the version compatibility test. Attempt to enable the version compatibility test.
and rephrased the code documentation, as suggested by @terrelln
1. Follow the scheme introduced in PR #2501 for both `zdict.h` and `zstd_errors.h`. 2. If the `*_VISIBLE` macro isn't set, but the `*_VISIBILITY` macro is, use that. Also make this change for `zstd.h`, since we probably shouldn't have changed that macro name without backward compatibility in the first place. 3. Change all references to `*_VISIBILITY` to `*_VISIBLE`. Fixes #3359.
Fix `zdict.h` static linking only section so if you include it twice it still exposes the static linking only symbols. E.g. this pattern: ``` ``` This can easily happen when a header you include includes `zdict.h`.
check potential overflow of compressBound()
fixed an overflow in an intermediate result on 32-bit platform. Checked that the new test catch this bug in 32-bit mode.
Reduce external matchfinder API overhead by 25%
In 32-bit mode, ZSTD_getOffsetInfo() can be called when nbSeq == 0, and in this case the offset table is uninitialized. The function should just return 0 for both values, because there are no sequences. Credit to OSS-Fuzz
- Do not use threaded AsyncIO when handling small files. - Some typo / doc fixes
Bumps [ilammy/msvc-dev-cmd](https://github.com/ilammy/msvc-dev-cmd) from 1.12.0 to 1.12.1. - [Release notes](https://github.com/ilammy/msvc-dev-cmd/releases) - [Commits](ilammy/msvc-dev-cmd@7315a94...cec98b9) --- updated-dependencies: - dependency-name: ilammy/msvc-dev-cmd dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>
…my/msvc-dev-cmd-1.12.1 Bump ilammy/msvc-dev-cmd from 1.12.0 to 1.12.1
such scenario can happen, for example, when trying a decompression-only benchmark on invalid data. Other possibilities include an allocation error in an intermediate step. So far, the benchmark would return immediately, but still return 0. On command line, this would be confusing, as the program appears successful (though it does not display any successful message). Now it returns !0, which can be interpreted as an error by command line.
for the decompression-only benchmark mode.
return error code when benchmark fails
impacts legacy decoder v0.3 in 32-bit mode
in case it would be applicable here too.
slightly different constraints on end of buffer conditions
in case it would be applicable for this legacy version too.
in case it would be applicable for this version too
in case it would applicable for this version
which uses a different technique to store literals, and therefore must check for potential overwrites.
fix oss-fuzz case 55714
This frame is invalid because the `Window_Size = 0`, and the `Block_Maximum_Size = min(128 KB, Window_Size) = 0`. But the empty compressed block has a `Block_Content` size of 2, which is invalid. The fix is to switch to using a `Window_Descriptor` instead of the `Single_Segment_Flag`. This sets the `Window_Size = 1024`. Hexdump before this PR: `28b5 2ffd 2000 1500 0000 00` Hexdump after this PR: `28b5 2ffd 0000 1500 0000 00` For issue #3482.
* change "external matchfinder" to "external sequence producer" * migrate contrib/ to new naming convention * fix contrib build * fix error message * update debug strings * fix def of invalid sequences in zstd.h * nit * update CHANGELOG * fix .gitignore
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
|
|
||
| jobs: | ||
| publish-release-artifacts: | ||
| permissions: | ||
| contents: read # to fetch code (actions/checkout) | ||
| actions: write # to attach binaries to release artifacts (skx/github-action-publish-binaries) |
Check failure
Code scanning / Scorecard
Token-Permissions
| @@ -61,7 +68,7 @@ | |||
| fi | |||
|
|
|||
| - name: Publish | |||
| uses: skx/github-action-publish-binaries@release-1.3 | |||
| uses: skx/github-action-publish-binaries@release-2.0 | |||
Check warning
Code scanning / Scorecard
Pinned-Dependencies
| @@ -1,4 +1,4 @@ | |||
| FROM circleci/buildpack-deps:bionic | |||
| FROM circleci/buildpack-deps:focal | |||
Check warning
Code scanning / Scorecard
Pinned-Dependencies
| run: | | ||
| sudo apt-get update | ||
| sudo apt-get -y install build-essential python3-pip ninja-build liblz4-dev | ||
| pip install --pre meson |
Check warning
Code scanning / Scorecard
Pinned-Dependencies
| - uses: actions/checkout@v2 | ||
| - name: Mingw long test | ||
| - uses: actions/checkout@ac593985615ec2ede58e132d2e21d2b1cbd6127c # tag=v3 | ||
| - uses: msys2/setup-msys2@v2 |
Check warning
Code scanning / Scorecard
Pinned-Dependencies
| - name: Add MSBuild to PATH | ||
| uses: microsoft/setup-msbuild@v1.0.2 | ||
| uses: microsoft/setup-msbuild@v1.3 |
Check warning
Code scanning / Scorecard
Pinned-Dependencies
| $env:PATH = $env:PATH_MINGW32 + ";" + $env:PATH_ORIGINAL | ||
| } | ||
| - uses: actions/checkout@ac593985615ec2ede58e132d2e21d2b1cbd6127c # tag=v3 | ||
| - uses: msys2/setup-msys2@v2 |
Check warning
Code scanning / Scorecard
Pinned-Dependencies
| - name: Add MSBuild to PATH | ||
| uses: microsoft/setup-msbuild@v1.0.2 | ||
| uses: microsoft/setup-msbuild@v1.3 |
Check warning
Code scanning / Scorecard
Pinned-Dependencies
| - name: Add MSBuild to PATH | ||
| uses: microsoft/setup-msbuild@v1.0.2 | ||
| uses: microsoft/setup-msbuild@v1.3 |
Check warning
Code scanning / Scorecard
Pinned-Dependencies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
perf: +20% faster huffman decompression for targets that can't compile x64 assembly (#3449, @terrelln)
perf: up to +10% faster streaming compression at levels 1-2 (#3114, @embg)
perf: +4-13% for levels 5-12 by optimizing function generation (#3295, @terrelln)
pref: +3-11% compression speed for
armtarget (#3199, #3164, #3145, #3141, #3138, @JunHe77 and #3139, #3160, @danlark1)perf: +5-30% faster dictionary compression at levels 1-4 (#3086, #3114, #3152, @embg)
perf: +10-20% cold dict compression speed by prefetching CDict tables (#3177, @embg)
perf: +1% faster compression by removing a branch in ZSTD_fast_noDict (#3129, @felixhandte)
perf: Small compression ratio improvements in high compression mode (#2983, #3391, @Cyan4973 and #3285, #3302, @daniellerozenblit)
perf: small speed improvement by better detecting
STATIC_BMI2forclang(#3080, @TocarIP)perf: Improved streaming performance when
ZSTD_c_stableInBufferis set (#2974, @Cyan4973)cli: Asynchronous I/O for improved cli speed (#2975, #2985, #3021, #3022, @yoniko)
cli: Change
zstdlessbehavior to align withzless(#2909, @binhdvo)cli: Keep original file if
-cor--stdoutis given (#3052, @dirkmueller)cli: Keep original files when result is concatenated into a single output with
-o(#3450, @Cyan4973)cli: Preserve Permissions and Ownership of regular files (#3432, @felixhandte)
cli: Print zlib/lz4/lzma library versions with
-vv(#3030, @terrelln)cli: Print checksum value for single frame files with
-lv(#3332, @Cyan4973)cli: Print
dictIDwhen present with-lv(#3184, @htnhan)cli: when
stderris not the console, disable status updates, but preserve final summary (#3458, @Cyan4973)cli: support
--bestand--no-nameingzipcompatibility mode (#3059, @dirkmueller)cli: support for
posixhigh resolution timerclock_gettime(), for improved benchmark accuracy (#3423, @Cyan4973)cli: improved help/usage (
-h,-H) formatting (#3094, @dirkmueller and #3385, @jonpalmisc)cli: Fix better handling of bogus numeric values (#3268, @ctkhanhly)
cli: Fix input consists of multiple files and
stdin(#3222, @yoniko)cli: Fix tiny files passthrough (#3215, @cgbur)
cli: Fix for
-ron empty directory (#3027, @brailovich)cli: Fix empty string as argument for
--output-dir-*(#3220, @embg)cli: Fix decompression memory usage reported by
-vv --long(#3042, @u1f35c, and #3232, @zengyijing)cli: Fix infinite loop when empty input is passed to trainer (#3081, @terrelln)
cli: Fix
--adaptdoesn't work when--no-progressis also set (#3354, @terrelln)api: Support for External Sequence Producer (#3333, @embg)
api: Support for in-place decompression (#3432, @terrelln)
api: New
ZSTD_CCtx_setCParams()function, set all parameters defined in aZSTD_compressionParametersstructure (#3403, @Cyan4973)api: Streaming decompression detects incorrect header ID sooner (#3175, @Cyan4973)
api: Window size resizing optimization for edge case (#3345, @daniellerozenblit)
api: More accurate error codes for busy-loop scenarios (#3413, #3455, @Cyan4973)
api: Fix limit overflow in
compressBoundanddecompressBound(#3362, #3373, Cyan4973) reported by @nigeltaoapi: Deprecate several advanced experimental functions: streaming (#3408, @embg), copy (#3196, @mileshu)
bug: Fix corruption that rarely occurs in 32-bit mode with wlog=25 (#3361, @terrelln)
bug: Fix for block-splitter (#3033, @Cyan4973)
bug: Fixes for Sequence Compression API (#3023, #3040, @Cyan4973)
bug: Fix leaking thread handles on Windows (#3147, @animalize)
bug: Fix timing issues with cmake/meson builds (#3166, #3167, #3170, @Cyan4973)
build: Allow user to select legacy level for cmake (#3050, @shadchin)
build: Enable legacy support by default in cmake (#3079, @niamster)
build: Meson build script improvements (#3039, #3120, #3122, #3327, #3357, @eli-schwartz and #3276, @neheb)
build: Add aarch64 to supported architectures for zstd_trace (#3054, @ooosssososos)
build: support AIX architecture (#3219, @qiongsiwu)
build: Fix
ZSTD_LIB_MINIFYbuild macro, which now reduces static library size by half (#3366, @terrelln)build: Fix Windows issues with Multithreading translation layer (#3364, #3380, @yoniko) and ARM64 target (#3320, @cwoffenden)
build: Fix
cmakescript (#3382, #3392, @terrelln and #3252 @Tachi107 and #3167 @Cyan4973)doc: Updated man page, providing more details for
--trainmode (#3112, @Cyan4973)doc: Add decompressor errata document (#3092, @terrelln)
misc: Enable Intel CET (#2992, #2994, @hjl-tools)
misc: Fix
contrib/seekable format (#3058, @yhoogstrate and #3346, @daniellerozenblit)misc: Improve speed of the one-file library generator (#3241, @wahern and #3005, @cwoffenden)