Fix lowLimit underflow in overflow correction #1957
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
After PR #1624 we no longer updated
lowLimit
every block. That meanslowLimit
only gets updated when the round buffer overlaps in single threaded mode, or when we start a new job in multithreaded mode. After that change (and maybe before too),lowLimit
can underflow. IflowLimit
underflows, then for the remainder of compression all matches are deemed out of bounds, so compression ratio plummets.This fixes the problem by ensuring
lowLimit
never underflows. We setlowLimit
anddictLimit
to 1 instead, and ensure that we aren't invalidating any of the window.I've modified two tests in
playTests.sh
to trigger overflow correction. Currently they don't because after PR #1658 we clear the context instead of overflow correction if we are starting within 16 MB of the correction point. Setting a larger window log ensures a larger job size, which doesn't fall within 16 MB of the correction point.enwik10
now compresses as expected: