You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When using zstd in a streaming context (file which gets extended at arbitrary times) it seems reasonable to structure a decompression loop like so:
size_tstreaming_decompress(State*state, char*out, size_tout_size) {
ZSTD_outBufferout= {out, out_size, 0};
while (1) {
ZSTD_inBufferin= {state->compressed_buf, state->compressed_buf_size, state->compressed_buf_pos};
size_trv=ZSTD_decompressStream(zds, &out, &in);
if (ZSTD_isError(rv)) {
// complain
}
if (out.pos==out.size) {
// exhausted output buffer, record progress through input buffer for next callstate->compressed_buf_pos=in.pos;
break;
}
// exhausted input buffer, try to read more data from fileif (try_refill_compressed_buf(state)) {
// got more compressed data, loop around
} else {
// no more data available at the momentbreak;
}
}
returnout.pos;
}
If no new data has come in for a while, each invocation of this function calls into libzstd (no-op) then tries to refill (also no-op). On the 16th attempt this fails to the ZSTD_NO_FORWARD_PROGRESS_MAX check. This took me a while to track down because:
A check like that was completely unexpected, resulting in some incredulity as to why this call would work some of the time then fail with the exact same parameters. I was going to post a minimal repro as a bug report when a colleague had to point out the existence of this check.
The error message Src size is incorrect was not helpful in figuring out the root cause.
I won't discuss the merit of the check beyond the above as you've obviously intentionally put it in, but my suggestion is to provide more explicit error messages, for example No forward progress for a while due to src.pos == src.size and similar for dst.
Thank you!
The text was updated successfully, but these errors were encountered:
Hello.
When using zstd in a streaming context (file which gets extended at arbitrary times) it seems reasonable to structure a decompression loop like so:
If no new data has come in for a while, each invocation of this function calls into libzstd (no-op) then tries to refill (also no-op). On the 16th attempt this fails to the ZSTD_NO_FORWARD_PROGRESS_MAX check. This took me a while to track down because:
Src size is incorrect
was not helpful in figuring out the root cause.I won't discuss the merit of the check beyond the above as you've obviously intentionally put it in, but my suggestion is to provide more explicit error messages, for example
No forward progress for a while due to src.pos == src.size
and similar for dst.Thank you!
The text was updated successfully, but these errors were encountered: