Skip to content

Commit

Permalink
[cherry-pick](branch-2.1) Pick "[Fix](LZ4 compression) Fix wrong LZ4 …
Browse files Browse the repository at this point in the history
…compression max input size limit (#41239)" (#41505)

## Proposed changes

LZ4 compression max supported value is LZ4_MAX_INPUT_SIZE, which is
0x7E000000(2,113,929,216 bytes). Doris use wrong max size INT_MAX, which
is 2,147,483,647, to check. If input data size is between this two size,
then it can pass the check but LZ4 compression will fail.

This PR fix it.

<!--Describe your changes.-->

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
  • Loading branch information
Yukang-Lian authored Oct 1, 2024
1 parent 806fe98 commit d33a3eb
Showing 1 changed file with 6 additions and 4 deletions.
10 changes: 6 additions & 4 deletions be/src/util/block_compression.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -127,11 +127,13 @@ class Lz4BlockCompression : public BlockCompressionCodec {
}

Status compress(const Slice& input, faststring* output) override {
if (input.size > INT_MAX) {
if (input.size > LZ4_MAX_INPUT_SIZE) {
return Status::InvalidArgument(
"LZ4 not support those case(input.size>INT_MAX), maybe you should change "
"fragment_transmission_compression_codec to snappy, size={}",
input.size);
"LZ4 not support those case(input.size>LZ4_MAX_INPUT_SIZE), maybe you should "
"change "
"fragment_transmission_compression_codec to snappy, input.size={}, "
"LZ4_MAX_INPUT_SIZE={}",
input.size, LZ4_MAX_INPUT_SIZE);
}

std::unique_ptr<Context> context;
Expand Down

0 comments on commit d33a3eb

Please sign in to comment.