Bump parquet2 to 0.10.3 & replace compression algo LZ4 with LZ4Raw

[dantengsky]

btw, the parquet created by databend can not parsed by pyarrow ,

import pyarrow.parquet as pq
pq.read_pandas("/usr/local/var/minio/databend/1/4/_b/2fbff1270aec4748941939bdb388be3a_v0.parquet")

got issue,

OSError: Corrupt Lz4 compressed data.

Seems that pyarrow no longer supports LZ4 (which is replaced by LZ4_RAW)

Seems that there are some compatibility issues (pyarrow do support LZ4, as the error message indicates clearly)

https://lists.apache.org/thread/l15qq12v38w9jnkd6p9mdd11kr0nq3gr

maybe LZ4_RAW should be used

https://issues.apache.org/jira/browse/PARQUET-2032

Originally posted by @dantengsky in #4654 (comment)

[BohuTANG]

#4689 have upgrade parquet2 to the latest version:

parquet2 = { version = "0.10.3", default_features = false }

[dantengsky]

#4689 have upgrade parquet2 to the latest version:

parquet2 = { version = "0.10.3", default_features = false }

there are some inconsistencies between the parquet2 v0.10.3 published in crates.io and the version of GitHub (I do not know why), the crates parquet-format-async-temp they depend on are of different versions:

• for the crate.io
  parquet-format-async-temp ^0.2.0 is needed
  https://crates.io/crates/parquet2/0.10.3/dependencies

• for the github.com
  which https://github.com/datafuse-extras/parquet2 based on
  parquet-format-async-temp = "0.3.0" is needed
  https://github.com/jorgecarleitao/parquet2/blob/f8eb2436d5b39614139fc728050a6f375781f813/Cargo.toml#L14-L15

even worse, some data types exported by parquet-format-async-temp is used directly in the code base...

I am trying to resolve the conflicts.

[BohuTANG]

there are some inconsistencies between the parquet2 v0.10.3 published in crates.io and the version of GitHub (I do not know why), the crates parquet-format-async-temp they depend on are of different versions:

@jorgecarleitao Perhaps can help here, thanks.

[jorgecarleitao]

Thanks for the ping!

• v0.10.3 points to 198637b68dfeb3ed8065545d18a5fb36b4e20632 which depends on parquet-format-async-temp 0.2: https://github.com/jorgecarleitao/parquet2/blob/198637b68dfeb3ed8065545d18a5fb36b4e20632/Cargo.toml#L15 . It is also the same code base that was released to crates as v0.10.3

• the main branch depends on parquet-format-async-temp 0.3 because it adds support for page filtering, but it has not been released to crates (yet).

• https://github.com/datafuse-extras/parquet2 points to the latest main of parquet2

It seems that you are depending on the latest parquet2. There are some breaking changes from 0.10 to the current main / future 0.11. That may explain the conflicts.

[dantengsky]

It seems that you are depending on the latest parquet2. There are some breaking changes from 0.10 to the current main / future 0.11. That may explain the conflicts.

@jorgecarleitao Thank you for your detailed explanations!

There are some breaking changes from 0.10 to the current main / future 0.11.

May I ask if there was a near-future plan for version 0.11? and will crate arrow2 be upgraded as well? (We are trying to integrate the Lz4Raw jorgecarleitao/parquet2#95 feature)

[jorgecarleitao]

Ahhh, got it. I am finishing the necessary changes in arrow2 to add support page filter offsets, since parquet2 adds support for that.

I suspect that it should be ready within 2-3 weeks

[dantengsky]

@jorgecarleitao roger & thanks