Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build: Revert Parquet to 1.13.1 #11462

Merged
merged 1 commit into from
Nov 4, 2024

Conversation

RussellSpitzer
Copy link
Member

apache/parquet-java#3040 -
Was discovered by @pan3793 in Parquet 1.14.(0,1,2,3). This is considered a critical data loss issue so we will pull back parquet to avoid the vulnerability.

@RussellSpitzer
Copy link
Member Author

Some Flink Tests Broken, Rerunning

@RussellSpitzer
Copy link
Member Author

Same test failed again,

> Task :iceberg-flink:iceberg-flink-1.18:test

TestMetadataTableReadableMetrics > testPrimitiveColumns() > catalogName=testhive, baseNamespace= FAILED
    java.lang.AssertionError: 
    Expecting actual:
      [+I[+I[+I[52, 4, 2, null, [-41, 93, 117], [-37, 109, -74]], +I[32, 4, 0, null, false, true], +I[85, 4, 1, null, 1.00, 2.00], +I[85, 4, 0, 1, 1.0, 2.0], +I[44, 4, 2, null, [-41, 93, 117], [-37, 109, -74]], +I[71, 4, 0, 2, 0.0, 0.0], +I[71, 4, 0, null, 1, 2], +I[79, 4, 0, null, 1, 2], +I[79, 4, 0, null, 1, 2]]]]
    to contain exactly in any order:
      [+I[+I[+I[55, 4, 2, null, [-41, 93, 117], [-37, 109, -74]], +I[36, 4, 0, null, false, true], +I[91, 4, 1, null, 1.00, 2.00], +I[91, 4, 0, 1, 1.0, 2.0], +I[47, 4, 2, null, [-41, 93, 117], [-37, 109, -74]], +I[77, 4, 0, 2, 0.0, 0.0], +I[77, 4, 0, null, 1, 2], +I[85, 4, 0, null, 1, 2], +I[85, 4, 0, null, 1, 2]]]]
    elements not found:
      [+I[+I[+I[55, 4, 2, null, [-41, 93, 117], [-37, 109, -74]], +I[36, 4, 0, null, false, true], +I[91, 4, 1, null, 1.00, 2.00], +I[91, 4, 0, 1, 1.0, 2.0], +I[47, 4, 2, null, [-41, 93, 117], [-37, 109, -74]], +I[77, 4, 0, 2, 0.0, 0.0], +I[77, 4, 0, null, 1, 2], +I[85, 4, 0, null, 1, 2], +I[85, 4, 0, null, 1, 2]]]]
    and elements not expected:
      [+I[+I[+I[52, 4, 2, null, [-41, 93, 117], [-37, 109, -74]], +I[32, 4, 0, null, false, true], +I[85, 4, 1, null, 1.00, 2.00], +I[85, 4, 0, 1, 1.0, 2.0], +I[44, 4, 2, null, [-41, 93, 117], [-37, 109, -74]], +I[71, 4, 0, 2, 0.0, 0.0], +I[71, 4, 0, null, 1, 2], +I[79, 4, 0, null, 1, 2], +I[79, 4, 0, null, 1, 2]]]]
        at org.apache.iceberg.flink.TestHelpers.assertRows(TestHelpers.java:162)
        at org.apache.iceberg.flink.source.TestMetadataTableReadableMetrics.testPrimitiveColumns(TestMetadataTableReadableMetrics.java:255)
        ```

@RussellSpitzer
Copy link
Member Author

Also fails locally

@Fokko
Copy link
Contributor

Fokko commented Nov 4, 2024

@RussellSpitzer Easiest is probably to revert this commit: b8c2b20

@RussellSpitzer
Copy link
Member Author

Thanks @Fokko ! Note that we should probably fix these tests so that they don't hardwire expected values irrelevant fields

This reverts commit b8c2b20.

apache/parquet-java#3040
Was discovered by @pan3793 in Parquet 1.14.(0,1,2,3).
@github-actions github-actions bot added the flink label Nov 4, 2024
@RussellSpitzer RussellSpitzer merged commit 7cc16fa into apache:main Nov 4, 2024
49 checks passed
@RussellSpitzer RussellSpitzer deleted the RevertParquet branch November 4, 2024 19:27
RussellSpitzer added a commit to RussellSpitzer/iceberg that referenced this pull request Nov 4, 2024
@RussellSpitzer
Copy link
Member Author

RussellSpitzer commented Nov 4, 2024 via email

RussellSpitzer added a commit that referenced this pull request Nov 4, 2024
Fokko added a commit to Fokko/iceberg that referenced this pull request Nov 8, 2024
Fokko added a commit that referenced this pull request Nov 20, 2024
* Revert "Revert "Build: Bump parquet from 1.13.1 to 1.14.3 (#11264)" (#11462)"

This reverts commit 7cc16fa.

* Bump to Parquet 1.14.4

* Lookup sizes instead

* Update build.gradle
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants