-
Notifications
You must be signed in to change notification settings - Fork 3.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix failure to read nullable time(6) columns #15606
Conversation
e39e974
to
074d2c9
Compare
Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you add a test in Iceberg connector as well?
Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla |
Added relevant test in the Iceberg connector. |
94a5a47
to
0660094
Compare
Rebased onto current master. |
This pull request has gone a while without any activity. Tagging the Trino developer relations team: @bitsondatadev @colebow @mosabua |
@raunaqmorarka are you able to review this? |
0660094
to
ec7b0a9
Compare
Rebasing to resolve merge conflict in TestIcebergOrcConnectorTest. Anything I can do to help get this merged? |
@joejensen looking into it now. |
ec7b0a9
to
86902f0
Compare
lib/trino-orc/src/main/java/io/trino/orc/reader/LongColumnReader.java
Outdated
Show resolved
Hide resolved
lib/trino-orc/src/main/java/io/trino/orc/reader/LongColumnReader.java
Outdated
Show resolved
Hide resolved
lib/trino-orc/src/main/java/io/trino/orc/metadata/statistics/TimeStatisticsBuilder.java
Outdated
Show resolved
Hide resolved
dd91469
to
27e6748
Compare
lib/trino-orc/src/main/java/io/trino/orc/metadata/statistics/TimeStatisticsBuilder.java
Outdated
Show resolved
Hide resolved
lib/trino-orc/src/main/java/io/trino/orc/metadata/statistics/TimeStatisticsBuilder.java
Outdated
Show resolved
Hide resolved
lib/trino-orc/src/main/java/io/trino/orc/OrcWriteValidation.java
Outdated
Show resolved
Hide resolved
@joejensen Could you rebase on master to resolve conflicts? |
27e6748
to
45122a1
Compare
Rebased on master to resolve conflicts and addressed comments except for #15606 (comment) |
45122a1
to
19f3c9f
Compare
19f3c9f
to
d8d700d
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This lgtm, but would be great to get another pair of eyes on it
...in/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergMinioOrcConnectorTest.java
Outdated
Show resolved
Hide resolved
...in/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergMinioOrcConnectorTest.java
Show resolved
Hide resolved
@raunaqmorarka we should have connector-level test coverage for this. |
d8d700d
to
f1341e3
Compare
That is because of a quirk in that test's data setup and the way orc reader works. There is only 1 null in the setup and that is in the first row. The orc reader reads in batches of 1, 2, 4 ... rows. So the first batch is all nulls, while the 2nd batch is all non-nulls. This bug requires a batch containing null as well as non-null values to cause failure. |
Description
Fixes #15603
Additional context and related issues
Fixes issue where ORC files containing nullable Iceberg TIME columns could throw an exception when being read. Also fixed an error in the ORC file validator where time columns did not validate correctly in order to add appropriate tests for the Time type.
Release notes
( ) This is not user-visible or docs only and no release notes are required.
( ) Release notes are required, please propose a release note for me.
(x) Release notes are required, with the following suggested text: