Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix orc_test.py on Databricks 14.3 #11528

Closed
razajafri opened this issue Sep 27, 2024 · 1 comment · Fixed by #11652
Closed

Fix orc_test.py on Databricks 14.3 #11528

razajafri opened this issue Sep 27, 2024 · 1 comment · Fixed by #11652
Assignees
@mythrocks

Comments

@razajafri
Copy link
Collaborator

Build the plugin against Databricks 14.3 cluster using #11467. Once built successfully run the ORC tests by TESTS=orc_test.py jenkins/databricks/test.sh

The following tests fail

[gw0] [ 77%] FAILED ../../src/main/python/orc_test.py::test_project_fallback_when_reading_hive_fixed_length_char
@razajafri razajafri changed the title Fix ORC tests Fix ORC tests on Databricks 14.3 Sep 27, 2024
@mythrocks mythrocks changed the title Fix ORC tests on Databricks 14.3 Fix orc_test.py on Databricks 14.3 Oct 23, 2024
@mythrocks mythrocks self-assigned this Oct 23, 2024
@mythrocks
Copy link
Collaborator

Investigating this one. This might just be another skip, just like on 13.3.

mythrocks added a commit to mythrocks/spark-rapids that referenced this issue Oct 24, 2024
@mythrocks
Fix skipping fixed_length_char ORC tests on [databricks] > 13.3
b54b266 
Fixes NVIDIA#11528.

This commit addresses the failure of
`orc_test.py::test_project_fallback_when_reading_hive_fixed_length_char`
when run on Databricks 13.3+.

This test was skipped with an incorrect reason statement in NVIDIA#9646, to
address a failure on Databricks 13.3.  That failure was not the result
of a data mismatch.  It failed because the ProjectExec didn't fall back
to CPU.  (In fact, with Databricks 13.3 onwards, the ProjectExec is not
involved in `SELECT *` queries.)

As an aside, the same test is now skipped on Databricks 13.3 and 14.3,
because the error condition it tests does not fire on those versions.

Signed-off-by: MithunR <mithunr@nvidia.com>
@mythrocks mythrocks mentioned this issue Oct 24, 2024
Merged
mythrocks added a commit that referenced this issue Nov 4, 2024
@mythrocks
Fix skipping fixed_length_char ORC tests on [databricks] > 13.3 (#11652)
f533fc9 
Fixes #11528.

This commit addresses the failure of
`orc_test.py::test_project_fallback_when_reading_hive_fixed_length_char`
when run on Databricks 13.3+.

This test was skipped with an incorrect reason statement in #9646, to
address a failure on Databricks 13.3.  That failure was not the result
of a data mismatch.  It failed because the ProjectExec didn't fall back
to CPU.  (In fact, with Databricks 13.3 onwards, the ProjectExec is not
involved in `SELECT *` queries.)

As an aside, the same test is now skipped on Databricks 13.3 and 14.3,
because the error condition it tests does not fire on those versions.

Signed-off-by: MithunR <mithunr@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mythrocks mythrocks

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix skipping fixed_length_char ORC tests on [databricks] > 13.3 mythrocks/spark-rapids
2 participants
@mythrocks @razajafri