Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix test failures for Spark 4.0.0 #11004

Open
18 of 26 tasks
razajafri opened this issue Jun 8, 2024 · 0 comments
Open
18 of 26 tasks

Fix test failures for Spark 4.0.0 #11004

razajafri opened this issue Jun 8, 2024 · 0 comments
Labels
bug Something isn't working Spark 4.0+ Spark 4.0+ issues

Comments

@razajafri
Copy link
Collaborator

razajafri commented Jun 8, 2024 •
edited by mythrocks
Loading

This is an epic that will contain all the issues for various integration failures and unit test failures

Tasks
Beta Give feedback
  1. Fix tests failures in parquet_test.py #11015
    Spark 4.0+ bug
    mythrocks
  2. Fix test failures in aqe_test.py #11005
    Spark 4.0+ bug
  3. Fix tests failures in cache_test.py #11010
    Spark 4.0+ bug
  4. Fix tests failures in conditionals_test.py #11012
    Spark 4.0+ bug
  5. Fix tests failures in orc_test.py #11013
    Spark 4.0+ bug
  6. Fix tests failures in csv_test.py #11016
    Spark 4.0+ bug
    rwlee
  7. Revisit the ANSI tests which is enabled by default in Spark 4.0.0 #11047
    0 of 3
    Spark 4.0+ bug
  8. Fix tests failures in join_test.py #11022
    Spark 4.0+ bug
    rwlee
  9. Fix tests failures in collection_ops_test.py #11011
    Spark 4.0+ bug
    mythrocks
  10. Fix tests failures in cast_test.py #11009
    Spark 4.0+ bug
  11. Fix tests failures in orc_cast_test.py #11021
    Spark 4.0+ bug
    rwlee
  12. Fix tests failures in parquet_write_test.py #11024
    Spark 4.0+ bug
    rwlee
  13. Fix test failures in arithmetic_ops_test.py #11006
    Spark 4.0+ bug
    razajafri
  14. Fix tests failures in ast_test.py #11008
    Spark 4.0+ bug
    razajafri
  15. Fix tests failures in array_test.py #11007
    Spark 4.0+ bug
    razajafri
  16. Fix tests failures in string_test.py #11030
    Spark 4.0+ bug
    mythrocks
  17. Fix tests failures in grouping_sets_test.py #11020
    Spark 4.0+ bug
    mythrocks
  18. Fix tests failures in dpp_test.py #11023
    Spark 4.0+ bug
    mythrocks
  19. Fix tests failures in window_function_test.py #11019
    Spark 4.0+ bug
    mythrocks
  20. Fix tests failures in date_time_test.py #11025
    Spark 4.0+ bug
    mythrocks
  21. Fix tests failures in map_test.py #11026
    Spark 4.0+ bug
    mythrocks
  22. Fix tests failures in sort_test.py #11027
    Spark 4.0+ bug
    mythrocks
  23. Fix tests failures in subquery_test.py #11029
    Spark 4.0+ bug
    mythrocks
  24. Fix tests failures in hash_aggregate_test.py #11018
    Spark 4.0+ bug
    mythrocks
  25. Fix tests failures in multiple files #11031
    Spark 4.0+ bug
    mythrocks
  26. Fix tests failures in url_test.py #11017
    Spark 4.0+ bug
    mythrocks
@razajafri razajafri added bug Something isn't working ? - Needs Triage Need team to review and classify Spark 4.0+ Spark 4.0+ issues labels Jun 8, 2024
@mattahrens mattahrens removed the ? - Needs Triage Need team to review and classify label Jun 11, 2024
@mythrocks mythrocks mentioned this issue Jun 24, 2024
Merged
@razajafri razajafri mentioned this issue Jun 28, 2024
Open
mythrocks added a commit to mythrocks/spark-rapids that referenced this issue Sep 27, 2024
@mythrocks
Spark 4: Fix parquet_test.py.
b712c2d 
Fixes NVIDIA#11015.
Contributes to NVIDIA#11004.

This commit addresses the tests that fail in parquet_test.py, when
run on Spark 4.

1. Some of the tests were failing as a result of NVIDIA#5114.  Those tests
have been disabled, at least until we get around to supporting
aggregations with ANSI mode enabled.

2. `test_parquet_check_schema_compatibility` fails on Spark 4 regardless
of ANSI mode, because it tests implicit type promotions where the read
schema includes wider columns than the write schema.  This will require
new code.  The test is disabled until NVIDIA#11512 is addressed.

3. `test_parquet_int32_downcast` had an erroneous setup phase that fails
   in ANSI mode.  This has been corrected. The test was refactored to
run in ANSI and non-ANSI mode.

Signed-off-by: MithunR <mithunr@nvidia.com>
@mythrocks mythrocks mentioned this issue Sep 27, 2024
Merged
mythrocks added a commit that referenced this issue Oct 8, 2024
@mythrocks
Spark 4: Fix parquet_test.py [databricks] (#11519)
5eeddc6 
* Spark 4:  Fix parquet_test.py.

Fixes #11015. (Spark 4 failure.)
Also fixes #11531. (Databricks 14.3 failure.)
Contributes to #11004.

This commit addresses the tests that fail in parquet_test.py, when
run on Spark 4.

1. Some of the tests were failing as a result of #5114.  Those tests
have been disabled, at least until we get around to supporting
aggregations with ANSI mode enabled.

2. `test_parquet_check_schema_compatibility` fails on Spark 4 regardless
of ANSI mode, because it tests implicit type promotions where the read
schema includes wider columns than the write schema.  This will require
new code.  The test is disabled until #11512 is addressed.

3. `test_parquet_int32_downcast` had an erroneous setup phase that fails
   in ANSI mode.  This has been corrected. The test was refactored to
run in ANSI and non-ANSI mode.

Signed-off-by: MithunR <mithunr@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working Spark 4.0+ Spark 4.0+ issues
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mattahrens @razajafri