[BUG] test_array_intersect failed in databricks 10.4 runtime and Spark 3.3+

[pxLi]

Describe the bug
related to #5958

[2022-08-03T05:20:04.980Z] FAILED ../../src/main/python/array_test.py::test_array_intersect[Short][INCOMPAT]
[2022-08-03T05:20:04.980Z] FAILED ../../src/main/python/array_test.py::test_array_intersect[Double][INCOMPAT]
[2022-08-03T05:20:04.980Z] FAILED ../../src/main/python/array_test.py::test_array_intersect[Integer][INCOMPAT]
[2022-08-03T05:20:04.980Z] FAILED ../../src/main/python/array_test.py::test_array_intersect[Long][INCOMPAT]
[2022-08-03T05:20:04.980Z] FAILED ../../src/main/python/array_test.py::test_array_intersect[Byte][INCOMPAT]
[2022-08-03T05:20:04.980Z] FAILED ../../src/main/python/array_test.py::test_array_intersect[Float][INCOMPAT]

AssertionError: CPU and GPU list have different lengths at [0, 'sort_array(array_intersect(a, b), true)'] CPU: 1 GPU: 0�

BTW databricks 9.1 runtime works fine.

[abellina]

This also failed in other CI jobs (non databricks) for 22.08 or 22.10.

[NVnavkumar]

This also failed in other CI jobs (non databricks) for 22.08 or 22.10.

confirmed the failure is in Spark 3.3.0

[abellina]

I am seeing a related failure in our Yarn test for test_array_union_before_spark313[Decimal(7,3)]:

cpu = [Decimal('2069.784')], gpu = None

[NVnavkumar]

This also failed in other CI jobs (non databricks) for 22.08 or 22.10.

confirmed the failure is in Spark 3.3.0

The array_intersect test failure looks like a bug in array_intersect that was introduced in Spark 3.3.0. Here is an example of a test failure with this strange output on the CPU:

Row(a=[-3872, -10423, -24617, -13585, -4093, -23070, -8059, -17080], b=[-17558, -29933, -8524, 30046, -24762, None, 23018, 1300, -23335, 13145, -24768], sort_array(array_intersect(a, b), true)=[], sort_array(array_intersect(b, a), true)=[None])

It probably came from here: apache/spark#34077

I'm guessing Databricks might have pulled in the same "fix"

I will look into the array_union issue but that looks like a separate issue from this one.

[NVnavkumar]

Filed https://issues.apache.org/jira/browse/SPARK-39976

This also failed in other CI jobs (non databricks) for 22.08 or 22.10.

confirmed the failure is in Spark 3.3.0

The array_intersect test failure looks like a bug in array_intersect that was introduced in Spark 3.3.0. Here is an example of a test failure with this strange output on the CPU:

Row(a=[-3872, -10423, -24617, -13585, -4093, -23070, -8059, -17080], b=[-17558, -29933, -8524, 30046, -24762, None, 23018, 1300, -23335, 13145, -24768], sort_array(array_intersect(a, b), true)=[], sort_array(array_intersect(b, a), true)=[None])

It probably came from here: apache/spark#34077

I'm guessing Databricks might have pulled in the same "fix"

I will look into the array_union issue but that looks like a separate issue from this one.

[NVnavkumar]

Hmm, I replicated the test scenario on the YARN cluster, and all the tests all passed with the same spark version and via YARN and UCX. This looks like some kind of ephemeral failure at this point.

I am seeing a related failure in our Yarn test for test_array_union_before_spark313[Decimal(7,3)]:

cpu = [Decimal('2069.784')], gpu = None

[NVnavkumar]

This failure occurred again. I'm going to file a new issue to track this one.

Hmm, I replicated the test scenario on the YARN cluster, and all the tests all passed with the same spark version and via YARN and UCX. This looks like some kind of ephemeral failure at this point.

I am seeing a related failure in our Yarn test for test_array_union_before_spark313[Decimal(7,3)]:

cpu = [Decimal('2069.784')], gpu = None

[NVnavkumar]

Filed #6249 for the array_union failure

This failure occurred again. I'm going to file a new issue to track this one.

Hmm, I replicated the test scenario on the YARN cluster, and all the tests all passed with the same spark version and via YARN and UCX. This looks like some kind of ephemeral failure at this point.

I am seeing a related failure in our Yarn test for test_array_union_before_spark313[Decimal(7,3)]:

cpu = [Decimal('2069.784')], gpu = None

[sameerz]

@NVnavkumar please file a follow up issue to revert the changes in #6223 when a fix goes into Spark, and then close this issue.

[NVnavkumar]

Filed a followup issue #6285 to revert the changes once the fix is made in Apache Spark.