[SPARK-49569][BUILD][FOLLOWUP] Exclude `spark-connect-shims` from `sql/core` module #48403

LuciferYang · 2024-10-09T18:55:59Z

What changes were proposed in this pull request?

This pr exclude spark-connect-shims from sql/core module for further fix maven daily test.

Why are the changes needed?

For fix maven daily test:

After #48399, although the Maven build was successful in my local environment, the Maven daily test pipeline still failed to build:

https://github.com/apache/spark/actions/runs/11255598249/job/31311358712

Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/ApproximatePercentileQuerySuite.scala:121: value makeRDD is not a member of org.apache.spark.SparkContext
Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:75: value id is not a member of org.apache.spark.rdd.RDD[org.apache.spark.sql.columnar.CachedBatch]
Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:82: value env is not a member of org.apache.spark.SparkContext
Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:88: value env is not a member of org.apache.spark.SparkContext
Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:185: value parallelize is not a member of org.apache.spark.SparkContext
Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:481: value cleaner is not a member of org.apache.spark.SparkContext
Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:500: value parallelize is not a member of org.apache.spark.SparkContext
Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:940: value addSparkListener is not a member of org.apache.spark.SparkContext
Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:943: value listenerBus is not a member of org.apache.spark.SparkContext
Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:947: value removeSparkListener is not a member of org.apache.spark.SparkContext
Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:1667: value listenerBus is not a member of org.apache.spark.SparkContext
Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:1668: value addSparkListener is not a member of org.apache.spark.SparkContext
Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:1673: value partitions is not a member of org.apache.spark.rdd.RDD[org.apache.spark.sql.Row]
Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:1674: value listenerBus is not a member of org.apache.spark.SparkContext
Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:1682: value partitions is not a member of org.apache.spark.rdd.RDD[org.apache.spark.sql.Row]
Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:1683: value listenerBus is not a member of org.apache.spark.SparkContext
Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:1687: value removeSparkListener is not a member of org.apache.spark.SparkContext
Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:1708: value partitions is not a member of org.apache.spark.rdd.RDD[org.apache.spark.sql.Row]
...

After using the mvn dependency:tree command to check, I found that sql/core cascadingly introduced org.apache.spark:spark-connect-shims_2.13:jar:4.0.0-SNAPSHOT:test through org.apache.spark:spark-sql-api_2.13:test-jar:tests:4.0.0-SNAPSHOT:test.

[INFO] ------------------< org.apache.spark:spark-sql_2.13 >-------------------
[INFO] Building Spark Project SQL 4.0.0-SNAPSHOT                        [18/42]
[INFO]   from sql/core/pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- dependency:3.6.1:tree (default-cli) @ spark-sql_2.13 ---
[INFO] org.apache.spark:spark-sql_2.13:jar:4.0.0-SNAPSHOT
...
[INFO] +- org.apache.spark:spark-catalyst_2.13:test-jar:tests:4.0.0-SNAPSHOT:test
[INFO] +- org.apache.spark:spark-sql-api_2.13:test-jar:tests:4.0.0-SNAPSHOT:test
[INFO] |  +- org.scala-lang.modules:scala-parser-combinators_2.13:jar:2.4.0:compile
[INFO] |  +- org.apache.spark:spark-connect-shims_2.13:jar:4.0.0-SNAPSHOT:test
[INFO] |  +- org.antlr:antlr4-runtime:jar:4.13.1:compile
[INFO] |  +- org.apache.arrow:arrow-vector:jar:17.0.0:compile
[INFO] |  |  +- org.apache.arrow:arrow-format:jar:17.0.0:compile
[INFO] |  |  +- org.apache.arrow:arrow-memory-core:jar:17.0.0:compile
[INFO] |  |  +- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar:2.18.0:compile
[INFO] |  |  \- com.google.flatbuffers:flatbuffers-java:jar:24.3.25:compile
[INFO] |  \- org.apache.arrow:arrow-memory-netty:jar:17.0.0:compile
[INFO] |     \- org.apache.arrow:arrow-memory-netty-buffer-patch:jar:17.0.0:compile

This should be unexpected.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Pass GitHub Actions
Pass maven test on GitHub Actions: https://github.com/LuciferYang/spark/runs/31314342332

All maven test passed

Was this patch authored or co-authored using generative AI tooling?

No

LuciferYang · 2024-10-09T18:57:53Z

.github/workflows/build_and_test.yml

@@ -306,6 +306,12 @@ jobs:
        name: unit-tests-log-${{ matrix.modules }}-${{ matrix.comment }}-${{ matrix.java }}-${{ matrix.hadoop }}-${{ matrix.hive }}
        path: "**/target/*.log"

+  maven-test:


Let's make another attempt. This is to confirm that the GA can be successfully built. Once it can be successfully built, we will delete this piece of code.

LuciferYang · 2024-10-10T02:42:07Z

all maven test passed

LuciferYang · 2024-10-10T03:11:19Z

cc @hvanhovell @HyukjinKwon @dongjoon-hyun FYI

HyukjinKwon · 2024-10-10T04:25:32Z

I think it needs @hvanhovell's look

LuciferYang · 2024-10-10T04:45:12Z

I think it needs @hvanhovell's look

OK

LuciferYang · 2024-10-10T15:33:06Z

Merged into master. Thanks @hvanhovell and @HyukjinKwon

LuciferYang · 2024-10-11T03:08:36Z

After merging this PR, Maven daily test has been restored:

Java 17: https://github.com/apache/spark/actions/runs/11274643792

Java 21: https://github.com/apache/spark/actions/runs/11275643152

I just thought of a question that needs confirmation. In which directory should the spark-connect-shims.jar be located in the distribution? Currently, after executing dev/make-distribution.sh --tgz, it exists in the jars directory but not in the jars/connect-repl directory. Is this expected? @hvanhovell @HyukjinKwon

HyukjinKwon · 2024-10-11T03:32:16Z

FWIW, I think it's gonna fix https://github.com/apache/spark/actions/runs/11259624487/job/31309026637 too

LuciferYang · 2024-10-11T05:59:06Z

FWIW, I think it's gonna fix https://github.com/apache/spark/actions/runs/11259624487/job/31309026637 too

Not successful？It seems that there is still a class conflict caused by classpath issues?

HyukjinKwon · 2024-10-11T07:08:56Z

actually yeah seems like all REPL are broken after Maven build. Taking a look.

LuciferYang · 2024-10-11T07:42:42Z

actually yeah seems like all REPL are broken after Maven build. Taking a look.

we should move spark-connect-shims.jar from jars to jars/connect-repl

HyukjinKwon · 2024-10-11T07:52:52Z

@LuciferYang mind creating a PR when you find some time? 🙏

LuciferYang · 2024-10-11T07:54:23Z

@LuciferYang mind creating a PR when you find some time? 🙏

OK ~

init

0a63e7c

github-actions bot added SQL BUILD INFRA labels Oct 9, 2024

LuciferYang commented Oct 9, 2024

View reviewed changes

Revert maven test

2cb3fec

github-actions bot removed the INFRA label Oct 10, 2024

LuciferYang changed the title ~~Test Maven Build~~ [SPARK-49569][BUILD][FOLLOWUP] Exclude spark-connect-shims from sql/core module Oct 10, 2024

hvanhovell approved these changes Oct 10, 2024

View reviewed changes

Merge branch 'apache:master' into test-maven-build

55b4e80

LuciferYang closed this in b056e0b Oct 10, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-49569][BUILD][FOLLOWUP] Exclude `spark-connect-shims` from `sql/core` module #48403

[SPARK-49569][BUILD][FOLLOWUP] Exclude `spark-connect-shims` from `sql/core` module #48403

LuciferYang commented Oct 9, 2024 •

edited

Loading

LuciferYang Oct 9, 2024

LuciferYang commented Oct 10, 2024

LuciferYang commented Oct 10, 2024

HyukjinKwon commented Oct 10, 2024

LuciferYang commented Oct 10, 2024

LuciferYang commented Oct 10, 2024

LuciferYang commented Oct 11, 2024 •

edited

Loading

HyukjinKwon commented Oct 11, 2024 •

edited

Loading

LuciferYang commented Oct 11, 2024

HyukjinKwon commented Oct 11, 2024

LuciferYang commented Oct 11, 2024

HyukjinKwon commented Oct 11, 2024

LuciferYang commented Oct 11, 2024

[SPARK-49569][BUILD][FOLLOWUP] Exclude spark-connect-shims from sql/core module #48403

[SPARK-49569][BUILD][FOLLOWUP] Exclude spark-connect-shims from sql/core module #48403

Conversation

LuciferYang commented Oct 9, 2024 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

LuciferYang Oct 9, 2024

Choose a reason for hiding this comment

LuciferYang commented Oct 10, 2024

LuciferYang commented Oct 10, 2024

HyukjinKwon commented Oct 10, 2024

LuciferYang commented Oct 10, 2024

LuciferYang commented Oct 10, 2024

LuciferYang commented Oct 11, 2024 • edited Loading

HyukjinKwon commented Oct 11, 2024 • edited Loading

LuciferYang commented Oct 11, 2024

HyukjinKwon commented Oct 11, 2024

LuciferYang commented Oct 11, 2024

HyukjinKwon commented Oct 11, 2024

LuciferYang commented Oct 11, 2024

[SPARK-49569][BUILD][FOLLOWUP] Exclude `spark-connect-shims` from `sql/core` module #48403

[SPARK-49569][BUILD][FOLLOWUP] Exclude `spark-connect-shims` from `sql/core` module #48403

LuciferYang commented Oct 9, 2024 •

edited

Loading

LuciferYang commented Oct 11, 2024 •

edited

Loading

HyukjinKwon commented Oct 11, 2024 •

edited

Loading