[CELEBORN-1413] Support Spark 4.0

[FMX]

What changes were proposed in this pull request?

To support Spark 4.0.0 preview.

Why are the changes needed?

1. Changed Scala to 2.13.
2. Introduce columnar shuffle module for spark 4.0.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Cluster test.

[github-actions]

This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[FMX]

Will update this PR next week.

[FMX]

The CI for dependency check is not related to this PR. I'll fix that in another PR.

[FMX]

Spark-4.0 profile can be compiled in JDK 17+ environment only.

[SteNicholas]

I have already tested the implementation of Spark 4.0 integration in internal environment of Spark 4.0 and worked well for Spark 4.0.

[SteNicholas]

LGTM.

[RexXiong]

cc @pan3793

[mridulm]

QQ: Why are we moving spark-3 to spark-3-4 ?
It is possible that spark-4 impl ends up evolving nontrivially in time, and in a manner which conflicts with spark-3.

Do we want to simply keep spark-3 (which eventually goes away when support for spark-3.5 is dropped), and start with a new spark-4 ?

I understand the duplication aspect could be nontrivial - wondering if there is a middle ground for 3.5 vs 4, which minimizes this.

This would also minimize impact on existing spark-3 users (from dep pov)

[mridulm]

Took a quick pass, will try to go over it in more details later during holidays.
But please dont block on my review.

[FMX]

QQ: Why are we moving spark-3 to spark-3-4 ? It is possible that spark-4 impl ends up evolving nontrivially in time, and in a manner which conflicts with spark-3.

Do we want to simply keep spark-3 (which eventually goes away when support for spark-3.5 is dropped), and start with a new spark-4 ?

I understand the duplication aspect could be nontrivial - wondering if there is a middle ground for 3.5 vs 4, which minimizes this.

This would also minimize impact on existing spark-3 users (from dep pov)

The main reason to add a module to share within spark 3.5 and spark 4 is because the shuffle APIs are stable. The unstable APIs are in the columnar shuffle module which is separated from the module of spark 3.5.