Fix spark400 build in datagen and tests

[jlowe]

Fixes #11372. Adds expression shims for code that's trying to build Column from expressions or extract an expression from a Column. Also unshims and removes an unnecessary base class that no longer needs to be shimmed.

[pxLi]

build

failed ci

[pxLi]

the change would fail shims other than 400 like spark320,

[2024-08-22T00:01:16.183Z] [INFO] --- scala-maven-plugin:4.9.1:testCompile (scala-test-compile-first) @ rapids-4-spark-tests_2.12 ---
[2024-08-22T00:01:16.183Z] [INFO] Compiler bridge file: /home/jenkins/agent/workspace/jenkins-rapids_premerge-github-9916/target/spark320/.sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.12-1.10.0-bin_2.12.15__52.0-1.10.0_20240505T232140.jar
[2024-08-22T00:01:28.363Z] [INFO] compiling 143 Scala sources and 2 Java sources to /home/jenkins/agent/workspace/jenkins-rapids_premerge-github-9916/tests/target/spark320/test-classes ...
[2024-08-22T00:01:28.363Z] [ERROR] [Error] /home/jenkins/agent/workspace/jenkins-rapids_premerge-github-9916/tests/src/test/scala/org/apache/spark/sql/rapids/ProjectExprSuite.scala:34: object DataGenExprShims is not a member of package org.apache.spark.sql.tests.datagen
[2024-08-22T00:01:28.363Z] [ERROR] [Error] /home/jenkins/agent/workspace/jenkins-rapids_premerge-github-9916/tests/src/test/scala/org/apache/spark/sql/rapids/ProjectExprSuite.scala:191: not found: value DataGenExprShims
[2024-08-22T00:01:28.363Z] [ERROR] [Error] /home/jenkins/agent/workspace/jenkins-rapids_premerge-github-9916/tests/src/test/scala/org/apache/spark/sql/rapids/ProjectExprSuite.scala:195: not found: value DataGenExprShims
[2024-08-22T00:01:28.363Z] [ERROR] [Error] /home/jenkins/agent/workspace/jenkins-rapids_premerge-github-9916/tests/src/test/scala/org/apache/spark/sql/rapids/ProjectExprSuite.scala:198: not found: value DataGenExprShims
[2024-08-22T00:01:28.363Z] [ERROR] [Error] /home/jenkins/agent/workspace/jenkins-rapids_premerge-github-9916/tests/src/test/scala/org/apache/spark/sql/rapids/ProjectExprSuite.scala:34: Unused import
[2024-08-22T00:01:28.363Z] [ERROR] [Error] /home/jenkins/agent/workspace/jenkins-rapids_premerge-github-9916/tests/src/test/scala/org/apache/spark/sql/timezone/TimeZonePerfUtils.scala:57: not found: value DataGenExprShims
[2024-08-22T00:01:28.364Z] [ERROR] [Error] /home/jenkins/agent/workspace/jenkins-rapids_premerge-github-9916/tests/src/test/scala/org/apache/spark/sql/timezone/TimeZonePerfUtils.scala:58: not found: value DataGenExprShims

[jlowe]

the change would fail shims other than 400 like spark320

Weird, the github workflow matrix builds for 320 passed, so wonder how the blossom CI failed to build. I also tested this locally before pushing the changes, and it compiled for spark320 and passed the test that is somehow failing to build in blossom CI.

[jlowe]

The CI failure is due to a bug where CI ends up downloading nightly snapshot artifacts to test against rather than the artifacts built from the PR premerge build. See #11382.

[razajafri]

build