Skip to content

Commit

Permalink
ODP-1302 [SPARK-43225][BUILD][SQL] Remove jackson-core-asl and jackso…
Browse files Browse the repository at this point in the history
…n-mapper-asl from pre-built distribution (#39)

- Remove `jackson-core-asl` from maven dependency.
- Change the scope of `jackson-mapper-asl` from compile to test.
- Replace all `Hive.get(conf)` with `Hive.getWithoutRegisterFns(conf)`.

To fix CVE issue: https://github.com/apache/spark/security/dependabot/50.

No.

manual test.

Closes apache#40893 from wangyum/SPARK-43225.

Lead-authored-by: Yuming Wang <wgyumg@gmail.com>
Co-authored-by: Yuming Wang <yumwang@ebay.com>
Signed-off-by: Sean Owen <srowen@gmail.com>
(cherry picked from commit 9c237d7)

[SPARK-43868][SQL][TESTS] Remove `originalUDFs` from `TestHive` to ensure `ObjectHashAggregateExecBenchmark` can run successfully on Github Action

This pr remove `originalUDFs` from `TestHive` to ensure `ObjectHashAggregateExecBenchmark` can run successfully on Github Action.

After SPARK-43225, `org.codehaus.jackson:jackson-mapper-asl` becomes a test scope dependency, so when using GA to run benchmark, it is not in the classpath because GA uses

https://github.com/apache/spark/blob/d61c77cac17029ee27319e6b766b48d314a4dd31/.github/workflows/benchmark.yml#L179-L183

iunstead of the sbt `Test/runMain`.

`ObjectHashAggregateExecBenchmark` used `TestHive`, and `TestHive` will always call `org.apache.hadoop.hive.ql.exec.FunctionRegistry#getFunctionNames` to init `originalUDFs` before this pr, so when we run `ObjectHashAggregateExecBenchmark` on GitHub Actions, there will be the following exceptions:

(cherry picked from commit 1c10e28)

# Conflicts:
#	pom.xml

Co-authored-by: Yuming Wang <wgyumg@gmail.com>
  • Loading branch information
senthh and wangyum authored Sep 18, 2024
1 parent 1003fa3 commit 625c313
Showing 1 changed file with 0 additions and 1 deletion.
1 change: 0 additions & 1 deletion dev/deps/spark-deps-hadoop-3-hive-2.3
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,6 @@ ini4j/0.5.4//ini4j-0.5.4.jar
istack-commons-runtime/3.0.8//istack-commons-runtime-3.0.8.jar
ivy/2.5.1//ivy-2.5.1.jar
jackson-annotations/2.14.2//jackson-annotations-2.14.2.jar
jackson-core-asl/1.9.13//jackson-core-asl-1.9.13.jar
jackson-core/2.14.2//jackson-core-2.14.2.jar
jackson-databind/2.14.2//jackson-databind-2.14.2.jar
jackson-dataformat-cbor/2.14.2//jackson-dataformat-cbor-2.14.2.jar
Expand Down

0 comments on commit 625c313

Please sign in to comment.