Skip to content

Commit

Permalink
[SPARK-26427][BUILD][BACKPORT-2.4] Upgrade Apache ORC to 1.5.4
Browse files Browse the repository at this point in the history
This is a backport of apache#23364.

To make Apache Spark 2.4.1 more robust, this PR aims to update Apache ORC dependency to the latest version 1.5.4 released at Dec. 20. ([Release Notes](https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12318320&version=12344187]))
```
[ORC-237] OrcFile.mergeFiles Specified block size is less than configured minimum value
[ORC-409] Changes for extending MemoryManagerImpl
[ORC-410] Fix a locale-dependent test in TestCsvReader
[ORC-416] Avoid opening data reader when there is no stripe
[ORC-417] Use dynamic Apache Maven mirror link
[ORC-419] Ensure to call `close` at RecordReaderImpl constructor exception
[ORC-432] openjdk 8 has a bug that prevents surefire from working
[ORC-435] Ability to read stripes that are greater than 2GB
[ORC-437] Make acid schema checks case insensitive
[ORC-411] Update build to work with Java 10.
[ORC-418] Fix broken docker build script
```

Build and pass Jenkins.

Closes apache#23646 from dongjoon-hyun/SPARK-26427-2.4.

Authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

Conflicts:
	pom.xml
  • Loading branch information
dongjoon-hyun authored and kai-chi committed Jul 23, 2019
1 parent c00c811 commit d69b43b
Show file tree
Hide file tree
Showing 4 changed files with 14 additions and 10 deletions.
6 changes: 3 additions & 3 deletions dev/deps/spark-deps-hadoop-2.6
Original file line number Diff line number Diff line change
Expand Up @@ -153,9 +153,9 @@ objenesis-2.5.1.jar
okhttp-3.8.1.jar
okio-1.13.0.jar
opencsv-2.3.jar
orc-core-1.5.2-nohive.jar
orc-mapreduce-1.5.2-nohive.jar
orc-shims-1.5.2.jar
orc-core-1.5.4-nohive.jar
orc-mapreduce-1.5.4-nohive.jar
orc-shims-1.5.4.jar
oro-2.0.8.jar
osgi-resource-locator-1.0.1.jar
paranamer-2.8.jar
Expand Down
6 changes: 3 additions & 3 deletions dev/deps/spark-deps-hadoop-2.7
Original file line number Diff line number Diff line change
Expand Up @@ -154,9 +154,9 @@ objenesis-2.5.1.jar
okhttp-3.8.1.jar
okio-1.13.0.jar
opencsv-2.3.jar
orc-core-1.5.2-nohive.jar
orc-mapreduce-1.5.2-nohive.jar
orc-shims-1.5.2.jar
orc-core-1.5.4-nohive.jar
orc-mapreduce-1.5.4-nohive.jar
orc-shims-1.5.4.jar
oro-2.0.8.jar
osgi-resource-locator-1.0.1.jar
paranamer-2.8.jar
Expand Down
6 changes: 3 additions & 3 deletions dev/deps/spark-deps-hadoop-3.1
Original file line number Diff line number Diff line change
Expand Up @@ -172,9 +172,9 @@ okhttp-2.7.5.jar
okhttp-3.8.1.jar
okio-1.13.0.jar
opencsv-2.3.jar
orc-core-1.5.2-nohive.jar
orc-mapreduce-1.5.2-nohive.jar
orc-shims-1.5.2.jar
orc-core-1.5.4-nohive.jar
orc-mapreduce-1.5.4-nohive.jar
orc-shims-1.5.4.jar
oro-2.0.8.jar
osgi-resource-locator-1.0.1.jar
paranamer-2.8.jar
Expand Down
6 changes: 5 additions & 1 deletion pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -132,7 +132,7 @@
<hive.version.short>3.0.0.1</hive.version.short>
<derby.version>10.12.1.1</derby.version>
<parquet.version>1.10.0</parquet.version>
<orc.version>1.5.2</orc.version>
<orc.version>1.5.4</orc.version>
<orc.classifier></orc.classifier>
<hive.parquet.version>1.6.0</hive.parquet.version>
<jetty.version>9.3.24.v20180605</jetty.version>
Expand Down Expand Up @@ -1814,6 +1814,10 @@
<classifier>${orc.classifier}</classifier>
<scope>${orc.deps.scope}</scope>
<exclusions>
<exclusion>
<groupId>javax.xml.bind</groupId>
<artifactId>jaxb-api</artifactId>
</exclusion>
<exclusion>
<groupId>io.hops</groupId>
<artifactId>hadoop-common</artifactId>
Expand Down

0 comments on commit d69b43b

Please sign in to comment.