Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-44277][BUILD] Upgrade to Avro 1.11.2 #41830

Closed
wants to merge 1 commit into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -81,14 +81,14 @@ private[sql] class AvroOptions(

/**
* Top level record name in write result, which is required in Avro spec.
* See https://avro.apache.org/docs/1.11.1/specification/#schema-record .
* See https://avro.apache.org/docs/1.11.2/specification/#schema-record .
* Default value is "topLevelRecord"
*/
val recordName: String = parameters.getOrElse(RECORD_NAME, "topLevelRecord")

/**
* Record namespace in write result. Default value is "".
* See Avro spec for details: https://avro.apache.org/docs/1.11.1/specification/#schema-record .
* See Avro spec for details: https://avro.apache.org/docs/1.11.2/specification/#schema-record .
*/
val recordNamespace: String = parameters.getOrElse(RECORD_NAMESPACE, "")

Expand Down
6 changes: 3 additions & 3 deletions dev/deps/spark-deps-hadoop-3-hive-2.3
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,9 @@ arrow-memory-core/12.0.1//arrow-memory-core-12.0.1.jar
arrow-memory-netty/12.0.1//arrow-memory-netty-12.0.1.jar
arrow-vector/12.0.1//arrow-vector-12.0.1.jar
audience-annotations/0.5.0//audience-annotations-0.5.0.jar
avro-ipc/1.11.1//avro-ipc-1.11.1.jar
avro-mapred/1.11.1//avro-mapred-1.11.1.jar
avro/1.11.1//avro-1.11.1.jar
avro-ipc/1.11.2//avro-ipc-1.11.2.jar
avro-mapred/1.11.2//avro-mapred-1.11.2.jar
avro/1.11.2//avro-1.11.2.jar
aws-java-sdk-bundle/1.12.367//aws-java-sdk-bundle-1.12.367.jar
azure-data-lake-store-sdk/2.3.9//azure-data-lake-store-sdk-2.3.9.jar
azure-keyvault-core/1.0.0//azure-keyvault-core-1.0.0.jar
Expand Down
4 changes: 2 additions & 2 deletions docs/sql-data-sources-avro.md
Original file line number Diff line number Diff line change
Expand Up @@ -417,7 +417,7 @@ applications. Read the [Advanced Dependency Management](https://spark.apache
Submission Guide for more details.

## Supported types for Avro -> Spark SQL conversion
Currently Spark supports reading all [primitive types](https://avro.apache.org/docs/1.11.1/specification/#primitive-types) and [complex types](https://avro.apache.org/docs/1.11.1/specification/#complex-types) under records of Avro.
Currently Spark supports reading all [primitive types](https://avro.apache.org/docs/1.11.2/specification/#primitive-types) and [complex types](https://avro.apache.org/docs/1.11.2/specification/#complex-types) under records of Avro.
<table class="table table-striped">
<thead><tr><th><b>Avro type</b></th><th><b>Spark SQL type</b></th></tr></thead>
<tr>
Expand Down Expand Up @@ -481,7 +481,7 @@ In addition to the types listed above, it supports reading `union` types. The fo
3. `union(something, null)`, where something is any supported Avro type. This will be mapped to the same Spark SQL type as that of something, with nullable set to true.
All other union types are considered complex. They will be mapped to StructType where field names are member0, member1, etc., in accordance with members of the union. This is consistent with the behavior when converting between Avro and Parquet.

It also supports reading the following Avro [logical types](https://avro.apache.org/docs/1.11.1/specification/#logical-types):
It also supports reading the following Avro [logical types](https://avro.apache.org/docs/1.11.2/specification/#logical-types):

<table class="table table-striped">
<thead><tr><th><b>Avro logical type</b></th><th><b>Avro type</b></th><th><b>Spark SQL type</b></th></tr></thead>
Expand Down
2 changes: 1 addition & 1 deletion pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -154,7 +154,7 @@
-->
<codahale.metrics.version>4.2.18</codahale.metrics.version>
<!-- Should be consistent with SparkBuild.scala and docs -->
<avro.version>1.11.1</avro.version>
<avro.version>1.11.2</avro.version>
<aws.kinesis.client.version>1.12.0</aws.kinesis.client.version>
<!-- Should be consistent with Kinesis client dependency -->
<aws.java.sdk.version>1.11.655</aws.java.sdk.version>
Expand Down
2 changes: 1 addition & 1 deletion project/SparkBuild.scala
Original file line number Diff line number Diff line change
Expand Up @@ -1110,7 +1110,7 @@ object DependencyOverrides {
dependencyOverrides += "com.google.guava" % "guava" % guavaVersion,
dependencyOverrides += "xerces" % "xercesImpl" % "2.12.2",
dependencyOverrides += "jline" % "jline" % "2.14.6",
dependencyOverrides += "org.apache.avro" % "avro" % "1.11.1")
dependencyOverrides += "org.apache.avro" % "avro" % "1.11.2")
}

/**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -900,7 +900,7 @@ class HiveClientSuite(version: String, allVersions: Seq[String])
test("Decimal support of Avro Hive serde") {
val tableName = "tab1"
// TODO: add the other logical types. For details, see the link:
// https://avro.apache.org/docs/1.11.1/specification/#logical-types
// https://avro.apache.org/docs/1.11.2/specification/#logical-types
val avroSchema =
"""{
| "name": "test_record",
Expand Down