v4.0.0

The 4.0.0 release adds support for Spark 2.2. The published artifact is compatible with both Spark 2.1 and 2.2.

New Features:

• Support for Spark 2.2 (#242)

Bug fixes:

• Fix name conflict in nested records (#249)

v3.2.0

⚠️ Important: If you are using Spark 1.x, then use v2.0.1 instead. 3.x releases of this library are only compatible with Spark 2.x.

The 3.2.0 release (which supports Spark 2.x) adds support for Spark 2.1.0+. The published artifact is compatible with both Spark 2.0.x and 2.1.x.

This will most likely be the last release to support Spark 2.0.x. New features will be released in version 4.0.0, which will only support Spark 2.1.0 and higher.

New Features:

• Support for Spark 2.1.0 (#206 and #212)

v3.1.0

⚠️ Important: If you are using Spark 1.x, then use v2.0.1 instead. 3.x releases of this library are only compatible with Spark 2.x.

The 3.1.0 release (which supports Spark 2.x) contains the following changes:

New Features:

• Custom schema support:
  • Support for user-defined Avro schemas when reading: using the avroSchema option, users can specify a custom Avro schema (as a JSON string) to use when reading Avro files (#160, #161). Default values specified in the Avro schema will be respected (#176, #195).
  • Improved handling of custom schemas specified via .schema(): clarified handling of additional / missing fields to support more types of schema evolution (#155, #96).
  • Together, these changes support several schema-evolution-related use-cases: #31, #165, #49.
• Improved UNION support: previously, this library only supported union(int, long), union(float, double) and union(*, null), but as of this release it now support all other union types ("complex unions") by converting them into structs containing at most one non-null field (#108, #117).

Bug fixes:

• Avro files are now splittable during reads: version 3.0 of this library broke the ability to read a large Avro file as multiple splits / partitions. As of 3.1.0, such splitting / partitioning is supported again (#179, #182).

v3.0.1

The 3.0.1 release (which is compatible with Spark 2.0.0 or higher) includes the following bug fixes:

• Fix NullPointerException when using Kryo serialization (#147, #148, #170).
• Close input streams to avoid a resource leak which could cause reads to hang (#156, #174)

In addition, the README examples have been updated to use SparkSession instead of SqlContext (which is deprecated as of Spark 2.0).