Spark 2.2.0 Support

[squito]

This adds support for spark 2.2.0. Primarily this is addressing the api change introduced by SPARK-19085 apache/spark@b3d3962. This fixes the issue in the most simplistic way: it copies the old conversion from InternalRow to Row. A better implementation would do something more efficient with InternalRow.

This keeps both write() methods, so it should be compatible with all spark 2+ versions.

It also fixes some dependency conflicts when running tests -- it seems curator has a conflicting version of guava with hadoop, but we don't actually need curator for tests.

Tested by running unit tests locally.

Fixes #240

[squito]

hmm, I guess travis configuration needs to be changed to build with jdk 8

[codecov-io]

Codecov Report

Merging #242 into master will increase coverage by 0.05%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master     #242      +/-   ##
==========================================
+ Coverage    90.4%   90.46%   +0.05%     
==========================================
  Files           5        5              
  Lines         323      325       +2     
  Branches       49       51       +2     
==========================================
+ Hits          292      294       +2     
  Misses         31       31

[marcintustin]

Would be wonderful if this could be merged

[marcintustin]

For now I'm running this with jitpack :(

[rxin]

It is currently failing codecov ...

[ritesh-dineout]

Is there any timeline by which this will be merged and this will be released? We started facing this issue in production, because of aws emr upgrading to spark 2.2.

[squito]

code coverage is now passing.

The added test is pretty lame to get code coverage. Probably the right thing to do is to start testing against spark 2.2, but that requires reconfiguring travis to use java 1.8, and I don't have permissions for that.

[vanzin]

Isn't all travis configuration in https://github.com/databricks/spark-avro/blob/branch-3.2/.travis.yml?

[squito]

oh good point. I actually meant codecoverage, but taking a closer look at the build process I guess that is automatic from travis. lets see what coverage I get from the latest changes

[rxin]

Alright thanks.

[vara-bonthu]

Just checking if this is expected to be released as 3.3.0 some time today? Thanks

[tgravescs]

Just curious when is 3.3 expected to release?

[praneetsharma]

Hi, I have a naive question on InternalRow to Row conversion. Since this conversion will happen for each row, can it result in performance degradation in case of millions of records when compared to spark-avro_3.2.1 (with Spark 2.1.0)?

[febinsathar]

when will 3.3 release? thank you

[gengliangwang]

@squito @marcintustin @ritesh-dineout @febinsathar spark-avro 4.0.0 is released :)