[SPARK-4520] [SQL] This pr fixes the ArrayIndexOutOfBoundsException as r...

[dhruvasood]

...aised in SPARK-4520.

The exception is thrown only for a thrift generated parquet file. The array element schema name is assumed as "array" as per ParquetAvro but for thrift generated parquet files, it is array_name + "_tuple". This leads to missing child of array group type and hence when the parquet rows are being materialized leads to the exception.

[AmplabJenkins]

Can one of the admins verify this patch?

[liancheng]

ok to test

[SparkQA]

Test build #26798 has started for PR 4148 at commit c5ccde8.

• This patch merges cleanly.

[liancheng]

Double checked parquet-avro, parquet-protobuf, and parquet-thrift. This solution LGTM as long as Jenkins agrees. Also, I think this worth a JIRA ticket for Parquet.

[liancheng]

Would like to add that https://github.com/apache/incubator-parquet-format/pull/17 is trying to add specs for Parquet LIST and MAP annotations. The Thrift _tuple issue is also mentioned.

[SparkQA]

Test build #26798 has finished for PR 4148 at commit c5ccde8.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26798/
Test PASSed.

[liancheng]

Merged this to master and branch-1.3. Thanks!

PS: My first merge as a committer :)