Upgraded version for parquet-hadoop #5469
Conversation
malhotrashivam
added
parquet
| @@ -289,7 +289,7 @@ class Classpaths { | |||
|
|
|||
| static void inheritParquetHadoop(Project p, String configName = JavaPlugin.IMPLEMENTATION_CONFIGURATION_NAME) { | |||
| Configuration config = p.configurations.getByName(configName) | |||
| addDependency(config, 'org.apache.parquet', 'parquet-hadoop', '1.13.1') | |||
| addDependency(config, 'org.apache.parquet', 'parquet-hadoop', '1.14.0') | |||
There was a problem hiding this comment.
We should investigate what the release notes means for:
PARQUET-1822 - Parquet without Hadoop dependencies
that would be quite nice if we don't need some of the hadoop deps.
There was a problem hiding this comment.
I was looking at the linked changes and the main commit seems like this: https://github.com/apache/parquet-mr/pull/1141/files#diff-b044ae9879a94e2b8a49d6e6911ea5498ef162df1373cc049ded6256980a7248
One interesting class they have now added are org.apache.parquet.conf.PlainParquetConfiguration to replace org.apache.hadoop.conf.Configuration.
Some other interesting classes that I noticed are
org.apache.parquet.hadoop.CodecFactorywhich can potentially replace the usage oforg.apache.hadoop.io.compress.CompressionCodecFactory.org.apache.parquet.hadoop.CodecFactory.HeapBytesCompressorwhich can replaceorg.apache.hadoop.io.compress.Compressororg.apache.parquet.hadoop.CodecFactory.HeapBytesDecompressorwhich can replaceorg.apache.hadoop.io.compress.Decompressor
I think there is enough here so we can get rid of the hadoop-common dependencies. We would still need parquet-hadoop though which internally would be using hadoop-common only.
Do you think that would be beneficial?
There was a problem hiding this comment.
Sounds good. Is it okay if I make a separate issue for this?
This would require some dedicated effort for the change as well as for the benchmarking.
There was a problem hiding this comment.
Created this issue: #5517
There was a problem hiding this comment.
Please create and link issue to here for follow up wrt https://issues.apache.org/jira/browse/PARQUET-1822
Upgrading to the latest parquet-mr release: https://github.com/apache/parquet-mr/blob/master/CHANGES.md#version-1140