Refactor Parquet compression handling to avoid native libraries #2397

niloc132 · 2022-05-17T21:40:57Z

This still needs some testing on arm platforms, but should not only remove the "parquet" codecs and replace with "hadoop-only" ones, but still let these codecs be used by both readers and writers.

Downstream builds as of this patch can no longer use Brotli, but demand for brotli seems spotty anyway. Adding the same unmaintained rdblue brolti codec jar to the classpath would automatically have it be picked up by the factory (confirmed by the test-only dependency), without leaving it on the classpath for platforms that don't support it. Follow-up work should selectively include that jar in x86 docker containers.

Configuration is stubbed in, but not well used yet, allowing certain codecs to be added ahead of others. This could for example let an implementation specify that even though there is a fallback lzo codec implemented in java, a particular native implementation should be used instead.

Fixes #2118

niloc132 · 2022-05-17T21:42:41Z

ParquetHadoop/build.gradle

-    runtimeOnly('com.hadoop.gplcompression:hadoop-lzo:0.4.20')  // For LZO codec.
-    runtimeOnly('org.lz4:lz4-java:1.8.0')                       // For LZ4 codec.
-    runtimeOnly('com.github.rdblue:brotli-codec:0.1.1')         // For Brotli codec.
+    runtimeOnly('com.fasterxml.woodstox:woodstox-core:5.2.1') {


#1162 will make this less terrible

niloc132 · 2022-05-17T21:42:42Z

buildSrc/src/main/groovy/io.deephaven.java-classpath-conventions.gradle

@@ -70,7 +70,7 @@ dependencies {
            'net.sf.trove4j:trove4j:3.0.3',
            'com.intellij:annotations:5.1',
            'commons-codec:commons-codec:1.11',
-            'org.apache.commons:commons-compress:1.18'
+            depCommonsCompress


...s/parquet/compression/src/main/java/io/deephaven/parquet/compress/NonClosingInputStream.java

.../parquet/compression/src/main/java/io/deephaven/parquet/compress/NonClosingOutputStream.java

extensions/parquet/base/src/main/java/io/deephaven/parquet/base/ParquetFileReader.java

niloc132 · 2022-05-17T21:47:47Z

extensions/parquet/base/src/main/java/io/deephaven/parquet/base/ParquetFileWriter.java

            final Map<String, String> extraMetaData) throws IOException {
        this.pageSize = pageSize;
        this.allocator = allocator;
        this.extraMetaData = new HashMap<>(extraMetaData);
        writeChannel = channelsProvider.getWriteChannel(filePath, false); // TODO add support for appending
        this.type = type;
        this.channelsProvider = channelsProvider;
-        CodecFactory codecFactory = new CodecFactory(new Configuration(), pageSize);
-        this.compressor = codecFactory.getCompressor(codecName);
+        DeephavenCodecFactory ccf = new DeephavenCodecFactory(Collections.emptyList());


ditto, should probably be singleton, so the configured classes are shared.

Note however that it has no state, except the CompressionCodecFactory, and that only has state of the codecs it was built with, no other state maintained along the way - perhaps default ctor instead would pick up the list from Configuration?

buildSrc/src/main/groovy/io.deephaven.java-repository-conventions.gradle

extensions/parquet/compression/src/main/java/io/deephaven/parquet/compress/codec/LzoCodec.java

extensions/parquet/compression/src/main/java/io/deephaven/parquet/compress/codec/ZstdCodec.java

extensions/parquet/compression/src/main/java/io/deephaven/parquet/compress/Compressor.java

extensions/parquet/compression/gradle.properties

extensions/parquet/base/src/main/java/io/deephaven/parquet/base/ColumnWriterImpl.java

extensions/parquet/base/src/main/java/io/deephaven/parquet/base/ColumnChunkReaderImpl.java

extensions/parquet/base/src/main/java/io/deephaven/parquet/base/ColumnPageReaderImpl.java

buildSrc/src/main/groovy/io.deephaven.java-repository-conventions.gradle

...s/parquet/compression/src/main/java/io/deephaven/parquet/compress/DeephavenCodecFactory.java

rcaudy

Currently, this fails when I try to run tests. For ParquetTableReadWriteTest:

Whichever one runs first (usually testParquetZstdCompressionCodec) fails with:

org.apache.hadoop.io.compress.CompressionCodec: Provider org.apache.hadoop.io.compress.BrotliCodec could not be instantiated
java.util.ServiceConfigurationError: org.apache.hadoop.io.compress.CompressionCodec: Provider org.apache.hadoop.io.compress.BrotliCodec could not be instantiated
	at java.base/java.util.ServiceLoader.fail(ServiceLoader.java:582)
	at java.base/java.util.ServiceLoader$ProviderImpl.newInstance(ServiceLoader.java:804)
	at java.base/java.util.ServiceLoader$ProviderImpl.get(ServiceLoader.java:722)
	at java.base/java.util.ServiceLoader$3.next(ServiceLoader.java:1395)
	at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:119)
	at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:180)
	at io.deephaven.parquet.compress.DeephavenCodecFactory.<init>(DeephavenCodecFactory.java:123)
	at io.deephaven.parquet.compress.DeephavenCodecFactory.<init>(DeephavenCodecFactory.java:119)
	at io.deephaven.parquet.compress.DeephavenCodecFactory.<clinit>(DeephavenCodecFactory.java:38)
	at io.deephaven.parquet.base.ParquetFileWriter.<init>(ParquetFileWriter.java:56)
	at io.deephaven.parquet.table.ParquetTableWriter.getParquetFileWriter(ParquetTableWriter.java:314)
	at io.deephaven.parquet.table.ParquetTableWriter.write(ParquetTableWriter.java:210)
	at io.deephaven.parquet.table.ParquetTableWriter.write(ParquetTableWriter.java:155)
	at io.deephaven.parquet.table.ParquetTableWriter.write(ParquetTableWriter.java:179)
	at io.deephaven.parquet.table.ParquetTools.writeParquetTableImpl(ParquetTools.java:670)
	at io.deephaven.parquet.table.ParquetTools.writeTable(ParquetTools.java:224)
	at io.deephaven.parquet.table.ParquetTools.writeTable(ParquetTools.java:140)
	at io.deephaven.parquet.table.ParquetTableReadWriteTest.compressionCodecTestHelper(ParquetTableReadWriteTest.java:265)
	at io.deephaven.parquet.table.ParquetTableReadWriteTest.testParquetZstdCompressionCodec(ParquetTableReadWriteTest.java:296)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
	at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
	at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:119)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182)
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:164)
	at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:414)
	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.UnsatisfiedLinkError: Couldn't load native library 'brotli'. [LoaderResult: os.name="Mac OS X", os.arch="aarch64", os.version="12.4", java.vm.name="OpenJDK 64-Bit Server VM", java.vm.version="11.0.14+9-LTS", java.vm.vendor="Azul Systems, Inc.", alreadyLoaded="null", loadedFromSystemLibraryPath="false", nativeLibName="libbrotli.dylib", temporaryLibFile="/var/folders/16/1ktb8rbx1rx2c17ykj42jxhr0000gn/T/brotli14466667012403530655/libbrotli.dylib", libNameWithinClasspath="/lib/darwin-aarch64/libbrotli.dylib", usedThisClassloader="false", usedSystemClassloader="false", java.library.path="/Users/rcaudy/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:."]
	at org.meteogroup.jbrotli.libloader.BrotliLibraryLoader.loadBrotli(BrotliLibraryLoader.java:35)
	at org.apache.hadoop.io.compress.BrotliCodec.<init>(BrotliCodec.java:40)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
	at java.base/java.util.ServiceLoader$ProviderImpl.newInstance(ServiceLoader.java:780)
	... 69 more

The remainder fail with:

Could not initialize class io.deephaven.parquet.compress.DeephavenCodecFactory
java.lang.NoClassDefFoundError: Could not initialize class io.deephaven.parquet.compress.DeephavenCodecFactory
	at io.deephaven.parquet.base.ParquetFileWriter.<init>(ParquetFileWriter.java:56)
	at io.deephaven.parquet.table.ParquetTableWriter.getParquetFileWriter(ParquetTableWriter.java:314)
	at io.deephaven.parquet.table.ParquetTableWriter.write(ParquetTableWriter.java:210)
	at io.deephaven.parquet.table.ParquetTableWriter.write(ParquetTableWriter.java:155)
	at io.deephaven.parquet.table.ParquetTableWriter.write(ParquetTableWriter.java:179)
	at io.deephaven.parquet.table.ParquetTableWriter.write(ParquetTableWriter.java:151)
	at io.deephaven.parquet.table.ParquetTools.writeParquetTableImpl(ParquetTools.java:666)
	at io.deephaven.parquet.table.ParquetTools.writeTable(ParquetTools.java:224)
	at io.deephaven.parquet.table.ParquetTools.writeTable(ParquetTools.java:153)
	at io.deephaven.parquet.table.ParquetTableReadWriteTest.groupingByLongKey(ParquetTableReadWriteTest.java:221)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
	at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
	at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:119)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182)
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:164)
	at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:414)
	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
	at java.base/java.lang.Thread.run(Thread.java:829)

extensions/parquet/compression/README.md

...s/parquet/compression/src/main/java/io/deephaven/parquet/compress/DeephavenCodecFactory.java

…uet/compress/DeephavenCodecFactory.java Co-authored-by: Ryan Caudy <rcaudy@gmail.com>

niloc132 · 2022-05-31T14:16:24Z

Okay, since brotli blows up in its constructor, we're a bit limited in what we can do around this. Option one, reimplement the actual BrotliCodec class (only about 100 lines long, none of it interesting) to fail later (lazily init the underlying native code). Option two, factor out the test and the wiring that runs it so that gradle only attempts to invoke it on the correct platform.

I think option two is better, letting the brotli codec remain self contained, and just keep it off the classpath except for this specific test.

Co-authored-by: Ryan Caudy <rcaudy@gmail.com>

...et/table/src/brotliTest/java/io/deephaven/parquet/table/BroltiParquetTableReadWriteTest.java

rcaudy · 2022-05-31T22:44:02Z

...et/table/src/brotliTest/java/io/deephaven/parquet/table/BroltiParquetTableReadWriteTest.java

+
+import static org.junit.Assert.assertTrue;
+
+public class BroltiParquetTableReadWriteTest {


Shouldn't we be an @Category(OutOfBandTest.class)?

No - the codec tests are basically instant, the read/write test should probably be split into two classes, and only run the slow tests as out of band?

Accidental approve

rcaudy · 2022-05-31T22:46:29Z

Still failing on Mac OS/M1:

Could not initialize class org.xerial.snappy.Snappy
java.lang.NoClassDefFoundError: Could not initialize class org.xerial.snappy.Snappy
	at org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressDirectBuf(SnappyCompressor.java:282)
	at org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:210)
	at org.apache.hadoop.io.compress.BlockCompressorStream.compress(BlockCompressorStream.java:149)
	at org.apache.hadoop.io.compress.BlockCompressorStream.finish(BlockCompressorStream.java:142)
	at org.apache.hadoop.io.compress.CompressionOutputStream.close(CompressionOutputStream.java:64)
	at org.apache.hadoop.io.compress.CompressorStream.close(CompressorStream.java:106)
	at java.base/java.nio.channels.Channels$WritableByteChannelImpl.implCloseChannel(Channels.java:476)
	at java.base/java.nio.channels.spi.AbstractInterruptibleChannel.close(AbstractInterruptibleChannel.java:112)
	at io.deephaven.parquet.base.ColumnWriterImpl.writeDictionaryPage(ColumnWriterImpl.java:131)
	at io.deephaven.parquet.base.ColumnWriterImpl.addDictionaryPage(ColumnWriterImpl.java:117)
	at io.deephaven.parquet.table.ParquetTableWriter.writeColumnSource(ParquetTableWriter.java:476)
	at io.deephaven.parquet.table.ParquetTableWriter.write(ParquetTableWriter.java:223)
	at io.deephaven.parquet.table.ParquetTableWriter.write(ParquetTableWriter.java:155)
	at io.deephaven.parquet.table.ParquetTableWriter.write(ParquetTableWriter.java:179)
	at io.deephaven.parquet.table.ParquetTools.writeParquetTableImpl(ParquetTools.java:670)
	at io.deephaven.parquet.table.ParquetTools.writeTable(ParquetTools.java:224)
	at io.deephaven.parquet.table.ParquetTools.writeTable(ParquetTools.java:153)
	at io.deephaven.parquet.table.ParquetTableReadWriteTest.flatTable(ParquetTableReadWriteTest.java:155)
	at io.deephaven.parquet.table.ParquetTableReadWriteTest.flatParquetFormat(ParquetTableReadWriteTest.java:199)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
	at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
	at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:119)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182)
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:164)
	at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:414)
	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
	at java.base/java.lang.Thread.run(Thread.java:829)

rcaudy · 2022-05-31T22:53:19Z

[Test worker] INFO io.deephaven.configuration.PropertyFile - io.deephaven.engine.rowset.impl.sortedranges.SortedRanges: property SortedRanges.debug = false
[Test worker] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new compressor [.snappy]

[FAILED_TO_LOAD_NATIVE_LIBRARY] no native library is found for os.name=Mac and os.arch=aarch64
org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] no native library is found for os.name=Mac and os.arch=aarch64
	at org.xerial.snappy.SnappyLoader.findNativeLibrary(SnappyLoader.java:361)
	at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:195)
	at org.xerial.snappy.SnappyLoader.loadSnappyApi(SnappyLoader.java:167)
	at org.xerial.snappy.Snappy.init(Snappy.java:69)
	at org.xerial.snappy.Snappy.<clinit>(Snappy.java:46)
	at org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressDirectBuf(SnappyCompressor.java:282)
	at org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:210)
	at org.apache.hadoop.io.compress.BlockCompressorStream.compress(BlockCompressorStream.java:149)
	at org.apache.hadoop.io.compress.BlockCompressorStream.finish(BlockCompressorStream.java:142)
	at org.apache.hadoop.io.compress.CompressionOutputStream.close(CompressionOutputStream.java:64)
	at org.apache.hadoop.io.compress.CompressorStream.close(CompressorStream.java:106)
	at io.deephaven.parquet.base.ColumnWriterImpl.writePage(ColumnWriterImpl.java:301)
	at io.deephaven.parquet.base.ColumnWriterImpl.writePage(ColumnWriterImpl.java:370)
	at io.deephaven.parquet.base.ColumnWriterImpl.addPageNoNulls(ColumnWriterImpl.java:91)
	at io.deephaven.parquet.table.ParquetTableWriter.writeColumnSource(ParquetTableWriter.java:523)
	at io.deephaven.parquet.table.ParquetTableWriter.write(ParquetTableWriter.java:223)
	at io.deephaven.parquet.table.ParquetTableWriter.write(ParquetTableWriter.java:155)
	at io.deephaven.parquet.table.ParquetTableWriter.write(ParquetTableWriter.java:179)
	at io.deephaven.parquet.table.ParquetTableWriter.write(ParquetTableWriter.java:151)
	at io.deephaven.parquet.table.ParquetTools.writeParquetTableImpl(ParquetTools.java:666)
	at io.deephaven.parquet.table.ParquetTools.writeTable(ParquetTools.java:224)
	at io.deephaven.parquet.table.ParquetTools.writeTable(ParquetTools.java:153)
	at io.deephaven.parquet.table.ParquetTableReadWriteTest.groupingByLongKey(ParquetTableReadWriteTest.java:222)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
	at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
	at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:119)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182)
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:164)
	at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:414)
	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
	at java.base/java.lang.Thread.run(Thread.java:829)

devinrsmith · 2022-06-01T18:03:13Z

Super happy we are reducing our external repositories. Created this for follow-up: #2459

niloc132 · 2022-06-03T14:24:42Z

Snappy should be fixed now, confirmed on arm by @lbooker42.

niloc132 requested review from rcaudy and jcferretti May 17, 2022 21:40

niloc132 added DocumentationNeeded parquet Related to the Parquet integration labels May 17, 2022

niloc132 added this to the May 2022 milestone May 17, 2022

niloc132 commented May 17, 2022

View reviewed changes

rcaudy reviewed May 25, 2022

View reviewed changes

niloc132 added 10 commits May 27, 2022 16:32

WIP

3dfb314

ryan.happy()

18e7c74

Add service file, two impls, this will be rewritten as configs instead

a42c229

WIP

846ee3a

Custom codec factory

de38bc2

Reimplement decompress

c70cb94

Add docs, remove unused imports and dead code

6a9015d

spotless

7060e0f

Review feedback

576ce31

Read codecs from configs, use a singleton, but allow users to replace

3c1a110

niloc132 force-pushed the 2118-platform-specific-libraries branch from 132c7bf to 3c1a110 Compare May 27, 2022 21:32

niloc132 commented May 27, 2022

View reviewed changes

buildSrc/src/main/groovy/io.deephaven.java-repository-conventions.gradle Show resolved Hide resolved

niloc132 force-pushed the 2118-platform-specific-libraries branch from 0cfbbf6 to 55cd5fe Compare May 27, 2022 21:43

rcaudy reviewed May 27, 2022

View reviewed changes

...s/parquet/compression/src/main/java/io/deephaven/parquet/compress/DeephavenCodecFactory.java Outdated Show resolved Hide resolved

niloc132 added 2 commits May 27, 2022 16:59

Restore missing kafka repo

60fdcfa

Change singleton, other review feedback

c8ff3ec

niloc132 force-pushed the 2118-platform-specific-libraries branch from 55cd5fe to c8ff3ec Compare May 27, 2022 21:59

niloc132 added 2 commits May 27, 2022 17:03

Indicate that failure is expected for brotli on some platforms

4ba49cf

spotless

b7fcf54

rcaudy reviewed May 28, 2022

View reviewed changes

extensions/parquet/compression/README.md Outdated Show resolved Hide resolved

...s/parquet/compression/src/main/java/io/deephaven/parquet/compress/DeephavenCodecFactory.java Outdated Show resolved Hide resolved

Update extensions/parquet/compression/src/main/java/io/deephaven/parq…

946a674

…uet/compress/DeephavenCodecFactory.java Co-authored-by: Ryan Caudy <rcaudy@gmail.com>

Move the broli test to its own sourceset/classpath

fdd0b42

Update extensions/parquet/compression/README.md

8d84827

Co-authored-by: Ryan Caudy <rcaudy@gmail.com>

rcaudy previously approved these changes May 31, 2022

View reviewed changes

Use aircompressor's snappy impl instead of the default

42fccf8

rcaudy previously approved these changes Jun 3, 2022

View reviewed changes

spotless is wrong

570191a

niloc132 dismissed rcaudy’s stale review via 570191a June 3, 2022 14:31

rcaudy previously approved these changes Jun 3, 2022

View reviewed changes

Fix typo, more docs, missing newline

6cfe9bb

niloc132 dismissed rcaudy’s stale review via 6cfe9bb June 3, 2022 15:57

rcaudy approved these changes Jun 3, 2022

View reviewed changes

jcferretti approved these changes Jun 3, 2022

View reviewed changes

niloc132 merged commit c289b18 into deephaven:main Jun 3, 2022

github-actions bot locked and limited conversation to collaborators Jun 3, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor Parquet compression handling to avoid native libraries #2397

Refactor Parquet compression handling to avoid native libraries #2397

niloc132 commented May 17, 2022

niloc132 May 17, 2022

niloc132 May 17, 2022

niloc132 May 17, 2022

niloc132 May 17, 2022

rcaudy left a comment

niloc132 commented May 31, 2022

rcaudy May 31, 2022

niloc132 Jun 3, 2022

rcaudy commented May 31, 2022

rcaudy commented May 31, 2022

devinrsmith commented Jun 1, 2022

niloc132 commented Jun 3, 2022


		import static org.junit.Assert.assertTrue;

		public class BroltiParquetTableReadWriteTest {

Refactor Parquet compression handling to avoid native libraries #2397

Refactor Parquet compression handling to avoid native libraries #2397

Conversation

niloc132 commented May 17, 2022

niloc132 May 17, 2022

Choose a reason for hiding this comment

niloc132 May 17, 2022

Choose a reason for hiding this comment

niloc132 May 17, 2022

Choose a reason for hiding this comment

niloc132 May 17, 2022

Choose a reason for hiding this comment

rcaudy left a comment

Choose a reason for hiding this comment

niloc132 commented May 31, 2022

rcaudy May 31, 2022

Choose a reason for hiding this comment

niloc132 Jun 3, 2022

Choose a reason for hiding this comment

rcaudy commented May 31, 2022

rcaudy commented May 31, 2022

devinrsmith commented Jun 1, 2022

niloc132 commented Jun 3, 2022