Reconsider Parquet read buffer size #11162
Labels
duplicate
This issue or pull request already exists
reliability
Features to improve reliability or bugs that severly impact the reliability of the plugin
ParquetPartitionReaderBase uses the parquet.read.allocation.size config to control the copy buffer used for reading from the Parquet input stream. The default is 8MB which is pretty big, especially when there are many reader threads via the multithreaded reader. With a large number of reader threads all performing Parquet reads, these per-thread buffers can add up to a nontrivial amount of heap memory which can trigger OOMs. We should do some empirical measurements in various environments to see if we can significantly lower the default for this.
The text was updated successfully, but these errors were encountered: