Added support to write metadata files in parquet #5105

malhotrashivam · 2024-02-01T18:48:53Z

Closes #5039, #958

Also,

Added support to do partitioned parquet writing.
Offset index information will be read from parquet files on demand and not while reading column chunks.

extensions/parquet/base/build.gradle

extensions/parquet/base/src/main/java/io/deephaven/parquet/base/MetadataFileWriter.java

extensions/parquet/table/src/main/java/io/deephaven/parquet/table/ParquetTableWriter.java

extensions/parquet/table/src/main/java/io/deephaven/parquet/table/ParquetTools.java

...nsions/parquet/table/src/test/java/io/deephaven/parquet/table/ParquetTableReadWriteTest.java

extensions/parquet/base/src/main/java/io/deephaven/parquet/base/MetadataFileWriter.java

rcaudy

Answered some questions

extensions/parquet/base/src/main/java/io/deephaven/parquet/base/MetadataFileWriter.java

extensions/parquet/table/src/main/java/io/deephaven/parquet/table/ParquetTableWriter.java

...ns/parquet/table/src/main/java/io/deephaven/parquet/table/ParquetMetadataFileWriterImpl.java

lbooker42

This looks good, a few comments about Javadoc parameters but this is easy to follow.

...nsions/parquet/table/src/test/java/io/deephaven/parquet/table/ParquetTableReadWriteTest.java

extensions/parquet/table/src/main/java/io/deephaven/parquet/table/ParquetTableWriter.java

lbooker42

LGTM

extensions/parquet/table/src/main/java/io/deephaven/parquet/table/ParquetTools.java

...nsions/parquet/table/src/test/java/io/deephaven/parquet/table/ParquetTableReadWriteTest.java

extensions/parquet/table/src/main/java/io/deephaven/parquet/table/ParquetTools.java

py/server/deephaven/parquet.py

chipkent

Python LGTM

extensions/parquet/table/src/main/java/io/deephaven/parquet/table/ParquetInstructions.java

...ns/parquet/table/src/main/java/io/deephaven/parquet/table/ParquetMetadataFileWriterImpl.java

extensions/parquet/table/src/main/java/io/deephaven/parquet/table/ParquetTableWriter.java

extensions/parquet/table/src/main/java/io/deephaven/parquet/table/ParquetTools.java

...nsions/parquet/table/src/test/java/io/deephaven/parquet/table/ParquetTableReadWriteTest.java

py/server/deephaven/parquet.py

extensions/parquet/table/src/main/java/io/deephaven/parquet/table/ParquetTools.java

py/server/deephaven/parquet.py

...parquet/table/src/main/java/io/deephaven/parquet/table/layout/ParquetMetadataFileLayout.java

deephaven-internal · 2024-04-02T20:54:30Z

Labels indicate documentation is required. Issues for documentation have been opened:

Community: deephaven/deephaven-docs-community#180

malhotrashivam added 2 commits January 31, 2024 18:15

Copied methods from inside parquet hadoop to write metadata files

f8ae4ef

Calling methods from inside ParquetHadoop for writing metadata files

0ef7f79

malhotrashivam self-assigned this Feb 1, 2024

malhotrashivam added feature request New feature or request parquet Related to the Parquet integration NoDocumentationNeeded NoReleaseNotesNeeded No release notes are needed. labels Feb 1, 2024

malhotrashivam added this to the 2. February 2024 (end of month) milestone Feb 1, 2024

malhotrashivam mentioned this pull request Feb 1, 2024

Fixed a bug in reading metadata files #5103

Merged

malhotrashivam requested a review from rcaudy February 1, 2024 19:09

malhotrashivam added 3 commits February 13, 2024 23:10

Cleaned up the code and added a lot of TODOs for review

44d031f

Merge branch 'main' into sm-pq-metadata

6394bac

Some more changes

6c6b075

malhotrashivam commented Feb 13, 2024

View reviewed changes

malhotrashivam changed the title ~~[WIP] Added support to write metadata files in parquet~~ Added support to write metadata files in parquet Feb 13, 2024

malhotrashivam commented Feb 13, 2024

View reviewed changes

extensions/parquet/base/src/main/java/io/deephaven/parquet/base/MetadataFileWriter.java Outdated Show resolved Hide resolved

rcaudy reviewed Feb 14, 2024

View reviewed changes

extensions/parquet/base/src/main/java/io/deephaven/parquet/base/MetadataFileWriter.java Outdated Show resolved Hide resolved

extensions/parquet/table/src/main/java/io/deephaven/parquet/table/ParquetTableWriter.java Outdated Show resolved Hide resolved

malhotrashivam added 2 commits February 14, 2024 23:44

WIP commit

9263aa0

Added a custom metadata file writer

8c887c4

malhotrashivam commented Feb 16, 2024

View reviewed changes

...ns/parquet/table/src/main/java/io/deephaven/parquet/table/ParquetMetadataFileWriterImpl.java Outdated Show resolved Hide resolved

malhotrashivam added 2 commits February 17, 2024 00:23

Merge branch 'main' into sm-pq-metadata

05400c5

Minor fix

b696649

malhotrashivam requested a review from lbooker42 February 16, 2024 18:59

malhotrashivam added 3 commits February 17, 2024 19:02

Fixed failing test

797b3dc

Moved some code around

d987a06

Minor change

d4da175

malhotrashivam force-pushed the sm-pq-metadata branch from 3087ea9 to d4da175 Compare February 21, 2024 12:32

lbooker42 reviewed Feb 21, 2024

View reviewed changes

Review comments

f33d712

lbooker42 previously approved these changes Feb 22, 2024

View reviewed changes

malhotrashivam commented Mar 22, 2024

View reviewed changes

chipkent reviewed Mar 22, 2024

View reviewed changes

py/server/deephaven/parquet.py Outdated Show resolved Hide resolved

py/server/deephaven/parquet.py Outdated Show resolved Hide resolved

py/server/deephaven/parquet.py Outdated Show resolved Hide resolved

malhotrashivam added 2 commits March 22, 2024 10:14

Review with Chip part 2

0d7c62e

Review with Chip and Jianfeng Part 3

5a3de8e

chipkent reviewed Mar 22, 2024

View reviewed changes

py/server/deephaven/parquet.py Outdated Show resolved Hide resolved

chipkent reviewed Mar 22, 2024

View reviewed changes

py/server/deephaven/parquet.py Outdated Show resolved Hide resolved

py/server/deephaven/parquet.py Outdated Show resolved Hide resolved

py/server/deephaven/parquet.py Outdated Show resolved Hide resolved

Review with Chip and Jianfeng continued

f68551f

chipkent previously approved these changes Mar 22, 2024

View reviewed changes

Added new APIs for managing indexes

b11ebd1

malhotrashivam mentioned this pull request Mar 25, 2024

Optimize creating indexes when writing partitioned table #5292

Open

malhotrashivam dismissed chipkent’s stale review via b11ebd1 March 25, 2024 22:15

Trigger CI jobs

3bdc92b

rcaudy reviewed Mar 26, 2024

View reviewed changes

malhotrashivam added 2 commits March 26, 2024 15:57

Review with Ryan

3dd6097

Added python support for writing indexes

d8bc0a5

rcaudy reviewed Mar 27, 2024

View reviewed changes

extensions/parquet/table/src/main/java/io/deephaven/parquet/table/ParquetTools.java Outdated Show resolved Hide resolved

py/server/deephaven/parquet.py Show resolved Hide resolved

Reordered comments

f700883

rcaudy previously approved these changes Mar 27, 2024

View reviewed changes

chipkent reviewed Mar 27, 2024

View reviewed changes

py/server/deephaven/parquet.py Outdated Show resolved Hide resolved

py/server/deephaven/parquet.py Show resolved Hide resolved

py/server/deephaven/parquet.py Outdated Show resolved Hide resolved

py/server/deephaven/parquet.py Outdated Show resolved Hide resolved

Added more details to python comments

6246289

malhotrashivam dismissed rcaudy’s stale review via 6246289 March 27, 2024 14:27

Moved from list to sequence

fa2b0c5

chipkent previously approved these changes Mar 28, 2024

View reviewed changes

malhotrashivam commented Mar 28, 2024

View reviewed changes

...parquet/table/src/main/java/io/deephaven/parquet/table/layout/ParquetMetadataFileLayout.java Show resolved Hide resolved

Added fixes and tests for windows

7909ea0

malhotrashivam dismissed chipkent’s stale review via 7909ea0 April 1, 2024 17:52

rcaudy approved these changes Apr 2, 2024

View reviewed changes

malhotrashivam merged commit a43948f into deephaven:main Apr 2, 2024
15 checks passed

github-actions bot locked and limited conversation to collaborators Apr 2, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added support to write metadata files in parquet #5105

Added support to write metadata files in parquet #5105

malhotrashivam commented Feb 1, 2024 •

edited

Loading

rcaudy left a comment

lbooker42 left a comment

lbooker42 left a comment

chipkent left a comment

deephaven-internal commented Apr 2, 2024

Added support to write metadata files in parquet #5105

Added support to write metadata files in parquet #5105

Conversation

malhotrashivam commented Feb 1, 2024 • edited Loading

rcaudy left a comment

Choose a reason for hiding this comment

lbooker42 left a comment

Choose a reason for hiding this comment

lbooker42 left a comment

Choose a reason for hiding this comment

chipkent left a comment

Choose a reason for hiding this comment

deephaven-internal commented Apr 2, 2024

malhotrashivam commented Feb 1, 2024 •

edited

Loading