New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: DH-18174: Delay reading from parquet file when creating table and column location #6590

Closed

malhotrashivam wants to merge 18 commits into deephaven:rc/v0.37.x from malhotrashivam:nightly/sm-DH-18174

Contributor

malhotrashivam commented Jan 23, 2025 •

edited

Loading

https://deephaven.atlassian.net/browse/DH-18174

This PR backports #6606 for v0.37.x

malhotrashivam added 5 commits

January 22, 2025 20:23


          Lazy initialization added for ParquetTableLocation

e0f4e0d


          Added lazy initialization for ParquetColumnLocation

f9b0916


          Synchronized the initialization logic

24a544a


          Updated some comments

dcb84ea


          Added unit tests

6b4f803

malhotrashivam added feature request parquet NoDocumentationNeeded ReleaseNotesNeeded labels

malhotrashivam added this to the 0.38.0 milestone

malhotrashivam self-assigned this

malhotrashivam requested review from devinrsmith, nbauernfeind, niloc132, rcaudy, cpwright, chipkent, jmao-denver and jcferretti as code owners

January 23, 2025 21:35

malhotrashivam changed the base branch from main to release/v0.37.4

January 23, 2025 21:35

malhotrashivam changed the title ~~DH-18174: Delay reading from parquet file when creating table and column location~~ feat: Delay reading from parquet file when creating table and column location

malhotrashivam marked this pull request as draft

January 23, 2025 21:36

malhotrashivam commented

View reviewed changes

...table/src/main/java/io/deephaven/engine/table/impl/locations/impl/AbstractTableLocation.java Outdated Show resolved Hide resolved


          Added a local variable along with the volatile

047896e

malhotrashivam changed the base branch from release/v0.37.4 to rc/v0.37.x

January 24, 2025 17:45

devinrsmith reviewed

View reviewed changes

...table/src/main/java/io/deephaven/engine/table/impl/locations/impl/AbstractTableLocation.java Outdated Show resolved Hide resolved

...table/src/main/java/io/deephaven/engine/table/impl/locations/impl/AbstractTableLocation.java Outdated Show resolved Hide resolved

...table/src/main/java/io/deephaven/engine/table/impl/locations/impl/AbstractTableLocation.java Outdated Show resolved Hide resolved


          Review with Devin

cc7ca09

malhotrashivam commented

View reviewed changes

...s/parquet/table/src/main/java/io/deephaven/parquet/table/location/ParquetColumnLocation.java Show resolved Hide resolved

malhotrashivam added 2 commits

January 27, 2025 19:12


          Minor tweaks

1e0e36f


          Removing redundant tests

b67b4f1

malhotrashivam marked this pull request as ready for review

January 27, 2025 20:27

devinrsmith requested changes

View reviewed changes

...table/src/main/java/io/deephaven/engine/table/impl/locations/impl/AbstractTableLocation.java Outdated Show resolved Hide resolved

...s/parquet/table/src/main/java/io/deephaven/parquet/table/location/ParquetColumnLocation.java Outdated Show resolved Hide resolved

...s/parquet/table/src/main/java/io/deephaven/parquet/table/location/ParquetColumnLocation.java Outdated Show resolved Hide resolved

...s/parquet/table/src/main/java/io/deephaven/parquet/table/location/ParquetColumnLocation.java Outdated Show resolved Hide resolved

...s/parquet/table/src/main/java/io/deephaven/parquet/table/location/ParquetColumnLocation.java Outdated Show resolved Hide resolved

...ns/parquet/table/src/main/java/io/deephaven/parquet/table/location/ParquetTableLocation.java Outdated Show resolved Hide resolved

...ns/parquet/table/src/main/java/io/deephaven/parquet/table/location/ParquetTableLocation.java Outdated Show resolved Hide resolved

...ns/parquet/table/src/main/java/io/deephaven/parquet/table/location/ParquetTableLocation.java Outdated Show resolved Hide resolved

...ns/parquet/table/src/main/java/io/deephaven/parquet/table/location/ParquetTableLocation.java Outdated Show resolved Hide resolved

...table/src/main/java/io/deephaven/engine/table/impl/locations/impl/AbstractTableLocation.java Outdated Show resolved Hide resolved

malhotrashivam added 3 commits

January 29, 2025 18:16


          Review with Devin

70788ab


          Removed ColumnLocationState


          Minor optimizations

a8330b7

malhotrashivam requested a review from devinrsmith

January 29, 2025 18:27

devinrsmith reviewed

View reviewed changes

...ns/parquet/table/src/main/java/io/deephaven/parquet/table/location/ParquetTableLocation.java Outdated Show resolved Hide resolved

...table/src/main/java/io/deephaven/engine/table/impl/locations/impl/AbstractTableLocation.java Outdated Show resolved Hide resolved

...s/parquet/table/src/main/java/io/deephaven/parquet/table/location/ParquetColumnLocation.java Outdated Show resolved Hide resolved

...s/parquet/table/src/main/java/io/deephaven/parquet/table/location/ParquetColumnLocation.java Outdated Show resolved Hide resolved

...s/parquet/table/src/main/java/io/deephaven/parquet/table/location/ParquetColumnLocation.java Outdated Show resolved Hide resolved

...ns/parquet/table/src/main/java/io/deephaven/parquet/table/location/ParquetTableLocation.java Outdated Show resolved Hide resolved


          Review with Devin contd.

80cd3df

malhotrashivam commented

View reviewed changes

...table/src/main/java/io/deephaven/engine/table/impl/locations/impl/AbstractTableLocation.java Show resolved Hide resolved

devinrsmith reviewed

View reviewed changes

...ns/parquet/table/src/main/java/io/deephaven/parquet/table/location/ParquetTableLocation.java Outdated Show resolved Hide resolved

...s/parquet/table/src/main/java/io/deephaven/parquet/table/location/ParquetColumnLocation.java Outdated Show resolved Hide resolved

...ns/parquet/table/src/main/java/io/deephaven/parquet/table/location/ParquetTableLocation.java Outdated Show resolved Hide resolved

...ns/parquet/table/src/main/java/io/deephaven/parquet/table/location/ParquetTableLocation.java Outdated Show resolved Hide resolved

...table/src/main/java/io/deephaven/engine/table/impl/locations/impl/AbstractTableLocation.java Show resolved Hide resolved

...table/src/main/java/io/deephaven/engine/table/impl/locations/impl/AbstractTableLocation.java Outdated Show resolved Hide resolved


          More review comments with Devin

2babf80

malhotrashivam requested a review from devinrsmith

January 30, 2025 20:05

malhotrashivam changed the base branch from rc/v0.37.x to main

January 30, 2025 20:08

malhotrashivam changed the base branch from main to rc/v0.37.x

January 30, 2025 20:09

malhotrashivam changed the base branch from rc/v0.37.x to main

January 30, 2025 20:15

malhotrashivam marked this pull request as draft

January 30, 2025 20:16

malhotrashivam changed the base branch from main to rc/v0.37.x

January 30, 2025 20:19

devinrsmith reviewed

View reviewed changes

...ns/parquet/table/src/main/java/io/deephaven/parquet/table/location/ParquetTableLocation.java Show resolved Hide resolved

...s/parquet/table/src/main/java/io/deephaven/parquet/table/location/ParquetColumnLocation.java Show resolved Hide resolved

...s/parquet/table/src/main/java/io/deephaven/parquet/table/location/ParquetColumnLocation.java Show resolved Hide resolved

...s/parquet/table/src/main/java/io/deephaven/parquet/table/location/ParquetColumnLocation.java Outdated Show resolved Hide resolved

...ns/parquet/table/src/main/java/io/deephaven/parquet/table/location/ParquetTableLocation.java Show resolved Hide resolved

...ns/parquet/table/src/main/java/io/deephaven/parquet/table/location/ParquetTableLocation.java Outdated Show resolved Hide resolved

devinrsmith changed the title ~~feat: Delay reading from parquet file when creating table and column location~~ feat: DH-18174: Delay reading from parquet file when creating table and column location


          Review contd.

1c8d7d0

malhotrashivam modified the milestones: 0.38.0, 0.37.5

malhotrashivam marked this pull request as ready for review

January 30, 2025 20:53

malhotrashivam added 3 commits

January 30, 2025 22:44


          More review comments

386038d


          Added a comment

9dce33d


          Review contd.

cdd2313

malhotrashivam marked this pull request as draft

January 30, 2025 22:20

Contributor Author

malhotrashivam commented Jan 30, 2025

Will add a separate PR for backporting changes once #6606 is merged

malhotrashivam closed this

github-actions bot locked and limited conversation to collaborators

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

devinrsmith devinrsmith requested changes

nbauernfeind Awaiting requested review from nbauernfeind

niloc132 Awaiting requested review from niloc132

rcaudy Awaiting requested review from rcaudy

cpwright Awaiting requested review from cpwright

chipkent Awaiting requested review from chipkent

jmao-denver Awaiting requested review from jmao-denver

jcferretti Awaiting requested review from jcferretti

Labels

feature request NoDocumentationNeeded parquet ReleaseNotesNeeded