Initial commit of Iceberg integration.

[lbooker42]

Highlights
Can select specific snapshots or automatically choose the latest when loading a table. Supports single-file, multi-file and key-value partitioned data.

TODO (known limits):

• Support delete files (for merge-on-read support)
• Refreshing tables (automatic when load latest snapshot?)

[rcaudy]

We should have a discussion with @malhotrashivam and @devinrsmith about configuration strategies to unify with Parquet and existing S3 support.

[rcaudy]

Do we require s3 instructions?

[lbooker42]

Need to verify that (e.g.) an all-AWS install would not need explicit instructions but could function with defaults.

[rcaudy]

.

[rcaudy]

I think I'm good with this, modulo a few quibbles. Maybe @niloc132 should look as a gradle expert.

[rcaudy]

Should it be iceberg/aws?

[niloc132]

nest subdir vs nested subproject - technically orthogonal. ryan's comment is asking for the former, which is a way to structure the directories (replace the string passed to the file() method). I would argue that not only do we want that, we also want nested subprojects, using : delimiters.

The Gradle convention is that nested directories also always means nested projects. Deephaven (starting with DHE, inherited in DHC) doesn't follow this as a rule, though DHC sometimes uses this if the parent directory is itself a project.

That said, using subprojects also conflicts a bit with our naming policies (which is "deephaven-" plus the project's own name), since nesting projects removes the parent project from the name.

(for the rest of this i'm assuming we're calling it s3 rather than aws, but aws-s3 might also make sense)

From talking to Larry it sounds like we anticipate other iceberg implementations that aren't s3 specific. So, I would propose something like:

extensions/
  iceberg/ # name this project deephaven-extensions-iceberg?
    build.gradle
    src/
      s3/ # name this project deephaven-iceberg-s3? 
         build.gradle
         src/
      r2/ # possible other impl?
        build.gradle
        src/

[chipkent]

I have only reviewed the gradle file in the py dir.

[niloc132]

I am concerned about these version numbers, since unlike maven versions, these are docker tags that are mutable. Typically we use our docker-registry project to pin to a particular image hash and ensure that even if a tag moves, our old builds will behave the same.

That said, Devin is out, and this is how he wrote the S3 testing code, so we should be consistent until he can tell us why we want it that way (or offer another way forward).

[niloc132]

nest subdir vs nested subproject - technically orthogonal. ryan's comment is asking for the former, which is a way to structure the directories (replace the string passed to the file() method). I would argue that not only do we want that, we also want nested subprojects, using : delimiters.

The Gradle convention is that nested directories also always means nested projects. Deephaven (starting with DHE, inherited in DHC) doesn't follow this as a rule, though DHC sometimes uses this if the parent directory is itself a project.

That said, using subprojects also conflicts a bit with our naming policies (which is "deephaven-" plus the project's own name), since nesting projects removes the parent project from the name.

(for the rest of this i'm assuming we're calling it s3 rather than aws, but aws-s3 might also make sense)

From talking to Larry it sounds like we anticipate other iceberg implementations that aren't s3 specific. So, I would propose something like:

extensions/
  iceberg/ # name this project deephaven-extensions-iceberg?
    build.gradle
    src/
      s3/ # name this project deephaven-iceberg-s3? 
         build.gradle
         src/
      r2/ # possible other impl?
        build.gradle
        src/

[chipkent]

Py dir changes look ok.

[deephaven-internal]

Labels indicate documentation is required. Issues for documentation have been opened:

Community: deephaven/deephaven-docs-community#226