Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEATURE] Nessie integration of Iceberg REST #7043

Merged
merged 28 commits into from
May 28, 2024
Merged

[FEATURE] Nessie integration of Iceberg REST #7043

merged 28 commits into from
May 28, 2024

Conversation

snazy
Copy link
Member

@snazy snazy commented Jun 14, 2023 •
edited
Loading

No description provided.

@snazy snazy added this to the 1.0.0 milestone Jun 14, 2023
@snazy snazy changed the title [Nessie Catalog Server] Feature branch [FEATURE] Nessie Catalog Server] Jun 14, 2023
@snazy snazy changed the title [FEATURE] Nessie Catalog Server] [FEATURE] Nessie Catalog Server Jun 14, 2023
@snazy snazy force-pushed the feature/nessie-catalog-server branch 3 times, most recently from ae85c09 to 5783a01 Compare July 11, 2023 15:44
@snazy snazy force-pushed the feature/nessie-catalog-server branch 4 times, most recently from 422dd34 to 9b4e0fb Compare July 19, 2023 17:13
@snazy snazy force-pushed the feature/nessie-catalog-server branch 7 times, most recently from 694c558 to 899ce14 Compare August 3, 2023 08:04
@snazy snazy changed the title [FEATURE] Nessie Catalog Server [FEATURE] TBD May 2, 2024
@snazy snazy closed this May 2, 2024
@snazy snazy force-pushed the feature/nessie-catalog-server branch from fd576c7 to d696495 Compare May 2, 2024 15:25
@snazy snazy reopened this May 2, 2024
@snazy snazy changed the title [FEATURE] TBD [FEATURE] Nessie integration of Iceberg REST May 13, 2024
@snazy snazy force-pushed the feature/nessie-catalog-server branch from c34104b to 47de890 Compare May 13, 2024 15:06
@snazy snazy requested review from adutra and dimas-b May 13, 2024 15:06
@snazy snazy marked this pull request as ready for review May 13, 2024 15:06
@snazy snazy force-pushed the feature/nessie-catalog-server branch 2 times, most recently from b8c6df1 to b1b16ff Compare May 17, 2024 07:57
@snazy snazy added pr-integrations NesQuEIT (Iceberg, Spark, Flink, Presto) pr-macos-win Test PR on macOS + Windows labels May 17, 2024
adutra and others added 17 commits May 27, 2024 20:55
@adutra @snazy
S3 auth tests: properly close S3 signer executor (#8520)
3a89577
@snazy
[Catalog] Update object-store related tests (#8519)
71a620b 
Really use a non-default warehouse, configure the warehouse _name_ via the misleadingly named property `CatalogProperties.WAREHOUSE_LOCATION`.

Also eliminate some duplicate code.
@adutra @snazy
Iceberg REST: force table / view location (#8516)
9330f7e
@snazy
[Catalog] Improve warehouse lookup a little (#8527)
8291803 
and add some unit tests

also remove unused `CatalogServerConfig`
@snazy
[Catalog] Minor fixes to recorded timestamps (#8537)
976119e
@snazy
[Catalog] Use production config+tree resources from Nessie Catalog (#…
e329ad4 
…8539)

The Nessie client used from Nessie Catalog is currently a "simple" one. This change lets it use a client that's configured against the actual config+tree resources, which also perform authZ.
@snazy
[Catalog] Move combined-cs + remove a build() variant (#8547)
73b05e3
@snazy
[Catalog] Collect table base locations in snapshots (#8538)
20213d8 
Start collecting all base locations in `NessieTableSnapshot`. This change is a start to collect _all_ (base) table locations, to handle cases when a table's (meta)data files are "spread" across multiple base locations. Ideally (and probably usually), each table has only one base location.
@snazy
[Catalog] Update behavior for default warehouse (#8530)
1430a44 
* Make `nessie.catalog.default-warehouse` just the optional name of the default warehouse (prevent duplicate configuration).
* Remove `WarehouseConfig.name()` - the name is implicitly defined via the map key of `CatalogConfig.warehouses()`
@adutra @snazy
[Catalog] Remove OAuth proxy (#8564)
52ade0e
@snazy
[Catalog] Bearer auth in Spark extensions (#8604)
d028924
@snazy
[Catalog] Refactor secrets handling (#8600)
73ac561 
The handling of secrets is a bit awkward at the moment. We declare "IDs" on the objects, which are to be looked up via the `SecretsProvider`, which is currently just a `Map<String, String>`.

This change changes the behavior: The (default) behavior to lookup via another map is gone.

Secrets are now generally looked up in the following order:
1. If the value is defined on the "per bucket" config, it is used as is.
2. If a secret for the bucket can be resolved, it is used.
3. If the value is defined on the "default" config, it is used as is.
4. If a secret for the "default" config can be resolved, it is used.
5. The secret is assumed to not exist.

The above logic is implemented in the new `SecretsHelper` class. It calls the `SecretProvider` using a collection of keys to look up, which shall return a map with the existing, resolved secrets.

Follow-up, after #7043 is merged to `main`: refactor secrets handlings in "Nessie Core", where/if necessary.
@adutra @snazy
[Catalog] add support for OAuth in catalog-full/docker-compose.yml (#…
a1e2f9e 
…8607)
@snazy
Fix image digests (#8614)
1179f02
@snazy @adutra
[Catalog] docker-compose: define 3 clients (#8616)
35ae80a 
Co-authored-by: Alexandre Dutra <adutra@users.noreply.github.com>
@snazy
[Catalog] Add simple authz model (#8617)
9987f17 
- Client `client1` has access to everything.
- Client `client2` has *NO* access to the `sales` namespace.
- Client `client3` has *NO* access to the `eng` namespace.

----------
Co-authored-by: Alexandre Dutra <adutra@users.noreply.github.com>
@snazy
Nessie CLI: Add initial Iceberg REST functionality (#8511)
49cf7f7 
`CONNECT TO` and `USE` working, `SHOW TABLE/VIEW` displays the table/view metadata.

Also adds the `ON initialReference` clause to `CONNECT TO`.
@snazy snazy force-pushed the feature/nessie-catalog-server branch from 5e176f9 to 49cf7f7 Compare May 27, 2024 18:55
@snazy
Copy link
Member Author

snazy commented May 27, 2024

@dimas-b @adutra Let's give this "small piece" a final cross-check before we finally merge it.

@dimas-b
Copy link
Member

dimas-b commented May 28, 2024

I ran a quick smoke test in my local on bd33b94 - it seems to work fine 🎉

adutra
adutra approved these changes May 28, 2024
View reviewed changes
Copy link
Contributor

@adutra adutra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving although I think we have at least one issue with base location that we just discussed offline.

@snazy snazy merged commit e4778c2 into main May 28, 2024
16 checks passed
@snazy snazy deleted the feature/nessie-catalog-server branch May 28, 2024 16:06
@dimas-b dimas-b mentioned this pull request Jun 3, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dimas-b dimas-b dimas-b approved these changes

@adutra adutra adutra approved these changes

Assignees
No one assigned
Labels
pr-docker Smoke test Docker images pr-helm Helm chart testing pr-integrations NesQuEIT (Iceberg, Spark, Flink, Presto) pr-macos-win Test PR on macOS + Windows pr-newer-java
Projects
None yet
Milestone
1.0.0
Development

Successfully merging this pull request may close these issues.
3 participants
@snazy @dimas-b @adutra