feat: add aws glue as an iceberg connection type #16824

Rowlandev · 2024-11-12T19:05:11Z

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

This pull request enables AWS Glue to be used as a connection type when creating an Apache Iceberg catalog.
Started work on this because I noticed there was support for REST & Hive, but not AWS Glue.
fixes: #[https://github.com/Feature: Support AWS Glue Catalog for Apache Iceberg #16793]

Tests

Unit Test
Logic Test
Benchmark Test
No Test - Explain why

Type of change

Bug Fix (non-breaking change which fixes an issue)
New Feature (non-breaking change which adds functionality)
Breaking Change (fix or feature that could cause existing functionality not to work as expected)
Documentation Update
Refactoring
Performance Improvement
Other (please describe):

This change is

drmingdrmer

A change to the proto requires a new version to be added in this following file. And corresponding tests should be added. Please refer to the following instruction and existing test case file such as https://github.com/datafuselabs/databend/blob/2b6431c3e39b25a41011172718ba471df7f77800/src/meta/proto-conv/tests/it/v110_database_meta_gc_in_progress.rs

https://github.com/datafuselabs/databend/blob/2b6431c3e39b25a41011172718ba471df7f77800/src/meta/proto-conv/src/util.rs#L142

(110, "2024-09-18: Add: database.proto: DatabaseMeta.gc\_in\_progress"),  
// Dear developer:  
//      If you're gonna add a new metadata version, you'll have to add a test for it.  
//      You could just copy an existing test file(e.g., \`../tests/it/v024\_table\_meta.rs\`)  
//      and replace two of the variable \`bytes\` and \`want\`.

@Xuanwo Could you please help review this new catalog option, about the usage and testing?

Reviewed 10 of 13 files at r1, all commit messages.
Reviewable status: 10 of 13 files reviewed, all discussions resolved

BohuTANG · 2024-11-13T03:39:30Z

For the check CI failed, we should keep the Cargo.toml crate is in order, the fix is(taplo fmt -f):

diff --git a/Cargo.toml b/Cargo.toml
index 71b5ec05d2..e82e52ac7e 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -308,9 +308,9 @@ humantime = "2.1.0"
 hyper = "1"
 hyper-util = { version = "0.1.9", features = ["client", "client-legacy", "tokio", "service"] }
 iceberg = { version = "0.3.0", git = "https://github.com/Xuanwo/iceberg-rust/", rev = "fe5df3f" }
+iceberg-catalog-glue = { version = "0.3.0", git = "https://github.com/Xuanwo/iceberg-rust/", rev = "fe5df3f" }
 iceberg-catalog-hms = { version = "0.3.0", git = "https://github.com/Xuanwo/iceberg-rust/", rev = "fe5df3f" }
 iceberg-catalog-rest = { version = "0.3.0", git = "https://github.com/Xuanwo/iceberg-rust/", rev = "fe5df3f" }
-iceberg-catalog-glue = { version = "0.3.0", git = "https://github.com/Xuanwo/iceberg-rust/", rev = "fe5df3f" }
 indexmap = "2.0.0"
 indicatif = "0.17.5"
 itertools = "0.13.0"
diff --git a/src/query/storages/iceberg/Cargo.toml b/src/query/storages/iceberg/Cargo.toml
index fc336ee6a0..add1c755fa 100644
--- a/src/query/storages/iceberg/Cargo.toml
+++ b/src/query/storages/iceberg/Cargo.toml
@@ -25,9 +25,9 @@ databend-storages-common-table-meta = { workspace = true }
 fastrace = { workspace = true }
 futures = { workspace = true }
 iceberg = { workspace = true }
+iceberg-catalog-glue = { workspace = true }
 iceberg-catalog-hms = { workspace = true }
 iceberg-catalog-rest = { workspace = true }
-iceberg-catalog-glue = { workspace = true }
 match-template = { workspace = true }
 ordered-float = { workspace = true }
 serde = { workspace = true }

Xuanwo · 2024-11-13T05:26:53Z

Hi, @Rowlandev, thank you so much for creating this—it's greatly appreciated! Most of the changes look good to me. The only missing part is integrating with our proto tests to ensure compatibility with our meta services.

Please add a new entry here to register a new proto version.

databend/src/meta/proto-conv/src/util.rs

Lines 142 to 147 in 7abfa96

    
               (110, "2024-09-18: Add: database.proto: DatabaseMeta.gc_in_progress"), 
        
               // Dear developer: 
        
               //      If you're gonna add a new metadata version, you'll have to add a test for it. 
        
               //      You could just copy an existing test file(e.g., `../tests/it/v024_table_meta.rs`) 
        
               //      and replace two of the variable `bytes` and `want`. 
        
           ];

Please add a new test case here to make sure our proto compatible

You can refer to

databend/src/meta/proto-conv/tests/it/v098_catalog_option.rs

Lines 34 to 59 in 7abfa96

    
           #[test] 
        
           fn test_decode_v98_catalog() -> anyhow::Result<()> { 
        
               let catalog_meta_v98 = vec![ 
        
                   18, 55, 26, 53, 18, 45, 10, 21, 104, 116, 116, 112, 58, 47, 47, 49, 50, 55, 46, 48, 46, 48, 
        
                   46, 49, 58, 57, 57, 48, 48, 18, 14, 115, 51, 58, 47, 47, 109, 121, 95, 98, 117, 99, 107, 
        
                   101, 116, 160, 6, 98, 168, 6, 24, 160, 6, 98, 168, 6, 24, 162, 1, 23, 50, 48, 49, 52, 45, 
        
                   49, 49, 45, 50, 56, 32, 49, 50, 58, 48, 48, 58, 48, 57, 32, 85, 84, 67, 160, 6, 98, 168, 6, 
        
                   24, 
        
               ]; 
        
               let want = || databend_common_meta_app::schema::CatalogMeta { 
        
                   catalog_option: CatalogOption::Iceberg(IcebergCatalogOption::Rest( 
        
                       IcebergRestCatalogOption { 
        
                           uri: "http://127.0.0.1:9900".to_string(), 
        
                           warehouse: "s3://my_bucket".to_string(), 
        
                           props: Default::default(), 
        
                       }, 
        
                   )), 
        
                   created_on: Utc.with_ymd_and_hms(2014, 11, 28, 12, 0, 9).unwrap(), 
        
               }; 
        
               common::test_pb_from_to(func_name!(), want())?; 
        
               common::test_load_old(func_name!(), catalog_meta_v98.as_slice(), 98, want())?; 
        
               Ok(()) 
        
           }

But changed with glue catalog.

Rowlandev · 2024-11-13T14:54:23Z

@Xuanwo, @BohuTANG & @drmingdrmer - thanks for the help! Let me know what else I can do to fix up this pull request.

drmingdrmer · 2024-11-13T14:59:49Z

@Xuanwo, @BohuTANG & @drmingdrmer - thanks for the help! Let me know what else I can do to fix up this pull request.

Thank you. Most to this PR looks good enough to me. As I mentioned in a previous comment, A new version and a corresponding test case should be added:

In this comment: #16824 (review)

Rowlandev · 2024-11-13T15:03:52Z

@drmingdrmer My apologies, using git on this repo with two different machines, thought I pushed up.

drmingdrmer · 2024-11-13T15:09:04Z

@drmingdrmer My apologies, using git on this repo with two different machines, thought I pushed up.

Does not matter :D

I'll receive a notification once there is a new push.

Rowlandev · 2024-11-13T16:21:10Z

Is it party time? @drmingdrmer

src/query/storages/iceberg/src/catalog.rs

drmingdrmer · 2024-11-13T23:33:27Z

Is it party time? @drmingdrmer

Not quite yet - still have some work to wrap up, though I'm flexible with the schedule:)

drmingdrmer

Reviewed 2 of 13 files at r1, 3 of 3 files at r2, 2 of 2 files at r3, 2 of 2 files at r4, all commit messages.
Reviewable status: all files reviewed, 2 unresolved discussions (waiting on @Rowlandev and @Xuanwo)

src/meta/proto-conv/tests/it/v111_add_glue_as_iceberg_catalog_option.rs line 52 at r4 (raw file):

        drop_on: None,
        gc_in_progress: true,
    };

This test should build an instance of IcebergCatalogOption` containing the new type to ensure it can be encoded and decoded correctly.

Code quote:

    let want = || mt::DatabaseMeta {
        engine: "44".to_string(),
        engine_options: btreemap! {s("abc") => s("def")},
        options: btreemap! {s("xyz") => s("foo")},
        created_on: Utc.with_ymd_and_hms(2014, 11, 28, 12, 0, 9).unwrap(),
        updated_on: Utc.with_ymd_and_hms(2014, 11, 29, 12, 0, 9).unwrap(),
        comment: "foo bar".to_string(),
        drop_on: None,
        gc_in_progress: true,
    };

drmingdrmer

Looks good, with one minor issue noted below:

Reviewed all commit messages.
Reviewable status: 14 of 15 files reviewed, 2 unresolved discussions (waiting on @Rowlandev and @Xuanwo)

src/meta/proto-conv/tests/it/v111_add_glue_as_iceberg_catalog_option.rs line 49 at r5 (raw file):

                address: "http://127.0.0.1:9900".to_string(),
                warehouse: "s3://my_bucket".to_string(),
                props: Default::default(),

Add several sample values to props to ensure it is actually encoded and decoded

drmingdrmer

LGTM. Thank you!

Reviewed 1 of 1 files at r6, all commit messages.
Reviewable status: all files reviewed, 2 unresolved discussions (waiting on @Rowlandev and @Xuanwo)

drmingdrmer

Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @Rowlandev and @Xuanwo)

Rowlandev

Reviewed 9 of 13 files at r1, 2 of 3 files at r2, 2 of 2 files at r3, 1 of 2 files at r4, 1 of 1 files at r6, all commit messages.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @Xuanwo)

src/query/storages/iceberg/src/catalog.rs

drmingdrmer

Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @Xuanwo)

src/query/storages/iceberg/src/catalog.rs

drmingdrmer

Reviewed 1 of 1 files at r7, all commit messages.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @Xuanwo)

drmingdrmer · 2024-11-15T13:31:49Z

It looks like the source code is not well formatted, which caused the check fails:
https://github.com/databendlabs/databend/actions/runs/11857098771/job/33044752102?pr=16824

You can run cargo fmt locally to fix this issue.

drmingdrmer

Reviewed 1 of 1 files at r8, 1 of 1 files at r9, all commit messages.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @Xuanwo)

drmingdrmer · 2024-11-16T01:16:42Z

Nice job! Thank you! Let's merge!

BohuTANG · 2024-11-18T01:01:47Z

@soyeric128 Please update the documentation. Thanks.

Xuanwo · 2024-11-26T13:33:01Z

Hi, @Rowlandev, thank you so much for creating this. Could you provide a real example of creating a Glue catalog (with the secrets redacted)?

Rowlandev · 2024-11-27T19:35:00Z

@Xuanwo Yes sir. This relies on environment variables being set in the contained environment:

CREATE CATALOG my_aws_glue_catalog
TYPE = ICEBERG
CONNECTION = (
    TYPE = 'glue',
    ADDRESS = 'https://glue.us-east-1.amazonaws.com',
    WAREHOUSE = 's3://your-s3-data'
);

github-actions bot added the pr-feature this PR introduces a new feature to the codebase label Nov 12, 2024

Rowlandev marked this pull request as ready for review November 12, 2024 22:28

Rowlandev requested a review from drmingdrmer as a code owner November 12, 2024 22:28

drmingdrmer reviewed Nov 13, 2024

View reviewed changes

BohuTANG requested a review from Xuanwo November 13, 2024 01:26

Rowlandev requested a review from drmingdrmer November 13, 2024 14:56

Xuanwo reviewed Nov 13, 2024

View reviewed changes

src/query/storages/iceberg/src/catalog.rs Outdated Show resolved Hide resolved

drmingdrmer requested a review from Xuanwo November 13, 2024 23:41

drmingdrmer requested changes Nov 13, 2024

View reviewed changes

drmingdrmer reviewed Nov 14, 2024

View reviewed changes

drmingdrmer approved these changes Nov 14, 2024

View reviewed changes

Rowlandev commented Nov 14, 2024

View reviewed changes

src/query/storages/iceberg/src/catalog.rs Outdated Show resolved Hide resolved

drmingdrmer requested changes Nov 14, 2024

View reviewed changes

src/query/storages/iceberg/src/catalog.rs Outdated Show resolved Hide resolved

drmingdrmer approved these changes Nov 15, 2024

View reviewed changes

chase.grainger and others added 5 commits November 16, 2024 08:35

added aws glue catalog as option for metastore

d79a687

linted

67eb15e

removed unused dependencies

f122cb7

ran lint

d22a670

added test

1703d77

Rowlandev added 6 commits November 16, 2024 08:35

put toml files in order

1843027

added proto test

4eba608

changed test to use glue as a catalog option

f38327b

added props for test

5b01206

got rid of extra block_in_place

663bbfb

linted

977a93b

drmingdrmer force-pushed the add-glue-catalog branch from fb9dc4b to 977a93b Compare November 16, 2024 00:35

drmingdrmer merged commit ec9b077 into databendlabs:main Nov 16, 2024
74 of 75 checks passed

Rowlandev deleted the add-glue-catalog branch November 16, 2024 18:51

Rowlandev mentioned this pull request Nov 16, 2024

Feature: Support AWS Glue Catalog for Apache Iceberg #16793

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add aws glue as an iceberg connection type #16824

feat: add aws glue as an iceberg connection type #16824

Rowlandev commented Nov 12, 2024 •

edited

Loading

drmingdrmer left a comment

BohuTANG commented Nov 13, 2024 •

edited

Loading

Xuanwo commented Nov 13, 2024

Rowlandev commented Nov 13, 2024

drmingdrmer commented Nov 13, 2024

Rowlandev commented Nov 13, 2024

drmingdrmer commented Nov 13, 2024

Rowlandev commented Nov 13, 2024

drmingdrmer commented Nov 13, 2024

drmingdrmer left a comment

drmingdrmer left a comment

drmingdrmer left a comment

drmingdrmer left a comment

Rowlandev left a comment

drmingdrmer left a comment

drmingdrmer left a comment

drmingdrmer commented Nov 15, 2024

drmingdrmer left a comment

drmingdrmer commented Nov 16, 2024

BohuTANG commented Nov 18, 2024

Xuanwo commented Nov 26, 2024

Rowlandev commented Nov 27, 2024

feat: add aws glue as an iceberg connection type #16824

feat: add aws glue as an iceberg connection type #16824

Conversation

Rowlandev commented Nov 12, 2024 • edited Loading

Summary

Tests

Type of change

drmingdrmer left a comment

Choose a reason for hiding this comment

BohuTANG commented Nov 13, 2024 • edited Loading

Xuanwo commented Nov 13, 2024

Rowlandev commented Nov 13, 2024

drmingdrmer commented Nov 13, 2024

Rowlandev commented Nov 13, 2024

drmingdrmer commented Nov 13, 2024

Rowlandev commented Nov 13, 2024

drmingdrmer commented Nov 13, 2024

drmingdrmer left a comment

Choose a reason for hiding this comment

drmingdrmer left a comment

Choose a reason for hiding this comment

drmingdrmer left a comment

Choose a reason for hiding this comment

drmingdrmer left a comment

Choose a reason for hiding this comment

Rowlandev left a comment

Choose a reason for hiding this comment

drmingdrmer left a comment

Choose a reason for hiding this comment

drmingdrmer left a comment

Choose a reason for hiding this comment

drmingdrmer commented Nov 15, 2024

drmingdrmer left a comment

Choose a reason for hiding this comment

drmingdrmer commented Nov 16, 2024

BohuTANG commented Nov 18, 2024

Xuanwo commented Nov 26, 2024

Rowlandev commented Nov 27, 2024

Rowlandev commented Nov 12, 2024 •

edited

Loading

BohuTANG commented Nov 13, 2024 •

edited

Loading