feat: support reading tables via Unity Catalog provided credentials #3078

hntd187 · 2024-12-22T17:39:10Z

Description

You can read tables via UC provided creds.

Related Issue(s)

Documentation

Temp Creds API

codecov · 2024-12-23T16:25:34Z

Codecov Report

Attention: Patch coverage is 3.97351% with 145 lines in your changes missing coverage. Please review.

Project coverage is 71.88%. Comparing base (7d29885) to head (52fa4a9).

Files with missing lines	Patch %	Lines
crates/catalog-unity/src/models.rs	1.88%	52 Missing ⚠️
crates/catalog-unity/src/datafusion.rs	0.00%	50 Missing ⚠️
crates/catalog-unity/src/lib.rs	10.63%	41 Missing and 1 partial ⚠️
crates/core/src/table/builder.rs	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3078      +/-   ##
==========================================
- Coverage   72.08%   71.88%   -0.21%     
==========================================
  Files         134      134              
  Lines       43360    43480     +120     
  Branches    43360    43480     +120     
==========================================
- Hits        31258    31255       -3     
- Misses      10085    10203     +118     
- Partials     2017     2022       +5

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

roeap

Just some minor nits to maybe look at?

One question. Do i read that right, that we are only supporting AWS for now?

crates/catalog-unity/src/datafusion.rs

crates/catalog-unity/src/lib.rs

roeap · 2025-01-17T08:43:51Z

the build failures should hopefully be addressed by #3138 - could you merge / rebase?

commit cd79543 Author: Stephen Carman <hntd187@users.noreply.github.com> Date: Thu Jan 16 16:05:10 2025 -0500 feat: support reading tables via Unity Catalog provided credentials Signed-off-by: Stephen Carman <hntd187@users.noreply.github.com> commit 5a9ab33 Merge: 9551413 d6e1fbf Author: Stephen Carman <hntd187@users.noreply.github.com> Date: Thu Jan 16 15:48:52 2025 -0500 Merge remote-tracking branch 'mine/unity-catalog-read' into unity-catalog-read # Conflicts: # crates/catalog-unity/Cargo.toml commit 9551413 Merge: ed8ac96 7f8b2ab Author: Stephen Carman <hntd187@users.noreply.github.com> Date: Thu Jan 16 15:47:41 2025 -0500 feat: support reading tables via Unity Catalog provided credentials Signed-off-by: Stephen Carman <hntd187@users.noreply.github.com> commit d6e1fbf Merge: ed8ac96 7f8b2ab Author: Stephen Carman <hntd187@users.noreply.github.com> Date: Thu Jan 16 15:47:41 2025 -0500 Merge remote-tracking branch 'mine/unity-catalog-read' into unity-catalog-read # Conflicts: # crates/catalog-unity/Cargo.toml commit ed8ac96 Author: Stephen Carman <hntd187@users.noreply.github.com> Date: Sun Dec 22 12:34:42 2024 -0500 feat: support reading tables via Unity Catalog provided credentials Signed-off-by: Stephen Carman <hntd187@users.noreply.github.com> commit bb81a51 Author: Stephen Carman <hntd187@users.noreply.github.com> Date: Sun Dec 22 12:34:42 2024 -0500 feat: support reading tables via Unity Catalog provided credentials Signed-off-by: Stephen Carman <hntd187@users.noreply.github.com> commit 17a9d1b Author: Stephen Carman <hntd187@users.noreply.github.com> Date: Sun Dec 22 12:34:42 2024 -0500 feat: support reading tables via Unity Catalog provided credentials Signed-off-by: Stephen Carman <hntd187@users.noreply.github.com> commit 7f8b2ab Merge: 9a6e48e 24bbcdb Author: Stephen Carman <hntd187@users.noreply.github.com> Date: Thu Dec 26 11:43:29 2024 -0500 Merge branch 'main' into unity-catalog-read commit 9a6e48e Author: Stephen Carman <hntd187@users.noreply.github.com> Date: Sun Dec 22 12:34:42 2024 -0500 feat: support reading tables via Unity Catalog provided credentials Signed-off-by: Stephen Carman <hntd187@users.noreply.github.com> commit 1c3fe85 Author: Stephen Carman <hntd187@users.noreply.github.com> Date: Sun Dec 22 12:34:42 2024 -0500 feat: support reading tables via Unity Catalog provided credentials Signed-off-by: Stephen Carman <hntd187@users.noreply.github.com> commit f7cc78b Author: Stephen Carman <hntd187@users.noreply.github.com> Date: Sun Dec 22 12:34:42 2024 -0500 feat: support reading tables via Unity Catalog provided credentials Signed-off-by: Stephen Carman <hntd187@users.noreply.github.com> Signed-off-by: Stephen Carman <hntd187@users.noreply.github.com>

Signed-off-by: Stephen Carman <hntd187@users.noreply.github.com>

crates/catalog-unity/Cargo.toml

ion-elgreco

Looks good, just some small nits. Also I suggest we add an integration test to the CI, this would also help with setting up python tests later on

crates/catalog-unity/src/datafusion.rs

crates/catalog-unity/src/models.rs

Signed-off-by: Stephen Carman <hntd187@users.noreply.github.com>

ion-elgreco · 2025-01-20T07:32:55Z

crates/catalog-unity/src/lib.rs

-            .unwrap();
-        assert!(matches!(get_table_response, GetTableResponse::Success(_)));
+            .await;
+        dbg!(&get_table_response);


Left a dbg! in there : )

Signed-off-by: Stephen Carman <hntd187@users.noreply.github.com>

ion-elgreco

Great work! :D

omkar-foss · 2025-01-21T15:49:13Z

Great work! :D

+1 Lovely PR @hntd187, I'll start using the temp creds function in PR #3113. Thanks! 😁

github-actions bot added the binding/rust Issues for the Rust crate label Dec 22, 2024

ion-elgreco mentioned this pull request Jan 10, 2025

feat(python): add capability to read unity catalog (uc://) uris #3113

Draft

roeap reviewed Jan 12, 2025

View reviewed changes

crates/catalog-unity/src/datafusion.rs Outdated Show resolved Hide resolved

crates/catalog-unity/src/datafusion.rs Outdated Show resolved Hide resolved

crates/catalog-unity/src/datafusion.rs Outdated Show resolved Hide resolved

crates/catalog-unity/src/lib.rs Outdated Show resolved Hide resolved

rtyler added this to the v0.24 milestone Jan 15, 2025

github-actions bot added the binding/python Issues for the Python package label Jan 16, 2025

hntd187 marked this pull request as ready for review January 16, 2025 21:06

hntd187 requested review from wjones127, rtyler, fvaleye and ion-elgreco as code owners January 16, 2025 21:06

hntd187 force-pushed the unity-catalog-read branch from cd79543 to e95b915 Compare January 16, 2025 21:09

github-actions bot removed the binding/python Issues for the Python package label Jan 16, 2025

hntd187 added 2 commits January 17, 2025 12:01

feat: Add UC support for all cloud providers

a2c910c

Signed-off-by: Stephen Carman <hntd187@users.noreply.github.com>

hntd187 force-pushed the unity-catalog-read branch from 0617459 to a2c910c Compare January 17, 2025 17:03

ion-elgreco reviewed Jan 19, 2025

View reviewed changes

crates/catalog-unity/Cargo.toml Show resolved Hide resolved

ion-elgreco reviewed Jan 19, 2025

View reviewed changes

crates/catalog-unity/Cargo.toml Show resolved Hide resolved

ion-elgreco requested changes Jan 19, 2025

View reviewed changes

crates/catalog-unity/src/datafusion.rs Outdated Show resolved Hide resolved

crates/catalog-unity/src/models.rs Show resolved Hide resolved

feat: Add UC support for all cloud providers

52fa4a9

Signed-off-by: Stephen Carman <hntd187@users.noreply.github.com>

ion-elgreco reviewed Jan 20, 2025

View reviewed changes

ion-elgreco mentioned this pull request Jan 20, 2025

feat: Experimental unity catalog client pola-rs/polars#20798

Merged

feat: Add UC support for all cloud providers

b28018e

Signed-off-by: Stephen Carman <hntd187@users.noreply.github.com>

ion-elgreco approved these changes Jan 20, 2025

View reviewed changes

ion-elgreco added this pull request to the merge queue Jan 20, 2025

Merged via the queue into delta-io:main with commit a73d646 Jan 20, 2025
21 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: support reading tables via Unity Catalog provided credentials #3078

feat: support reading tables via Unity Catalog provided credentials #3078

hntd187 commented Dec 22, 2024

codecov bot commented Dec 23, 2024 •

edited

Loading

roeap left a comment

roeap commented Jan 17, 2025

ion-elgreco left a comment

ion-elgreco Jan 20, 2025

ion-elgreco left a comment

omkar-foss commented Jan 21, 2025

feat: support reading tables via Unity Catalog provided credentials #3078

feat: support reading tables via Unity Catalog provided credentials #3078

Conversation

hntd187 commented Dec 22, 2024

Description

Related Issue(s)

Documentation

codecov bot commented Dec 23, 2024 • edited Loading

Codecov Report

roeap left a comment

Choose a reason for hiding this comment

roeap commented Jan 17, 2025

ion-elgreco left a comment

Choose a reason for hiding this comment

ion-elgreco Jan 20, 2025

Choose a reason for hiding this comment

ion-elgreco left a comment

Choose a reason for hiding this comment

omkar-foss commented Jan 21, 2025

codecov bot commented Dec 23, 2024 •

edited

Loading