Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

store userMetadata on write operations #1990

Closed
resulyrt93 opened this issue Dec 22, 2023 · 5 comments · Fixed by #1994
Closed

store userMetadata on write operations #1990

resulyrt93 opened this issue Dec 22, 2023 · 5 comments · Fixed by #1994
Labels
binding/python Issues for the Python package enhancement New feature or request

Comments

@resulyrt93
Copy link

Description

With spark deltalake we can store userMetadata with write operations, thus we can access them in delta history. It can be an arbitrary json data. Is there a way to store it with delta-rs python binding? Doc link

Use Case
In our case, we want to store user data with write operation to match transaction with user.

Related Issue(s)
#133
@resulyrt93 resulyrt93 added the enhancement New feature or request label Dec 22, 2023
@rtyler rtyler added the binding/python Issues for the Python package label Dec 22, 2023
@ion-elgreco
Copy link
Collaborator

Should be trivial to expose, I can take a look at it after new years eve

@r3stl355
Copy link
Contributor

Can userMetadata be added to app_metadata before being passed to commit (unless it is already the same thing and just needs to be exposed to Python)? Adding a new parameter to commit would touch many operations.
I am working on #1957 so can also look into this after I'm done there

@cekicbaris
Copy link

cekicbaris commented Dec 27, 2023 •
edited
Loading

this is exactly what we need.
+1

@ion-elgreco ion-elgreco mentioned this issue Dec 27, 2023
Merged
@ion-elgreco
Copy link
Collaborator

@r3stl355 it's the same thing, see my PR: #1994

image

@r3stl355
Copy link
Contributor

cool, so this doesn't touch Rust at all, nice

ion-elgreco added a commit that referenced this issue Jan 2, 2024
@ion-elgreco
feat(python): expose custom metadata to writers (#1994)
093a756 
# Description
- exposes the custom_metadata to pyarrow and rust writer
- addresses a bug in the create operation, we were not passing the
app_metadata to the actual commit

# Related Issue(s)
- closes #1990
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
binding/python Issues for the Python package enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat(python): expose custom metadata to writers ion-elgreco/delta-rs
5 participants
@rtyler @r3stl355 @ion-elgreco @cekicbaris @resulyrt93