Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support pickling & always manage store prefix #185

Merged
merged 21 commits into from
Jan 31, 2025

Conversation

kylebarron
Copy link
Member

@kylebarron kylebarron commented Jan 31, 2025

See #184 for background

Changes

  • Remove separate PrefixStore. Always manage prefix for AWS, GCP, Azure.

TODO:

  • Document pickle support. Any middlewares in the future won't be able to be pickled.
  • Add dev doc to repo

Closes #125

@kylebarron kylebarron mentioned this pull request Jan 31, 2025
2 tasks
@kylebarron kylebarron marked this pull request as ready for review January 31, 2025 05:27
@kylebarron kylebarron changed the title Prototype: support pickling via always managing store prefix Support pickling & always manage store prefix Jan 31, 2025
@maxrjones
Copy link
Member

Thanks for your work on this @kylebarron!

How do I get the prefix to be recognized when using S3Store.from_url:

e.g.,

from obstore.store import S3Store
import obstore as obs

bucket = "nasa-veda-scratch"
path = "zarr-obstore-test/max/zarr-v3/test-era5-v3-919"

# Open store with obstore
s3store = S3Store.from_url(f"s3://{bucket}/{path}")
stream = obs.list(s3store, chunk_size=10)
for list_result in stream:
    print(list_result[0])
    break

Currently returns:

{'path': 'icechunk-test/max/met/manifests/4RCPNN1YNY6REGPDAQBG', 'last_modified': datetime.datetime(2025, 1, 27, 21, 8, 48, tzinfo=datetime.timezone.utc), 'size': 214, 'e_tag': '"5622f6e96f657cfc1089e5c5b53f923c"', 'version': None}

@kylebarron
Copy link
Member Author

Hmm this might be a bug in my prefix handling

Copy link
Member

@maxrjones maxrjones left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to implement __getnewargs_ex__ for LocalStore?

obstore/python/obstore/store/_aws.pyi Show resolved Hide resolved
obstore/python/obstore/store/_aws.pyi Show resolved Hide resolved
@kylebarron kylebarron enabled auto-merge (squash) January 31, 2025 21:08
@kylebarron kylebarron merged commit d924c28 into main Jan 31, 2025
4 checks passed
@kylebarron kylebarron deleted the kyle/pickle-always-prefix-store branch January 31, 2025 21:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support for pickling
2 participants