Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Listable V3 Stores #1634

Merged
merged 3 commits into from
Feb 7, 2024
Merged

Listable V3 Stores #1634

merged 3 commits into from
Feb 7, 2024

Conversation

jhamman
Copy link
Member

@jhamman jhamman commented Jan 16, 2024

This goes on top of #1590

The core of this PR is to add list_* methods to the v3 stores. The Store interface is very much still in flux but this should be enough to pick up #1590 again.

TODO:

  • Add unit tests and/or doctests in docstrings
  • Add docstrings and API docs for any new/modified user-facing classes and functions
  • New/modified features documented in docs/tutorial.rst
  • Changes documented in docs/release.rst
  • GitHub Actions have all passed
  • Test coverage is 100% (Codecov passes)

@pep8speaks
Copy link

pep8speaks commented Jan 16, 2024

Hello @jhamman! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

Line 34:44: E203 whitespace before ':'
Line 56:30: E203 whitespace before ':'

Comment last updated at 2024-02-07 04:46:23 UTC

@jhamman jhamman changed the title V3 list store Listable V3 Stores Jan 16, 2024
Copy link
Member Author

@jhamman jhamman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few comments on the state of things here.

async def get(self, key: str) -> bytes:
async def get(
self, key: str, byte_range: Optional[Tuple[int, Optional[int]]] = None
) -> Optional[bytes]:
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question for discussion. Do we want to have the byte_range parameter in the single key get method or just in get_partial_values? I lean toward this API but it is somewhat duplicative and goes against the suggestion in the spec:

get - Retrieve the value associated with a given key.

Parameters: key
Output: value

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My instinct would be to ignore the abstract store interface and just do what makes sense locally.

I don't understand why the spec describes both get(key) -> value and get_partial_values(key_range) -> List[Optional[value]], as opposed to get(key, range) -> Optional[value] and letting clients handle iteration themselves.

Copy link
Member Author

@jhamman jhamman Feb 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thinking about this more, we are definitely going to want to be able to pass the store a batch of values keys/ranges to the store. This will let us push optimizations down to lower level code or let us do optimizations like coalescing requests.

I still feel like its worth supporting single partial requests though so I'm going to include that in the abc.

Comment on lines 79 to 80
except (FileNotFoundError, IsADirectoryError, NotADirectoryError):
return None
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found this behavior in Zarrita to be surprising. @normanrz - can you comment on the motivation here? In V2 we would have just raised a KeyError.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I opted for using None as non-existent. Raising and catching a KeyError would also work.

@@ -0,0 +1,95 @@
from __future__ import annotations
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have not really worked on this store yet.

@@ -1,87 +1,96 @@
import array
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Big diff in this file as I decided to comment everything out and bring it back piece by piece. Here's the current test run:

=================================================================================== 13 passed in 0.08s ===================================================================================
❯ pytest zarr/tests/test_storage.py -vvv
================================================================================== test session starts ===================================================================================
platform darwin -- Python 3.11.6, pytest-7.4.3, pluggy-1.3.0 -- /Users/jhamman/miniforge3/envs/zarr-dev/bin/python3.11
cachedir: .pytest_cache
rootdir: /Users/jhamman/Library/CloudStorage/Dropbox/src/zarr-python
configfile: pyproject.toml
plugins: anyio-4.0.0, asyncio-0.23.3
asyncio: mode=Mode.STRICT
collected 13 items                                                                                                                                                                       

zarr/tests/test_storage.py::test_kvstore_repr PASSED                                                                                                                               [  7%]
zarr/tests/test_storage.py::test_ensure_store PASSED                                                                                                                               [ 15%]
zarr/tests/test_storage.py::test_capabilities PASSED                                                                                                                               [ 23%]
zarr/tests/test_storage.py::TestMappingStore::test_get_set_del_contains PASSED                                                                                                     [ 30%]
zarr/tests/test_storage.py::TestMappingStore::test_set_invalid_content PASSED                                                                                                      [ 38%]
zarr/tests/test_storage.py::TestMappingStore::test_writeable_values PASSED                                                                                                         [ 46%]
zarr/tests/test_storage.py::TestMappingStore::test_pickle PASSED                                                                                                                   [ 53%]
zarr/tests/test_storage.py::TestMappingStore::test_hierarchy PASSED                                                                                                                [ 61%]
zarr/tests/test_storage.py::TestDirectoryStore::test_get_set_del_contains PASSED                                                                                                   [ 69%]
zarr/tests/test_storage.py::TestDirectoryStore::test_set_invalid_content PASSED                                                                                                    [ 76%]
zarr/tests/test_storage.py::TestDirectoryStore::test_writeable_values PASSED                                                                                                       [ 84%]
zarr/tests/test_storage.py::TestDirectoryStore::test_pickle PASSED                                                                                                                 [ 92%]
zarr/tests/test_storage.py::TestDirectoryStore::test_hierarchy PASSED  

- removed abcs for groups/arrays
- improved return types in group.py
- warn (temporarily) when an implicit group is found
- add attributes.py with Attributes class

add test file

wip
fixes after rebas
e

make all tests pass
@jhamman
Copy link
Member Author

jhamman commented Feb 7, 2024

I'm going to merge this into the V3 branch to make way for tomorrow's sprint. I expect at least one more major rev on the store interface in the coming weeks.

@jhamman jhamman merged commit 3bc305e into zarr-developers:v3 Feb 7, 2024
7 checks passed
@jhamman jhamman mentioned this pull request Mar 30, 2024
6 tasks
@jhamman jhamman added this to the 3.0.0.alpha milestone Apr 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

4 participants