Support of alternative array classes: ndarray-like

[madsbk]

This PR implements support of alternative array types (other than NumPy arrays) by introducing the concept ndarray-like.

The idea is to differentiate between the need of a numpy.ndarray and an array that behaves like a NumPy array, such as a CuPy array. Calling the old compat functions will still return regular NumPy arrays but two new compat functions now accept any NumPy array-like arrays:

# Ensure `buf` is a NumPy array
def ensure_ndarray(buf) -> np.ndarray: ...
def ensure_contiguous_ndarray(buf, max_buffer_size=None) -> np.array: ...

# Ensure `buf` is a NumPy array-like
def ensure_ndarray_like(buf) -> NDArrayLike:
def ensure_contiguous_ndarray_like(buf, max_buffer_size=None) -> NDArrayLike:

NumPy array like

I suggest that we use Python3.8's typing.Protocol to define what we consider ndarray-like:

@runtime_checkable
class DType(Protocol):
    itemsize: int
    name: str
    kind: str

@runtime_checkable
class FlagsObj(Protocol):
    c_contiguous: bool
    f_contiguous: bool
    owndata: bool

@runtime_checkable
class NDArrayLike(Protocol):
    dtype: DType
    shape: Tuple[int, ...]
    strides: Tuple[int, ...]
    ndim: int
    size: int
    itemsize: int
    nbytes: int
    flags: FlagsObj
    def __len__(self) -> int: ...
    def __getitem__(self, key) -> Any: ...
    def __setitem__(self, key, value): ...
    def tobytes(self, order: Optional[str] = ...) -> bytes: ...
    def reshape(self, *shape: int, order: str = ...) -> "NDArrayLike": ...
    def view(self, dtype: DType = ...) -> "NDArrayLike": ...

This requires Python v3.8. If this is a problem, we can use the typing_extensions module as a fallback:

if sys.version_info >= (3, 8):
    from typing import Protocol, runtime_checkable
else:
    from typing_extensions import Protocol, runtime_checkable

Motivation

This is the first step towards support of CuPy arrays in Zarr.

TODO:

• Closes WIP: Allow CuPy #212 and Coerce other data to bytes in ensure_text #213
• Unit tests and/or doctests in docstrings
• tox -e py39 passes locally
• Docstrings and API docs for any new/modified user-facing classes and functions
• Changes documented in docs/release.rst
• tox -e docs passes locally
• GitHub Actions CI passes
• Test coverage to 100% (Coveralls passes)

cc. @jakirkham

[jakirkham]

@joshmoore do you have thoughts here? 🙂

[joshmoore]

Beyond one minor question on my part (and @jakirkham's about installing typing_extensions, it looks good. The plan here would be to try to consume it from zarr-developers/zarr-python#934?

[rabernat]

I don't have any thing to add to the review, but just wanted to chime in to say I think this is a great feature. Supporting different array-like things is really important for future interoperability of zarr / numcodecs.

👏

[madsbk]

Anything else we need in this PR? Else, I think it is ready to be merged.

[madsbk]

@joshmoore, @jakirkham anything else we need to do here? I don't think the CI error relates to this PR.

[jakirkham]

Regarding CI. Fixing in PR ( #311 )

[jakirkham]

Thanks for working on this Mads! 😄 Also appreciate Ryan and Josh taking the time to look.

Would you be able to fix up the merge conflicts, Mads? 🙂

Generally LGTM. Though would be great if someone other than me merged (assuming people have no further questions)

[madsbk]

Would you be able to fix up the merge conflicts, Mads? slightly_smiling_face

Sure, it is merged :)

[jakirkham]

Thanks Mads! 😀

@joshmoore or @rabernat if you are comfortable with the changes here, please feel free to merge (if not please let us know what is still needed)? Thanks!

[joshmoore]

Nothing else from my side. Thoughts on getting this out as either 0.10 or is there a preference for another pre-release?

(@MSanKeys963 would likely also ask if a blog post is in order :) If so, text blurbs welcome)

[MSanKeys963]

(@MSanKeys963 would likely also ask if a blog post is in order :) If so, text blurbs welcome)

Happy to write a blog post on this as this focuses on interoperability.

@madsbk would you like to help me with the blog post? If yes, I can post a few questions and you can answer them.

[madsbk]

(@MSanKeys963 would likely also ask if a blog post is in order :) If so, text blurbs welcome)

Happy to write a blog post on this as this focuses on interoperability.

@madsbk would you like to help me with the blog post? If yes, I can post a few questions and you can answer them.

Sure, that sounds great!

[joshmoore]

👍 all around. Any thoughts on the release style?

[madsbk]

+1 all around. Any thoughts on the release style?

I think including it in the v0.10 release is good but can we merge it now? It will help me finishing zarr-developers/zarr-python#934.

[jakirkham]

Ok with either release style. No strong preferences. This is more internally facing (though would be used by Zarr too) and the same APIs Zarr uses are present. So would expect things to still work

Edit: Also sorry for being somewhat quiet here, was out sick most of the week.

[jakirkham]

Thanks all! 😄

[madsbk]

Thanks!

[joshmoore]

Building now as v0.10.0.a3

[jakirkham]

Thanks Josh! 😄