Skip to content

Commit

Permalink
formatting
Browse files Browse the repository at this point in the history
  • Loading branch information
normanrz committed Dec 20, 2024
1 parent 875e8ea commit 44e768a
Show file tree
Hide file tree
Showing 2 changed files with 34 additions and 34 deletions.
60 changes: 30 additions & 30 deletions docs/user-guide/extending.rst
Original file line number Diff line number Diff line change
Expand Up @@ -13,66 +13,66 @@ Custom stores
Custom codecs
-------------

There are three types of codecs in Zarr: array-to-array, array-to-bytes, and bytes-to-bytes.
Array-to-array codecs are used to transform the n-dimensional array data before serializing
There are three types of codecs in Zarr: array-to-array, array-to-bytes, and bytes-to-bytes.
Array-to-array codecs are used to transform the n-dimensional array data before serializing
to bytes. Examples include delta encoding or scaling codecs. Array-to-bytes codecs are used
for serializing the array data to bytes. In Zarr, the main codec to use for numeric arrays
is the :class:`zarr.codecs.BytesCodec`. Bytes-to-bytes transform the serialized bytestreams
of the array data. Examples include compression codecs, such as
:class:`zarr.codecs.GzipCodec`, :class:`zarr.codecs.BloscCodec` or
:class:`zarr.codecs.ZstdCodec`, and codecs that add a checksum to the bytestream, such as
for serializing the array data to bytes. In Zarr, the main codec to use for numeric arrays
is the :class:`zarr.codecs.BytesCodec`. Bytes-to-bytes transform the serialized bytestreams
of the array data. Examples include compression codecs, such as
:class:`zarr.codecs.GzipCodec`, :class:`zarr.codecs.BloscCodec` or
:class:`zarr.codecs.ZstdCodec`, and codecs that add a checksum to the bytestream, such as
:class:`zarr.codecs.Crc32cCodec`.

Custom codecs for Zarr are implemented by subclassing the relevant base class, see
:class:`zarr.abc.codec.ArrayArrayCodec`, :class:`zarr.abc.codec.ArrayBytesCodec` and
:class:`zarr.abc.codec.BytesBytesCodec`. Most custom codecs should implemented the
``_encode_single`` and ``_decode_single`` methods. These methods operate on single chunks
Custom codecs for Zarr are implemented by subclassing the relevant base class, see
:class:`zarr.abc.codec.ArrayArrayCodec`, :class:`zarr.abc.codec.ArrayBytesCodec` and
:class:`zarr.abc.codec.BytesBytesCodec`. Most custom codecs should implemented the
``_encode_single`` and ``_decode_single`` methods. These methods operate on single chunks
of the array data. Alternatively, custom codecs can implement the ``encode`` and ``decode``
methods, which operate on batches of chunks, in case the codec is intended to implement
methods, which operate on batches of chunks, in case the codec is intended to implement
its own batch processing.

Custom codecs should also implement the following methods:

- ``compute_encoded_size``, which returns the byte size of the encoded data given the byte
size of the original data. It should raise ``NotImplementedError`` for codecs with
- ``compute_encoded_size``, which returns the byte size of the encoded data given the byte
size of the original data. It should raise ``NotImplementedError`` for codecs with
variable-sized outputs, such as compression codecs.
- ``validate``, which can be used to check that the codec metadata is compatible with the
- ``validate``, which can be used to check that the codec metadata is compatible with the
array metadata. It should raise errors if not.
- ``resolve_metadata`` (optional), which is important for codecs that change the shape,
- ``resolve_metadata`` (optional), which is important for codecs that change the shape,
dtype or fill value of a chunk.
- ``evolve_from_array_spec`` (optional), which can be useful for automatically filling in
- ``evolve_from_array_spec`` (optional), which can be useful for automatically filling in
codec configuration metadata from the array metadata.

To use custom codecs in Zarr, they need to be registered using the
To use custom codecs in Zarr, they need to be registered using the
`entrypoint mechanism <https://packaging.python.org/en/latest/specifications/entry-points/>`_.
Commonly, entrypoints are declared in the ``pyproject.toml`` of your package under the
``[project.entry-points."zarr.codecs"]`` section. Zarr will automatically discover and
Commonly, entrypoints are declared in the ``pyproject.toml`` of your package under the
``[project.entry-points."zarr.codecs"]`` section. Zarr will automatically discover and
load all codecs registered with the entrypoint mechanism from imported modules.

.. code-block:: toml
[project.entry-points."zarr.codecs"]
"custompackage.fancy_codec" = "custompackage:FancyCodec"
New codecs need to have their own unique identifier. To avoid naming collisions, it is
strongly recommended to prefix the codec identifier with a unique name. For example,
New codecs need to have their own unique identifier. To avoid naming collisions, it is
strongly recommended to prefix the codec identifier with a unique name. For example,
the codecs from ``numcodecs`` are prefixed with ``numcodecs.``, e.g. ``numcodecs.delta``.

.. note::
Note that the extension mechanism for the Zarr version 3 is still under development.
Requirements for custom codecs including the choice of codec identifiers might
Note that the extension mechanism for the Zarr version 3 is still under development.
Requirements for custom codecs including the choice of codec identifiers might
change in the future.

It is also possible to register codecs as replacements for existing codecs. This might be
useful for providing specialized implementations, such as GPU-based codecs. In case of
It is also possible to register codecs as replacements for existing codecs. This might be
useful for providing specialized implementations, such as GPU-based codecs. In case of
multiple codecs, the :mod:`zarr.core.config` mechanism can be used to select the preferred
implementation.
implementation.

.. note::
This sections explains how custom codecs can be created for Zarr version 3. For Zarr
version 2, codecs should subclass the
`numcodecs.abc.Codec <https://numcodecs.readthedocs.io/en/stable/abc.html#numcodecs.abc.Codec>`_
base class and register through
version 2, codecs should subclass the
`numcodecs.abc.Codec <https://numcodecs.readthedocs.io/en/stable/abc.html#numcodecs.abc.Codec>`_
base class and register through
`numcodecs.registry.register_codec <https://numcodecs.readthedocs.io/en/stable/registry.html#numcodecs.registry.register_codec>`_.


Expand Down
8 changes: 4 additions & 4 deletions docs/user-guide/v3_migration.rst
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ The Array class
1. Disallow direct construction - use :func:`zarr.open_array` or :func:`zarr.create_array`
instead of directly constructing the :class:`zarr.Array` class.

2. Defaulting to ``zarr_format=3`` - newly created arrays will use the version 3 of the
2. Defaulting to ``zarr_format=3`` - newly created arrays will use the version 3 of the
Zarr specification. To continue using version 2, set ``zarr_format=2`` when creating arrays.

The Group class
Expand Down Expand Up @@ -137,16 +137,16 @@ Dependencies Changes
Configuration
~~~~~~~~~~~~~

There is a new configuration system based on `donfig <https://github.com/pytroll/donfig>`_,
which can be accessed via :mod:`zarr.core.config`.
There is a new configuration system based on `donfig <https://github.com/pytroll/donfig>`_,
which can be accessed via :mod:`zarr.core.config`.
Configuration values can be set using code like the following:

.. code-block:: python
import zarr
zarr.config.set({"array.order": "F"})
Alternatively, configuration values can be set using environment variables,
Alternatively, configuration values can be set using environment variables,
e.g. ``ZARR_ARRAY__ORDER=F``.

Configuration options include the following:
Expand Down

0 comments on commit 44e768a

Please sign in to comment.