Update multiscale_spatial_image.py

[dzenanz]

RC4 is available, update the code accordingly

[thewtex]

Please also:

• Remove itk_helper.py
• Update the dependency version in pyproject.toml to 5.3rc4.post2 or newer

[tbirdso]

Failures appear to be NGFF releated:

test/test_multiscale_spatial_image.py ........                           [ 66%]
test/test_ngff_validation.py FFFF                                        [100%]

=================================== FAILURES ===================================
_____________________________ test_y_x_valid_ngff ______________________________

    def test_y_x_valid_ngff():
        array = np.random.random((32, [16](https://github.com/spatial-image/multiscale-spatial-image/runs/7772527038?check_suite_focus=true#step:8:17)))
        image = to_spatial_image(array)
        multiscale = to_multiscale(image, [2, 4])
    
>       check_valid_ngff(multiscale)

test/test_ngff_validation.py:53: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
test/test_ngff_validation.py:43: in check_valid_ngff
    validator.validate(ngff)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = Draft[20](https://github.com/spatial-image/multiscale-spatial-image/runs/7772527038?check_suite_focus=true#step:8:21)2012Validator(schema={'$defs': {'axes': {'contains': {'properties': {'name': {'type': 'string'}, 'type': {'enum...ma', '$schema': 'https://json...020-12/schema', 'description': 'JSON from OME-NGFF .zattrs', ...}, format_checker=None)
args = ({},), kwargs = {}
error = <ValidationError: "'multiscales' is a required property">

[tbirdso]

I am having difficulty recreating the NGFF failure, with itk==v5.3rc04.post2 on both Linux and Windows I am seeing 12 of 12 tests passing.

@dzenanz do you have a Linux machine where you may be able to reproduce the errors?

Python package versions (Windows and Linux):

dask                          2022.8.0
dask-image                    2021.12.0
...
itk                           5.3rc4.post2
itk-core                      5.3rc4.post2
itk-filtering                 5.3rc4.post2
itk-io                        5.3rc4.post2
itk-numerics                  5.3rc4.post2
itk-registration              5.3rc4.post2
itk-segmentation              5.3rc4.post2
...
json5                         0.9.9
jsonschema                    4.12.1
...
numpy                         1.20.2
...
typing-extensions             3.10.0.2
...
zarr                          2.12.0

[dzenanz]

When I try test instructions, I run into:

dzenan@corista:~/zarr/multiScale$ ipfs get -o ./test/data -- $cid
Error: ipld: could not find bafybeihjwerpt2nxihajhco5ukl2sjatvhoerx7qn2clriwti7zjs7tvg4
dzenan@corista:~/zarr/multiScale$ pip3 show itk-core
Name: itk-core
Version: 5.3rc4.post2

[tbirdso]

The CID looks correct to me, I was able to get content in testing yesterday. Perhaps an IPFS question for @thewtex ?

[thewtex]

@dzenanz is ipfs-desktop running? Please try again.

[dzenanz]

I now run into

dzenan@corista:~/zarr/multiScale$ ipfs get -o ./test/data -- $cid
Error: block was not found locally (offline): ipld: could not find bafybeihjwerpt2nxihajhco5ukl2sjatvhoerx7qn2clriwti7zjs7tvg4
dzenan@corista:~/zarr/multiScale$

while ipfs-desktop is running. I have even pinned the cid in question. If I "download" it via ipfs-desktop's GUI, and then run pytest, I get errors, head and tail here:

dzenan@corista:~/zarr/multiScale$ pytest
===================================================================== test session starts ======================================================================
platform linux -- Python 3.10.4, pytest-7.1.2, pluggy-0.13.0
rootdir: /home/dzenan/zarr/multiScale
plugins: nbmake-1.3.4, napari-plugin-engine-0.2.0, napari-0.4.16, mypy-0.9.1, npe2-0.6.0
collected 12 items                                                                                                                                             

test/test_multiscale_spatial_image.py EEEEEEEE                                                                                                           [ 66%]
test/test_ngff_validation.py FFFF                                                                                                                        [100%]

============================================================================ ERRORS ============================================================================
______________________________________________________________ ERROR at setup of test_base_scale _______________________________________________________________

cls = <class 'xarray.backends.zarr.ZarrStore'>, store = <zarr.storage.DirectoryStore object at 0x7fdf54b71180>, mode = 'r', synchronizer = None, group = None
consolidated = None, consolidate_on_close = False, chunk_store = None, storage_options = None, append_dim = None, write_region = None, safe_chunks = True
stacklevel = 5

    @classmethod
    def open_group(
        cls,
        store,
        mode="r",
        synchronizer=None,
        group=None,
        consolidated=False,
        consolidate_on_close=False,
        chunk_store=None,
        storage_options=None,
        append_dim=None,
        write_region=None,
        safe_chunks=True,
        stacklevel=2,
    ):
    
        # zarr doesn't support pathlib.Path objects yet. zarr-python#601
        if isinstance(store, os.PathLike):
            store = os.fspath(store)
    
        open_kwargs = dict(
            mode=mode,
            synchronizer=synchronizer,
            path=group,
        )
        open_kwargs["storage_options"] = storage_options
    
        if chunk_store:
            open_kwargs["chunk_store"] = chunk_store
            if consolidated is None:
                consolidated = False
    
        if consolidated is None:
            try:
>               zarr_group = zarr.open_consolidated(store, **open_kwargs)

../../.local/lib/python3.10/site-packages/xarray/backends/zarr.py:384:

(a lot more similar warnings/errors here)

======================================================================= warnings summary =======================================================================
test/test_multiscale_spatial_image.py::test_base_scale
test/test_multiscale_spatial_image.py::test_isotropic_scale_factors
test/test_multiscale_spatial_image.py::test_gaussian_isotropic_scale_factors
test/test_multiscale_spatial_image.py::test_label_gaussian_isotropic_scale_factors
test/test_multiscale_spatial_image.py::test_anisotropic_scale_factors
test/test_multiscale_spatial_image.py::test_gaussian_anisotropic_scale_factors
test/test_multiscale_spatial_image.py::test_label_gaussian_anisotropic_scale_factors
test/test_multiscale_spatial_image.py::test_from_itk
  /home/dzenan/zarr/multiScale/test/test_multiscale_spatial_image.py:25: RuntimeWarning: Failed to open Zarr store with consolidated metadata, falling back to try reading non-consolidated metadata. This is typically much slower for opening a dataset. To silence this warning, consider:
  1. Consolidating metadata in this existing store with zarr.consolidate_metadata().
  2. Explicitly setting consolidated=False, to avoid trying to read consolidate metadata, or
  3. Explicitly setting consolidated=True, to raise an error in this case instead of falling back to try reading non-consolidated metadata.
    image_ds = xr.open_zarr(store)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=================================================================== short test summary info ====================================================================
FAILED test/test_ngff_validation.py::test_y_x_valid_ngff - jsonschema.exceptions.ValidationError: 'multiscales' is a required property
FAILED test/test_ngff_validation.py::test_z_y_x_valid_ngff - jsonschema.exceptions.ValidationError: 'multiscales' is a required property
FAILED test/test_ngff_validation.py::test_z_y_x_c_valid_ngff - jsonschema.exceptions.ValidationError: 'multiscales' is a required property
FAILED test/test_ngff_validation.py::test_t_z_y_x_c_valid_ngff - jsonschema.exceptions.ValidationError: 'multiscales' is a required property
ERROR test/test_multiscale_spatial_image.py::test_base_scale - zarr.errors.GroupNotFoundError: group not found at path ''
ERROR test/test_multiscale_spatial_image.py::test_isotropic_scale_factors - zarr.errors.GroupNotFoundError: group not found at path ''
ERROR test/test_multiscale_spatial_image.py::test_gaussian_isotropic_scale_factors - zarr.errors.GroupNotFoundError: group not found at path ''
ERROR test/test_multiscale_spatial_image.py::test_label_gaussian_isotropic_scale_factors - zarr.errors.GroupNotFoundError: group not found at path ''
ERROR test/test_multiscale_spatial_image.py::test_anisotropic_scale_factors - zarr.errors.GroupNotFoundError: group not found at path ''
ERROR test/test_multiscale_spatial_image.py::test_gaussian_anisotropic_scale_factors - zarr.errors.GroupNotFoundError: group not found at path ''
ERROR test/test_multiscale_spatial_image.py::test_label_gaussian_anisotropic_scale_factors - zarr.errors.GroupNotFoundError: group not found at path ''
ERROR test/test_multiscale_spatial_image.py::test_from_itk - zarr.errors.GroupNotFoundError: group not found at path ''
=========================================================== 4 failed, 8 warnings, 8 errors in 2.65s ============================================================
dzenan@corista:~/zarr/multiScale$

[thewtex]

FAILED test/test_ngff_validation.py::test_y_x_valid_ngff - jsonschema.exceptions.ValidationError: 'multiscales' is a required property

This is addressed with: aef02bb

Error: block was not found locally (offline): ipld: could not find bafybeihjwerpt2nxihajhco5ukl2sjatvhoerx7qn2clriwti7zjs7tvg4

This still indicates the daemon is not available. However, the testing data is now managed in a different way in via #59 .

Please try rebasing on main.

[dzenanz]

Mostly works now. Is mvn command for Maven? I don't have it installed - I never needed it before.

dzenan@corista:~/zarr/multiScale$ pytest --nbmake --nbmake-timeout=3000 examples/*ipynb -W:Deprecation
===================================================================== test session starts ======================================================================
platform linux -- Python 3.10.4, pytest-7.1.2, pluggy-0.13.0
rootdir: /home/dzenan/zarr/multiScale
plugins: nbmake-1.3.4, napari-plugin-engine-0.2.0, napari-0.4.16, mypy-0.9.1, npe2-0.6.0
collected 5 items                                                                                                                                              

examples/ConvertImageioImageResource.ipynb .                                                                                                             [ 20%]
examples/ConvertITKImage.ipynb .                                                                                                                         [ 40%]
examples/ConvertPyImageJDataset.ipynb F                                                                                                                  [ 60%]
examples/ConvertTiffFile.ipynb .                                                                                                                         [ 80%]
examples/HelloMultiscaleSpatialImageWorld.ipynb .                                                                                                        [100%]

=========================================================================== FAILURES ===========================================================================
______________________________________________ /home/dzenan/zarr/multiScale/examples/ConvertPyImageJDataset.ipynb ______________________________________________
---------------------------------------------------------------------------
# initialize ImageJ2
ij = imagej.init('2.5.0')
print(f"ImageJ2 version: {ij.getVersion()}")
---------------------------------------------------------------------------
ExecutableNotFound                        Traceback (most recent call last)
Input In [3], in <cell line: 2>()
      1 # initialize ImageJ2
----> 2 ij = imagej.init('2.5.0')
      3 print(f"ImageJ2 version: {ij.getVersion()}")

File ~/.local/lib/python3.10/site-packages/imagej/__init__.py:1496, in init(ij_dir_or_version_or_endpoint, mode, add_legacy, headless)
   1493     raise EnvironmentError("Sorry, the interactive mode is not available on macOS.")
   1495 if not sj.jvm_started():
-> 1496     success = _create_jvm(ij_dir_or_version_or_endpoint, mode, add_legacy)
   1497     if not success:
   1498         raise RuntimeError("Failed to create a JVM with the requested environment.")

File ~/.local/lib/python3.10/site-packages/imagej/__init__.py:1701, in _create_jvm(ij_dir_or_version_or_endpoint, mode, add_legacy)
   1698 sj.config.endpoints.extend(original_endpoints)
   1700 try:
-> 1701     sj.start_jvm()
   1702 except subprocess.CalledProcessError as e:
   1703     # Check to see if initialization failed due to "un-managed" imagej-legacy
   1704     err_lines = []

File ~/.local/lib/python3.10/site-packages/scyjava/__init__.py:116, in start_jvm(options)
    114     endpoints = endpoints[:1] + sorted(endpoints[1:])
    115     _logger.debug('Using endpoints %s', endpoints)
--> 116     _, workspace = jgo.resolve_dependencies(
    117         '+'.join(endpoints),
    118         m2_repo=scyjava.config.get_m2_repo(),
    119         cache_dir=scyjava.config.get_cache_dir(),
    120         manage_dependencies=scyjava.config.get_manage_deps(),
    121         repositories=repositories,
    122         verbose=scyjava.config.get_verbose()
    123     )
    124     jpype.addClassPath(os.path.join(workspace, '*'))
    126 # initialize JPype JVM

File ~/.local/lib/python3.10/site-packages/jgo/jgo.py:635, in resolve_dependencies(endpoint_string, cache_dir, m2_repo, link_type, update_cache, force_update, manage_dependencies, repositories, shortcuts, verbose)
    627 mvn_args = (
    628     ["-B"]
    629     + ["-f", pom_path, "dependency:resolve"]
    630     + (["-U"] if force_update else [])
    631     + (["-X"] if verbose > 1 else [])
    632 )
    634 try:
--> 635     mvn = executable_path_or_raise("mvn")
    636     mvn_out = run_and_combine_outputs(mvn, *mvn_args)
    637 except subprocess.CalledProcessError as e:

File ~/.local/lib/python3.10/site-packages/jgo/jgo.py:192, in executable_path_or_raise(tool)
    190 path = executable_path(tool)
    191 if path is None:
--> 192     raise ExecutableNotFound(tool, os.getenv("PATH"))
    193 return path

ExecutableNotFound: mvn not found on path /home/dzenan/.nvm/versions/node/v16.13.1/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin:/home/dzenan/.local/bin
=================================================================== short test summary info ====================================================================
FAILED examples/ConvertPyImageJDataset.ipynb:: - ExecutableNotFound: mvn not found on path /home/dzenan/.nvm/versions/node/v16.13.1/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin:/home/dzenan/.local/bin
=========================================================== 1 failed, 4 passed in 120.52s (0:02:00) ============================================================
dzenan@corista:~/zarr/multiScale$

[thewtex]

Is mvn command for Maven? I don't have it installed - I never needed it before.

Yes, this is just for the PyImageJ notebook. It is tested by CI -- you could skip it locally.

Addressed

[thewtex]

Thanks!