[WIP]: Add Pyodide support and CI jobs for Zarr #1903
Conversation
|
Zarr v3 has |
|
Update: we added both Next, when Zarr v3 (stable) comes out on PyPI in the next month or so, I shall bump the version/recipe for the in-tree Pyodide build as well. It would be great to have the out-of-tree build, i.e., this PR, ready for review/merging by then, and I will be committed to putting my best efforts in doing so – thanks! |
|
The Pyodide 0.26.0 release is out, and we have been lucky to have got both |
agriyakhetarpal
force-pushed
the
emscripten
branch
from
f23d1d9 to
cdf0bb2
Compare
agriyakhetarpal
force-pushed
the
emscripten
branch
from
b52b6c4 to
7ae9a97
Compare
This is because they test async code and also use threads. See the following issues: pyodide/pyodide#2221 pyodide/pyodide#237
agriyakhetarpal
force-pushed
the
emscripten
branch
from
6c6d5fa to
a6565de
Compare
agriyakhetarpal
force-pushed
the
emscripten
branch
2 times, most recently
from
4ca4aef to
9278c65
Compare
agriyakhetarpal
force-pushed
the
emscripten
branch
from
9278c65 to
eb36d40
Compare
| @@ -18,6 +19,7 @@ | |||
| from zarr.store import StorePath | |||
|
|
|||
|
|
|||
| @pytest.mark.skipif(IS_WASM, reason="Can't test async code in WASM") | |||
There was a problem hiding this comment.
This is going to be a problem for moving this PR forward. The entire design of V3 depends on being able to execute async. If threading and async are a show stopper, I think we should evaluate whether this can work another way.
There was a problem hiding this comment.
I agree, I think about half of the test suite is being skipped because of these tests under v3. Both threading and async code are probably not a priority at this time for Pyodide (perhaps @hoodmane will have more insights).
There was a problem hiding this comment.
From the latest run on my fork:
14 failed, 1606 passed, 1572 skipped, 9 xfailed, 93 warnings in 104.70s (0:01:44)There was a problem hiding this comment.
I assume 99% of the passing tests are coming from the v2 test suite. I wouldn't put much weight on that. We need tests/v3 to be passing for this to work.
There was a problem hiding this comment.
Yes. If there is a way to run asynchronous tests in a manner that is not asynchronous, then there might be a way forward, but if Zarr's underlying functionality is going to rely on this, I am not sure if there is a solution. The first way would be to get pytest-asyncio both packaged and running under Pyodide.
Edit: the related issue is pyodide/pyodide#2221.
There was a problem hiding this comment.
It is at this point possible to run async tests in Pyodide using stack switching. Just have to implement it.
There was a problem hiding this comment.
Thanks for pushing this forward @agriyakhetarpal.
Description
Tip
This PR is a rework of #1902, which targeted the
mainbranch. This one targets thev3branch, as suggested by @jhamman.This description is a stub and will be updated in due course.
TODO:
Additional context
Here are a few xrefs to some other PRs that have been implemented for packages in the Scientific Python ecosystem:
scikit-image): Build and test PyWavelets Pyodide wheels in CI PyWavelets/pywt#701 and Upload dev wheels to Anaconda.org + revamp wheels publishing workflow PyWavelets/pywt#714pandasrepository BLD, TST: Build and test Pyodide wheels forpandasin CI pandas-dev/pandas#57896scikit-imagescikit-image/scikit-image#7350The greater, long-term goal towards this change is to implement Sphinx-based interactive documentation via JupyterLite and WASM-powered, in-browser kernels, as referenced in Quansight-Labs/czi-scientific-python-mgmt#19, see also: Quansight-Labs/czi-scientific-python-mgmt#18. A pilot that can be readily tried out is available for the "API reference" pages under the PyWavelets documentation. This will be preceded by configuring a place to publish these WebAssembly/Emscripten wheels nightly or on a scheduled cadence for Zarr (which could be a third-party, PyPI-like index such as Anaconda.org) and then integrating Sphinx extensions such as
jupyterlite-sphinxfor hosted documentation.