Update to Python 3.11

[rth]

The question is whether we update to Python 3.11 in the next release (0.22) or wait for one extra release for the packages we ship to get a compatible release. Anyway, this was already discussed, just opening a dedicated issue about it.

I think it's worth trying for this release if there are no major issues with packages. That would allow shorter iteration with work @tiran is doing upstream in CPython.

This way we could also maintain Python/Emscripten in sync update policy already starting from this release.

xref: pyscript/pyscript#610

[rth]

@hoodmane I think you had a branch with it a few months ago? (I can't find that discussion)

[tiran]

What's the planned release date of 0.22? Python 3.11.0 final is scheduled for Oct 3, little more than a month from now.

[rth]

Probably around 3 months from now, so after the final Python 3.11.0 release.

[hoodmane]

I was thinking about maybe doing another short release for 0.22 to get #2976 released.

Though I would be fine with keeping it as an alpha release and using the alpha in CI. IMO the command line runner is such a large improvement for testing that I'm reluctant to try adding Pyodide to more package CI before getting it finished.

I don't have a super strong opinion about early update to Python 3.11. Personally I like always having the newest versions of everything but there are reasons we normally don't do that... I lean in favor of this suggestion though.

[hoodmane]

I did have a couple of Python 3.11 branches, I can revive them. For core Python it should be pretty simple. Dunno if the packages will work.

[ryanking13]

I'm +1 with having Python 3.11 early as it has many performance benefits and WASM works thanks to Christian.
Though I'm not sure all packages we have will support Python 3.11 regarding our release cycle.

[rth]

another short release for 0.22 to get #2976 released

Definitely +1 for an alpha release there. For a final release with it very soon, I don't know maybe keeping it as an alpha a bit longer wouldn't hurt either so we can play with the CLI API (and use it in projects tests e.g. numpy) more before freezing it. As once it's released it would be harder to change.

[hoodmane]

Sounds good to me.

[stonebig]

would love to see an alpha of pyodide with cpython-3.11 tomorrow, so we can see if it brings speed-up in pyodide too.

[henryiii]

Here's the 3.11 wheel status:

Not found (probably not packages):

CLAPACK not found
_hashlib not found
boost-cpp not found
cffi_example not found
cpp-exceptions-test not found
distutils not found
fpcast-test not found
libde265 not found
libgmp not found
libgsl not found
libhdf5 not found
libheif not found
libiconv not found
liblzma not found
libmpc not found
libmpfr not found
libproj not found
libwebp not found
libxml not found
libxslt not found
libyaml not found
lzma not found
openssl not found
pyb2d not found
sharedlib-test not found
sharedlib-test-py not found
sqlite3 not found
suitesparse not found
test not found
zlib not found

Found only 3.10 wheels:

MarkupSafe only has Python 3.10 wheels
RobotRaconteur only has Python 3.10 wheels
biopython only has Python 3.10 wheels
bitarray only has Python 3.10 wheels
brotli only has Python 3.10 wheels
cytoolz only has Python 3.10 wheels
galpy only has Python 3.10 wheels
gmpy2 only has Python 3.10 wheels
h5py only has Python 3.10 wheels
msgpack only has Python 3.10 wheels
msprime only has Python 3.10 wheels
nlopt only has Python 3.10 wheels
numcodecs only has Python 3.10 wheels
pyclipper only has Python 3.10 wheels
pyerfa only has Python 3.10 wheels
pyheif only has Python 3.10 wheels
pyrsistent only has Python 3.10 wheels
python-sat only has Python 3.10 wheels
python_solvespace only has Python 3.10 wheels
scikit-image only has Python 3.10 wheels
statsmodels only has Python 3.10 wheels
swiglpk only has Python 3.10 wheels
tskit only has Python 3.10 wheels
wordcloud only has Python 3.10 wheels
wrapt only has Python 3.10 wheels

Found 3.11 wheels:

astropy is ready!
boost-histogram is ready!
cffi is ready!
cftime is ready!
coverage is ready!
gsw is ready!
iminuit is ready!
kiwisolver is ready!
lazy-object-proxy is ready!
lxml is ready!
matplotlib is ready!
numpy is ready!
pandas is ready!
pillow is ready!
pydantic is ready!
pyproj is ready!
pywavelets is ready!
pyyaml is ready!
regex is ready!
scikit-learn is ready!
scipy is ready!
shapely is ready!
sqlalchemy is ready!
traits is ready!
yt is ready!

Script & nox entry:

@nox.session(reuse_venv=True, python="3.11")
def pyupgrade_readiness(session: nox.Session) -> None:
    session.install("pyyaml", "ruamel.yaml", "pypi-command-line", "typer<0.6")
    session.run("python", "tools/check_pyverwheel.py")

And the script itself:

from pathlib import Path
import asyncio
import shutil
import sys

packages = [str(p.parent.name) for p in Path("packages").glob("*/meta.yaml")]


async def check_pkg(package: str) -> dict[str, bool]:
    proc = await asyncio.subprocess.create_subprocess_exec(
        "pypi", "wheels", package, stdout=asyncio.subprocess.PIPE
    )
    stdout = await proc.stdout.read()
    res = stdout.decode()

    return {
        "found": "Package Type:" in res,
        "universal": "none-any.whl" in res,
        "py310": "cp310" in res,
        "py311": "cp311" in res,
    }


async def run_all(packages: list[str]) -> list[str]:
    async with asyncio.TaskGroup() as g:
        tasks = [g.create_task(check_pkg(p)) for p in packages]
    return [t.result() for t in tasks]


packages_out = asyncio.run(run_all(packages))

ans = dict(sorted(zip(packages, packages_out)))

for name, pkg in ans.items():
    match pkg:
        case {"found": False}:
            print(name, "not found")
for name, pkg in ans.items():
    match pkg:
        case {"found": True, "py310": True, "py311": False}:
            print(name, "only has Python 3.10 wheels")
for name, pkg in ans.items():
    match pkg:
        case {"found": True, "py310": True, "py311": True}:
            print(name, "is ready!")

[ryansobol]

Now that Pyodide 0.22.0 has been released, what's the chance that work will resume on updating to Python 3.11? Are there anymore wheels missing 3.11 support that this is blocking on?

[henryiii]

I think it's being worked on right now in #3252...

[hoodmane]

Freesasa and wordcloud have fixed Python 3.11 support but haven't made a release with the fix yet. This isn't a problem, I think we can go ahead and disable them, we can reenable when they make a release supporting Python 3.11.

We could consider doing a short Pyodide 0.23 release which is mostly just the update to Python 3.11. Not sure what @ryanking13 and @rth think.

[rth]

a short Pyodide 0.23 release which is mostly just the update to Python 3.11.

+1 as well. Though ideally I would like to also include some of the pyc compilation PRs (disabled by default) and figure out whether we could deploy both versions. At least for package wheels that's easy, and it's going to have a larger impact on load time than the 3.10->3.11 update I guess.

[hoodmane]

deploy both versions

Ideally I think it ought to be a setting for loadPyodide whether to use the pyc files or the full ones. We should also probably look further into the possibilities for fixing up tracebacks, if it's reasonable we might offer an API for it.

[ryanking13]

Though ideally I would like to also include some of the pyc compilation PRs (disabled by default) and figure out whether we could deploy both versions.

+1 for it. I'll open a issue to listup things we want to include in the alpha release.

[hoodmane]

Closed by #3252.