wasm support

[samuelcolvin]

I really want pydantic-core to support wasm. This is mostly so that the examples in pydantic's docs can be edited and run in the browser, but also for wider use of pydantic.

As per PyO3/pyo3#2412 (comment), it looks like it should be possible.

But I'm not sure how to integrate that with maturin github actions. @messense any pointers? Or would you be willing to submit a PR?

Also, as well as getting wheels to build, what more do we need to do to get pydantic-core working with pyiodide?

[hoodmane]

The next release of Pyodide will be able to load binary wheels with micropip. Hopefully we'll have an alpha release in the next few days.

[samuelcolvin]

Woop Woop, that's great news. Thanks @hoodmane.

[hoodmane]

I am happy to help out with CI. I am still trying to get improvements for the emscripten target into Rust. If they merge a few more PRs it will significantly simplify the complexity of the flags needed to build Rust code. Though I suppose that is all handled by maturin so it probably won't affect you directly.
https://github.com/rust-lang/rust/pulls?q=author%3Ahoodmane

There are also some warts in Pyodide, namely that we make it strangely hard to install a wheel from the local Emscripten filesystem with micropip which is really desirable for running unit tests.

pyodide/pyodide#2731

[messense]

#110 added the emscripten wheel building part, I'd wait for a new pyodide release to add the pytests part.

[samuelcolvin]

#110 looks great, just one tiny suggestion and otherwise we can merge it.

Since pypi won't at present accept wasm wheels, could we add those files (and indeed all wheels) to the release in github? Then it looks from micropip's docs like we could install pydantic-core from the github URL?

@hoodmane do you know of any other projects running unit tests with wasm/pyiodide? Might be a useful starting point to add the target to the test step of CI.

[samuelcolvin]

Is there anything else we need to do in this repo for this?

[hoodmane]

How about let's keep this open until we release the stable Pyodide v0.21.0. I expect to come back with some improvements in v0.21.0a3.

[samuelcolvin]

Sounds good.

[hoodmane]

Fix for the terminal output:
emscripten-core/emscripten#17320

[hoodmane]

I got all the plugins working:

[samuelcolvin]

amazing, what did you do about pytest-benchmark?

[hoodmane]

pytest-benchmark needs a patch to remove py-cpuinfo:
https://github.com/pyodide/pyodide/pull/2799/files#diff-cd8ee95e7e4c9293b1b333d19377558df5b66b57f57e6daf44726f6f01469653
py-cpuinfo is just used for reporting, it doesn't provide any critical functionality.

[samuelcolvin]

I'm just fixing wasm tests in #148, tests are now passing on CI, but I get the following error on macos M1:

============================= test session starts ==============================
platform emscripten -- Python 3.10.2, pytest-7.1.2, pluggy-1.0.0
rootdir: /test_dir
plugins: hypothesis-6.49.1, speed-0.3.1
collecting ... Pyodide has suffered a fatal error. Please report this to the Pyodide maintainers.
The cause of the fatal error was:
RangeError: Maximum call stack size exceeded
    at getter_call_trampoline (/Users/samuel/code/pydantic-core/tests/node_modules/pyodide/pyodide.asm.js:7647:36)
    at wasm://wasm/0244357a:wasm-function[1069]:0x137410
    at wasm://wasm/0244357a:wasm-function[2223]:0x17ff94
    at wasm://wasm/0244357a:wasm-function[1927]:0x16e527
    at wasm://wasm/0244357a:wasm-function[1929]:0x16e7d5
    at wasm://wasm/0244357a:wasm-function[703]:0x11c70f
    at wasm://wasm/0244357a:wasm-function[702]:0x11c6bb
    at wasm://wasm/0244357a:wasm-function[701]:0x11c542
    at wasm://wasm/0244357a:wasm-function[2960]:0x1e0ea6
    at wasm://wasm/0244357a:wasm-function[1851]:0x16b264 {
  pyodide_fatal_error: true
}
RangeError: Maximum call stack size exceeded
    at getter_call_trampoline (/Users/samuel/code/pydantic-core/tests/node_modules/pyodide/pyodide.asm.js:7647:36)
    at wasm://wasm/0244357a:wasm-function[1069]:0x137410
    at wasm://wasm/0244357a:wasm-function[2223]:0x17ff94
    at wasm://wasm/0244357a:wasm-function[1927]:0x16e527
    at wasm://wasm/0244357a:wasm-function[1929]:0x16e7d5
    at wasm://wasm/0244357a:wasm-function[703]:0x11c70f
    at wasm://wasm/0244357a:wasm-function[702]:0x11c6bb
    at wasm://wasm/0244357a:wasm-function[701]:0x11c542
    at wasm://wasm/0244357a:wasm-function[2960]:0x1e0ea6
    at wasm://wasm/0244357a:wasm-function[1851]:0x16b264 {
  pyodide_fatal_error: true
}

Not sure if this is expected, @hoodmane should I report it?

[hoodmane]

I get the following error on macos M1:

No it's not expected. Someone else reported a problem with this but it was a pretty low quality bug report so I don't think I responded. Which browser do you see it on?

[hoodmane]

Which browser do you see it on?

Oh right it's on node.

[samuelcolvin]

Another issue, as per this build, hypothesis has discover an overflow issue with datetime:

datetime.fromtimestamp(-2147483649) is causing OverflowError: timestamp out of range for platform time_t. Outside pyiodide, is runs fine and returns datetime.datetime(1901, 12, 13, 20, 9, 6).

cc @Zac-HD if you're interested, not sure if anyone else is using hypothesis to test code in pyiodide.

See pyodide/pyodide#2841.

[samuelcolvin]

No it's not expected.

I've submitted an issue pyodide/pyodide#2840.

[Zac-HD]

Hmm, not quite sure what Hypothesis should do here, I need to check how CPython handles such platforms. Definitely glad to have the report though, and will probably add Pyodide to our CI 😊

[samuelcolvin]

@Zac-HD I assume hypothesis is doing nothing wrong, you can get the error with just datetime.fromtimestamp(-2147483649), just thought you'd be interested.

[samuelcolvin]

Closing this, tests have been passing on wasm for a while and we have the demo of tests running in the browser.

[simonegiacomelli]

Hello, I used pydantic 1.x in pyodide before but I would like to use the 2.x.

I know it is available from the previous message of @samuelcolvin.
I was not able to install any of the version 2.x in pyodide (using micropip).

Is it because the wheels are not published?

It looks like that for the automated tests in pyodide, pydantic is being installed from an url.

Is there any chance to have the 2.x wheels branch to pypi?

If given some directions, I could contribute the code.

[hoodmane]

It's not currently allowed to upload Pyodide wheels to pypi.

[simonegiacomelli]

Thank you @hoodmane, I didn't know.
After a little digging I found https://github.com/pyodide/pyodide-lock and I now know there is more depth to this topic than I imagined.
I will stick with version 1.x for now.