WASM and Python

[AndrewJSchoen]

I just finished successfully building a rust/pyo3 python package with a bunch of custom exported structs, and realized that it would actually be very beneficial to also export something with wasm for use on the web. Currently, the package is structured like this:

A main struct/class that is instantiated with an xml-formatted string and a number of additional optional parameters (e.g. Option<Vec<CustomEnum>> or Option<CustomStruct>). It also has a method method1 that accepts similar types of arguments. That looks something like this:

#[pymethods]
impl MainClass {
    #[new]
    fn new(
        xml: String, 
        param1: Vec<CustomEnum1>,
        param2: Option<Vec<[f64; 2]>>,
        param3: Option<Vec<CustomEnum2>>,
        param4: Option<CustomStruct>
    ) -> Self {
    ...
    }

   fn method1(
        &mut self,
        param1: Option<Vec<Option<CustomEnum3>>>,
        param2: Option<Vec<Option<f64>>>,
        param3: f64,
        param4: Option<Vec<CustomEnum2>>
    ) -> PyResult<CustomStruct> {
    ...
    }
}

Each of the enums enclose a variety of structs (e.g. CustomStruct1, CustomStruct2, .... CustomStruct14). I suppose I could enclose those in a Box as well, but I ended up using enums instead for the time being.
So, in the lib.rs file, I construct the module by adding all those different structs as classes, including the ones I enclose in the enums.

Talking on the gitter channel, it seems like it might be possible to handle this in part with some support for WASM that pyo3 has, but I wasn't sure how to make use of that, and what changes I may have to make. Any help/advice would be appreciated, and let me know if you need any more info!

[davidhewitt]

Thanks for kicking off the discussion! I am woefully lacking in knowledge about how Python is used on the web (with wasm) at the moment, so you'll have to bear with me while I ask silly questions. Hopefully together we can figure out what the missing pieces (if any) are.

beneficial to also export something with wasm for use on the web

How do you plan this to be used with wasm? Is it going into an executable program you control? Are you hosting a Python playground for users to try out your package on the web? Or do you want to ship it as a package for others to use your bundle, like we would do fornormal Python packages distributed via PyPI?

support for WASM that pyo3 has

Afaik we should build correctly for the wasm32-wasi target, however I don't know much at all about where to get a wasm build of Python to link against. How to do this might vary depending on how you want to "export" as per questions above.

[AndrewJSchoen]

How do you plan this to be used with wasm? Is it going into an executable program you control? Are you hosting a Python playground for users to try out your package on the web? Or do you want to ship it as a package for others to use your bundle, like we would do fornormal Python packages distributed via PyPI?

While I do have a specific target application in mind, the goal would be to make it available on NPM for general use. Ideally I would like make the two interfaces (WebAssembly/Python) essentially congruent and avoid duplicating my own code.

[davidhewitt]

Oh I see! I'm not aware of any packages which allow creating bindings for Wasm and Python in a single framework.

You could use #[wasm_bindgen] to create Wasm bindings, but there will be some duplication.

I tend to find that the language interfaces do have some code specific to the target languages, so some code "duplication" can be a good thing anyway to make the bindings more idiomatic.

Not saying that a package that solves both with minimal duplication wouldn't be cool!

[davidhewitt]

(see e.g. #780 and #1003 for some pitfalls with the #[cfg] approach)

[AndrewJSchoen]

I ended up going with the newtype approach, and just finished rewriting everything in that mind. However, when now trying to run wasm-pack build I get the following error:

error: failed to run custom build command for `pyo3 v0.14.5`

Caused by:
  process didn't exit successfully: `/path/to/project/target/release/build/pyo3-53865f813ccac69f/build-script-build` (exit status: 1)
  --- stdout
  cargo:rerun-if-env-changed=PYO3_CROSS
  cargo:rerun-if-env-changed=PYO3_CROSS_LIB_DIR
  cargo:rerun-if-env-changed=PYO3_CROSS_PYTHON_VERSION

  --- stderr
  error: The PYO3_CROSS_LIB_DIR environment variable must be set when cross-compiling
warning: build failed, waiting for other jobs to finish...
error: build failed
Error: Compiling your crate to WebAssembly failed
Caused by: failed to execute `cargo build`: exited with exit status: 101
  full command: "cargo" "build" "--lib" "--release" "--target" "wasm32-unknown-unknown"

I am currently building on a Mac, and I haven't seen much in terms of how this configuration goes on Mac, or if this error makes any sense given what I am trying to do. Any suggestions would be appreciated!

[davidhewitt]

You probably need to make pyo3 an optional feature and disable it when building for wasm?

[AndrewJSchoen]

Ah, that makes sense. It seems like the following works, in case anyone tries to do the same thing:

In the cargo.toml file add these lines and specify dependencies as such:

[features]
pybindings = ["pyo3"]
jsbindings = ["wasm-bindgen"]
...
[dependencies]
wasm-bindgen = { version = "0.2", optional = true }
pyo3 = {version = "0.14.5", features = ["extension-module"], optional = true}

In the setup.py file, add the needed feature as an argument to the RustExtension:

rust_extensions=[RustExtension("my_lib.my_lib", binding=Binding.PyO3, quiet=True, features=['pybindings'])],

And if running wasm-pack, use the passthrough to build commands:

wasm-pack build -- --features jsbindings

Then, for any pybinding stuff, just add the cfg annotation as such:

#[cfg(feature = "pybindings")]
#[pyclass(name="MyClass")] 
#[derive(Clone,Debug)]
pub struct PyMyClass(MyClass);

I haven't fully tested the wasm bindings stuff, so I may encounter further things that need tweaking, but I can keep the above up-to-date as I do.

[davidhewitt]

For what it's worth you don't need the pybindings feature - with optional pyo3 dependency you can just use --features=['pyo3'] to activate the pyo3 dependency.

[messense]

There is https://github.com/ethanhs/python-wasm now.