PyO3 submodules in Rust/Python mixed projects

[ChristopherRabotin]

Hi there,

I have a Rust project that defines a few Python submodules. I can import these without a problem with from nyx_space.orbit_determination import GaussMarkov for example (and I do so in a few Python tests). However, if I try to switch the project to a Rust/Python mixed project, then only the Python submodules seem to be available. If I try to add the from .nyx_space imoprt cosmic (for example) in the __init__.py in python/nyx_space/, then Python complains about a circular import.

What am I missing ? Thanks for your help.

pyproject.toml:

╰$ cat pyproject.toml 
[build-system]
requires = ["maturin>=0.14,<0.15"]
build-backend = "maturin"

[project]
name = "nyx_space"
requires-python = ">=3.7"
classifiers = [
    "Development Status :: 4 - Beta",
    "Topic :: Scientific/Engineering",
    "Topic :: Scientific/Engineering :: Physics",
    "Intended Audience :: Science/Research",
    "License :: OSI Approved :: GNU Affero General Public License v3 or later (AGPLv3+)"
]
dependencies = ["pandas==1.4.4", "plotly==5.10.0"]

[tool.yapf]
based_on_style = "google"
spaces_before_comment = 4
column_limit = 100

[tool.maturin]
python-source = "python"
module-name = "nyx_space._nyx_space"

Example from src/python/cosmic.rs:

// (...)
pub(crate) fn register_cosmic(py: Python<'_>, parent_module: &PyModule) -> PyResult<()> {
    let sm = PyModule::new(py, "nyx_space.cosmic")?;
    sm.add_class::<Cosm>()?;
    sm.add_class::<Bodies>()?;
    sm.add_class::<Frame>()?;
    sm.add_class::<Orbit>()?;
    sm.add_class::<Spacecraft>()?;
    sm.add_class::<SrpConfig>()?;
    sm.add_class::<DragConfig>()?;
    sm.add_class::<Thruster>()?;
    sm.add_class::<GuidanceMode>()?;

    py_run!(py, sm, "import sys; sys.modules['nyx_space.cosmic'] = sm");
    parent_module.add_submodule(sm)?;
    Ok(())
}
// (...)

I'm trying to add some Python only functions to the package.

Here is my tree:

$ tree -L 2
.
├── build.rs
├── Cargo.lock
├── Cargo.toml
├── CHANGELOG.md
├── examples
├── LICENSE
├── pyproject.toml
├── python
│   ├── nyx_space
│   ├── plotting.py
│   ├── plot_traj.py
│   ├── plt_utils.py
│   └── __pycache__
├── README.md
├── src
│   ├── bin
│   ├── cosmic
│   ├── dynamics
│   ├── errors.rs
│   ├── io
│   ├── lib.rs
│   ├── mc
│   ├── md
│   ├── od
│   ├── polyfit
│   ├── propagators
│   ├── python
│   ├── python.orig
│   ├── tools
│   └── utils.rs
└── tests
    ├── cosmic
    ├── GMAT_scripts
    ├── gm.rs
    ├── lib.rs
    ├── mission_design
    ├── monte_carlo
    ├── orbit_determination
    ├── Papers
    ├── propagation
    ├── propulsion
    ├── python
    └── spiceypy

[messense]

If I try to add the from .nyx_space imoprt cosmic (for example) in the __init__.py in python/nyx_space/, then Python complains about a circular import.

I think you want to use from ._nyx_space import cosmic since you have set module-name = "nyx_space._nyx_space" in pyproject.toml.

[messense]

What is the version of maturin you are using?

[ChristopherRabotin]

I was using version 0.14.13 and upgrading to 0.14.16 exhibits the same behavior.

[messense]

module-name was added in 0.14.16 so it can't worked for you before.

Please make sure

1. Set module-name = "nyx_space._nyx_space" in pyproject.toml
2. Use _nyx_space as #[pymodule] function name
3. Use from ._nyx_space import cosmic in python/nyx_space/__init__.py
4. Inspect the dylib filenames in python/nyx_space folder and make sure it only contains a _nyx_space.*.so.

If it still won't work for you, please provide a public repository for repro.

[messense]

You could also try from . import _nyx_space first, there is a known issue that might be related: PyO3/pyo3#759

[ChristopherRabotin]

Thank you very much for your instantaneous help, this worked! I summarized the solution below, feel free to edit it for clarity of course.

[ChristopherRabotin]

Summary of the solution:

1. Using maturin 0.14.16 or higher
2. Set module-name = "nyx_space._nyx_space" in pyproject.toml
3. Use _nyx_space as #[pymodule] function name
4. Use from ._nyx_space import cosmic in python/nyx_space/init.py
5. In the register_ functions that add the submodule, make sure to use the package name without the underscore, e.g. py_run!(py, sm, "import sys; sys.modules['nyx_space.cosmic'] = sm");

Now, you can expand the __init__.py file to include additional attributes like __all__:

from . import _nyx_space

from nyx_space import time, cosmic, mission_design, orbit_determination

__all__ = ["time", "cosmic", "mission_design", "orbit_determination"]

    Finished dev [unoptimized + debuginfo] target(s) in 0.35s
📦 Built wheel for CPython 3.10 to /tmp/.tmpIiTMV7/nyx_space-2.0.0a2-cp310-cp310-linux_x86_64.whl
🛠 Installed nyx_space-2.0.0-alpha.2
Python 3.10.9 (main, Dec 08 2022, 14:49:06) [GCC] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from nyx_space import *
>>> time.Epoch
<class 'builtins.Epoch'>
>>>