Extend Python Class from Rust

[davidhewitt]

A user asked a question on Gitter just now about extending a Python class from Rust.

While I think it's possible to do this by hand with a lot of unsafe code by implementing PyTypeInfo for the base type manually, it's pretty complicated.

This is an issue to think one day about how to make this easier. Possibly it's good enough to document the current process, or maybe there are design changes we can make internally.

[nekitdev]

I had an initial idea of calling PyType_Type, just like from python, in order to construct new classes with multiple inheritance:

class A:
    a = 13

class B:
    b = 42

C = type("C", (A, B), {})

[davidhewitt]

Interesting. You might find #1152 interesting which I think might use that or something similar (haven't had a chance to review it yet)

[chinedufn]

Possibly it's good enough to document the current process

Are there any existing examples that you can point me to of how this is done now?

Thanks!

[LegNeato]

I recently hit this and asked in gitter too

"How do I use abstract base classes / make my Rust code with the same api as a built in class pass an isinstance check? I
can't seem to find it in the docs or any examples. I specifically want my Rust class to pass isinstance(i, uuid.UUID) where i is a instance created by my Rust code.
(if it isn't supported, happy to put up a PR if someone will point me in the right direction and it is reasonable for a pyo3 beginner)"

David Hewitt @davidhewitt Feb 20 22:50
@LegNeato you're looking at a special case of #991. Unfortunately this is not supported at the moment. It's quite a hard issue to solve, but if you're interested in implementing it I can discuss the design and implementation steps with you on that issue.

@davidhewitt I likely won't have time soon due to work so don't sweat it, but if you find some cycles and can throw some information here I can possibly take a crack at it in the future when work settles down.

[LegNeato]

Potentially relevant: https://www.python.org/dev/peps/pep-0253/

[calbaker]

I tried using subclass in a pyclass in https://github.com/NREL/fastsim (can grant access to this private repo if needed), and when I tried:

class ChildVehicle(fsr.RustVehicle):
    def get_veh_kg(self) -> float:
        return self.veh_kg

veh = ChildVehicle.from_file(str(fsim.package_root() / 'resources/vehdb/2012_Ford_Fusion.yaml'))

veh.get_veh_kg()

this happened:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In [12], line 7
      3         return self.veh_kg
      5 veh = ChildVehicle.from_file(str(fsim.package_root() / 'resources/vehdb/2012_Ford_Fusion.yaml'))
----> 7 veh.get_veh_kg()

AttributeError: 'fastsimrust.RustVehicle' object has no attribute 'get_veh_kg'

Note that on the Rust side, we use a proc macro to add the pyo3 attributes so it'll look a little wonky to anyone who looks into the Rust code.

[birkenfeld]

@calbaker please open this as a separate discussion/issue - this issue is about extending the other way.

[caniko]

I had this issue about half a year ago, and came up with an idea (link to pydantic-core issue) where we use BaseModel from Pydantic as an interface between the two languages.

Half a year later, I need this feature again in a completely different project; sad to see it being so far away (very understandable).

A big +1 from me.

[ghost]

How could this be implemented? Wouldn't it require evaluating the python code with an from module import SomeClass and generating a rust struct that's then imported into rust?

Or would the macro generate the equivalent CPython commands to import the class and then subclass it dynamically, thus making the code unsafe?

[caniko]

How could this be implemented? Wouldn't it require evaluating the python code with an from module import SomeClass and generating a rust struct that's then imported into rust?

Or would the macro generate the equivalent CPython commands to import the class and then subclass it dynamically, thus making the code unsafe?

You are welcome to join the discussion @LoveIsGrief

[ghost]

Just a dump of my thoughts @nekitdev already found it, but classes / types in python can be created dynamically #991 (comment). (relevant python doc). They generate a PyTypeObject (documentation on creating new types in C).

So it should be possible to dynamically create types in rust from imported python classes. The downside is they will be dynamic / at runtime and thus cannot be used for static typing elsewhere in Rust code. As a first step, that might be OK and it might even already be possible (still have to try it out to confirm).

Making it static though would be the most difficult, I imagine. One would have to somehow evaluate the python code before compilation or during compilation to generate either:

• a C representation of the python class with headers, then creating a binding to use in rust
• a rust representation of the python class
  In either case, I think the generated library would be hard bound to the python code it was generated from e.g generating a subclass of SomeClass from pylib-v1.1.0 could have unexpected behavior is used at run time with SomeClass pylib-v1.2.0.

@caniko I'm not sure why this should be dependent on pydantic. Could shed some light on that? Shouldn't PyO3 be independent of third-party python libs?

[caniko]

Making it static though would be the most difficult

@LoveIsGrief, you are already thinking about the issue. The Pydantic way would make it static. 👬

[davidhewitt]

@LoveIsGrief agreed that making dynamic subtyping is a fine first step, I think all we would need to do to achieve that is give #[pyclass] some way to load the base type, probably by a PyTypeInfo implementation. There is also work that would need to be done in the pyclass internals. If you're interested in helping I can try to write up what's needed.

To make it static I think we can borrow inspiration from Duchess which has a java_package! macro. I understand this can do compile-time reflection using javap; we could consider invoking Python in the same way to introspect.

@caniko I think the pydantic approach you propose may be suitable for some use-cases but I don't think it's necessary for a barebones implementation here.

[AtomicGamer9523]

I have a similar issue, however mine might be even more complex.
Basically:
mymodule.pyi:

import abc as _abc
class Entity(_abc.ABC):
    """An entity"""
    @_abc.abstractproperty
    def uuid(self) -> str:
        """Unique ID of this entity"""
        ...

And then I also have implementations of that class (also in .pyi).
But the users might want to create their own class (in python), that extends the Entity class,
and define the property for it. And my rust code needs to handle that accordingly. How would I go about doing this?

Edit:
I will attempt to acknowledge the issue of abstract classes in this repo.

[ghost]

@davidhewitt time flew by! A quick click through the #[pyclass] macro had my head spinning. However, if you have time to write up what's needed, I'd be happy to give this a shot in 2024 if I find the time to!

[kythyria]

In my case I'm generating Rust-facing wrappers via macro anyway, so the boilerplate being boilerplatey is less of an issue than the lack of documentation on what it needs to be: I've narrowed it down to needing PyTypeInfo and PyClassBaseImpl on the wrapper, but don't understand the latter enough to know what it is that's actually causing me problems.