Porting from python 3.9 to micropython

[AutoPilotjn]

Dear all,
First of all congratulations for the huge work done, it is impressive.
I have to port a medium size application (#10.000 lines of python code) written
in python 3.9 and up.
When porting the 3.9-3.10 code to micropython I am facing these main issues:

• most of the data structures are decorated with @DataClass
• use of Generic in type hints
• use of interfaces through ABC class inheritance and abstractmethod.
• there are some Literal declarations, but I can manage to get rid of them

The target is an STM32F7xx with 2Mbytes of Flash and 512Kbytes of Ram. The code is
validated on a Host.
I would prefer not downgrading the code: replacing dataclass with dictionnaries
and removing Generics. In addition, interfaces through ABC and abstractmethod
is a must (btw I added class ABC(): pass in abc.py). mypy is a mandatory step in our
validation process.
If it makes sense, with pointers and guidelines I could implement a subset of dataclass
including few options such as slots=True. Among difficulties, I am aware there are
cascading dependancies such as the re-regex module.
What would you advise ?

[robert-hh]

Unless most of the code lines in your package are comments or docstrings, the RAM & flash size of the target may be too small to handle a 10.000 LOC script, not considering the data size and the language incompatibilities. Pre-compiling code may help somewhat.

[dpgeorge]

10k lines of Python code should be fine if it's frozen. Even as precompiled .mpy it should be OK. (I've seen larger projects.)

[dpgeorge]

I can imaging there could be compatibility issues with new syntax features that are not implemented in MicroPython. Also limitations of the class model (eg multiple inheritance, inheriting from a native type).

Can you give explicit examples of code that does not compile/run under MicroPython, for all 4 of your points? I would have thought that generic type hints would work.

[dpgeorge]

You can implement the [] syntax on a class, for example:

class GenericClass:
    def __getitem__(self, key):
        pass

Generic = GenericClass()

[AutoPilotjn]

I have implemented the suggested 'getitem()' :
Nota: I have also been inspired by https://github.com/pfalcon/pycopy-lib/blob/master/typing/typing.py .

class _Subscriptable:
    def __getitem__(self, sub):
        pass

_SubSingleton = _Subscriptable()

Literal = _SubSingleton
Generic = _SubSingleton

Literal statements does not throws error, neither TypeVar

t_OK_KO = Literal["Ok", "Ko"]

def TypeVar(name, *constraints, bound=None, covariant=False, contravariant=False):
    return "TypeVar string"  # or return None, same issue with Generic, see below

G_Type = TypeVar("G_Type")

However the following code throws an error at run-time

class GetValue(Generic[G_Type]):
    pass

TypeError: can't create 'NoneType' instances

When declaring a 'class_getitem()':

class _SubscriptableClass:
    def __class_getitem__(self, sub):
        return None
_SubSingletonClass = _SubscriptableClass()
Generic = _SubSingletonClass

I get the following run-time error:

TypeError: '_SubscriptableClass' object isn't subscriptable

In both cases 'getitem()' and 'class_getitem()' when removing the generic type parameter:

class GetValue(Generic):
    pass

There is no error.

I am a bit lost. I read in

micropython/micropython-lib#190 (dated [8 Aug 2018])
[...] This is because Micropython does not support the class_getitem(cls, key) special function that allows the creation of generic class objects. @dpgeorge would it be possible to add that feature?

Any pointers would be very much appreciated.

[cwalther]

Your _Subscriptable is equivalent to

class _Subscriptable:
    def __getitem__(self, sub):
        return None

and that is the None that it’s complaining about (you are effectively trying class GetValue(None): which doesn’t work becaue None is not a valid base class).

Try this instead:

class _Subscriptable:
    def __getitem__(self, sub):
        return object

[AutoPilotjn]

Thanks, works with 'object'.
I have now an updated dummy implementation of 'typing.py'. Shall I share it ?
Please note I am a newbie to git and github, but willing to learn.

[dpgeorge]

I have now an updated dummy implementation of 'typing.py'. Shall I share it ?

If you'd like to open a PR at https://github.com/micropython/micropython-lib with it, we can consider merging it there.

[AutoPilotjn]

Update.
It was not possible get rid of advanced type hints such as Generic[T] in a fake typing module, as class_getitem() is not implemented in MicroPython (no metaclass mecanism implemented).
Therefore the solution was to clean all type hints in the source code by the means of AST module, as described in https://stackoverflow.com/questions/42733877/remove-type-hints-in-python-source-programmatically.
Then compiling the stripped code with MicroPython works fine.
The development workflow is as follows:

1. The typed source code is written with VS code and Pylance running in the background for checking the type hints across all code base (as well as Mypy)
2. The typed source code is tested on a host with CPython V3.9 or up and dedicated modules simulating IOs. During this step, it shall be taken care of MicroPython differences from CPython
3. The stripped code (without type hints) runs on the host with the Unix MicroPython compiler with dedicated modules simulating IOs
4. The stripped code is then loaded and run on the STMF7xx target

This brings all innovative features of the Python ecosystem, before running on the target.

[winni2k]

This sounds great! I would also like to see dataclasses as part of micropython. Any chance you could share your implementation?

[AutoPilotjn]

I have not implemented the dataclass decorator in MicroPython. Instead I used the regular TypedDict, which is clean and clear enough for structuring data in my application.