Python bindings for the core library

[xvzf]

Hi everyone,

I'd really like to start building python bindings for the library as dronekit comes more and more obsolete. I was also thinking about using the new asyncio framework in order to support the fancy async/await keywords.

Is there already an outline or project-structure which advises where to put the python wrapper?

Cheers,
Matthias

[julianoes]

@xvzf that would be great!

There is:
https://github.com/DroneCore/DroneCore-Python
and @JonasVautherin can probably best tell you what the state is and what's to do next.

[xvzf]

@julianoes Great! I'll have a look! I never worked with gRPC but it seems to provide a solid base!

Looking forward to get in touch with @JonasVautherin!

[JonasVautherin]

Hello @xvzf!

So we have made very good progress on the iOS bindings, but there are still quite a few things to improve on the C++ side, so for Python it would be on the prototyping side, I would say. I will see to update the python repo in the next few days, and then we could talk about how we want to start.

About asyncio, I was actually thinking about using RxPython. First of all because I love the reactive extensions, and because we already use RxSwift and we will quite definitely use RxJava. And I would prefer to keep the bindings consistent between the different languages. Do you know RxPython? What would be the advantage of asyncio over it?

[xvzf]

Hi there,

Unfortunately I can't help on the C++ side, haven't had time to dig into C++11 yet.

As it comes to asyncio ad RxPython - I have used RxPython before, it is nice working with it but I am one of them who thinks callbacks are not really pythonic and do not improve code readability. Also I find it somewhat more easy to work with coroutines in asyncio! I also like the in-language keywords which make the general workflow much easier and faster.

Just a quick example:

import asyncio
...
loop = asyncio.get_event_loop()
...
async def test_coroutine():
    while True:
        # Blocks this coroutine and hands over to other coroutines,
        # until data becomes available
        battery_voltage = await get_battery_voltage()
        print(f"Battery voltage:{battery_voltage}")
        # This coroutine does want to sleep for 5 seconds
        # so the eventloop switches over to another coroutine
        await asyncio.sleep(5)

# Just an example function
async def get_battery_voltage():
    # Instead of creating a request and setting a callback
    # await blocks this coroutine until data is received
    # therefore code flow is not interrupted
    await system_0.get_battery_voltage()
    # Now do something with the received data, this would
    # be a callback function with RxPython

loop.create_task(test_coroutine()) # Keeps running
loop.run_forever() # Start the eventloop and run forever

Of course it would be better to use RxPython if we need to support Python 2.x and Python 3.4, asyncio is available since Python 3.5 and will receive a feature increase in the upcoming 3.7 release. A small bonus for asyncio - though not relevant as it is an additional dependency - is uvloop. When using uvloop it is possible to achive almost Go-like performence for networking tasks (could potentially be useful in applications that use DroneCore)

[hamishwillee]

FWIW It will probably make things easier for documentation if all the front ends use similar paradigms (i.e. RxPython), but as a rule it is better to make it easier for end users. I don't know what is more familiar to Python developers - I haven't had any experience of either framework

[JonasVautherin]

Thanks for the example and for your contribution, @xvzf :-)!

It will probably make things easier for documentation if all the front ends use similar paradigms

Documentation, development, maintenance and support. If all projects end up using the reactive extensions together with gRPC, they will essentially do the exact same thing, but in different languages. So most of what we learn from one project will apply to the others. People used to the swift SDK will be able to understand code written with the python/java SDK. If we do a completely different interface, they become different projects, and probably different maintainers.

asyncio is available since Python 3.5

I don't have experience about Python 2 vs Python 3... How limiting is that? Can we assume every new project is running Python 3.5?

I am one of them who thinks callbacks are not really pythonic and do not improve code readability.

How do you deal with an infinite stream without callbacks? Your example seems to be polling the battery every 5 seconds. But taking the telemetry, for instance, that's not how it works in mavlink, and that's not how it is implemented in our gRPC implementation. So my question is the following: would it imply that the python frontend would need to save all the values it receives, and wait for the client to poll for them?

The way it is implemented right now is using gRPC streams, so essentially callbacks. And that's why I feel like RxPy fits naturally there: it is a library for dealing with streams. Translating our streams to something that can be polled by asyncio seems like it is more complex and therefore prone to error. And not consistent with the other projects, which makes it more difficult to maintain.

What is your opinion on that, @xvzf? Since you've been using both RxPy and asyncio, I am very interested in your opinion :-).

[hamishwillee]

I don't have experience about Python 2 vs Python 3... How limiting is that? Can we assume every new project is running Python 3.5?

IMO being restricted to Python 3 is not the problem it was 3 or 4 years. Most really useful 3rd party libraries support both Python 2 and 3, and some big projects are moving exclusively to Python 3 - e.g. Django.

I personally would prefer we support both, but would not put it as a requirement on the project.

[xvzf]

I don't have experience about Python 2 vs Python 3... How limiting is that? Can we assume every new project is running Python 3.5?

At work we switched to Python 3 two years ago, Python 2.7 will get updates, but not new features and is (related to the syntax) sort of incompatible with Python 3. Ubuntu 16.04 was already using python 3.5, the latest release is using 3.6.

would it imply that the python frontend would need to save all the values it receives, and wait for the client to poll for them?

Sort of - I was thinking about a direct call and a non blocking socket, but when we are handling with streams and therefore this would not work

What is your opinion on that, @xvzf? Since you've been using both RxPy and asyncio, I am very interested in your opinion :-).

We should go with RxPy, the gRPC client for python does not support asyncio (grpc/grpc#6046) therefore we'd have to develope some workarounds just for getting asyncio working with gRPC.
A "shared" framework would also be beneficial!

[JonasVautherin]

We should go with RxPy, the gRPC client for python does not support asyncio (grpc/grpc#6046) therefore we'd have to develope some workarounds just for getting asyncio working with gRPC.

Well, that makes the choice easier then :-).

A "shared" framework would also be beneficial!

What do you mean by that?

[JonasVautherin]

BTW I would advise you start having a look at the current Swift implementation, @xvzf, probably beginning with Core and Action.

Also, I'd like to get your opinion on the current structure of the Python repo. My idea was to have one python package per plugin, but I'm not sure if I did that the right way :-).

[xvzf]

What do you mean by that?

While using Rx, development wit DroneCore will be the same in every language!

BTW I would advise you start having a look at the current Swift implentation, @xvzf, probably beginning with Core and Action.

Already had a look, the gRPC interface together wit the Rx Framework looks promising and quite easy to implement!

Also, I'd like to get your opinion on the current structure of the Python repo. My idea was to have one python package per plugin, but I'm not sure if I did that the right way :-).

I'd rather go with one package for DroneCore and using modules to implement Core, Action, etc. I'll do a PR of how I would consider the structure as future proof this afternoon! It'll likely be sort of the same as the Swift implementation!

[JonasVautherin]

What's the difference between a package and a module in Python? The idea is that you can install and use the plugins separately. Because maybe a third-party will want to create and publish his own my_custom_action, and I believe that would be a separate package, right?

We could say we have one package for the "official modules" and different packages for the third-party plugins, but I was thinking that maybe, having one package for each plugin would just make it more consistent.

Again, you tell me, I'm not very used to Python packaging :-).

[xvzf]

A python package (a directory) is a collection of modules (let's just say .py files for simplicity), that's it. Inheritance is easier than e.g. in Java and there is no need for extensions of DroneCore to be in the same package!

E.g. for importing the Core or Action it would be:

from dronecore import Core, Action

or

import dronecore

# Access to the core
test = dronecore.Core()

I just started restructuring the repo and push to my fork, I'll drop a link here in a few minutes when I'm done and we can discuss whether it is worth a PR.

An extension could be named e.g. dronecore_testextension. That's quite common in a popular webframework for Python and I quite like this approach for naming extensions

[xvzf]

Here the link to the branch:
https://github.com/xvzf/DroneCore-Python/tree/restructure

Can we agree on using PEP8 guidelines for code formating when implementing the library? My vim setup doesn't like it at all if files are not following PEP8 ;-)

[julianoes]

@xvzf yes I vote PEP8!

[JonasVautherin]

I'm happy to use standard style guidelines. If we can have a style checker, that's even better!

[xvzf]

Pylint is pretty handy, as well as autopep8!

Just made a PR this morning! What do you think about the structure?

[JonasVautherin]

I merged it already :-). Thanks a lot for your help! I guess the next step would be to look at RxPy and try to make a simple call with it (e.g. Action.arm()), in a similar way as what is implemented in DroneCore-Swift.

For now, you can just start the backend separately on your computer, and connect the Python bindings to it.

I'll close this issue for now and will contact you on Slack (PX4) to discuss what could be the next step!