Passing frames to python

[mmyros]

Hello, thanks for sharing this, great software! Do you think it is feasible to have frames available in python? Any pointers on the best way to implement?

[jonnew]

Thanks! Hmm, good question. I think that if was I was going to implement this it would be either to:

1. Create a new component; oat-framesock
2. Generalize oat-posisock to publish any token (frame or position) to the network.

Basically, either option would marry a ZMQ socket to Oat's synchronization structure and data types. This would make Frames available to any language with a decent ZMQ binding and there are a lot of those. Also, because ZMQ is amazing, you could stream frames to python via several different transports: inter-process communication for very high performance, TCP over the network if your python script was running on a different computer.

Because ZMQ msg_t's data specified as avoid * and size_t (number of bytes) pair, we would need to think of a way to serialize oat::Frames to pack them into a ZMQ message. There are lots a good ways to do that:

• https://github.com/google/flatbuffers
• https://github.com/google/protobuf

come to mind. So basically, in summary, I think the easiest way is to make a new component called oat-framesock that essentially copies /src/positionsocket. All the Position specific stuff would be exchaged for Frame analogs. Finally, to create ZMQ messages from Frames, a third party serialization library is used.

If you just want to hack this, you can get the raw image data using oat::Frame::data public member which provides a pointer to the raw pixel values. You can cast to void * and send, but this will not contain any metadata.

Does any of this sounds like it might be something you want to try? I'm happy to guide you if you want to give it a shot. If it results in something useful am I happy to pull it into the codebase.

[mmyros]

Thanks for a detailed response! This is helpful. I was thinking of hacking this on python's end, but only if there is a library that can do this somewhat easily. Unfortunately, I'm not well-versed in C enough to make this quick enough, and I am in a somewhat tight timeframe - we already received animals to train on a maze. Maybe I can contribute in other ways.