Model server inference using Python Mediapipe

[sayanmutd]

How can i loop over a set of detection results to run classifier over detected regions using mediapipe graph. It will be helpful if i get a graph example using openvino inference calculator

[mzegla]

Could you share more details? You mention python in the title. Do you mean python mediapipe package or do you want to execute python code in mediapipe graph in OVMS?

I assume the flow you want to achieve would look like:

1. Put image on the input of the graph
2. Run it through detection model
3. Extract detected regions and run them through classifier
4. Return some kind of combined results

Is that right?

[sayanmutd]

@mzegla Thanks for your prompt response. I want to excecute python code alongwith media pipe graph. I want to iterate through OVMS_PY_TENSOR containing detected regions inside the mediapipe graph to facilitate classification over those detected regions.

[mzegla]

Well, python execution is enabled via separate node in the mediapipe graph (there is a dedicated calculator for that).
Checkout: https://github.com/openvinotoolkit/model_server/blob/main/docs/python_support/quickstart.md

I suppose the easiest solution would be to have your whole processing done in Python - so you would have a single node that loads your detection and classification model in initialize method of OvmsPythonModel and then run detection, extraction and classification in the execute method.
See: https://github.com/openvinotoolkit/model_server/blob/main/docs/python_support/reference.md

Note that you will likely need to extend docker image with layers containing python packages you need like numpy, opencv etc.
(openvino is already available) - https://github.com/openvinotoolkit/model_server/blob/main/docs/python_support/reference.md#building-docker-image

The other solution is to have multiple nodes: detection, extraction and classification where only extraction is done in Python and detection/classification is executed via CPP based calculators. This approach also requires converter nodes between nodes mentioned earlier.

See CLIP demo for reference:
https://github.com/openvinotoolkit/model_server/tree/main/demos/python_demos/clip_image_classification