Object Detection

[willcray]

Hey @mrousavy,

I forked this repo to create an object detection plugin for object tracking + bounding boxes. I'm hoping to add it to your Community Plugins page once it's working. However, I'm running into a small issue.

In my plugin, the call to .process() always returns an empty array of objects. If I switch the client to an image labeling model (following your plugin), it successfully returns labels, and I'm able to extract that data and pass it back to the JS layer. I'm running into the same issue on iOS, although it might be complicated by this issue.

Do you have any idea what's wrong? Any help is appreciated! I'm trying to create something that enriches the ecosystem of react-native-vision-camera, so hopefully I'm not wasting your time.

[mrousavy]

This might be one of the two things:

• Maybe the format passed into the <Camera> component has an invalid pixelFormat (e.g. x420, which is 10-Bit HDR, often not yet supported by some native funcs like MLKit)
• Maybe your MLKit Model is not downloaded (Make sure you added the Pod via pod install and follow their instructions)

[willcray]

This might be one of the two things:

• Maybe the format passed into the <Camera> component has an invalid pixelFormat (e.g. x420, which is 10-Bit HDR, often not yet supported by some native funcs like MLKit)
• Maybe your MLKit Model is not downloaded (Make sure you added the Pod via pod install and follow their instructions)

I think your pixelFormat pointer will likely fix the issue on iOS. I'm working through that in mrousavy/vision-camera-image-labeler/#12.

I'm trying to work through the Android issue to no avail in googlesamples/mlkit/#/423. The jist of the Android issue is that your image labeler code is working well for me on Android. However, when I switch the model to be Object Detection or Face Detection, I receive java.lang.IllegalStateException: Image is already closed on the first call of the plugin. It continues to throw the same exception on each subsequent call of .process(). I've tried both excluding the .onCompleteListener() with the call to close the ImageProxy, as the MLKit guides mention here. I've also tried excluding both of those entirely, as you did in your image labeler plugin.

Any ideas would be hugely helpful!

[mrousavy]

Hmm, if the Image is already closed that means that your function maybe isn't synchronous, e.g. if it's using callbacks

[willcray]

I switched to a synchronous function, which eliminated the exceptions. Thanks for the heads up on that.

Then, I ran into the issue that I was encountering previous to the exception: no objects were detected ever, despite aiming the camera at a myriad of objects.

I was able to start detected objects by changing the frameProcessorFps CameraProp from 5 to auto (the default). I now am able to detect some objects, albeit inconsistently. This is an improvement, but its results are far from being useful.

I'm not sure why changing the frameProcessorFps would yield such a substantial difference in results. I'm also not sure what other CameraProps to tweak in order to detect objects consistently and well. It seems to be a limitation in the frame processor implementation, because MLKit's demo object detection app works superbly for me on both iOS and Android.

[akhilkrishnan1268]

I have used a TFLITE model for object detection and output is just array of 4 objects.What am i missing.How to get detected objects classnames or labels and bounding box cordinates?