Motion detection not working when using Tensorflow and event filter

[ducadk]

Hi,

I'm a big fan of your HA Shinobi integration :)
I recently modified my Shinobi setup to use Tensorflow (as detection engine) and added a filter, so it only will trigger if a 'Person' is identified. The integration with HA is still working except that when a motion detection is triggered in Shinobi it is not triggered in HA.
I do see the following in HA (when motion detection is triggered in Shinobi and debug log is on in HA):
2023-01-07 13:07:30.993 DEBUG (MainThread) [custom_components.shinobi.component.api.websocket] Payload received, Data: {'f': 'detector_trigger', 'id': 'terrasse', 'ke': 'sC0jRzUi9E', 'details': {'plug': 'Tensorflow', 'name': 'Tensorflow', 'reason': 'object', 'matrices': [{'x': 489.9746322631836, 'y': 41.84635877609253, 'width': 126.097412109375, 'height': 218.58217477798462, 'tag': 'person', 'confidence': 0.8130321502685547}], 'imgHeight': 480, 'imgWidth': 640, 'time': 203}, 'doObjectDetection': False}

2023-01-07 13:07:30.993 DEBUG (MainThread) [custom_components.shinobi.component.api.websocket] Firing event shinobi/object, Payload: {'f': 'detector_trigger', 'id': 'terrasse', 'ke': 'sC0jRzUi9E', 'details': {'plug': 'Tensorflow', 'name': 'Tensorflow', 'reason': 'object', 'matrices': [{'x': 489.9746322631836, 'y': 41.84635877609253, 'width': 126.097412109375, 'height': 218.58217477798462, 'tag': 'person', 'confidence': 0.8130321502685547}], 'imgHeight': 480, 'imgWidth': 640, 'time': 203}, 'doObjectDetection': False}

2023-01-07 13:08:00.888 DEBUG (MainThread) [custom_components.shinobi.component.api.websocket] No message handler available, Message: 456-["f",{"f":"video_build_success","hrefNoAuth":"/videos/sC0jRzUi9E/terrasse/2023-01-07T13-07-24.mp4","filename":"2023-01-07T13-07-24.mp4","mid":"terrasse","ke":"sC0jRzUi9E","ext":"mp4","time":"2023-01-07T12:07:24.000Z","size":3941256,"end":"2023-01-07T12:07:54.879Z","objects":"person","events":[{"f":"trigger","id":"terrasse","ke":"sC0jRzUi9E","details":{"plug":"Tensorflow","name":"Tensorflow","reason":"object","matrices":[{"x":489.9746322631836,"y":41.84635877609253,"width":126.097412109375,"height":218.58217477798462,"tag":"person","confidence":0.8130321502685547}],"imgHeight":480,"imgWidth":640,"time":203},"frame":{"_placeholder":true,"num":0},"pluginKey":"491df0a3c4518eab92e78bf69ee90d89c4bc8c23ca62c250c6814cd5ecb5","plug":"Tensorflow","doObjectDetection":false,"mid":"terrasse","currentTime":"2023-01-07T12:07:30.862Z","currentTimestamp":"2023-01-07T13:07:30+01:00","screenshotName":"object_Terrasse_terrasse_sC0jRzUi9E_2023-01-07T13-07-30","screenshotBuffer":null},{"f":"trigger","id":"terrasse","ke":"sC0jRzUi9E","details":{"plug":"Tensorflow","name":"Tensorflow","reason":"object","matrices":[{"x":425.4217529296875,"y":49.87807273864746,"width":162.87757873535156,"height":330.8500099182129,"tag":"person","confidence":0.5065410733222961}],"imgHeight":480,"imgWidth":640,"time":226},"frame":{"_placeholder":true,"num":1},"pluginKey":"491df0a3c4518eab92e78bf69ee90d89c4bc8c23ca62c250c6814cd5ecb5","plug":"Tensorflow"},{"f":"trigger","id":"terrasse","ke":"sC0jRzUi9E","details":{"plug":"Tensorflow","name":"Tensorflow","reason":"object","matrices":[{"x":407.5551986694336,"y":33.83763313293457,"width":165.68626403808594,"height":335.7494831085205,"tag":"person","confidence":0.973068118095398}],"imgHeight":480,"imgWidth":640,"time":229},"frame":{"_placeholder":true,"num":2},"pluginKey":"491df0a3c4518eab92e78bf69ee90d89c4bc8c23ca62c250c6814cd5ecb5","plug":"Tensorflow"},{"f":"trigger","id":"terrasse","ke":"sC0jRzUi9E","details":{"plug":"Tensorflow","name":"Tensorflow","reason":"object","matrices":[{"x":389.66522216796875,"y":21.461176872253418,"width":194.9323272705078,"height":358.30957889556885,"tag":"person","confidence":0.9273003339767456}],"imgHeight":480,"imgWidth":640,"time":220},"frame":{"_placeholder":true,"num":3},"pluginKey":"491df0a3c4518eab92e78bf69ee90d89c4bc8c23ca62c250c6814cd5ecb5","plug":"Tensorflow"},{"f":"trigger","id":"terrasse","ke":"sC0jRzUi9E","details":{"plug":"Tensorflow","name":"Tensorflow","reason":"object","matrices":[{"x":390.4310607910156,"y":20.805974006652832,"width":125.38726806640625,"height":364.33011531829834,"tag":"person","confidence":0.9167547821998596}],"imgHeight":480,"imgWidth":640,"time":226},"frame":{"_placeholder":true,"num":4},"pluginKey":"491df0a3c4518eab92e78bf69ee90d89c4bc8c23ca62c250c6814cd5ecb5","plug":"Tensorflow"},{"f":"trigger","id":"terrasse","ke":"sC0jRzUi9E","details":{"plug":"Tensorflow","name":"Tensorflow","reason":"object","matrices":[{"x":406.4086151123047,"y":52.66399383544922,"width":160.6890869140625,"height":312.3201370239258,"tag":"person","confidence":0.9617406725883484}],"imgHeight":480,"imgWidth":640,"time":228},"frame":{"_placeholder":true,"num":5},"pluginKey":"491df0a3c4518eab92e78bf69ee90d89c4bc8c23ca62c250c6814cd5ecb5","plug":"Tensorflow"}]}]
Would it be possible to trigger a motion detection event in HA based on above input?
Thanks in advance

[ronluna]

were you able to get motion triggers working in home assistance by any chance? I'm having the same problem.

[elad-bar]

There are several type of events resulted from detection - motion, soundChange, object, face, etc...

For the motion detection to work in HA, you will need the Shinobi Video to publish 'motion' event, in logs above I see just object, means that no motion was identified by Shinobi.

Shinobi (if configured) is sending frames to detector for furst motion detection then to all plugins (it can be forced to have first motion to allow plugins to process the frame as well - again, configuration).

Can you pls share screenshot of the motion detection and object detection?

Thanks

[ronluna]

Here it is:

[elad-bar]

Ok, seems that the plugin is not sending the right event name, for motion it should send motion not object

[ronluna]

Currently I don't see a way to send the frame for motion when object detection is enable.

[elad-bar]

are you trying to use tensor for motion detection or for object detection? if you want it for motion detection, can you pls share the motivation? (asking because a year ago I thought about it too)

thanks

[ronluna]

Well, I actually thought that using Tensorflow for Object and Motion detection could offload the CPU as my understanding is that the built-in motion detection (Pixel Array) could spike the CPU usage when running for multiple cameras.

Now I'm wondering if you could consider the possibility of use the object detection flag as motion detection or simply add a new entity for object detection that could be used to trigger things in HA?

[ducadk]

Hi, the usecase for using Tensorflow for me is the same as ronluna, to be able to benefit from the GPUs available in my system when detecting motion/objects.
Regarding the motion detection prior to object detection, then I believe that was removed as an option in the new version of Shinobi (it was at least a checkmark option in the older versions).
Hope you will be able to find a way to filter for these events anyway :o)

[elad-bar]

Make sense, that was my use case for exploring that option, but I stayed with the default one as I compared the results for 16 camera and the impact was minimal,
If you would like to have tensorflow supported as real motion detection you need to ask developer of Shinobi (discord) to support it or allow pixel engine (default) to work with GPU,
you can ask it over the discord channel, if he will agree I will try to assist him.

Will look at the default code to find out how can it be done using GPU, to support motion detection by external plugin, platform should notify plugin in which mode to run - as motion detection or object detection (currently the assumption is only motion detection).

Hope you find it helpful

[ducadk]

Hi,
Thanks for your reply!
I fully understand your arguments, but would it, as a workaround, be possible to look for the output generated by the Tensorflow engine;
[
"f",
{
"f":"video_build_success",
"hrefNoAuth":"/videos/sC0jRzUi9E/terrasse/2023-01-07T13-07-24.mp4",
.....
"objects":"person",
"events":[
{
"f":"trigger",
"id":"terrasse",
"ke":"sC0jRzUi9E",
"details":{
"plug":"Tensorflow",
"name":"Tensorflow",
"reason":"object",
"matrices":[
{
"x":489.9746322631836,
"y":41.84635877609253,
"width":126.097412109375,
"height":218.58217477798462,
"tag":"person", ......

And use that as indicator, that a motion detection has been triggered?
Once again, I appreciate you're taking your time to reply and look into this question, thanks in advance!

[elad-bar]

I don't think that the integration of Shinobi in HA to be familiar with set of additional events, that's the reason I'm forwarding that event as HA event so you will be able to perform custom actions (shinobi/object).

I went trhough exactly same process and ideas as you said, but.. what would be considered as motion detection - person, cat, dog, refrigerator? Just an event of object detection? What if not objects identified, what if someone will create a plugin of objects instead object, should I support it as well?
All those questions led me to make it simple with extension point (HA event) so people would do what they want if they are using plugin and not out of the box functionality (which is also the concept of the plugin)

[ducadk]

Hi,
I fully understand the uncertainty on how to make sure you're dealing with a motion detection and the thoughts you have had.
But would Tensorflow not only trigger a new event if it detect a new object in the provided frame (hence a something changed, equals motion)? Or will it keep trigger events if it keeps detects fx a refrigerator?

Another option would be, that you are able to define a "Event filter" in Shinobi, where you can specify which object types that should trigger an event. So the events triggered by Tensorflow might always be considered a motion detection?

[elad-bar]

no, it's hard coded as object trigger and not dynamically based on detection type, which is what should get fixed in Shinobi,
Once that will happen, this integration will know how to handle it as you expect without additional code change

[elad-bar]

Request should be addressed to Shinobi developer, closing the issue