Tracking in video annotation "draging" when hitting border

[marinmarcillat]

Hello !

I'm trying to do some video annotation with auto-tracking, but while it's working very well in some cases, in other (especially when the movement of the camera is a bit strange like forward and left at the same time), the tracking is done quite well until the tracked object hit the border, and then the tracking just follow the frame border (see gif).

I wasn't able to understand fully what cause this. I believe that the threshold "object not within the frame anymore" might be too low, but I wasn't able to analyze the code for auto-tracking because l don't know where it is located. Maybe I can have a look and try to debug it myself if you can give me some hints ?

I'm working on Ifremer's instance of Biigle, on this video: https://filesender.renater.fr/?s=download&token=3aae9728-9913-4194-aeb8-f9737814af63. The annotation that you are seeing in the gif were made at 15"36

Thank you for your help !

[dlangenk]

Hey Marinmarcillat,

unfortunately this is a known problem with the underlying algorithm(s). The tracker is initialized here:

core/resources/scripts/ObjectTracker.py

Line 17 in 2547db9

self.tracker = cv2.TrackerCSRT_create()

We tried different tracking algorithms implemented in OpenCV. The CSRT Tracker was the best regarding tracking performance and detecting objects exiting the video. Unfortunately there is no threshold (at least not to my knowledge).
Here you can find a detailed blog post about the tracking algorithms in OpenCV.
The MMTracking library implements some more recent algorithms, you can have a look, but I don't think that we have the ressources right now to integrate these into Biigle in the near future. However if you want to work on this let us know and we can likely assist. But probably @mzur can say more about this (however he is ill at the moment).

[marinmarcillat]

Hello Daniel

Thanks for you quick and detailed answer.
Looking into the python code (and my data), I might have found a quick fix.

At this line, the program evaluate if the tracking center is still within the frame border:

def center_out_of_frame(self, center): return center[0] <= 0 or center[1] <= 0 or center[0] >= self.width or center[1] >= self.height

The problem is that with my annotation tracking, I regularly fall one pixel short from the frame edge.
I tried to modify the line to :

def center_out_of_frame(self, center): return center[0] <= 1 or center[1] <= 1 or center[0] >= self.width - 1 or center[1] >= self.height - 1

and it work much better ! The tracking stop instantaneously when hitting the border !
This could be a fix.

The MMT looks quite impressive as well, but much harder to implement. Maybe we may discuss this with Martin once he will be healed.

[dlangenk]

It seems reasonable to add a small margin. But we can wait for Martin to finally decide if we change that. He has done more work on tracking.

[mzur]

@marinmarcillat Thanks for bringing this up. I wanted to look into this issue for a long time now and the solution you found looks good! Please send a pull request.

I'd be open for experiments with MMTracking. We'll probably port MAIA to PyTorch/MMDetection in the next months (biigle/maia#96). Once this is finished, all the necessary dependencies for MMTracking should be available. Does this only work with GPU support? I'd be hesitant to include stuff that requires GPUs in biigle/core. Maybe it could be a separate module that would override the OpenCV tracker. Anyway, this would have to be a community contribution, as we (the BIIGLE team) will focus on other things for now.

[marinmarcillat]

Done, just added the pull request.

I'm glad to see that you have some python code in Biigle, I should be able to contribute a bit. I'm going to do some test (with and without GPU if possible) with MMTracking, I will keep you up informed with the results !

[mzur]

Great, thanks! I'll close this issue with the PR. Please reopen a new issue or PR if you want to show anything.

There is even more Python code in the biigle/maia module but as I said we will soon be working on that one ourselves. So that's not a good place for contributions right now. A better object tracker would be cool, too.