DOPE detection cube not detected on custom symmetrical object (bowl) with symmetrical texture

[saratrajput]

I have trained the DOPE network on a bowl with symmetrical pattern. It was trained for 220 epochs with 7000 domain randomized images; learning rate changed from 0.001 to 0.0001 after 150 epochs. This was done taking reference from https://www.youtube.com/watch?v=g1adPMSmrXY.

I wanted to know if the DOPE model faces any difficulty with detecting and drawing cubes for a symmetrical object with symmetrical pattern.

[TontonTremblay]

When dealing with symmetrical objects you have to be careful with the location of the keypoints. If you look at the output you shared for the belief map, you see that the centroid is correct, but not the rest, each display a circle. This means the neural network learn to output the distribution of where the point might be on the object. Since it is symmetrical, there is no signal to correlate the location of a keypoint and the object and thus outputting a distribution. The easiest way to deal with symmetries is to fix the axe of rotation for that object, this means for any point of view the rotation value on the axe of symmetry is always the same.

Look at Sec. 3.3 of the paper "BB8: A Scalable, Accurate, Robust to Partial Occlusion Method for Predicting the 3D Poses of Challenging Objects without Using Depth"

Please note that NDDS does not provide such feature. You will have to manually compute them by reprojecting the keypoints.

[saratrajput]

@TontonTremblay Thanks!

[aditya2592]

@TontonTremblay, keeping the requirement for symmetrical objects in mind, can we still use the FAT dataset to train DOPE to predict the 024_bowl YCB object?

[TontonTremblay]

Yeah you can, but you will have to play around in the data loader to take those symmetries in check. I was able to do solid colored cubes with dope and NDDS a while back. So I do not see why you could not detect the bowl. You might have to find a canonical view and reproject the keypoints. This is what I would start looking into.

[aditya2592]

Thanks for the help!

[aditya2592]

Hi @TontonTremblay . Does DOPE learning anything from an image where the object of interest is not present or those images are automatically excluded by the training script?

[TontonTremblay]

They are not ignored, the dataloader should return an empty black image for both affinities and beliefs. You need to show it some empty images otherwise you would get a lot of false positives.

[aditya2592]

Thanks for the info. But how many images like that should be shown? Because according to the FAT paper there are around 10k images per object, so is it needed to show 50k non object images? Assuming we are taking left camera images only

[TontonTremblay]

I am not sure, but a small percentage <5% should be shown I would say.

[Chi-Ruei]

have you solved your problem?