Ground truth boxes for evaluation?

[Moqeet]

Hi Bowen,

I really like that you made a metric which focuses on the boundary quality rather than overall IOU. I was applying your method on my models and was not sure exactly how to use the ground truth boxes for evaluation (Section 6.2, Table 4). I have the intuition that instead of taking the region predicted by RPN, you use the ground truth boxes' region and apply a ROIAlign operation to get the region of interest. Am I correct?

Also, can you suggest a clean way of doing this?

Thanks in advance.

Best,
Hamd ul Moqeet Riaz

[bowenc0221]

You do not necessarily need to use ground truth box for evaluation, you can simply calculate the Boundary IoU between your prediction masks and ground truth masks.

The reason that we use ground truth boxes in our paper is only for analysis purpose: there are many factors contribute to the performance of an instance segmentation algorithm besides mask quality, the localization and classification errors are also penalized in the AP metric. In order to study the segmentation (mask) quality on its own, we feed ground truth box to the mask branch of Mask R-CNN to rule out the effect of localization and classification error.

Regarding your question. As I mentioned earlier, if you simply want to use Boundary IoU to evaluate your method, there is not need to use ground truth boxes. You can simply take the prediction masks with boxes from RPN predictions.

[Moqeet]

Thanks for your answer. Actually, the model that I trained was generating more false positives and negatives predictions compared to the baseline. The mask quality however is looking better than the baseline. Therefore, I exactly wanted to test what you mentioned in your answer i.e the mask quality independent of classification and localization errors.
I guess I already got the answer that you feed ground truth boxes instead predictions from bounding boxes to the mask branch during evaluation. I will do that for my model as well.
Once again, thanks for your help.