NMS parameters for NuScenes panoptic fusion

[qchenclaire]

Hi, thank you for your interesting paper and releasing the code. I got a question about panoptic fusion on NuScenes. What NMS parameters for panoptic fusion you use for baseline PolarNet + CBGS in table 3? I use group nms following the multi-group head and nms parameters per group is
nms_pre_max_size=1000, nms_post_max_size=83, nms_iou_threshold=0.2, score_threshold=0.1
But I got panoptic quality as 31 instead of 67 for PolarNet + CBGS. I got decent SQ, but for RQ there are a lot of false positives for things.

I followed the evaluation code here https://github.com/edwardzhou130/Panoptic-PolarNet/blob/main/utils/eval_pq.py and change min_points to 20 as stated in your paper.

[edwardzhou130]

I used the pretrained model and the default parameters setting provided in OpenPCDet. The validation result I got is almost the same as listed in their repo. And in the fusion, I assigned a unique instance label to the matched points in each bbox. If you are getting a lot of false positives, maybe you can check the number of the different instance labels in both your prediction and gt. It should be similar to the number of bboxs.

[qchenclaire]

How did you generate groundtruth instance ids to the points? By checking if a point is inside the ground truth box of the same semantic class?

[edwardzhou130]

Some points can actually be outside a bbox. So I use the distance to the closest bbox center to assign the instance id. There are also some standalone points that don't have a bbox. I remove those points that are more than 5m apart from the nearest bbox.

[qchenclaire]

Some points can actually be outside a bbox. So I use the distance to the closest bbox center to assign the instance id. There are also some standalone points that don't have a bbox. I remove those points that are more than 5m apart from the nearest bbox.

You remove the points outside any box & 5m apart from nearest box? or if a point is inside a box but 5m apart from the center, do you also remove it?

[edwardzhou130]

I did the removal and finding the closest bbox steps only for those points outside any box. If a point is inside any box, it will have the instance id of that bbox.

[qchenclaire]

Have you tested the upper bound of NuScenes panoptic quality? I mean by using ground truth boxes and ground truth segmentation recovered from the network. I recovered the ground truth boxes from the target assigner and ground truth segmentation and got det map as 98.7 and seg miou as 95.3. They are not 100 because some boxes may be assigned to the same anchor or they are filtered after nms. For segmentation the point predictions are reconstructed from voxel predictions so seg miou is limited by voxel resolution. With these ground truth boxes and segmentation, I got PQ 83.2, SQ 93.8 and RQ 88.1, which is the upper bound of CBGS + PolarNet. I noticed you got RQ 84.6 for CBGS + PolarNet. That was very high and close to the upper bound. I think I might have missed something in predicting instance ids or evaluation. What is the upper bound you get?

[edwardzhou130]

I haven't tested the upper bound. The RQ I got for CBGS + PolarNet is 78.0%. 84.6% is the SQ.

[qchenclaire]

Did you filter gt boxes out of 50m when generating instance ids? I found my upper bound PQ is low because the gt boxes I recovered does not contain boxes out of 50m. NuScenes detection does not evaluate those boxes so det map is still close to 100. After I filtered boxes and points out of 50m, the upper bound PQ reached 91. I saw in your paper your range for NuScenes is [0,50]m so yours was not able to get heat map out of 50m either?

[qchenclaire]

I finally reproduced PQ for CBGS + PolarNet as 67 by filtering gt boxes and points within these ranges:
class_range = { "car": 50, "truck": 50, "bus": 50, "trailer": 50, "construction_vehicle": 50, "pedestrian": 40, "motorcycle": 40, "bicycle": 40, "traffic_cone": 30, "barrier": 30 }
and 50m for stuff points

[edwardzhou130]

Sorry for the late response. I used all the gt bbox given from nuScenes API. I did not filter out any bbox during the aforementioned gt generation procedure except that I removed some "thing" points. The CBGS baseline network will limit the point cloud to be inside the +-51.2m range. And for the Panoptic-PolarNet, I assigned all the out-of-range points to its closest edge. The center heatmap only generates predictions within the 50m range.

[qchenclaire]

Sorry for the late response. I used all the gt bbox given from nuScenes API. I did not filter out any bbox during the aforementioned gt generation procedure except that I removed some "thing" points. The CBGS baseline network will limit the point cloud to be inside the +-51.2m range. And for the Panoptic-PolarNet, I assigned all the out-of-range points to its closest edge. The center heatmap only generates predictions within the 50m range.

I also got around PQ 67 without filtering according to distance. I think that's because boxes out of 50m usually contains fewer than 20 points. I think the key is to filter instances with fewer than 20 points when generating instance ids. I'm not sure how min_points in https://github.com/edwardzhou130/Panoptic-PolarNet/blob/main/utils/eval_pq.py work but I did not want to change this code and just set min_points=20 as well.