Evaluation metrics

[kulkarnikeerti]

Hi @lingorX @wenguanwang, @tfzhou

Thank you for the great paper.

I have a question regarding the evaluation metrics. The metrics used in this paper is the mIoU.

1. So this mIoU is calculated for the original leaf nodes or does this also contain the parent nodes (which label_Ids are not present in the dataset)?
2. If I want to compare the segmentation models without and with the hierarchy, is mIoU appropriate metrics for that? If yes, how does this metrics behave? I mean should this increase if we have a hierarchy?

Thanks in advance

[joshmyersdean]

I am also curious about this.

[kulkarnikeerti]

1. It calculates only for the leaf nodes and does not consider the parent nodes into account while calculating the mIoU
2. This current implementation does not include the mIoU calculation for parent nodes, so if you just want to compare the leaf nodes you can use it.