Add SemanticSegmentationEvaluator #238

yuyu2172 · 2017-06-01T15:01:47Z

merge after #217.

The current evaluate.py is wrong because n_positive adds up for pixels with "ignore" GT labels.
I made the following modification.

    for batch in it:
        img, gt_label = concat_examples(batch, args.gpu)
        img = chainer.Variable(img, volatile=True)
        pred_label = F.argmax(F.softmax(model(img)), axis=1)
        pred_label = cuda.to_cpu(pred_label.data)
        gt_label = cuda.to_cpu(gt_label)
        for cls_i in range(n_class):
            if cls_i in ignore_labels:
                continue
            pred_label = pred_label[gt_label != -1]
            gt_label = gt_label[gt_label != -1]
            n_positive[cls_i] += np.sum(pred_label == cls_i)
            n_true[cls_i] += np.sum(gt_label == cls_i)
            n_true_positive[cls_i] += np.sum(
                (pred_label == cls_i) * (gt_label == cls_i))

I set batchsize to one. This was critical to reproduce the results with SemanticSegmentationEvaluator.

                    Sky : 0.878988525
               Building : 0.668390287
                   Pole : 0.192338392
                   Road : 0.873874189
               Pavement : 0.642089768
                   Tree : 0.622743688
             SignSymbol : 0.189341867
                  Fence : 0.213735296
                    Car : 0.635499263
             Pedestrian : 0.273885917
              Bicyclist : 0.241545763
==================================
               mean IoU : 0.493857541
 Class average accuracy : 0.670468304
Global average accuracy : 0.826623129

With SemanticSegmentationEvaluator, the scores were as follows. This is identical to the score above.

                    Sky : 0.878988525
               Building : 0.668390287
                   Pole : 0.192338392
                   Road : 0.873874189
               Pavement : 0.642089768
                   Tree : 0.622743688
             SignSymbol : 0.189341867
                  Fence : 0.213735296
                    Car : 0.635499263
             Pedestrian : 0.273885917
              Bicyclist : 0.241545763
==================================
               mean IoU : 0.493857541
 Class average accuracy : 0.670468304
Global average accuracy : 0.826623129

yuyu2172 · 2017-06-06T08:15:57Z

Since the content of this PR is not purely about SemanticSegmentationEvaluator, I split irrelevant part into #249.

…aluators

…on-evaluator

…yuyu2172/chainercv into semantic-segmentation-evaluator

yuyu2172 · 2017-06-22T02:01:27Z

I changed directory structure of extensions. This is the reason why I added no-compat label.

yuyu2172 · 2017-07-03T06:12:19Z

@Hakuyume
Can you check this?

Hakuyume · 2017-07-03T07:31:33Z

@yuyu2172 OK

Hakuyume · 2017-07-03T07:39:36Z

chainercv/extensions/evaluator/semantic_segmentation_evaluator.py

+            following tuple :obj:`img, label`.
+            :obj:`img` is an image, :obj:`label` is pixel-wise label.
+        target (chainer.Link): A semantic segmentation link. This link should
+            have :meth:`predict` method which takes a list of images and


which -> that? #229 (comment)

Hakuyume · 2017-07-03T07:40:28Z

chainercv/extensions/evaluator/semantic_segmentation_evaluator.py

+            returns :obj:`labels`.
+        label_names (iterable of strings): An iterable of names of classes.
+            If this value is specified, IoU and class accuracy for each class
+            is also reported with the keys


is -> are

Hakuyume · 2017-07-03T07:44:01Z

docs/source/reference/extensions.rst

-DetectionVisReport
-~~~~~~~~~~~~~~~~~~
-.. autofunction:: DetectionVisReport
-
 DetectionVOCEvaluator
 ~~~~~~~~~~~~~~~~~~~~~
 .. autofunction:: DetectionVOCEvaluator


I think it is a class (not a function)

We've been using autofunction for many classes (e.g. Dataset class).

They should be fixed too.

Hakuyume · 2017-07-03T07:46:36Z

tests/extensions_tests/evaluator_tests/test_semantic_segmentation_evaluator.py

+        np.testing.assert_equal(eval_['target/pixel_accuracy'], 1.)
+        np.testing.assert_equal(eval_['target/mean_class_accuracy'], 1.)
+        np.testing.assert_equal(eval_['target/iou/a'], 1.)
+        np.testing.assert_equal(eval_['target/iou/b'], 1.)


Could you make iou/a and iou/b different from each other?

yuyu2172 · 2017-07-03T09:36:47Z

Thanks for reviewing.

Hakuyume

LGTM

yuyu2172 changed the title ~~Add SemanticSegmentationEvaluator~~ [WIP] Add SemanticSegmentationEvaluator Jun 1, 2017

yuyu2172 force-pushed the semantic-segmentation-evaluator branch 3 times, most recently from 9d808de to f8d0014 Compare June 1, 2017 15:19

yuyu2172 mentioned this pull request Jun 6, 2017

Fix SegNet example's evaluation protocol #249

Merged

use semantic_segmentation_evaluator inside train

2ccc93f

yuyu2172 force-pushed the semantic-segmentation-evaluator branch from 9011c1e to 2ccc93f Compare June 6, 2017 08:22

yuyu2172 added 2 commits June 6, 2017 18:01

change semantic_segmentation_evaluator to be consistent with other ev…

1b0348a

…aluators

follow style of detection_voc_evaluator

9506e19

Hakuyume mentioned this pull request Jun 13, 2017

Segnet train.py no longer reports pixel_accuracy after v2 update #273

Closed

yuyu2172 added 3 commits June 14, 2017 09:47

merge

6b5c270

fix __init__.py

b68b2e8

cosmetic

3b298b2

yuyu2172 changed the title ~~[WIP] Add SemanticSegmentationEvaluator~~ Add SemanticSegmentationEvaluator Jun 14, 2017

yuyu2172 assigned Hakuyume Jun 14, 2017

yuyu2172 added 3 commits June 14, 2017 09:57

fix train.py

2def1dc

Merge remote-tracking branch 'origin/master' into HEAD

a5d64eb

fix train.py

3fd76c1

yuyu2172 unassigned Hakuyume Jun 16, 2017

Hakuyume mentioned this pull request Jun 21, 2017

Update train.py in the SegNet example for v2 #292

Closed

yuyu2172 added 7 commits June 21, 2017 22:09

Merge remote-tracking branch 'origin/master' into semantic-segmentati…

65a431b

…on-evaluator

use new evaluation API

e5954e9

Merge branch 'semantic-segmentation-evaluator' of https://github.com/…

eb8eebc

…yuyu2172/chainercv into semantic-segmentation-evaluator

fix documentations for semantic segmentation evaluator

7e7cab5

change directory structure

002a4b5

fix doc

c048884

report mean_class_accuracy

9fef692

yuyu2172 assigned Hakuyume Jun 22, 2017

yuyu2172 added the feature label Jun 22, 2017

yuyu2172 added the no-compat label Jun 22, 2017

fix train

7273241

Hakuyume reviewed Jul 3, 2017

View reviewed changes

yuyu2172 added 4 commits July 3, 2017 18:05

Merge remote-tracking branch 'origin/master' into HEAD

46e4add

autofunctions->autofunction

41be470

fix doc

60d42f7

make iou/a and iou/b different

ac50340

yuyu2172 mentioned this pull request Jul 3, 2017

Use autoclass for datasets doc #308

Merged

Hakuyume approved these changes Jul 3, 2017

View reviewed changes

Hakuyume merged commit dda1576 into chainer:master Jul 3, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add SemanticSegmentationEvaluator #238

Add SemanticSegmentationEvaluator #238

yuyu2172 commented Jun 1, 2017 •

edited

Loading

yuyu2172 commented Jun 6, 2017

yuyu2172 commented Jun 22, 2017

yuyu2172 commented Jul 3, 2017

Hakuyume commented Jul 3, 2017

Hakuyume Jul 3, 2017

Hakuyume Jul 3, 2017

Hakuyume Jul 3, 2017

yuyu2172 Jul 3, 2017

yuyu2172 Jul 3, 2017

yuyu2172 Jul 3, 2017

Hakuyume Jul 3, 2017

yuyu2172 commented Jul 3, 2017

Hakuyume left a comment

Add SemanticSegmentationEvaluator #238

Add SemanticSegmentationEvaluator #238

Conversation

yuyu2172 commented Jun 1, 2017 • edited Loading

yuyu2172 commented Jun 6, 2017

yuyu2172 commented Jun 22, 2017

yuyu2172 commented Jul 3, 2017

Hakuyume commented Jul 3, 2017

Hakuyume Jul 3, 2017

Choose a reason for hiding this comment

Hakuyume Jul 3, 2017

Choose a reason for hiding this comment

Hakuyume Jul 3, 2017

Choose a reason for hiding this comment

yuyu2172 Jul 3, 2017

Choose a reason for hiding this comment

yuyu2172 Jul 3, 2017

Choose a reason for hiding this comment

yuyu2172 Jul 3, 2017

Choose a reason for hiding this comment

Hakuyume Jul 3, 2017

Choose a reason for hiding this comment

yuyu2172 commented Jul 3, 2017

Hakuyume left a comment

Choose a reason for hiding this comment

yuyu2172 commented Jun 1, 2017 •

edited

Loading