PaddlePaddle · michaelowenliu · Feb 22, 2021 · Jan 14, 2021 · Jan 22, 2021 · Jan 22, 2021
diff --git a/README.md b/README.md
@@ -93,6 +93,7 @@ python train.py --config configs/quick_start/bisenet_optic_disc_512x512_1k.yml
 * [API Tutorial](https://aistudio.baidu.com/aistudio/projectdetail/1339458)
 * [Data Preparation](./docs/data_prepare.md)
 * [Training Configuration](./configs/)
+* [Loss Usage](./docs/loss_usage.md)
 * [API References](./docs/apis)
 * [Add New Components](./docs/add_new_model.md)
 
@@ -111,7 +112,7 @@ If you find our project useful in your research, please consider citing:
 
 ```latex
 @misc{liu2021paddleseg,
-      title={PaddleSeg: A High-Efficient Development Toolkit for Image Segmentation}, 
+      title={PaddleSeg: A High-Efficient Development Toolkit for Image Segmentation},
       author={Yi Liu and Lutao Chu and Guowei Chen and Zewu Wu and Zeyu Chen and Baohua Lai and Yuying Hao},
       year={2021},
       eprint={2101.06175},

diff --git a/README_CN.md b/README_CN.md
@@ -95,6 +95,7 @@ python train.py --config configs/quick_start/bisenet_optic_disc_512x512_1k.yml
 * [API使用教程](https://aistudio.baidu.com/aistudio/projectdetail/1339458)
 * [数据集准备](./docs/data_prepare.md)
 * [配置项](./configs/)
+* [Loss使用](./docs/loss_usage.md)
 * [API参考](./docs/apis)
 * [添加新组件](./docs/add_new_model.md)
 
@@ -114,7 +115,7 @@ python train.py --config configs/quick_start/bisenet_optic_disc_512x512_1k.yml
 
 ```latex
 @misc{liu2021paddleseg,
-      title={PaddleSeg: A High-Efficient Development Toolkit for Image Segmentation}, 
+      title={PaddleSeg: A High-Efficient Development Toolkit for Image Segmentation},
       author={Yi Liu and Lutao Chu and Guowei Chen and Zewu Wu and Zeyu Chen and Baohua Lai and Yuying Hao},
       year={2021},
       eprint={2101.06175},

diff --git a/configs/ocrnet/ocrnet_hrnetw18_cityscapes_1024x512_160k_lovasz_softmax.yml b/configs/ocrnet/ocrnet_hrnetw18_cityscapes_1024x512_160k_lovasz_softmax.yml
@@ -0,0 +1,35 @@
+_base_: '../_base_/cityscapes.yml'
+
+batch_size: 2
+iters: 160000
+
+model:
+  type: OCRNet
+  backbone:
+    type: HRNet_W18
+    pretrained: https://bj.bcebos.com/paddleseg/dygraph/hrnet_w18_ssld.tar.gz
+  backbone_indices: [0]
+
+optimizer:
+  type: sgd
+
+learning_rate:
+  value: 0.01
+  decay:
+    type: poly
+    power: 0.9
+
+
+loss:
+  types:
+    - type: MixedLoss
+      losses:
+        - type: CrossEntropyLoss
+        - type: LovaszSoftmaxLoss
+      coef: [0.8, 0.2]
+    - type: MixedLoss
+      losses:
+        - type: CrossEntropyLoss
+        - type: LovaszSoftmaxLoss
+      coef: [0.8, 0.2]
+  coef: [1, 0.4]
diff --git a/configs/ocrnet/ocrnet_hrnetw18_road_extraction_768x768_15k.yml b/configs/ocrnet/ocrnet_hrnetw18_road_extraction_768x768_15k.yml
@@ -0,0 +1,48 @@
+
+
+batch_size: 4
+iters: 15000
+
+train_dataset:
+  type: MiniDeepGlobeRoadExtraction
+  dataset_root: data/MiniDeepGlobeRoadExtraction
+  transforms:
+    - type: ResizeStepScaling
+      min_scale_factor: 0.5
+      max_scale_factor: 2.0
+      scale_step_size: 0.25
+    - type: RandomPaddingCrop
+      crop_size: [768, 768]
+    - type: RandomHorizontalFlip
+    - type: Normalize
+  mode: train
+
+val_dataset:
+  type: MiniDeepGlobeRoadExtraction
+  dataset_root: data/MiniDeepGlobeRoadExtraction
+  transforms:
+    - type: Normalize
+  mode: val
+
+model:
+  type: OCRNet
+  backbone:
+    type: HRNet_W18
+    pretrained: https://bj.bcebos.com/paddleseg/dygraph/hrnet_w18_ssld.tar.gz
+  backbone_indices: [0]
+
+optimizer:
+  type: sgd
+
+learning_rate:
+  value: 0.01
+  decay:
+    type: poly
+    power: 0.9
+
+
+loss:
+  types:
+    - type: CrossEntropyLoss
+    - type: CrossEntropyLoss
+  coef: [1, 0.4]
diff --git a/configs/ocrnet/ocrnet_hrnetw18_road_extraction_768x768_15k_lovasz_hinge.yml b/configs/ocrnet/ocrnet_hrnetw18_road_extraction_768x768_15k_lovasz_hinge.yml
@@ -0,0 +1,55 @@
+
+
+batch_size: 4
+iters: 15000
+
+train_dataset:
+  type: MiniDeepGlobeRoadExtraction
+  dataset_root: data/MiniDeepGlobeRoadExtraction
+  transforms:
+    - type: ResizeStepScaling
+      min_scale_factor: 0.5
+      max_scale_factor: 2.0
+      scale_step_size: 0.25
+    - type: RandomPaddingCrop
+      crop_size: [768, 768]
+    - type: RandomHorizontalFlip
+    - type: Normalize
+  mode: train
+
+val_dataset:
+  type: MiniDeepGlobeRoadExtraction
+  dataset_root: data/MiniDeepGlobeRoadExtraction
+  transforms:
+    - type: Normalize
+  mode: val
+
+model:
+  type: OCRNet
+  backbone:
+    type: HRNet_W18
+    pretrained: https://bj.bcebos.com/paddleseg/dygraph/hrnet_w18_ssld.tar.gz
+  backbone_indices: [0]
+
+optimizer:
+  type: sgd
+
+learning_rate:
+  value: 0.01
+  decay:
+    type: poly
+    power: 0.9
+
+loss:
+  types:
+    - type: MixedLoss
+      losses:
+        - type: CrossEntropyLoss
+        - type: LovaszHingeLoss
+      coef: [1, 0.01]
+    - type: MixedLoss
+      losses:
+        - type: CrossEntropyLoss
+        - type: LovaszHingeLoss
+      coef: [1, 0.01]
+  coef: [1, 0.4]
diff --git a/docs/apis/README.md b/docs/apis/README.md
@@ -5,4 +5,5 @@
 * [paddleseg.datasets](./datasets.md)
 * [paddleseg.models](./models.md)
 * [paddleseg.models.backbones](./backbones.md)
+* [paddleseg.models.losses](./losses.md)
 * [paddleseg.transforms](./transforms.md)
diff --git a/docs/apis/losses.md b/docs/apis/losses.md
@@ -0,0 +1,33 @@
+# [paddleseg.models.losses](../../paddleseg/models/losses)
+
+## LovaszSoftmaxLoss
+> CLASS paddleseg.models.losses.LovaszSoftmaxLoss(ignore_index=255, classes='present')
+
+    Multi-class Lovasz-Softmax loss.
+
+> > Args
+> > > - **ignore_index** (int64): Specifies a target value that is ignored and does not contribute to the input gradient. Default ``255``.
+> > > - **classes** (str|list): 'all' for all, 'present' for classes present in labels, or a list of classes to average.
+
+
+## LovaszHingeLoss
+> CLASS paddleseg.models.losses.LovaszHingeLoss(ignore_index=255)
+
+    Binary Lovasz hinge loss.
+
+> > Args
+> > > - **ignore_index** (int64): Specifies a target value that is ignored and does not contribute to the input gradient. Default ``255``.
+
+
+## MixedLoss
+> CLASS paddleseg.models.losses.MixedLoss(losses, coef)
+
+    Weighted computations for multiple Loss.
+    The advantage is that mixed loss training can be achieved without changing the networking code.
+
+> > Args
+> > > - **losses** (list of nn.Layer): A list consisting of multiple loss classes
+> > > - **coef** (float|int): Weighting coefficient of multiple loss
+
+> > Returns
+> > > - A callable object of MixedLoss.
diff --git a/docs/images/Lovasz_Hinge_Evaluate_mIoU.png b/docs/images/Lovasz_Hinge_Evaluate_mIoU.png
diff --git a/docs/images/Lovasz_Softmax_Evaluate_mIoU.png b/docs/images/Lovasz_Softmax_Evaluate_mIoU.png
diff --git a/docs/images/deepglobe.png b/docs/images/deepglobe.png
diff --git a/docs/loss_usage.md b/docs/loss_usage.md
@@ -0,0 +1,4 @@
+# Loss usage
+
+- [Lovasz loss](lovasz_loss.md)
+- To be continued
diff --git a/docs/lovasz_loss.md b/docs/lovasz_loss.md
@@ -0,0 +1,125 @@
+# Lovasz loss
+对于图像分割任务中，经常出现类别分布不均匀的情况，例如：工业产品的瑕疵检测、道路提取及病变区域提取等。我们可使用lovasz loss解决这个问题。
+
+Lovasz loss基于子模损失(submodular losses)的凸Lovasz扩展，对神经网络的mean IoU损失进行优化。Lovasz loss根据分割目标的类别数量可分为两种：lovasz hinge loss和lovasz softmax loss. 其中lovasz hinge loss适用于二分类问题，lovasz softmax loss适用于多分类问题。该工作发表在CVPR 2018上，可点击[参考文献](#参考文献)查看具体原理。
+
+
+## Lovasz loss使用指南
+接下来介绍如何使用lovasz loss进行训练。需要注意的是，通常的直接训练方式并一定管用，我们推荐另外2种训练方式：
+- （1）与cross entropy loss或bce loss(binary cross-entropy loss)加权结合使用。
+- （2）先使用cross entropy loss或bce loss进行训练，再使用lovasz softmax loss或lovasz hinge loss进行finetuning.
+
+以方式（1）为例，通过`MixedLoss`类选择训练时的损失函数， 通过`coef`参数对不同loss进行权重配比，从而灵活地进行训练调参。如下所示：
+
+```yaml
+loss:
+  types:
+    - type: MixedLoss
+      losses:
+        - type: CrossEntropyLoss
+        - type: LovaszSoftmaxLoss
+      coef: [0.8, 0.2]
+```
+
+```yaml
+loss:
+  types:
+    - type: MixedLoss
+      losses:
+        - type: CrossEntropyLoss
+        - type: LovaszHingeLoss
+      coef: [1, 0.02]
+```
+
+
+## Lovasz softmax loss实验对比
+
+接下来以经典的[Cityscapes](https://www.cityscapes-dataset.com/)数据集为例应用lovasz softmax loss. Cityscapes数据集共有19类目标，其中的类别并不均衡，例如类别`road`、`building`很常见，`fence`、`motocycle`、`wall`则较为罕见。我们将lovasz softmax loss与softmax loss进行了实验对比。这里使用OCRNet模型，backbone为HRNet w18.
+
+
+* 数据准备
+
+见[数据集准备教程](data_prepare.md)
+
+* Lovasz loss训练
+```shell
+CUDA_VISIBLE_DEVICES=0,1,2,3 python -u -m paddle.distributed.launch train.py \
+--config configs/ocrnet/ocrnet_hrnetw18_cityscapes_1024x512_160k_lovasz_softmax.yml \
+--use_vdl  --num_workers 3 --do_eval
+```
+
+* Cross entropy loss训练
+```shell
+CUDA_VISIBLE_DEVICES=0,1,2,3 python -u -m paddle.distributed.launch train.py \
+--config configs/ocrnet/ocrnet_hrnetw18_cityscapes_1024x512_160k.yml \
+--use_vdl  --num_workers 3 --do_eval
+```
+
+* 结果比较
+
+实验mIoU曲线如下图所示。
+<p align="center">
+  <img src="./images/Lovasz_Softmax_Evaluate_mIoU.png" hspace='10' /> <br />
+ </p>
+
+
+|Loss|best mIoU|
+|-|-|
+|cross entropy loss|80.46%|
+|lovasz softmax loss + cross entropy loss|81.53%|
+
+图中蓝色曲线代表lovasz softmax loss + cross entropy loss，绿色曲线代表cross entropy loss，相比提升1个百分点。
+
+可看出使用lovasz softmax loss后，精度曲线基本都高于原来的精度。
+
+## Lovasz hinge loss实验对比
+
+我们以道路提取任务为例应用lovasz hinge loss.
+基于MiniDeepGlobeRoadExtraction数据集与cross entropy loss进行了实验对比。
+该数据集来源于[DeepGlobe CVPR2018挑战赛](http://deepglobe.org/)的Road Extraction单项，训练数据道路占比为 4.5%. 道路在整张图片中的比例很小，是典型的类别不均衡场景。图片样例如下：
+<p align="center">
+  <img src="./images/deepglobe.png" hspace='10'/> <br />
+ </p>
+
+这里使用OCRNet模型，backbone为HRNet w18.
+
+* 数据集
+我们从DeepGlobe比赛的Road Extraction的训练集中随机抽取了800张图片作为训练集，200张图片作为验证集，
+制作了一个小型的道路提取数据集[MiniDeepGlobeRoadExtraction](https://paddleseg.bj.bcebos.com/dataset/MiniDeepGlobeRoadExtraction.zip)。
+运行训练脚本将自动下载该数据集。
+
+* Lovasz loss训练
+```shell
+CUDA_VISIBLE_DEVICES=0,1,2,3 python -u -m paddle.distributed.launch train.py \
+--config configs/ocrnet/ocrnet_hrnetw18_road_extraction_768x768_15k_lovasz_hinge.yml \
+--use_vdl  --num_workers 3 --do_eval
+```
+
+* Cross entropy loss训练
+```shell
+CUDA_VISIBLE_DEVICES=0,1,2,3 python -u -m paddle.distributed.launch train.py \
+--config configs/ocrnet/ocrnet_hrnetw18_road_extraction_768x768_15k.yml \
+--use_vdl  --num_workers 3 --do_eval
+```
+
+* 结果比较
+
+实验mIoU曲线如下图所示。
+<p align="center">
+  <img src="./images/Lovasz_Hinge_Evaluate_mIoU.png" hspace='10'/> <br />
+ </p>
+
+
+|Loss|best mIoU|
+|-|-|
+|cross entropy loss|78.69%|
+|lovasz softmax loss + cross entropy loss|79.18%|
+
+图中紫色曲线为lovasz hinge loss + cross entropy loss，蓝色曲线为cross entropy loss，相比提升0.5个百分点。
+
+可看出使用lovasz hinge loss后，精度曲线全面高于原来的精度。
+
+
+
+## 参考文献
+[Berman M, Rannen Triki A, Blaschko M B. The lovász-softmax loss: a tractable surrogate for the optimization of the intersection-over-union measure in neural networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018: 4413-4421.](http://openaccess.thecvf.com/content_cvpr_2018/html/Berman_The_LovaSz-Softmax_Loss_CVPR_2018_paper.html)
diff --git a/paddleseg/cvlibs/config.py b/paddleseg/cvlibs/config.py
@@ -211,7 +211,9 @@ def loss(self) -> dict:
                 if key == 'types':
                     self._losses['types'] = []
                     for item in args['types']:
-                        item['ignore_index'] = self.train_dataset.ignore_index
+                        if item['type'] != 'MixedLoss':
+                            item['ignore_index'] = \
+                                self.train_dataset.ignore_index
                         self._losses['types'].append(self._load_object(item))
                 else:
                     self._losses[key] = val

diff --git a/paddleseg/datasets/__init__.py b/paddleseg/datasets/__init__.py
@@ -18,3 +18,4 @@
 from .ade import ADE20K
 from .optic_disc_seg import OpticDiscSeg
 from .pascal_context import PascalContext
+from .mini_deep_globe_road_extraction import MiniDeepGlobeRoadExtraction