segmentation

Bonnetal Semantic Segmentation

This readme contains simple usage explanations of how the training, evaluation, test inference, and export to ONNX/TensorRT are performed.

>>> shrek = network.infer(wolf)

---

Train a Network

This part of the framework provides a set of apps to train semantic segmentation CNN. It uses a backbone from our library of backbones (or your own), attaches a decoder from our selection of decoders (or your own) and puts a pixel-wise classification layer in the end. Training is performed by a weighted cross entropy loss of the softmax-ed pixel logits.

The training parameters are selected through a cfg.yaml file containing

• train: Dictionary with training parameters, such as learning rate, momentum, batch size, warmup phase, learn rate decay, weight decay, model averaging, etc.
• backbone: Dictionary with backbone selector and configurator, such as dropout in convolutional layers, batch normalization decay, output stride, etc. This part is common to all tasks using the common backbones.
• decoder: Dictionary with decoder selector and configurator, such as dropout in convolutional layers, batch normalization decay, number of layers, skip connections, etc.
• head: Segmentation head config, such as dropout before the linear layer.
• dataset: Name and location of the dataset, sizes of images, means and standard deviations for input normalization, class definition, and class weights (if imbalanced).

To train a network

• For help:

  $ ./train.py --help

• From scratch (if the weights exist and the images are RGB, the backbones download imagenet weights from the internet, so it is not really from scratch):

  $ ./train.py -c /path/to/cfg.yaml -l /path/to/put/log/

• From a pretrained model (can be a different task, doesn't have to be segmentation):

  $ ./train.py -c /path/to/cfg.yaml -p /path/to/pretrained/ -l /path/to/put/log/

• From scratch for real (no imagenet backbone, you need to fake it):

  $ ./train.py -c /path/to/cfg.yaml -p /dev/null -l /path/to/put/log/

To evaluate a network

• It uses the same script as the train but with an extra flag:

  $ ./train.py -p /path/to/pretrained/ --eval

---

Test Inference

These are some scripts to infer images and videos (and generate nice paper images) with the trained CNNs. These scripts also work for checking that the C++ deployment will work with the "deploy ready" directories.

Infer images

• Help:

  $ ./infer_img.py --help

• General options

  $ ./infer_img.py --verbose # will print images on screen
  $ ./infer_img.py -l /path/to/log # will save the masks to /path/to/log

• With native pytorch implementation:

  $ ./infer_img.py -p /path/to/pretrained/ -i /path/to/images # can be multiple images
  $ ./infer_img.py -p /path/to/pretrained/ -i /path/to/images -b native # native is default

• With a different backend (we have implemented picking up a traced pytorch model, an ONNX with Caffe2, and TensorRT). This is here to try if the homologous backbone will work during C++ inference. The make_deploy_model.py script traces the CNN and gives us binaries to pick up with different backbones. Furthermore, when using the infer_img.py file, the TensorRT engine that was created from the ONNX model is saved in the inference folder, because it takes a while to create, and you may want to use it later. If you already ran it, and want to change computers, DELETE model.trt:

  # get ready for inference
  $ ./make_deploy_model -p /path/to/pretrained/ -l /path/to/save/inference/ready
  # infer from inference model
  $ ./infer_img.py -p /path/to/save/inference/ready -i /path/to/images -b pytorch # or
  $ ./infer_img.py -p /path/to/save/inference/ready -i /path/to/images -b caffe2 # or
  $ ./infer_img.py -p /path/to/save/inference/ready -i /path/to/images -b tensorrt

Infer video/webcam

• Help:

  $ ./infer_video.py --help

• General options

  $ ./infer_video.py --verbose # will print images on screen
  $ ./infer_video.py -l /path/to/log # will save the masks to /path/to/log

• With native pytorch implementation:

  $ ./infer_video.py -p /path/to/pretrained/ -v /path/to/video
  $ ./infer_video.py -p /path/to/pretrained/ -v /path/to/video -b native # same thing

• Infer Webcam instead of video:

    $ ./infer_video.py -p /path/to/pretrained/ # simply drop the video and it will look for /dev/video0

• With a different backend (we have implemented picking up a traced pytorch model, an ONNX with Caffe2, and TensorRT). This is here to try if the homologous backbone will work during C++ inference. The make_deploy_model.py script traces the CNN and gives us binaries to pick up with different backbones. Furthermore, when using the infer_img.py file, the TensorRT engine that was created from the ONNX model is saved in the inference folder, because it takes a while to create, and you may want to use it later. If you already ran it, and want to change computers, DELETE model.trt:

  # get ready for inference
  $ ./make_deploy_model -p /path/to/pretrained/ -l /path/to/save/inference/ready
  # infer from inference model
  $ ./infer_video.py -p /path/to/save/inference/ready -v /path/to/video -b pytorch # or
  $ ./infer_video.py -p /path/to/save/inference/ready -v /path/to/video -b caffe2 # or
  $ ./infer_video.py -p /path/to/save/inference/ready -v /path/to/video -b tensorrt

Make inference model

• Get ready for C++ deployment:

  $ ./make_deploy_model -p /path/to/pretrained/ -l /path/to/save/inference/ready

  This generates a Pytorch model and an ONNX model. The TensorRT one gets created and profiled for efficiency from the ONNX during the first run, either here or in the C++ interface. Once built, the interfaces save it as an engine that get's deserialized the next execution, so if you change computers, delete this file, as it will not be the proper format/optimal.

• Change the size of the image for inference, if you want to infer on difference size than what the network was trained on (if scale of objects didn't change, it shouldn't be a problem for accuracy). Because this script saves the modified config file with the new size, the config dictionaries will be scrambled:

  $ ./make_deploy_model -p /path/to/pretrained/ -l /path/to/save/inference/ready --new_w XXX --new_h YYY