[Feature]: Support deployment of panoptic segmentation models

[RunningLeon]

Motivation

Support the deployment of panoptic segmentation models like PanopticFPN, MaskFormer, Mask2Former on ONNX Runtime backend and TensorRT backends.

Modification

Supported models

Model	Task	OnnxRuntime	TensorRT	ncnn	PPLNN	OpenVINO
Panoptic FPN	Panoptic Segmentation	Y	Y	N	N	N
MaskFormer	Panoptic Segmentation	Y	Y	N	N	N
Mask2Former*	Panoptic Segmentation	Y	Y	N	N	N

Reminder

• Mask2Former should use TensorRT>=8.6.1 for dynamic shape inference.
• Should use mmdet after this PR: Support torch2onnx for maskformer series mmdetection#10782

BC-breaking (Optional)

None

Use cases (Optional)

If this PR introduces a new feature, it is better to list some use cases here, and update the documentation.

Checklist

1. Pre-commit or other linting tools are used to fix the potential lint issues.
2. The modification is covered by complete unit tests. If not, please add more unit tests to ensure the correctness.
3. If the modification has a dependency on downstream projects of a newer version, this PR should be tested with all supported versions of downstream projects.
4. The documentation has been modified accordingly, like docstring or example tutorials.

[CLAassistant]

All committers have signed the CLA.

[RunningLeon]

@daigo0927 hi, your commits fomr this PR #649 will be included in this PR, could you kindly sign the CLA? Thanks.

[daigo0927]

@daigo0927 hi, your commits fomr this PR #649 will be included in this PR, could you kindly sign the CLA? Thanks.

Thank you for the notification and I have signed the CLA. I'm really happy to hear the update 🎉

[AllentDan]

Got two errors:

1. batch_dets, batch_labels, batch_masks = outputs[:3]
   ValueError: not enough values to unpack (expected 3, got 2)
2. [TRT] [E] 1: [defaultAllocator.cpp::deallocate::42] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
   panoptic_fpn with TensorRT 8.4.1.5

[AllentDan]

I used the following commands:

python tools/deploy.py configs/mmdet/panoptic-seg/panoptic-seg_maskformer_tensorrt_dynamic-320x512-1344x1344.py ../mmdetection/configs/panoptic_fpn/panoptic-fpn_r50_fpn_1x_coco.py https://download.openmmlab.com/mmdetection/v2.0/panoptic_fpn/panoptic_fpn_r50_fpn_1x_coco/panoptic_fpn_r50_fpn_1x_coco_20210821_101153-9668fd13.pth ../mmdetection/demo/demo.jpg --work-dir work_dirs/tensorrt/det/panoptic_fpn --device cuda --show --dump-info

[RunningLeon]

closes #298 #1303 #1880

[RunningLeon]

I used the following commands:

python tools/deploy.py configs/mmdet/panoptic-seg/panoptic-seg_maskformer_tensorrt_dynamic-320x512-1344x1344.py ../mmdetection/configs/panoptic_fpn/panoptic-fpn_r50_fpn_1x_coco.py https://download.openmmlab.com/mmdetection/v2.0/panoptic_fpn/panoptic_fpn_r50_fpn_1x_coco/panoptic_fpn_r50_fpn_1x_coco_20210821_101153-9668fd13.pth ../mmdetection/demo/demo.jpg --work-dir work_dirs/tensorrt/det/panoptic_fpn --device cuda --show --dump-info

for panoptic-fpn model, should use configs/mmdet/panoptic-seg/panoptic-seg_panoptic-fpn_xxx.py

[AllentDan]

LGTM

[taifyang]

closes #298 #1303 #1880

Will mmdeploy SDK support MaskFormer, Mask2Former panoptic-seg for onnx and trt? I convert pth to onnx successfully but sdk inference failed. @RunningLeon

[RunningLeon]

closes #298 #1303 #1880

Will mmdeploy SDK support MaskFormer, Mask2Former panoptic-seg for onnx and trt? I convert pth to onnx successfully but sdk inference failed. @RunningLeon

@taifyang hi, sorry for the late reply. SDK has not supported for mask2former series. I'm afraid you have to implement it by yourself. Thanks for your understanding.