Skip to content

Latest commit

 

History

History
192 lines (137 loc) · 7.87 KB

mmaction2.md

File metadata and controls

192 lines (137 loc) · 7.87 KB
Raw

MMAction2 Deployment

MMAction2 is an open-source toolbox for video understanding based on PyTorch. It is a part of the OpenMMLab project.

Installation

Install mmaction2

Please follow the installation guide to install mmaction2.

Install mmdeploy

There are several methods to install mmdeploy, among which you can choose an appropriate one according to your target platform and device.

Method I： Install precompiled package

You can refer to get_started

Method II： Build using scripts

If your target platform is Ubuntu 18.04 or later version, we encourage you to run scripts. For example, the following commands install mmdeploy as well as inference engine - ONNX Runtime.

git clone --recursive -b main https://github.com/open-mmlab/mmdeploy.git
cd mmdeploy
python3 tools/scripts/build_ubuntu_x64_ort.py $(nproc)
export PYTHONPATH=$(pwd)/build/lib:$PYTHONPATH
export LD_LIBRARY_PATH=$(pwd)/../mmdeploy-dep/onnxruntime-linux-x64-1.8.1/lib/:$LD_LIBRARY_PATH

Method III: Build from source

If neither I nor II meets your requirements, building mmdeploy from source is the last option.

Convert model

You can use tools/deploy.py to convert mmaction2 models to the specified backend models. Its detailed usage can be learned from here.

When using tools/deploy.py, it is crucial to specify the correct deployment config. We've already provided builtin deployment config files of all supported backends for mmaction2, under which the config file path follows the pattern:

{task}/{task}_{backend}-{precision}_{static | dynamic}_{shape}.py

其中：

  • {task}: task in mmaction2.
  • {backend}: inference backend, such as onnxruntime, tensorrt, pplnn, ncnn, openvino, coreml etc.
  • {precision}: fp16, int8. When it's empty, it means fp32
  • {static | dynamic}: static shape or dynamic shape
  • {shape}: input shape or shape range of a model
  • {2d/3d}: model type

In the next part，we will take tsn model from video recognition task as an example, showing how to convert them to onnx model that can be inferred by ONNX Runtime.

Convert video recognition model

cd mmdeploy

# download tsn model from mmaction2 model zoo
mim download mmaction2 --config tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb --dest .

# convert mmaction2 model to onnxruntime model with dynamic shape
python tools/deploy.py \
    configs/mmaction/video-recognition/video-recognition_2d_onnxruntime_static.py \
    tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb \
    tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb_20220906-cd10898e.pth \
    tests/data/arm_wrestling.mp4 \
    --work-dir mmdeploy_models/mmaction/tsn/ort \
    --device cpu \
    --show \
    --dump-info

Model specification

Before moving on to model inference chapter, let's know more about the converted model structure which is very important for model inference.

The converted model locates in the working directory like mmdeploy_models/mmaction/tsn/ort in the previous example. It includes:

mmdeploy_models/mmaction/tsn/ort
├── deploy.json
├── detail.json
├── end2end.onnx
└── pipeline.json

in which,

  • end2end.onnx: backend model which can be inferred by ONNX Runtime
  • *.json: the necessary information for mmdeploy SDK

The whole package mmdeploy_models/mmaction/tsn/ort is defined as mmdeploy SDK model, i.e., mmdeploy SDK model includes both backend model and inference meta information.

Model Inference

Backend model inference

Take the previous converted end2end.onnx mode of tsn as an example, you can use the following code to inference the model and visualize the results.

from mmdeploy.apis.utils import build_task_processor
from mmdeploy.utils import get_input_shape, load_config
import numpy as np
import torch

deploy_cfg = 'configs/mmaction/video-recognition/video-recognition_2d_onnxruntime_static.py'
model_cfg = 'tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb'
device = 'cpu'
backend_model = ['./mmdeploy_models/mmaction2/tsn/ort/end2end.onnx']
image = 'tests/data/arm_wrestling.mp4'

# read deploy_cfg and model_cfg
deploy_cfg, model_cfg = load_config(deploy_cfg, model_cfg)

# build task and backend model
task_processor = build_task_processor(model_cfg, deploy_cfg, device)
model = task_processor.build_backend_model(backend_model)

# process input image
input_shape = get_input_shape(deploy_cfg)
model_inputs, _ = task_processor.create_input(image, input_shape)

# do model inference
with torch.no_grad():
    result = model.test_step(model_inputs)

# show top5-results
pred_scores = result[0].pred_scores.item.tolist()
top_index = np.argsort(pred_scores)[::-1]
for i in range(5):
    index = top_index[i]
    print(index, pred_scores[index])

SDK model inference

Given the above SDK model of tsn you can also perform SDK model inference like following,

Video recognition SDK model inference

from mmdeploy_runtime import VideoRecognizer
import cv2

# refer to demo/python/video_recognition.py
# def SampleFrames(cap, clip_len, frame_interval, num_clips):
#  ...

cap = cv2.VideoCapture('tests/data/arm_wrestling.mp4')

clips, info = SampleFrames(cap, 1, 1, 25)

# create a recognizer
recognizer = VideoRecognizer(model_path='./mmdeploy_models/mmaction/tsn/ort', device_name='cpu', device_id=0)
# perform inference
result = recognizer(clips, info)
# show inference result
for label_id, score in result:
    print(label_id, score)

Besides python API, mmdeploy SDK also provides other FFI (Foreign Function Interface), such as C, C++, C#, Java and so on. You can learn their usage from demos.

MMAction2 only API of c, c++ and python for now.

Supported models

Model TorchScript ONNX Runtime TensorRT ncnn PPLNN OpenVINO
TSN Y Y Y N N N
SlowFast Y Y Y N N N
TSM Y Y Y N N N
X3D Y Y Y N N N