tiovxdlpreproc

TIOVX DL PreProc

tiovxdlpreproc is used to Preprocesses a video for conventional deep learning algorithms. This plugin uses OpenVX base classes.

Hierarchy

GObject
 ╰──GInitiallyUnowned
     ╰──GstObject
         ╰──GstElement
             ╰──GstBaseTransform
                 ╰──GstTIOVXSiso
                     ╰──GstTIOVXDLPreProc

Pad templates

sink

      video/x-raw
                 format: { (string)RGB, (string)NV12, (string)NV21 }
                  width: [ 1, 8192 ]
                 height: [ 1, 8192 ]
              framerate: [ 0/1, 2147483647/1 ]
      video/x-raw(memory:batched)
                 format: { (string)RGB, (string)NV12, (string)NV21 }
                  width: [ 1, 8192 ]
                 height: [ 1, 8192 ]
              framerate: [ 0/1, 2147483647/1 ]
           num-channels: [ 1, 16 ]

Availability – always

Direction – sink

Object type – GstPad

src

      application/x-tensor-tiovx
               num-dims: 3
              data-type: [ 2, 10 ]
          channel-order: { (string)NCHW, (string)NHWC }
          tensor-format: { (string)RGB, (string)BGR }
           tensor-width: [ 1, 8192 ]
          tensor-height: [ 1, 8192 ]
      application/x-tensor-tiovx(memory:batched)
               num-dims: 3
              data-type: [ 2, 10 ]
          channel-order: { (string)NCHW, (string)NHWC }
          tensor-format: { (string)RGB, (string)BGR }
           tensor-width: [ 1, 8192 ]
          tensor-height: [ 1, 8192 ]
           num-channels: [ 1, 16 ]

Availability – always

Direction – src

Object type – GstPad

Properties

channel-order

Channel order for the tensor dimensions

flags: readable, writable, controllable

Enum "GstTIOVXDLPreProcChannelOrder"

Default: 0, "nchw"

(0): nchw - NCHW channel order

(1): nhwc - NHWC channel order

data-type

Data Type of tensor at the output

flags: readable, writable, controllable

Enum "GstTIOVXDLPreProcDataType"

Default: 10, "float32"

(2): int8 - VX_TYPE_INT8

(3): uint8 - VX_TYPE_UINT8

(4): int16 - VX_TYPE_INT16

(5): uint16 - VX_TYPE_UINT16

(6): int32 - VX_TYPE_INT32

(7): uint32 - VX_TYPE_UINT32

(10): float32 - VX_TYPE_FLOAT32

in-pool-size

Number of buffers to allocate in input pool

flags: readable, writable

Unsigned Integer Range: 2 - 16

Default : 2

mean-0

Mean pixel to be subtracted for the first plane

flags: readable, writable

Float Range: 0 - 255

Default : 0

mean-1

Mean pixel to be subtracted for the second plane

flags: readable, writable

Float Range: 0 - 255

Default : 0

mean-2

Mean pixel to be subtracted for the third plane

flags: readable, writable

Float Range: 0 - 255

Default : 0

model

TIDL Model directory with params, model and artifacts

flags: readable, writable

String

Default : NULL

name

The name of the object

flags: readable, writable

String

Default : "tiovxdlpreproc0"

out-pool-size

Number of buffers to allocate in output pool

flags: readable, writable

Unsigned Integer Range: 2 - 16

Default : 2

parent

The parent of the object

flags: readable, writable

GstObject

Default : N/A

qos

Handle Quality-of-Service events

flags: readable, writable

Boolean

Default : false

scale-0

Scaling value for the first plane

flags: readable, writable

Float Range: 0 - 1

Default : 1

scale-1

Scaling value for the second plane

flags: readable, writable

Float Range: 0 - 1

Default : 1

scale-2

Scaling value for the third plane

flags: readable, writable

Float Range: 0 - 1

Default : 1

target

TIOVX target to use by this element

flags: readable, writable, controllable

Enum "GstTIOVXDLPreProcTarget"

Default: 5, "MPU-0"

(5): MPU-0 - A72 instance 1, assigned to A72_0 core

tensor-format

Tensor format at the output

flags: readable, writable, controllable

Enum "GstTIOVXDLPreProcTensorFormat"

Default: 0, "rgb"

(0): rgb - RGB plane format

(1): bgr - BGR plane format

Defining preprocessing properties

Various pre-processing properties like channel-order, tensor-format, tensor-datatype, mean, and scale can be either defined explicitly or through a model directory containing params.yaml file.

Pipeline examples

Input with explicitly defined preprocessing properties

gst-launch-1.0                                                                                                       \
videotestsrc num-buffers=1 is-live=true ! "video/x-raw,format=NV12,width=1280,height=720,framerate=(fraction)30/1" ! \
tiovxdlpreproc data-type=uint8 channel-order=nchw tensor-format=bgr ! "application/x-tensor-tiovx" !                 \
filesink location=dl-pre-proc-output_1280x720_u8bit_nchw.bgr -e

Input with model directory

MODEL_DIR=/opt/model_zoo/ONR-OD-8200-yolox-nano-lite-mmdet-coco-416x416

gst-launch-1.0                                                                                                       \
videotestsrc num-buffers=1 is-live=true ! "video/x-raw,format=NV12,width=416,height=612,framerate=(fraction)30/1" !  \
tiovxdlpreproc model=$MODEL_DIR ! "application/x-tensor-tiovx" !                                                     \
filesink location=dl-pre-proc-output_1280x720_u8bit_nchw.bgr -e

Batched processing

gst-launch-1.0                                                                                                      \
videotestsrc is-live=true num-buffers=10 pattern=0 ! "video/x-raw, width=640, height=480, format=NV12" ! mux.       \
videotestsrc is-live=true num-buffers=10 pattern=ball ! "video/x-raw, width=640, height=480, format=NV12" ! mux.    \
tiovxmux name=mux ! "video/x-raw(memory:batched), width=640, height=480, format=NV12" !                             \
tiovxdlpreproc data-type=int8 channel-order=nchw tensor-format=bgr ! "application/x-tensor-tiovx(memory:batched)" ! \
tiovxdemux name=demux                                                                                               \
demux. ! queue ! multifilesink location=/tmp/image_a_%02d.bgr                                                       \
demux. ! queue ! multifilesink location=/tmp/image_b_%02d.bgr