[1.x] TensorRT: add INT8 with calibration

[Kh4L]

Description

This PR adds INT8 with calibration support to MXNet-TensorRT.
It enables TensorRT internal optimization to create an INT8 engine (that will contain some INT8 kernels, if they are faster than the FP16 or FP32 ones).
In this first version, the quantization and de-quantization values are computed during the calibration phase. During this phase (of a number of iterations set by the calibration_iters), the user is expect to provide samples representing the inference data, used to calibrate the engine. The inference model is slower during this phase.
Once the calibration is done, the MXNet-TensorRT inference model is ready for fast inference with INT8.

Saving and loading of the calibration tables will be added in a later PR.

Usage

We set calibration_iters to the number of batches we can feed with the calibration dataset.
For instance:

calibration_iters = 500

We call optimize_for:
Symbolic

trt_sym = sym.optimize_for('TensorRT', args=arg_params, aux=aux_params, ctx=mx.gpu(0),
                           int8_mode=True,
                           calibration_iters=calibration_iters)

Gluon

model.optimize_for(input_data, backend='TensorRT', static_alloc=True, static_shape=True,
                   backend_opts={'int8_mode':True, 'calibration_iters':calibration_iters})

We create the executor and we feed the calibration data:

# (len(calibration_dataset) / batch_size)  == calibration_iters
for d in calibration_dataset:
    executor.forward(is_train=False, data=data)

The calibration is slower than regular inference. Once it's done, we get a info message on stdout.
The executor with TRT INT8 engines is ready!

[mxnet-bot]

Hey @Kh4L , Thanks for submitting the PR
All tests are already queued to run once. If tests fail, you can trigger one or more tests again with the following commands:

• To trigger all jobs: @mxnet-bot run ci [all]
• To trigger specific jobs: @mxnet-bot run ci [job1, job2]

---

CI supported jobs: [website, centos-cpu, unix-gpu, miscellaneous, windows-gpu, centos-gpu, clang, unix-cpu, windows-cpu, edge, sanity]

---

Note:
Only following 3 categories can trigger CI :PR Author, MXNet Committer, Jenkins Admin.
All CI tests must pass before the PR can be merged.

[marcoabreu]

@leezu could you assist reviewing this PR? I think the structure wrt installation of packages and creating a new job might not be in line.

[szha]

this address returns 404 for me: https://developer.download.nvidia.com/compute/redist

[Kh4L]

https://developer.download.nvidia.com/compute/redist isn't supposed to be accessed by itself
The pip install line is the standard way of installing DALI:
https://docs.nvidia.com/deeplearning/dali/user-guide/docs/installation.html

[marcoabreu]

@KellenSunderland

[Kh4L]

@mxnet-bot run ci [sanity]

[mxnet-bot]

Jenkins CI successfully triggered : [sanity]

[Kh4L]

@mxnet-bot run ci [unix-gpu]

[mxnet-bot]

Jenkins CI successfully triggered : [unix-gpu]

[samskalicky]

Reviewed subgraph property portions, looks good to me. Please find someone to review the TRT-specific changes (maybe @ptrendx or @KellenSunderland) and the CI changes (maybe @josephevans?) too.

[szha]

in mxnet we switched to pytest and are no longer using nose.

[samskalicky]

I missed that discussion, can you point me to the RFC so I can catch up?

[Kh4L]

I forgot to add the [1.x] tag, this is 1.x PR, where we still use nosetests

[Kh4L]

@szha is this fine to leave it as nosetests as this is the 1.8 branch?

[ptrendx]

DALI has this script, why including it here too?

[ptrendx]

LGTM