NVIDIA Triton Inference Server

LATEST RELEASE: You are currently on the master branch which tracks under-development progress towards the next release. The latest release of the Triton Inference Server is 1.13.0 and is available on branch r20.03.1.

Triton V2: Starting with the 20.06 release, Triton moves to version 2. The master branch currently tracks V2 development and is likely to be more unstable than usual due to the significant changes during the transition from V1 to V2. A legacy V1 version of Triton will be released from the master-v1 branch. The V1 version of Triton is deprecated and no releases beyond 20.06 are planned. More information on the V1 and V2 transition is available in Roadmap.

NVIDIA Triton Inference Server provides a cloud inferencing solution optimized for NVIDIA GPUs. The server provides an inference service via an HTTP/REST or GRPC endpoint, allowing remote clients to request inferencing for any model being managed by the server. For edge deployments, Triton Server is also available as a shared library with an API that allows the full functionality of the server to be included directly in an application. Triton Server provides the following features:

• Multiple framework support. The server can manage any number and mix of models (limited by system disk and memory resources). Supports TensorRT, TensorFlow GraphDef, TensorFlow SavedModel, ONNX, PyTorch, and Caffe2 NetDef model formats. Also supports TensorFlow-TensorRT and ONNX-TensorRT integrated models. Variable-size input and output tensors are allowed if supported by the framework. See Capabilities for detailed support information for each framework.
• Concurrent model execution support. Multiple models (or multiple instances of the same model) can run simultaneously on the same GPU.
• Batching support. For models that support batching, Triton Server can accept requests for a batch of inputs and respond with the corresponding batch of outputs. Triton Server also supports multiple scheduling and batching algorithms that combine individual inference requests together to improve inference throughput. These scheduling and batching decisions are transparent to the client requesting inference.
• Custom backend support. Triton Server allows individual models to be implemented with custom backends instead of by a deep-learning framework. With a custom backend a model can implement any logic desired, while still benefiting from the GPU support, concurrent execution, dynamic batching and other features provided by the server.
• Ensemble support. An ensemble represents a pipeline of one or more models and the connection of input and output tensors between those models. A single inference request to an ensemble will trigger the execution of the entire pipeline.
• Multi-GPU support. Triton Server can distribute inferencing across all system GPUs.
• Triton Server provides multiple modes for model management. These model management modes allow for both implicit and explicit loading and unloading of models without requiring a server restart.
• Model repositories may reside on a locally accessible file system (e.g. NFS), in Google Cloud Storage or in Amazon S3.
• HTTP/REST and GRPC inference protocols based on the community developed KFServing protocol.
• Readiness and liveness health endpoints suitable for any orchestration or deployment framework, such as Kubernetes.
• Metrics indicating GPU utilization, server throughput, and server latency.
• C library inferface allows the full functionality of Triton Server to be included directly in an application.

The current release of the Triton Inference Server is 1.13.0 and corresponds to the 20.03.1 release of the tensorrtserver container on NVIDIA GPU Cloud (NGC). The branch for this release is r20.03.1.

Backwards Compatibility

Version 2 of Triton does not generally maintain backwards compatibility with version 1. This section will be updated with specifics as part of the first release of version 2.

Roadmap

The 20.03.1 release of Triton consists of a single server/container that supports both the existing version 1 APIs and protocols and the new version 2 APIs and protocols. For version 2 the release is beta quality and includes the new HTTP/REST and GRPC protocols and corresponding new C++ and Python client libraries. Version 2 also includes a beta release of the new server C API defined in tritionserver.h.

The upcoming 20.06 release of Triton will include two separate server containers:

• A legacy V1 version of Triton will be released from the master-v1 branch. The NGC container for the V1 version of Triton will be called tritonserver:20.06-v1-py3. The V1 version of Triton is deprecated and no releases beyond 20.06 are planned. The V1 version of Triton maintains backwards compatibility with prior V1 versions in both the server APIs and in the C++ and Python libraries. See the master-v1 branch README for more information.
• The new V2 version of Triton will be released from the master branch and will include the new GRPC and HTTP protocols based on inference protocols that have been proposed by the KFServing project. Version 2 of Triton will also have a new C API and new C++ and Python client libraries. The NGC container for version 2 of Triton will be called tritonserver:20.06-py3.

For both V1 and V2 the model repository struture and custom backend APIs will remain unchanged so that any existing model repository and custom backends will continue to work with Triton Server.

In the 20.06 V2 release there will be some changes to the tritonserver command-line executable arguments to remove deprecated arguments and adjust defaults. The changes will be detailed as part of the 20.06 release. It will be necessary to revisit and possibly adjust invocations of tritonserver executable.

Documentation

The User Guide, Developer Guide, and API Reference documentation for the current release provide guidance on installing, building, and running Triton Inference Server.

You can also view the documentation for the master branch and for earlier releases.

An FAQ provides answers for frequently asked questions.

READMEs for deployment examples can be found in subdirectories of deploy/, for example, deploy/single_server/README.rst.

The Release Notes and Support Matrix indicate the required versions of the NVIDIA Driver and CUDA, and also describe which GPUs are supported by Triton Server.

Presentations and Papers

• High-Performance Inferencing at Scale Using the TensorRT Inference Server.
• Accelerate and Autoscale Deep Learning Inference on GPUs with KFServing.
• Deep into Triton Inference Server: BERT Practical Deployment on NVIDIA GPU.
• Maximizing Utilization for Data Center Inference with TensorRT Inference Server.
• NVIDIA TensorRT Inference Server Boosts Deep Learning Inference.
• GPU-Accelerated Inference for Kubernetes with the NVIDIA TensorRT Inference Server and Kubeflow.

Contributing

Contributions to Triton Inference Server are more than welcome. To contribute make a pull request and follow the guidelines outlined in the Contributing document.

Reporting problems, asking questions

We appreciate any feedback, questions or bug reporting regarding this project. When help with code is needed, follow the process outlined in the Stack Overflow (https://stackoverflow.com/help/mcve) document. Ensure posted examples are:

• minimal – use as little code as possible that still produces the same problem
• complete – provide all parts needed to reproduce the problem. Check if you can strip external dependency and still show the problem. The less time we spend on reproducing problems the more time we have to fix it
• verifiable – test the code you're about to provide to make sure it reproduces the problem. Remove all other problems that are not related to your request/question.