What Is TensorRT?

TensorRT is NVIDIA’s library and runtime for high-performance inference, the job of running a trained AI model to make predictions. It compiles a model for the exact GPU it will run on, applying techniques such as layer fusion and reduced precision, so the model runs faster and uses less memory than it would in a general training framework.

TensorRT is usually the last step before a model is deployed on NVIDIA hardware. A model is trained in PyTorch or TensorFlow on a server, exported, converted into a TensorRT engine for the target device and then run by the TensorRT runtime inside the application. It runs on every current NVIDIA GPU, from Jetson modules to data centre cards, and ships as part of JetPack on Jetson.

TensorRT applies several optimizations when it builds an engine.

The usual path starts by exporting the trained model to ONNX. TensorRT’s ONNX parser reads the file, and the builder, either through the API or the trtexec command-line tool, produces an optimized engine, saved as a .engine or .plan file. The application loads that engine with the TensorRT runtime and feeds it data. PyTorch users can also use Torch-TensorRT, which compiles parts of a model from inside PyTorch.

An engine is tied to the GPU model and the TensorRT version it was built with. Building on an Orin NX and running on an AGX Orin, or upgrading JetPack, means rebuilding the engine. Most deployments build engines on the target the first time the application runs, or as part of the production image. Keep the ONNX file alongside the engine, so a new engine can be built after any software update.

Choosing the precision is the biggest decision when building an engine. Lower precision runs faster and uses less memory, but needs checking for accuracy loss.

INT8 needs a calibration set of real inputs so TensorRT can choose the right scaling for each layer. Our explainer on INT8 quantisation covers how that works and how much accuracy it typically costs.

On Jetson, TensorRT is installed with JetPack and is the standard way to deploy a model. It can also target the deep learning accelerators (DLAs) on the Orin NX and AGX Orin, running one model on a DLA while another runs on the GPU, which raises total throughput at low power. Layers the DLA cannot run fall back to the GPU automatically. On our Jetson AGX Orin computers, the two DLAs together add a useful share of the module’s rated TOPS for models that suit them, such as standard convolutional detectors.

NVIDIA’s DeepStream SDK for video analytics, and Isaac ROS for robotics, both run their models through TensorRT. Our edge AI computers use TensorRT wherever an NVIDIA GPU or Jetson module does the inference, and our guide to deploying rugged edge computers for AIoT shows where it fits in a wider system.

Several NVIDIA tools build on TensorRT. TensorRT-LLM optimizes large language models, with techniques such as paged key-value caching and in-flight batching. TensorRT Model Optimizer handles quantisation and sparsity before the engine is built. Triton Inference Server serves TensorRT engines, alongside other model formats, to many clients at once, which suits edge servers that run several models for a site and need to share one GPU between them.

We build TensorRT engines for the target module as part of taking a model to a deployable system, and validate their accuracy and frame rate before the system ships, as standard. Send us your model and target frame rate, and we will recommend the NVIDIA hardware to run it.