UK What NVIDIA VPI does
NVIDIA VPI (Vision Programming Interface) is a computer-vision and image-processing library. It provides accelerated implementations of common vision operations, such as filtering, rescaling, stereo disparity and feature tracking, and can run them on whichever processor is best suited: the GPU, the CPU, or dedicated vision accelerators where present.
NVIDIA introduced VPI in 2020 as part of JetPack, the Jetson software stack, and it is now installed with JetPack on every current Jetson module. It also runs on x86 Linux computers with an NVIDIA GPU, using the CPU and CUDA back-ends, so a pipeline can be developed on a workstation and moved to a Jetson.
The key idea is the back-end. A developer writes the algorithm call once and chooses where it runs with a flag. Changing a filter from the GPU to a vision accelerator is a one-word change, not a rewrite.
Why it matters for edge AI
AI vision is not only the neural network; there is usually a pipeline of conventional image processing around it: undistorting the lens, resizing, colour conversion and tracking objects between frames. On a Jetson that pipeline can take as much time as the inference itself. VPI accelerates it and spreads it across the available processors, freeing the GPU to concentrate on inference. For a camera-heavy edge AI system it is a useful part of the stack.
-
//
Vision algorithms. Filtering, rescaling, lens correction, stereo, optical flow, feature detection and tracking, and more.
-
//
Multiple back-ends. The same call can run on GPU, CPU or a vision accelerator, chosen by the developer.
-
//
Frees the GPU. Offloading pre-processing leaves more GPU time for inference.
-
//
Zero-copy interop. VPI images can wrap OpenCV, NumPy and camera buffers, so data is not copied between libraries.
VPI back-ends and the hardware they use
Each back-end maps to a different block of silicon. Which ones are present depends on the Jetson module.
| Back-end | Hardware | Where available | Typical use |
|---|---|---|---|
| CPU | Arm or x86 CPU cores | All platforms | Fallback, testing, light work |
| CUDA | NVIDIA GPU | All Jetson modules and NVIDIA GPUs | Most algorithms, heavy filtering |
| PVA | Programmable Vision Accelerator | Jetson AGX Orin and Orin NX, not Orin Nano | Low-power tracking, corner detection, filtering |
| VIC | Video Image Compositor | Jetson modules | Rescaling, format conversion, lens remap |
| OFA | Optical Flow Accelerator | Jetson Orin modules | Dense optical flow, stereo disparity |
Availability varies by algorithm as well as by module, so NVIDIA’s VPI documentation lists which back-ends each algorithm supports. VPI is built on CUDA for its GPU work, and our guide to the NVIDIA JetPack software stack shows where it sits beside the other libraries.
Common VPI algorithms
VPI covers the steps that most camera pipelines share. The main groups are below; each lists the back-ends it supports in NVIDIA’s documentation.
-
//
Image conversion and scaling. Colour format conversion, rescaling and image pyramids, so camera frames match the input the AI model expects.
-
//
Geometric correction. Remap, perspective warp and lens distortion correction for wide-angle and fisheye cameras.
-
//
Filtering. Box, Gaussian and bilateral filters, convolution, histogram equalisation and background subtraction.
-
//
Features and tracking. Harris corners, ORB features and the KLT tracker, used to follow points or objects from frame to frame.
-
//
Motion and depth. Pyramidal and dense optical flow, plus stereo disparity estimation for depth from two cameras.
-
//
Frequency domain. Fast Fourier transforms for filtering and image analysis.
How a VPI pipeline works
VPI organises work into streams. The application creates a stream, submits a sequence of algorithms to it, each with its chosen back-end, and VPI runs them in order without the CPU waiting on each step. Several streams can run at once, for example one per camera, and events synchronise them where one result feeds another.
A typical machine vision pipeline on a Jetson might use VIC to convert and resize each camera frame, PVA to track features between frames, and CUDA for a denoising filter, while the GPU’s Tensor Cores and TensorRT run the detection model. The work is spread across four blocks of silicon instead of queuing on one.
VPI vs OpenCV
OpenCV is the general-purpose computer-vision library most developers already know, with a far larger set of algorithms. VPI is narrower but built for NVIDIA hardware, and it can use the PVA, VIC and OFA blocks that OpenCV cannot reach. The two work together: VPI can wrap OpenCV images directly, so a team can keep OpenCV for most of the application and move the slow steps to VPI. For a stereo camera, for example, VPI’s disparity estimator on the OFA frees the GPU that OpenCV’s version would occupy.
Quick answers on NVIDIA VPI
NVIDIA VPI: common questions
Is VPI free?
Yes. VPI is included with JetPack at no cost and is also available for x86 Linux systems with NVIDIA GPUs.
Does VPI run on the Jetson Orin Nano?
Yes, with fewer back-ends. The Orin Nano has no PVA, so VPI uses its CPU, GPU, VIC and optical flow hardware instead.
Can VPI be used from Python?
Yes. VPI has C and C++ APIs and Python bindings that work with NumPy arrays, which makes it quick to prototype a pipeline.
Is VPI the same as DeepStream?
No. DeepStream is NVIDIA’s framework for complete video analytics pipelines, from decode to inference to output. VPI is a lower-level library of vision algorithms that can be used inside such a pipeline or on its own.
Does VPI run neural networks?
No. VPI covers classical computer vision and image processing. Neural networks run through TensorRT or another inference runtime.
VPI is one of the libraries we draw on when building a camera or vision-heavy edge AI platform, alongside the deep-learning stack. See our guide to common terms in AI GPU computing. Our Jetson AGX Orin computers carry every VPI back-end, including the PVA. We supply Jetson systems with JetPack, VPI and the camera drivers installed and validated as standard.