What Is CUDA?

CUDA (Compute Unified Device Architecture) is NVIDIA’s platform and programming model for running general-purpose code on a GPU instead of only on the CPU. NVIDIA announced it in 2006 and released it in 2007, and it now runs on every NVIDIA GPU, from Jetson modules to data centre accelerators.

CUDA exposes the thousands of parallel cores in the GPU to ordinary software, which is what makes GPU acceleration possible. It is the layer that everything else in the NVIDIA stack runs on: deep learning libraries, inference engines, video and vision libraries, and your own GPU code. Without it, the GPU in an edge AI computer only draws the screen.

A CUDA program has two parts. Host code runs on the CPU, sets up data and launches work. Device code, written as functions called kernels, runs on the GPU. When a kernel launches, the GPU runs it across thousands of threads at once, each working on a different piece of the data, such as one pixel of an image or one element of a matrix.

Threads are grouped into blocks, and blocks into a grid. The GPU schedules blocks onto its streaming multiprocessors (SMs), each of which holds many CUDA cores and, on recent GPUs, Tensor Cores. On a discrete graphics card, data has to be copied between system memory and the card’s memory; on a Jetson module the CPU and GPU share one pool of memory, which removes most of those copies. Our comparison of GPUs and CPUs explains why this parallel design suits AI and video so well.

Most developers never write a kernel. They use the libraries that ship with, or build on, the CUDA Toolkit.

Two numbers decide whether code runs on a GPU. The CUDA version is the toolkit release, such as CUDA 12 or CUDA 13. The compute capability identifies the GPU architecture and the features it supports. Each toolkit supports a range of compute capabilities, and code compiled for one architecture may need recompiling for another.

Newer toolkits drop the oldest architectures, so a long-life product should fix its CUDA version alongside its driver and operating system. On Jetson, the CUDA version is set by the JetPack release flashed onto the module.

An edge AI computer pairs a CPU with an NVIDIA GPU, either integrated on a Jetson module or fitted as a card. CUDA is what lets the application offload heavy, parallel work, such as image pre-processing and neural network maths, onto that GPU, so a compact, low-power computer can run models a CPU of the same size could not.

Our NVIDIA Jetson computers run CUDA on the module’s integrated GPU at a few tens of watts, while industrial GPU computers carry discrete RTX cards for heavier models and more streams. Our guide to common terms in AI GPU computing covers the wider vocabulary.

CUDA only runs on NVIDIA GPUs. AMD’s equivalent is ROCm, with the HIP programming interface designed to make porting CUDA code straightforward. Intel offers oneAPI and SYCL for its GPUs and accelerators. OpenCL and Vulkan compute are open, cross-vendor standards from the Khronos Group, but they have far less AI library support. In practice most AI frameworks treat CUDA as the default GPU back end, which is one reason NVIDIA hardware dominates edge AI. Porting a mature CUDA application to another vendor is possible, but budget time to revalidate its performance and accuracy.

We supply Jetson and GPU computers with the NVIDIA driver, CUDA and the matching AI libraries installed and validated as standard, at the versions your application needs. Tell us your software stack, and we will build the system to match it.