CUDA Cores vs Tensor Cores: What Is the Difference?

CUDA cores are the general-purpose arithmetic cores in an NVIDIA GPU, and Tensor Cores are specialised matrix-maths blocks that sit alongside them. Both live inside the GPU’s streaming multiprocessors (SMs). On recent Ampere and Ada Lovelace designs each SM holds 128 CUDA cores and four Tensor Cores, so a GPU always has far fewer Tensor Cores than CUDA cores.

Neural networks are built on matrix multiplication. CUDA cores can do that maths, but one multiply-add per core per clock. A Tensor Core multiplies and adds whole small matrices per clock, at reduced precision, so it gets far more AI work done per cycle and per watt. The two types work together rather than competing.

A CUDA core executes one floating-point or integer instruction at a time, much like a very simple CPU core. Thousands of them run the same instruction across different data in parallel, which is what makes a GPU fast at graphics, video processing, image pre-processing and general scientific code. Anything written for the GPU through the CUDA platform runs on these cores unless a library routes it to Tensor Cores.

In an AI pipeline, CUDA cores typically handle the work around the model: decoding and resizing frames, colour conversion, non-maximum suppression after a detector, and any layers that Tensor Cores cannot run.

A Tensor Core performs a fused matrix multiply and accumulate: it multiplies two small matrices and adds the result to a third in one operation. NVIDIA introduced them with the Volta architecture in 2017, and each generation since has added faster throughput and lower-precision formats. Lower precision means more operations per clock, which is why AI performance figures are usually quoted at INT8 or FP8.

Data centre parts sometimes gain a format a generation early, as Hopper did with FP8 in 2022, so check the datasheet for the exact card. INT8 quantisation is the usual way to put a trained model onto the faster, lower-precision paths.

You rarely choose between the two by hand. NVIDIA’s libraries, such as cuBLAS for matrix maths, cuDNN for neural network layers and TensorRT for inference, send work to Tensor Cores automatically when the data type and matrix shapes allow it, and to CUDA cores when they do not. On Ampere and later GPUs, matrix maths that would run at FP32 can be routed through Tensor Cores in TF32 format with no code change.

For training, frameworks such as PyTorch offer automatic mixed precision, which runs most layers at FP16 or BF16 on Tensor Cores and keeps sensitive steps at FP32. For inference, building the model at FP16, INT8 or FP8 is what moves the bulk of the work onto Tensor Cores, so the precision you deploy at matters as much as the hardware you buy.

Core counts only make sense when you know which work each type does. Our RTX 2000 Ada card pairs 2,816 CUDA cores with 88 fourth-generation Tensor Cores, while a Jetson Orin Nano 8GB module has 1,024 CUDA cores and 32 third-generation Tensor Cores. You cannot compare counts across generations directly, because each newer Tensor Core does more work per clock and supports lower precisions.

At the edge, the choice is usually between an integrated GPU on a Jetson module and a discrete card. Compact workstation GPUs such as the RTX 2000 Ada fit low-profile slots in industrial PCs, MXM GPU modules bring the same Tensor Core generations to rugged and conduction-cooled systems, and Jetson Orin computers combine CUDA and Tensor Cores with the CPU on one low-power module. Our guide to common terms in AI GPU computing explains the wider vocabulary.

We fit and validate NVIDIA GPUs and Jetson modules in industrial and edge AI computers as standard, with drivers, CUDA and TensorRT installed. Tell us about your model and frame rate, and we will recommend the GPU that matches.