What Is GPU Computing? Key Terms in AI GPU Computing Explained

GPU computing is the use of a graphics processing unit’s thousands of parallel cores for general-purpose work, such as training and running AI models, simulation, signal processing and video analytics, rather than only for drawing graphics. It is also called GPGPU, general-purpose computing on graphics processing units, and it is programmed through frameworks such as CUDA, OpenCL and ROCm.

GPU computing is what made modern AI practical. Neural networks are built from huge numbers of identical multiply-and-add operations, exactly the kind of work a GPU spreads across its cores. When the AlexNet image classifier was trained on two NVIDIA graphics cards in 2012 and beat every previous approach, deep learning moved onto GPUs and has stayed there. Today the same principle runs in data centres and in edge AI computers such as NVIDIA Jetson systems on factory floors, vehicles and robots.

This guide explains how GPU computing works and how it compares with a CPU, then acts as a term map for the language around it: CUDA and Tensor Cores, TOPS and TFLOPS, precision, machine learning, deep learning, training, inference and sensor fusion. Each term links to a short explainer, so this page gives the overview and the FAQs give the detail. It finishes with GPU options for edge and industrial systems.

A CPU is built to run a few threads very quickly, with large caches and sophisticated control logic. A GPU is built to run thousands of threads at once, each simpler and slower, and to hide memory delays by switching between them. The table compares typical embedded and edge parts; figures are examples from published specifications.

The two are complementary. In a vision system the CPU handles the cameras’ network traffic, the application and the database, and the GPU decodes video and runs the model. Our explainers on GPU vs CPU and integrated vs discrete GPUs go further, and our comparison of NPUs and GPUs for edge AI covers the newer low-power alternative.

A GPU program is split into two parts. The host code runs on the CPU, prepares data and launches work. The kernels run on the GPU: small functions that the GPU executes across thousands of threads, each working on a different piece of the data, such as one pixel, one matrix element or one neuron. Threads are scheduled in groups that execute the same instruction together, which is why GPUs excel at uniform work and slow down when threads take different branches.

Data has to reach the GPU before it can be processed. A discrete GPU has its own memory, so data is copied across the PCIe bus, and that copy can take longer than the computation if the work is small. Jetson modules and integrated GPUs share memory with the CPU, which avoids the copy but shares the bandwidth. Our explainer on PCIe covers the link a discrete GPU sits on.

Few teams write kernels by hand for AI. Frameworks such as PyTorch and TensorFlow call GPU libraries for each layer, and inference engines such as TensorRT optimise a trained model for a specific GPU. The programming model still matters when choosing hardware, because software written for one framework does not always run on another.

These are the hardware and software terms that appear on GPU data sheets and in AI project discussions. Each links to a short explainer.

Artificial intelligence has been discussed since the 1950s, with highs and lows in popularity. A useful distinction is between strong AI, which could do anything a person can and exists only in theory, and weak or narrow AI, which is already used in many applications, where it can equal or outperform people in specific tasks.

Machine learning: pattern recognition

Machine learning is a branch of AI in which a system learns patterns from data instead of being explicitly programmed. Given enough labelled examples, it can learn to recognise objects such as traffic signs and apply that knowledge to data it has not seen before. The weakness is that its behaviour is only as good as its training data: if the data does not represent reality, or a situation was never included, such as a vandalised traffic sign, the system may not respond correctly. Our explainer on machine learning vs deep learning sets out the difference.

Deep learning: artificial neural networks

Deep learning is a form of machine learning that uses neural networks with many layers. Shown a set of images labelled as containing a car or not, a network learns in stages: early layers respond to brightness and edges, middle layers to shapes, and deeper layers to parts such as wheels and headlights, until the network can recognise a car. Unlike traditional machine learning, the network learns these features itself rather than relying on features designed by hand. Convolutional neural networks are the standard design for images; see deep learning and convolutional neural networks.

Supervised, unsupervised and reinforcement learning

In supervised learning the data is labelled before training, for example images tagged as cars, trucks and motorcycles, so the model learns to assign new images to the right category. Public data sets support this: the MNIST database of handwritten digits has 60,000 training examples and 10,000 test examples. In unsupervised learning there are no labels, and the model finds structure on its own; it is widely used for clustering, anomaly detection on machine data and cyber security. In reinforcement learning an agent learns by trial and error, guided by rewards and penalties. Game-playing programs are the classic example: DeepMind’s AlphaGo combined supervised and reinforcement learning. In robotics it trains control policies, often in simulation first; see reinforcement learning in robotics.

Perception and sensor fusion

Perception is a machine’s understanding of its surroundings from sensor data. Sensor fusion combines data from several sensors, such as LiDAR, radar, ultrasound, microphones and cameras, into one consistent view of the environment, which AI models running on parallel processors then use to make decisions in real time. Perception and fusion enable vehicles and robots to operate with partial or full autonomy, or with advanced driver assistance. See sensor fusion and computer vision.

Training builds a model: the network sees large amounts of labelled data and adjusts its weights until its answers are accurate. Inference uses the finished model on new data, such as a camera frame, and returns an answer. Training is done once or occasionally, usually on data centre GPUs; inference runs continuously wherever the data is, which at the edge means a Jetson, a GPU computer or an NPU. Older texts sometimes call this step “inferencing”. Our explainers on AI inference and inference vs training cover both in more depth.

At the edge, GPU computing happens inside a computer that may sit in a cabinet, on a vehicle or outdoors, without a server room’s cooling. Three GPU formats cover most industrial systems, and each sets a different power and cooling budget.

Jetson. Jetson Orin modules put an Ampere GPU, Arm CPU and memory on one board at 7–60 W, so a complete AI computer can run fanless. Jetson Thor adds a Blackwell GPU and 128 GB of memory on the T5000 for robots running perception and language models together. Our guide to NVIDIA Jetson explains the family.

MXM. MXM GPU modules put a discrete NVIDIA GPU on a small board with a cold plate, so a rugged computer can carry it without a full-height card. The RTX PRO 2000 Blackwell module brings ECC GDDR7 memory at 384 GB/s within 60 W.

PCIe cards. Industrial GPU computers take standard cards, from the 72 W, passively cooled NVIDIA L4 to RTX PRO Blackwell workstation cards. Our project on MXM GPU computing on board a digital train shows the embedded route in practice.

A GPU adds heat and power draw, so the computer around it has to be designed for both. These are the checks we make.

Our explainers on thermal throttling and TDP explain why power and cooling set the usable performance.

For training, almost always. Training a neural network is dominated by matrix arithmetic on large batches, which a GPU runs many times faster than a CPU, and the main frameworks are built around GPU libraries. For inference the answer depends on the model and the site. A GPU is the flexible choice for large or changing models, many camera streams and video pipelines. A CPU can be enough for small models at low rates, and an AI accelerator or NPU can run supported models at a fraction of the power. Many edge systems combine them, which is why we size the CPU, GPU and any accelerator together.

Most industrial GPU computing is inference on sensor data, close to where it is captured. The table shows common applications and the GPU option that usually fits; the right choice depends on the model, the number of streams and the site.

How much GPU do you need? Start from the inference rate: cameras × frames per second × models per frame. Eight cameras at 30 fps with one detector need 240 inferences per second. Then measure the model on candidate hardware at the precision you will deploy. If a GPU runs the model in about 4 ms per frame, it offers roughly 250 inferences per second, which leaves almost no headroom for decode, tracking or a second model, so the next GPU up is the safer choice. The figures here are illustrative; the method is what matters, and we run the same measurement on your model before recommending hardware. Our guide to robot computers applies the same sizing to mobile machines.