NPU vs GPU: Which for Edge AI?

NPU vs GPU is the choice between two kinds of AI accelerator: a GPU runs neural networks on thousands of programmable cores and handles training, large models and many video streams, while an NPU runs trained models on fixed, low-precision hardware at a fraction of the power. For edge AI the question is rarely which is better in general. It is which one the model, the number of cameras and the power and heat budget of the installation call for, and whether the system needs both.

The two are converging. Intel Core Ultra and AMD Ryzen AI processors now carry an NPU and a capable integrated GPU on the same die as the CPU, NVIDIA Jetson modules combine a GPU with dedicated deep learning accelerators, and M.2 accelerators such as Hailo-8 add an NPU to almost any computer. That gives more options and more ways to choose badly.

This guide compares GPUs and NPUs on the terms that decide an edge AI build: throughput, efficiency, precision, software and deployment effort, power and cooling. It includes a workload decision table and worked examples. For what an NPU is and how it works inside, see our guide to NPUs; for the general processor comparison, see GPU vs CPU.

The table compares the accelerators you meet in edge AI computers. Figures are typical ranges from manufacturers’ published specifications; individual models vary.

NVIDIA Jetson sits between the two columns either side of it. Its main AI engine is an NVIDIA GPU, and the larger Orin modules add deep learning accelerators (DLA) that behave like an NPU for supported layers. For how the Jetson range lines up, see which Jetson module to choose.

A GPU is a large array of programmable cores built to run the same instruction across thousands of data elements at once. Modern NVIDIA GPUs add tensor cores that accelerate the matrix multiplications at the heart of neural networks, as our explainer on CUDA cores vs tensor cores describes. Because the cores are programmable, a new model type or a new layer usually runs on day one, and the same hardware can train a model as well as run it.

An NPU fixes more of the work in hardware. It arranges multiply-accumulate engines and on-chip memory so that data flows through the layers of a network with as little movement to and from external memory as possible, which is where most of the energy goes. That makes each inference far cheaper in watts, but it means the model has to be compiled for the NPU, usually quantised to 8-bit or 4-bit integers, and built from the operations the NPU supports. Our explainer on NPU vs CPU covers the same contrast against a general-purpose processor.

The comparison is not only NPU against a discrete GPU. The integrated graphics in current x86 processors is a capable inference engine in its own right. Intel’s Arc graphics in Core Ultra processors and AMD’s Radeon 780M and 890M graphics in Ryzen 7040 and Ryzen AI 300 processors run FP16 and INT8 models through OpenVINO or the Ryzen AI software, and for some models they deliver more throughput than the NPU on the same chip, at a higher power draw.

That makes the integrated GPU a useful middle step. A model that does not compile cleanly for the NPU, or needs FP16 for accuracy, can often run on the integrated GPU without adding a graphics card, and the system stays fanless. Our explainers on Intel Arc and Intel AI Boost, Intel’s name for its NPU, cover the Intel side. The limit is memory: an integrated GPU shares system memory with the CPU, so memory bandwidth, not compute, is often what caps throughput.

Accelerators are sold on TOPS, trillions of operations per second, but the headline figure hides choices that make two TOPS numbers hard to compare. Our explainer on what TOPS means covers the measure itself.

Examples from our range show how figures are quoted. NVIDIA rates the Jetson AGX Orin at up to 275 TOPS and the Orin NX 16GB Super at up to 157 TOPS, both sparse INT8 figures. Intel rates the NPU in the Core Ultra 5 225H at 13 TOPS, while the Core Ultra 200S processors in the Vecow ECX-4000 series reach up to 36 TOPS from the NPU and integrated graphics together. AMD rates the NPU in Ryzen AI 300 processors at 50 TOPS, and a Hailo-8 M.2 module adds 26 TOPS.

The reliable comparison is frames or tokens per second on your own model at the precision you will deploy, measured on the target hardware at the temperature it will run at. Our explainers on INT8 quantisation and TFLOPS and TOPS explain the precision trade-off.

Per inference, usually yes. An NPU spends less energy moving data and runs at low precision, so it delivers more inferences per watt on the models it supports. A GPU earns its power draw when the model is too large, too new or too varied for an NPU, or when the throughput needed is beyond what an NPU offers.

Watts per TOPS is only part of efficiency. A system that keeps the whole pipeline (decode, pre-processing, inference and post-processing) on one accelerator often uses less energy than one that copies frames between a CPU, an NPU and memory. That is one reason Jetson performs well in camera-heavy systems: video decode, the GPU and the accelerators share one memory pool. The NVIDIA L4 explainer covers the server-card end of the range.

Most edge AI projects can be placed in one row of this table before any benchmarking. It assumes current models and typical camera resolutions; your model and frame rate move the boundaries.

A worked example: a depot gate with four 1080p cameras running a YOLO-class detector at 15 frames per second, in a cabinet that reaches 50 °C, on a 24 V supply. An integrated NPU alone would be tight at that stream count, a discrete GPU would force active cooling, and a fanless box with a Hailo-8 module or a Jetson Orin NX sits in the middle. The choice between those two comes down to software: if the team already works in CUDA and DeepStream, Jetson; if the application runs on x86 under Windows, a Hailo-8 in an x86 box. Our guide to how many TOPS an NPU needs gives rules of thumb for sizing.

Yes, and in many edge systems they should. The CPU, GPU and NPU on a current processor are designed to share work, and a pipeline can place each stage where it runs best.

The Premio KCO-6000-ARL shows the approach: a Core Ultra 200S processor with an Intel NPU and integrated graphics for lighter inference, plus PCIe Gen5 slots for a discrete GPU when a model needs it. Intel’s OpenVINO toolkit can target the CPU, GPU or NPU from the same model, which makes it practical to move a stage between them during testing. On AMD Ryzen AI processors, the Ryzen AI software stack does the same for the NPU and integrated GPU, as our explainer on AMD Ryzen AI explains.

This is where projects lose most time, and where the ranking pages on this topic say least. A model trained in PyTorch runs on a GPU almost unchanged. Running it on an NPU means exporting it, usually to ONNX, quantising it, compiling it with the NPU vendor’s tools and checking that accuracy holds.

Three checks save weeks. First, confirm every layer in the model is supported by the NPU, because unsupported layers fall back to the CPU and can make the whole pipeline slower than running on the CPU alone. Second, measure accuracy after INT8 or INT4 quantisation on your own data, not a public benchmark; quantisation-aware training can recover accuracy where it drops. Third, check the version support of the toolchain against the operating system image the computer will ship with. We build and test that image as part of the computer, so the drivers, runtime and model format are fixed before deployment. Our guide to Hailo-8 accelerators covers that toolchain in more detail.

The accelerator often decides the enclosure. An NPU-based system can stay within the 15–35 W that a compact fanless chassis can shed, so it can be sealed against dust and moisture and mounted in a cabinet or on a DIN rail. A discrete GPU adds tens to hundreds of watts, which needs fans or a much larger heatsink, a bigger power supply and more space. Our explainer on iGPU vs dGPU covers the integrated and discrete trade-off.

Heat also limits sustained performance. An accelerator that reaches its rated throughput on a bench at 25 °C may throttle in a cabinet at 50 °C, as our guide to thermal throttling explains. Where a GPU is needed in a compact space, MXM GPU computers put a discrete GPU on a module that can be cooled through the chassis, and Jetson Super mode shows how a power mode trades throughput against heat.

These systems cover the main options in the table above. We fit and configure the accelerators, install the drivers and runtimes, image the operating system with your model format in mind and test the computer under load before it ships.

For other options, browse our edge AI computers, which include Jetson, Hailo, Core Ultra and Ryzen AI systems, and our NVIDIA L4 computers for GPU inference in a compact server. For the processor roadmap behind the integrated NPUs, see our guide to CPU generations and embedded processors.