What Is an NPU? Neural Processing Units for Edge AI

An NPU, or neural processing unit, is a processor core built to run neural networks: it performs the huge numbers of low-precision multiply-accumulate operations that AI inference needs, using arrays of arithmetic engines fed from on-chip memory, at a small fraction of the power a CPU or GPU would use for the same work. NPUs run trained models rather than train them, and they are now built into most new laptop, embedded and edge processors.

If you are asking what an NPU is in a computer, the short answer is that it is the part of the processor that runs AI features locally. In an industrial or edge system that means object detection on a camera feed, people counting, defect checks, speech commands or a small language model, running on the device instead of in the cloud. Intel calls its NPU AI Boost, AMD uses the XDNA architecture under the Ryzen AI name, Qualcomm has the Hexagon NPU and NVIDIA builds deep learning accelerators into its larger Jetson modules.

This guide explains how an NPU works, which processors and accelerators have one and how fast they are, how NPUs compare with CPUs, GPUs and TPUs, what they are used for at the edge and when a system needs one. For a head-to-head on choosing an accelerator, see our guide to NPU vs GPU for edge AI.

A neural network is mostly matrix arithmetic. Each layer multiplies an input, such as the pixels of a camera frame or the output of the previous layer, by a set of learned weights and adds the results. A CPU does that a few numbers at a time. An NPU is built to do thousands of those multiply-accumulate (MAC) operations in parallel, every clock cycle.

The common thread is data movement. In a neural network, fetching a weight from external memory can cost many times more energy than multiplying it, so NPU designs keep as much of the model and its intermediate results on the chip as they can. That is also why an NPU’s rated TOPS are only reached when the model fits its memory and every layer is supported.

Inference, not training

Training a model means running it forwards and backwards millions of times in high precision to learn the weights. That is GPU and data-centre work. AI inference runs the finished model on new data, and that is what an NPU is for. Before it runs on an NPU, a model is usually converted to 8-bit integers, as our explainer on INT8 quantisation describes, and compiled for that NPU.

NPUs started in smartphones. In 2017 Huawei’s Kirin 970 and Apple’s A11 Bionic each added a dedicated neural engine to run camera, face recognition and voice features on the phone. Arm SoC makers followed, and NXP, Rockchip and Qualcomm brought NPUs to embedded and industrial processors over the following years: the NXP i.MX 8M Plus, Rockchip RK3588 and Qualcomm QCS6490 are all common in industrial boards today.

x86 caught up in 2023, when AMD added an XDNA NPU to its Ryzen 7040 processors and Intel put an NPU into the first Core Ultra processors. In 2024 Microsoft set 40 TOPS of NPU performance as the threshold for its Copilot+ PCs, and the next generations jumped to 45–50 TOPS. Embedded versions of the same processors reach industrial computers a little later, which is why NPUs are now a standard line on edge computer datasheets. Our guide to Intel Core Ultra covers Intel’s side of that change.

The table lists the NPUs you are most likely to meet in industrial and embedded computers, with manufacturers’ published ratings. Ratings are for the NPU alone unless stated; most are INT8 figures, and some vendors quote INT4, so treat them as a guide rather than a like-for-like comparison.

Two points stand out. First, the NPU is not always the largest AI engine on the chip: on Intel Core Ultra 200S processors the NPU and integrated graphics together reach up to 36 TOPS, and on Jetson the GPU provides most of the module’s rating. Second, the Arm SoCs used in industrial boards carry smaller NPUs than the newest laptop processors, but they run within a few watts and on long-lifecycle parts. Our explainers on the Qualcomm Hexagon NPU, AMD Ryzen AI, Intel AI Boost and the Hailo-10H cover each one in more depth.

An NPU reaches a computer in one of three ways, and the route affects cost, power, software and how easily a design can be upgraded.

Add-on accelerators are often the practical route in industrial projects, because they let a proven x86 or Arm computer gain AI without changing the processor, the operating system image or its certifications. The Hailo range covers M.2 modules and systems built around them, and our guide to Hailo-8 accelerators explains how they integrate.

All four can run a neural network. They differ in how much of the work is fixed in hardware, and so in flexibility and efficiency.

In practice an edge computer uses them together. The CPU runs the application and the operating system, the GPU decodes video and draws the screen, and the NPU runs always-on inference. Our explainers on NPU vs CPU and NPU vs TPU compare each pair, and what an AI accelerator is covers the wider family.

NPUs are rated in TOPS, trillions of operations per second, where one multiply-accumulate counts as two operations. The figure is a peak at a stated precision, usually INT8, and assumes every MAC engine is busy every cycle. Real models never quite reach it, and two NPUs with the same TOPS can differ widely on the same model. Our explainer on what TOPS means covers the arithmetic.

A more useful measure is inferences or frames per second on a named model, such as a YOLO-class detector at a set resolution, and the power drawn while doing it. For sizing, our guide to how many TOPS an NPU needs gives rules of thumb by workload: a single camera running a small detector can fit within a few TOPS, while several streams or larger models need tens of TOPS or more.

An NPU’s efficiency comes from fixing work in hardware, and that sets limits worth knowing before a design depends on one.

• It cannot train models. Training needs high-precision arithmetic and backpropagation, which NPUs do not provide.
• It runs supported layers only. A new layer type may not compile, and a layer that falls back to the CPU can slow the whole model.
• Large models strain it. When weights do not fit on-chip memory, the NPU waits on system memory and its rated TOPS are out of reach.
• Compiled models are not portable. A model compiled for one vendor’s NPU has to be recompiled, and often requantised, for another.
• It is not general-purpose compute. Video decode, image pre-processing and the application itself still run on the GPU and CPU.

None of these rule an NPU out; they decide which models and projects it suits. Where a model is too large, too new or needs FP16, a GPU or a Jetson module is usually the answer, as our GPU vs CPU explainer and the NPU vs GPU guide describe.

In industrial and commercial systems, NPUs run the AI that needs to happen close to the sensor: fast, private and without a network round trip. That is the core of edge AI, and our explainer on edge AI vs cloud AI covers the trade-off.

The common factor is a trained model that runs continuously. An NPU suits that well, because it can run all day at a few watts while the CPU stays free for the application, the database and the network. Many of these jobs start with computer vision models built on convolutional neural networks, which NPUs handle particularly well.

Industrial computers are often fanless, sealed and powered from a vehicle or a cabinet supply. A discrete GPU adds heat, power and size that those installations cannot always take. An NPU brings useful inference into a compact, fanless computer, keeps data on the device, and can be added to an existing design. That is the difference between adding AI to a computer already on the line and specifying a separate GPU computer.

Three industrial factors shape the choice. Lifecycle: embedded versions of Core Ultra, Ryzen and the Arm SoCs above are built for years, so a model qualified on one can be rebuilt on the same hardware. Temperature: the NPU sits inside the processor or on an M.2 module rated with the computer, so wide-temperature builds are possible, such as the Vecow EIC-2000 on the NXP i.MX 8M Plus, rated from −25 to 70 °C. Software: the NPU’s toolchain has to be supported on the operating system image the computer ships with, which we fix and test as part of the build. For how NPUs fit into processor choices more widely, see our guide to CPU generations and embedded processors and what a system on chip is.

Not every edge computer needs one. The table is a quick test.

If the answer is not obvious, test the model. Compile it for the candidate NPU, run it on recorded data from the site and measure frames per second, accuracy and power. That costs days, and it avoids choosing hardware that cannot run the model.

These systems show the main ways an NPU reaches an edge computer. We configure them with the NPU drivers and runtime your model needs, image the operating system and test the complete computer before it ships.

For processor-integrated NPUs, see our fanless PCs on Core Ultra and Ryzen, and for the Arm side our Arm SBCs.