What Does TOPS Mean in AI?

TOPS, or tera operations per second, is a measure of how many trillion basic maths operations a processor can perform each second. In AI, it is the peak throughput of the hardware that runs neural networks: an NPU, a GPU, a dedicated AI accelerator or a whole system-on-module.

Vendors use TOPS because it gives one number to compare very different chips. An NVIDIA Jetson module, a Hailo M.2 accelerator and the NPU inside an AMD Ryzen AI processor can all be compared on a datasheet this way. TOPS is a useful first filter for edge AI hardware, but it is a theoretical maximum, not a measure of how fast your model will run.

Neural networks are built mostly from multiply-accumulate (MAC) operations: multiply two numbers and add the result to a running total. AI hardware contains thousands of MAC circuits working in parallel. Each MAC counts as two operations, so the headline figure is simply the number of MAC circuits, times two, times the clock speed.

For example, an accelerator with 4,096 MAC circuits running at 1 GHz performs about 8.2 trillion operations per second, or 8.2 TOPS. Raising the clock speed or the power limit raises TOPS on the same silicon, which is how NVIDIA’s Super modes lift Jetson Orin Nano and Orin NX figures. Our Jetson Super Mode explainer covers those power modes.

Two TOPS figures are only comparable if they were measured the same way. Check these points before comparing datasheets.

The table lists published peak figures for processors used in edge AI computers. They come from the chip makers and use each vendor’s own basis, so read them as a guide to scale rather than a benchmark.

Our Orin Nano vs Orin NX comparison shows how those figures translate into module choice, and our Hailo AI accelerators and computers add TOPS to an x86 system through an M.2 slot.

Two devices with the same TOPS can perform very differently on the same model. Memory bandwidth often limits real throughput, because the processor waits for data rather than doing maths. Software matters as much: a model compiled with the vendor’s own toolchain, such as TensorRT for Jetson or the Hailo Dataflow Compiler, can run several times faster than an unoptimised one. Some layers in a network also run poorly on an NPU and fall back to the CPU.

TOPS per watt is often the more useful figure for edge AI computers, where a sealed enclosure or a vehicle supply sets the power budget. The reliable test is to benchmark your own model at your own resolution and frame rate on the target hardware. The fanless GPU computer case study chose between Orin Nano and Orin NX on exactly that balance of TOPS and power.

TOPS works best as a way to narrow the field, not to make the final choice. Start from the workload: the model family, the input resolution, the number of camera streams and the frame rate each one needs. Published benchmarks for similar models on each candidate give a first estimate, and TOPS shows how much headroom is left for a second model or more cameras later. Then check the power budget, the operating temperature and the software stack, because a module that runs your model well in a lab may throttle in a sealed enclosure on a hot day. Finally, benchmark the real model on the shortlisted hardware before committing to a build.

We build edge AI computers around Jetson, Hailo, Intel and AMD processors, install the AI software stack as standard and can benchmark your model on the shortlisted hardware before you commit.