Hailo-8 AI Accelerators: Benefits and Integration for Edge Computing

The Hailo-8 is an AI accelerator chip that runs deep learning inference at up to 26 TOPS while drawing around 2.5 W, so a compact edge computer can run real-time vision models without a GPU. It sits beside the host processor on an M.2 or mini PCIe module and takes the neural network workload off the CPU.

Hailo was founded in Tel Aviv in 2017 to design processors for AI at the edge, and the Hailo-8 is its most widely deployed chip. It is used wherever video has to be analysed where it is captured: traffic junctions, factory lines, rail vehicles, shops and security systems. This guide covers how it works, how it compares with the Hailo-8L and Hailo-10H and with Jetson and on-chip NPUs, the module formats and PCIe lanes, the software path from a trained model to the chip, power and cooling, and how to build it into an industrial computer.

Running inference at the edge rather than in the cloud gives:

• Real-time decisions, with no round trip to a data centre
• Lower bandwidth, because only results or metadata leave the site
• Better data privacy, because images stay on the device
• Operation that continues when the network does not
• Lower running costs than streaming video to cloud inference

Our explainer on edge AI vs cloud AI sets out the trade-offs in more detail.

Hailo now offers three accelerators for edge computers. They share a software stack, so a model can move between them as the application changes. Figures are from Hailo’s published specifications; check the current data sheet for the exact module.

For most industrial vision work the Hailo-8 is the default: it has the widest choice of modules and the most systems built around it. Choose the Hailo-8L when the model is small and cost or power is tight, and the Hailo-10H when the application needs a language or vision language model to run on site. See every system we build around them on our Hailo AI accelerator computers page.

They are different tools rather than rivals. A Hailo-8 is an accelerator that adds AI to a computer you choose for other reasons; a Jetson Orin Nano is a complete computer with a GPU, Arm CPU and memory on one module. An NVIDIA GPU or Jetson Orin Nano offers more flexibility and a wider choice of software and models, at higher power. The NPU built into recent Intel Core Ultra and AMD Ryzen AI processors needs no extra hardware, but has less performance for multi-camera work. The Hailo-8 sits between them: dedicated performance at a few watts, added to almost any x86 or Arm computer.

Choose a Hailo-8 when you want to keep an x86 application, Windows, or a specific industrial computer and add vision AI with little extra heat. Choose a Jetson when the models change often, need FP16 or a GPU, or the project needs CUDA. Choose the on-chip NPU when one or two light models are enough. For a fuller comparison, read our comparison of GPUs and NPUs for edge AI, our explainer on what an NPU does and our guide to how many TOPS an application needs.

A CPU or GPU fetches data from shared memory, processes it and writes it back, over and over. For neural networks that constant traffic costs time and power. The Hailo-8 instead uses a dataflow architecture: the compiler maps each layer of the network onto its own set of compute, memory and control resources on the chip, and data flows through them in sequence.

Compute

Many parallel processing elements are allocated to each layer according to how much work it needs, so full HD video streams can be processed in real time without down-sampling.

Memory

Weights and intermediate results are held in memory beside the compute elements that use them. The Hailo-8 needs no external DRAM, which keeps the module small and its power low. It also sets a limit: a model too large for the on-chip memory is split into contexts that the chip switches between, which costs throughput.

Control

A control fabric schedules the flow of data between layers and manages the PCIe link to the host, which only has to send frames in and read results out.

For larger workloads, several Hailo-8 chips can run together on multi-chip PCIe cards, sharing a model or running separate models side by side. Our explainer on AI accelerators places the dataflow approach alongside GPUs and other NPUs.

Hailo-8 modules come in several formats. The chip talks to the host over PCIe Gen3 with up to four lanes, and the slot decides how many lanes it gets. The M key module uses all four lanes and gives the full 26 TOPS on heavy multi-stream work; B+M, A+E and mini PCIe slots usually carry fewer lanes, which suits lighter models or fewer cameras, and fits smaller boards and systems.

Lane count matters most when a lot of data crosses the link: many high-resolution streams, large input images, or a model split into contexts that reloads weights. A single-lane slot can still give near-full throughput on a small detector with a modest input size. Our guide to M.2 slots and keys explains which modules fit which slots, and our explainer on M.2 keying covers the notch positions. Several systems in our range trade an NVMe socket for a Hailo module: the SolidRun RAI300 takes up to two Hailo-8 or Hailo-10 modules in its M.2 M key 2242 sockets.

Hailo supplies the tools to take a trained model to a running application on the device.

• Dataflow Compiler: converts a trained model from common frameworks such as TensorFlow and PyTorch (via ONNX) into a program for the chip, quantising it and mapping each layer to on-chip resources.
• HailoRT: the runtime and driver on the host, with C++ and Python APIs and GStreamer support for video pipelines on x86 and Arm.
• Model Zoo: pre-trained and pre-compiled models for detection, segmentation, pose estimation and classification, including popular YOLO networks, as a starting point for custom work.
• TAPPAS: example application pipelines that show multi-stream detection and tracking end to end.

From ONNX to HEF. The compiled output is a Hailo Executable Format (HEF) file that HailoRT loads onto the chip. The usual path has six steps.

What models does Hailo-8 support? Convolutional networks for detection, classification, segmentation, pose estimation and re-identification are its home ground, and the Model Zoo also includes some transformer-based vision models. The quantisation step is the same idea as INT8 quantization on other accelerators. Large language models are not a fit, because the Hailo-8 has no external memory; that is the job of the Hailo-10H.

We supply Hailo-8 systems with the driver and runtime installed on a Windows or Linux image, so development starts on a working computer.

Hailo quotes around 2.5 W typical for the Hailo-8, which is why it fits fanless systems. Power rises with load, so a module running a large model on several streams draws more than one running a small detector, and short peaks can exceed the typical figure. Plan the heat path for several watts per module, and multiply by the number of modules in the system.

In a fanless computer, that heat has to reach the chassis. The module needs a heatsink or a thermal pad pressed against the case or an internal heat spreader, and the system has to be rated for the site temperature with the accelerator working. If the chip gets too hot it throttles to protect itself, and throughput falls. That is why we test Hailo systems under sustained load at temperature rather than on the bench alone; our explainer on thermal throttling explains the mechanism.

Our guide to how a fanless PC works covers the cooling design behind these ratings.

The accelerator is the easy part. Whether a Hailo-8 system keeps running on site depends on the computer around it. We work through four areas before recommending a computer.

Environment

Industrial-grade Hailo-8 modules are rated from −40 °C to 85 °C, but the system must match: a fanless, sealed chassis keeps out dust and moisture, and the temperature range, sealing and shock rating should fit the site. For trains, vehicles and outdoor enclosures, see our rugged edge computers.

Performance

Match the host processor to everything that runs beside the model: decoding, pre-processing, the application and any control software. The Hailo-8 connects over PCIe Gen3 with up to four lanes; a slot with fewer lanes limits throughput on heavy multi-stream work. Processors with their own NPU, such as AMD Ryzen 8000 parts with a 16 TOPS NPU, can share the load.

Expansion

Check that the system has the right M.2 key or mini PCIe slot for the module, plus the camera ports the application needs: PoE for IP cameras, or USB, GMSL or frame grabbers for machine vision cameras.

Lifecycle

Industrial deployments run for years, so choose a computer with long availability and a stable software image, and plan how models will be updated in the field.

We configure, image and test every Hailo system: the module fitted and cooled, HailoRT and drivers installed in the operating system image, your HEF files loaded where you supply them, and the system run under load before it ships.