What Is ONNX?

ONNX, short for Open Neural Network Exchange, is an open standard for describing a trained AI model in a single file. It was launched in 2017 by Microsoft and Facebook (now Meta), was joined by most of the chip and cloud vendors, and is now governed by LF AI & Data, part of the Linux Foundation.

Training frameworks and inference engines each have their own formats, which can lock a model to one vendor. ONNX sits between them: export once from the framework the data scientists use, then hand the file to the toolkit that suits the target hardware. That is why ONNX is so often the first step in moving a model from a training server onto an edge computer.

An ONNX file, with the extension .onnx, stores the model as a computation graph. Each node in the graph is an operator, such as a convolution, a matrix multiply or an activation, and the edges are the tensors that flow between them. The trained weights are stored alongside the graph, and the whole thing is serialised with Google’s Protocol Buffers.

The operators are defined in numbered opsets. Each ONNX release adds or refines operators, and a model declares which opset it uses. An inference engine supports a range of opsets, so the opset you export at has to fall within what your target toolkit accepts. Exporting is usually one call, such as torch.onnx.export in PyTorch or the tf2onnx converter for TensorFlow and Keras.

ONNX is only a format, so something has to execute it. ONNX Runtime is Microsoft’s open-source engine for that job. It can run a model on the CPU, or hand parts of the graph to hardware-specific back ends called execution providers, so one application can target an NVIDIA GPU, an Intel NPU or a Qualcomm NPU by changing a setting. Many vendor toolkits also read ONNX directly.

Support for individual operators varies between engines, so check that every layer in your model is supported by the target toolkit before you commit to the hardware.

ONNX earns its place in a deployment pipeline for a few practical reasons.

Most export problems come from a few places. Custom or very new layers may have no ONNX operator yet, so the exporter fails or splits them into many small operators that run slowly. Dynamic input sizes need to be declared at export, or the model is fixed to the shape used during export. Pre- and post-processing written in Python, such as image resizing or decoding a detector’s output, is often not part of the graph and has to be rebuilt on the target.

Check the exported model against the original on a set of real inputs before optimizing it. Small numerical differences are normal, but a large drop in accuracy usually points to an unsupported operator or a wrong opset.

On an edge computer the typical path is: train in PyTorch, export to ONNX, check the graph with a viewer such as Netron, then optimize it for the target. That last step often includes INT8 quantisation, which shrinks the model and speeds it up on NPUs and accelerators. The ONNX file then becomes a TensorRT engine on a Jetson, an OpenVINO model on an Intel system, or a compiled binary on a Hailo module.

Because the same ONNX file feeds all three, you can benchmark one model on Intel Core Ultra computers with a built-in NPU, on Hailo AI computers and on edge AI computers with NVIDIA GPUs, then choose on measured frames per second and power. Our guide to the benefits of Hailo-8 AI accelerators shows where a dedicated accelerator fits in that comparison.

We supply edge AI computers with the runtime for their hardware, such as TensorRT, OpenVINO or the Hailo software, installed and validated as standard. Send us your ONNX model and target frame rate, and we will benchmark it on the systems that fit.