US What AI inference means
AI inference is the process of running a trained machine learning model on new data to get a result. The model’s weights are fixed; it simply applies what it learned. A defect classifier looks at a fresh image and returns pass or fail. A detector finds the people in a frame. A language model reads a prompt and writes a reply. Each of those is one inference.
Training happens once, or occasionally, and is about learning. Inference happens every time the model is used, often thousands of times a second across a fleet of cameras, so it is where most of an AI system’s running cost, power and latency sit. Our comparison of AI inference vs training covers the two stages side by side; this page explains how inference itself works and how to run it well.
How AI inference works
Every inference request follows the same steps, whether the model is a small detector or a large language model.
-
//
Input preparation. Raw data is converted into the form the model expects: frames are resized and normalised, audio is turned into features, text is split into tokens.
-
//
Forward pass. The input flows once through the network’s layers, and each layer applies its learned weights. No weights change, which is why inference needs far less compute per sample than training.
-
//
Output decoding. Raw scores become something usable: a class label, boxes after overlap removal, or the next word of a reply.
-
//
Generative models. A language model first processes the whole prompt in one pass, called prefill, then generates the answer one token at a time, called decode. Decode repeats until the reply is complete.
-
//
Action. The result drives a reject gate, an alert, a robot move or a response on screen.
Measuring inference performance
There is no single speed figure for inference. The right metric depends on whether one answer must arrive quickly or many answers must be processed in total.
| Metric | What it measures | Matters most for |
|---|---|---|
| Latency | Time from input to result, usually in milliseconds | Robots, safety systems, inspection gates |
| Throughput | Results per second, such as frames or requests | Multi-camera video, busy servers |
| Time to first token | Delay before a generative model starts replying | Chat and voice assistants |
| Tokens per second | How fast a language model writes once started | Long answers and reports |
| Performance per watt | Work done for the power drawn | Fanless, battery and vehicle systems |
| Accuracy after optimization | Quality lost by quantising or pruning | Every deployment, checked on real data |
Batching several inputs together raises throughput but adds waiting time, so real-time systems usually run small batches and accept lower peak throughput.
Where AI inference runs
Inference can run in a cloud data centre, on a server in your building, on an edge computer next to the cameras or inside the device itself. Cloud inference suits large models, unpredictable demand and work that can tolerate network delay. Edge AI moves inference close to the data, which cuts latency, keeps footage and records on site, and keeps working when the connection drops. Most industrial systems run time-critical inference at the edge and send summaries or exceptions upstream. Inference inside the device itself, on a smart camera or a sensor with a built-in NPU, suits simple and fixed tasks but limits the size of model you can run and how easily it can be updated.
Making inference faster
The same model can run several times faster on the same hardware once it is optimized for inference. These are the usual steps.
-
//
Lower precision. Converting weights from 32-bit floating point to FP16, INT8 quantisation or 4-bit formats cuts memory and speeds up maths, usually with a small accuracy loss that should be measured.
-
//
Graph optimization. Inference engines such as NVIDIA TensorRT fuse layers, remove unused operations and choose the fastest kernels for the target GPU.
-
//
Smaller models. Pruning, distillation and efficient architectures reduce the work per inference.
-
//
Serving techniques. Dynamic batching, KV caching for language models and speculative decoding keep the hardware busy and shorten replies.
Inference hardware
CPUs can run light models, but most production inference uses parallel hardware: GPUs with Tensor Cores, NPUs built into newer processors and dedicated AI accelerators. On the edge, NVIDIA Jetson modules combine a GPU, CPU and video engines in one package; our guide to choosing a Jetson module matches them to workloads. For heavier on-site inference, NVIDIA L4 computers carry low-profile data centre GPUs designed for inference, while Hailo AI accelerator systems add efficient vision inference to low-power x86 and Arm designs.
Quick answers on AI inference
AI inference: common questions
What does inference mean in AI?
Using a trained model. In AI, inference means applying a trained model to new inputs to produce predictions or generated output, as opposed to training the model.
What is an AI inference server?
A computer dedicated to running models. It hosts one or more trained models and answers requests from cameras, applications or users, usually with GPUs and serving software such as NVIDIA Triton Inference Server.
What is inference in generative AI?
Generating the output. Each time a generative model writes text or creates an image from a prompt, it is performing inference, usually one token or step at a time.
What makes a GPU suitable for AI inference?
Parallel maths and memory. Many cores with support for low-precision formats such as FP16 and INT8, enough memory to hold the model, and high memory bandwidth.
Is inference cheaper than training?
Per run, yes. A single inference costs a tiny fraction of training, but it runs continuously, so over a system’s life inference often accounts for most of the compute and power.
We build edge AI computers and on-site inference servers with the GPU, accelerator or Jetson module, storage and operating system image fitted and tested as standard. Tell us your models, inputs and latency target, and we will size the system.