UK How object tracking works
Object tracking is the computer vision task of following an object through a sequence of video frames. A detector looks at one frame at a time and says what is there and where. A tracker adds memory: it decides which detection in this frame is the same object as one in the last frame, predicts where it will be next and keeps a consistent identity for it, even when it is briefly hidden. The output is a track, the object’s position, size and ID over time, from which software can work out speed, direction and path.
Single object tracking keeps a lock on one target an operator selects, which suits steering a PTZ camera or a gimbal. Multi-object tracking gives every person or vehicle in view its own ID, which suits counting, traffic analysis and autonomous vehicles. Both have to cope with targets that turn, change size or disappear behind something.
Detection vs tracking
Detection and tracking work together, and most modern systems use a method called tracking by detection. Each frame, the detector finds candidate objects. The tracker predicts where each existing track should be, often with a Kalman filter that models position and velocity, then matches the new detections to those predictions, typically with the Hungarian algorithm. Matched detections update their tracks, unmatched detections start new ones, and tracks that go unmatched for too long are closed.
Tracking adds three things detection alone cannot: a stable ID for each object, so a person is counted once rather than every frame; smoother positions, because noisy detections are filtered; and continuity through short gaps, when the detector misses an object for a few frames. That is why counting, line crossing and dwell-time analytics all depend on the tracker, as our video analytics FAQ explains.
Common tracking algorithms
Trackers trade accuracy against processing load. These are the families most often met in practice.
| Tracker | Type | How it links objects | Strengths | Typical use |
|---|---|---|---|---|
| SORT | Multi-object | Kalman filter and box overlap | Very fast, simple | Counting on low-power devices |
| DeepSORT | Multi-object | Adds an appearance model for re-identification | Fewer ID switches when paths cross | People tracking, retail, security |
| ByteTrack | Multi-object | Also matches low-confidence detections | Holds tracks through partial occlusion | Crowds, traffic |
| Correlation filter (e.g. KCF) | Single object | Learns the target’s appearance and searches nearby | Light, no detector needed | Operator-selected targets, gimbals |
| Siamese and transformer trackers | Single object | A neural network compares the target template with each frame | Robust to change of scale and appearance | Long-range EO/IR, drones, defence |
On NVIDIA Jetson, the DeepStream SDK includes several ready-made trackers, such as NvDCF and NvDeepSORT, that run alongside the detector in the same pipeline.
What makes tracking hard
Real scenes break trackers in predictable ways, and good systems are designed around them.
-
//
Occlusion. Targets pass behind other objects and must be picked up again with the same ID.
-
//
Image quality. Heat haze, fog, rain and low light degrade the picture and the detections.
-
//
Platform movement. Cameras on vehicles, masts and drones move as well as the target, so the tracker has to separate the two motions.
-
//
Similar objects. People in the same uniform or identical vehicles cause ID switches when paths cross.
-
//
Latency. Video has to be captured and processed fast enough to keep the lock, especially when the tracker steers the camera.
Where object tracking is used
Any system that needs to know how something moves, not just that it is there, relies on tracking.
-
//
Security and surveillance. Following intruders across a site, steering PTZ cameras and handing a target from one camera to the next.
-
//
Traffic and transport. Counting vehicles by lane and direction, measuring speed and queue length, and spotting wrong-way drivers.
-
//
Retail and venues. Counting people once each, measuring dwell time and mapping paths through a store or station.
-
//
Robots and autonomous vehicles. Predicting where people and vehicles will move next, so the machine can plan a safe path around them.
-
//
Defence and counter-drone. Holding a lock on small, fast targets at long range with EO/IR cameras on gimbals.
Tracking at the edge
When a tracker steers a camera, every millisecond of delay lets the target drift further from the centre of the frame. That is why tracking for PTZ cameras, gimbals and drones runs on a computer beside the camera rather than across a network. It needs low-latency video capture, enough AI processing for the detector and tracker together, and outputs to drive the pan-tilt head, often over RS-485. Tracks from cameras can also be combined with radar or LiDAR through sensor fusion.
Our object tracking case study runs a deep learning single-object tracker in vehicles on a fanless Jetson AGX Xavier computer, with RS-485 PTZ control and AHD, SDI and IP video inputs. Our rugged Jetson computers are built for that kind of installation.
Quick answers on object tracking
Object tracking: common questions
What is the difference between object detection and object tracking?
Memory. Detection finds objects in a single frame. Tracking links detections across frames and keeps a consistent ID for each object.
What is multi-object tracking?
Tracking every object in view at once. Each person or vehicle gets its own ID, which counting, traffic analysis and autonomy depend on.
What is single object tracking used for?
Following one chosen target. An operator selects it, and the tracker steers a PTZ camera or gimbal to keep it centred.
Is object tracking AI?
Usually, today. Detectors and appearance models are neural networks, while the motion prediction is often classic filtering such as a Kalman filter.
Can object tracking run in real time?
Yes, on suitable hardware. Edge AI computers run detection and tracking at full video frame rate on several streams.
We build rugged edge AI computers for tracking as standard, with the capture cards, serial ports and AI runtime installed and tested before the system ships.