Object Tracking Basics

Why this matters

Object tracking lets you follow the same physical object across video frames. As a Computer Vision Engineer, you will:

• Count and follow people or vehicles across cameras for analytics.
• Stabilize IDs for action recognition, sports analytics, and retail flows.
• Enable real-time systems (dashcams, robots) to predict where things will be next.
• Reduce detector load by predicting positions between detection frames.

Who this is for

Beginners to intermediate engineers in video vision who know detection but are new to tracking.

Prerequisites

• Basic linear algebra and probability (vectors, covariance, Gaussian noise).
• Familiarity with object detection outputs (bounding boxes, confidence scores).
• Comfort with arrays and simple algorithms (sorting, distance metrics).

Concept explained simply

Tracking is keeping the same ID for the same object across frames. Most modern systems use tracking-by-detection: run a detector each frame (or every few frames), then match detections to existing tracks.

Mental model

Imagine each object has a tiny "prediction bubble" that moves forward each frame. New detections land on the scene like pins. You connect each bubble to the closest pin that makes sense. If there is no pin, the bubble drifts forward a bit before you decide it disappeared. If a new pin appears without a bubble, you start a new bubble (a new track).

Core building blocks

• State: What you store per track. Common: position and velocity (e.g., x, y, vx, vy) and sometimes box size (w, h).
• Prediction: Where the object is expected now (often a Kalman Filter).
• Association: Which detection matches which track. Common costs: IoU distance, Mahalanobis distance, appearance similarity.
• Assignment: Solve the global matching using the Hungarian algorithm (minimize total cost).
• Lifecycle: Create, confirm, update, and delete tracks using rules (min hits to confirm, max age before delete).

Matching costs

• IoU-based: Good when boxes overlap well and motion is smooth.
• Motion-based: Mahalanobis distance using the predicted covariance (good for gating and occlusion handling).
• Appearance-based: Embeddings from a re-identification model to reduce ID switches.

Handling reality

• Occlusion: Keep tracks alive for a few frames without detections (max age). Re-link when the object reappears.
• False positives: Use confidence thresholds and confirmation logic (e.g., need N hits before declaring a real track).
• Real-time: Run the detector every N frames; predict in-between to save compute.

Worked examples

Exercises

These mirror the graded exercises below. Complete them here, then check solutions.

Exercise 1 — Build a simple IoU tracker (no filter)

You have 5 frames of detections (x, y, w, h). Use IoU matching with threshold 0.3, delete tracks if missed for 2 consecutive frames, and create new tracks for unmatched detections. Use a greedy assignment with highest IoU per track (ties broken arbitrarily). Report track IDs per frame.

• IoU threshold: 0.3
• max_age (misses allowed): 2
• Report mapping like: F1 T1→A, T2→B; F2 T1→A, T2→B; ...

• Checklist:
  • New track IDs created on first appearance.
  • Track for B survives one missing frame (F3) and re-associates on F4.
  • No extra IDs appear for A.

Exercise 2 — Associate with the Hungarian algorithm (by hand)

Given the cost matrix C for 3 tracks (rows T1..T3) and 3 detections (columns D1..D3), find the minimum-cost assignment. Costs are 1 - IoU, lower is better.

T1: [1.00, 1.00, 0.10]
T2: [0.15, 1.00, 1.00]
T3: [1.00, 0.20, 1.00]

• Checklist:
  • Each track is assigned to at most one detection.
  • Total cost is minimized.
  • No unassigned pairs have lower alternative cost than chosen pairs.

Common mistakes and self-check

• Using IoU alone for fast or small objects: Add motion gating (Mahalanobis) or appearance features.
• Too low IoU threshold: Causes ID switches and false links. Start with 0.3–0.5 and tune.
• No lifecycle rules: Without min_hits and max_age you get flickering IDs.
• Ignoring frame rate: dt must reflect actual time; otherwise predictions drift.
• Wrong box format: Mixing [x,y,w,h] vs [x1,y1,x2,y2] produces bad IoU; standardize early.
• Unbounded search: Not gating by distance lets far-away detections match; always gate.

Practical projects

• Webcam multi-object tracker:
  • Run a lightweight detector every 2–3 frames.
  • Between detections, predict with a Kalman filter.
  • Use IoU+gating to associate; display stable IDs.
• Traffic camera analytics:
  • Count vehicles by tracking them across a virtual line.
  • Handle stops and starts with max_age and min_hits.
  • Export per-track trajectories for speed estimation.
• Sports player tracking:
  • Combine motion and color histograms as appearance features.
  • Evaluate IDF1 and tune thresholds to reduce ID switches.

Learning path

1. Start: IoU-based matching with track lifecycle (this lesson).
2. Add motion: Kalman filter with Mahalanobis gating.
3. Scale to MOT: Hungarian assignment and birth/death logic.
4. Improve IDs: Appearance embeddings and re-identification.
5. Evaluate: MOTA, IDF1, HOTA; inspect errors where IDs switch.
6. Optimize: Run detector sparsely; consider SORT/DeepSORT/ByteTrack ideas.

Mini challenge

Design a tracker for people in a hallway camera at 15 FPS on a CPU-only device. Detector can run at most every 3 frames. How will you keep IDs stable at doorways where occlusions happen?

Next steps

• Implement a Kalman-based MOT with Hungarian assignment.
• Add re-identification features and compare IDF1 before/after.
• Practice on different scenes (crowds, traffic, indoor) to tune thresholds robustly.

Quick Test note

The Quick Test below is available to everyone. Sign in to save your progress and resume later.