Edge Deployment Basics

Why this matters

As a Computer Vision Engineer, you often deploy models on cameras, mobile devices, drones, kiosks, or robots. These edge environments have strict limits: low power, small memory, intermittent connectivity, and the need for real-time responses. Mastering edge deployment lets you deliver fast, reliable experiences without depending on the cloud.

• Run object detection on-camera to trigger alerts instantly.
• Count people locally in a retail kiosk without sending video to servers.
• Enable offline inspection on factory lines with predictable latency.

Concept explained simply

Edge deployment means running your ML model on a device near where data is produced (camera, phone, or embedded board), not in the cloud. You trade raw compute for lower latency, privacy, and reliability.

Mental model

Think of a triangle of constraints: latency, memory, and power. Improving one side often stresses another. Your job is to reshape the model and pipeline so the triangle fits your device.

Quick mental checklist to frame your solution

• Latency target: e.g., under 30 ms/frame for 30 FPS, or under 100 ms for interactive UX.
• Budget: CPU-only? GPU/NPU available? How much RAM/VRAM?
• Precision trade-offs: FP32 vs FP16 vs INT8 quantization.
• I/O scale: Resolution, frame rate, and pre/post-processing load.

Key constraints and design checklist

• Compute: CPU-only vs GPU/NPU accelerators; batch size usually 1.
• Memory: Model size and peak activation memory; avoid oversized input resolutions.
• Power/Thermals: Sustained performance may throttle; design for steady-state.
• Storage: Keep models and assets compact; prefer compressed formats.
• Connectivity: Assume no network or high jitter; avoid server round-trips.
• Security/Privacy: Keep data on-device when possible; sanitize logs.

Tooling and formats you will meet

• Intermediate formats: ONNX (general), TensorFlow Lite (mobile/embedded), Core ML (iOS), OpenVINO IR (Intel), TensorRT engine (NVIDIA).
• Runtimes: ONNX Runtime, TFLite Interpreter, Core ML runtime, TensorRT, OpenVINO, OpenCV DNN.
• Optimizations: Quantization (PTQ/QAT), pruning, operator fusion, static input shapes, reduced channels/width multipliers.

Edge deployment workflow in 6 steps

1. Profile your baseline: Measure latency, memory, and accuracy on a reference device or closest proxy.
2. Pick a deployment format: ONNX/TFLite/Core ML/OpenVINO/TensorRT based on device.
3. Convert the model: Export from PyTorch/TensorFlow; fix ops and static shapes where needed.
4. Optimize: Quantize, prune lightly, reduce input size, and fuse ops.
5. Integrate pre/post-processing: Keep it on-device; use the same resize/normalize as training.
6. Test and monitor: Verify accuracy drop is acceptable; measure real-world latency over time.

Worked examples

Example 1: CPU-only ARM camera (ONNX Runtime)

1. Target: 15 FPS on 640×480, CPU-only, 1 GB RAM.
2. Model: Lightweight detector (e.g., tiny/fast variant); export to ONNX with static shape 1×3×320×320.
3. Optimize: Reduce input to 320×320, apply INT8 PTQ if acceptable, ensure NMS is efficient.
4. Integration: Use a single frame buffer; reuse pre-allocated tensors to avoid allocations.

# Pseudocode
# 1) Export to ONNX with static shape
# 2) Run with ONNX Runtime on ARM CPU
# 3) Time per frame and memory

What you should observe

• Latency improves significantly after input downscale and INT8 conversion.
• Accuracy drop is small if calibration images match deployment domain.

Example 2: NVIDIA Jetson (TensorRT)

1. Target: 30 FPS at 640×480 with GPU acceleration.
2. Flow: PyTorch → ONNX → TensorRT engine (FP16).
3. Optimize: Use layer fusion and FP16; pin memory and use CUDA streams for pre/post.
4. Integration: Zero-copy where possible; batch size 1; cap power mode to avoid throttling.

Tip

Static input shapes often yield faster engines. Avoid dynamic shapes unless needed.

Example 3: Mobile phone (TFLite/Core ML)

1. Target: Smooth AR overlay at 30 FPS on mid-range device.
2. Flow: TensorFlow/Keras → TFLite (INT8) for Android or convert to Core ML for iOS.
3. Optimize: Use GPU delegate (Android) or Neural Engine (iOS) if compatible; minimize post-processing on CPU.
4. Integration: Reuse camera buffers; keep image conversion cheap (e.g., RGB, not multi-step conversions).

Debugging performance

• Measure per-stage times: capture, pre-process, inference, post-process, render.
• Try lower resolution first; then scale up until target FPS is met.

Exercises you can do today

These mirror the exercises below. Write down your results; small, reproducible reports help you iterate fast.

1. Exercise 1: Export a small classifier to ONNX with static shape and measure CPU latency at 224×224. Record model size, accuracy delta, and average latency over 200 frames.
2. Exercise 2: Convert a Keras model to TFLite and apply INT8 post-training quantization. Measure size reduction and latency change. Compare accuracy on a 100-image sample.

• Checklist:
  • Fixed input shape used in export.
  • Calibration images representative of deployment environment.
  • Per-stage timings captured (pre, infer, post).
  • Accuracy measured on a small but relevant sample.

Common mistakes and self-check

• Mistake: Using training-time image normalization at inference incorrectly (channel order or scaling). Self-check: Print first pixel values before/after normalize and compare to training pipeline.
• Mistake: Dynamic input shapes causing slow kernels. Self-check: Enforce static shape and compare latency.
• Mistake: Quantizing without proper calibration images. Self-check: Recalibrate with real-device frames and re-measure accuracy.
• Mistake: Doing heavy post-processing on CPU. Self-check: Profile NMS and sorting; use simpler algorithms or device-accelerated ops.
• Mistake: Ignoring warm-up. Self-check: Run 20–50 warm-up iterations before timing.

Practical projects

• Smart door counter: Deploy a tiny detector to count entries locally on a Raspberry Pi-class device.
• On-device defect flagger: Run a classifier on a micro workstation at a conveyor belt; show a red/green light instantly.
• Mobile AR hinting: Perform keypoint detection on-device to overlay simple guidance at 30 FPS.

Who this is for

Engineers who train CV models and need to ship them on real devices with tight latency and memory budgets.

Prerequisites

• Comfort with Python and either PyTorch or TensorFlow/Keras.
• Basic understanding of convolutional networks, preprocessing, and evaluation metrics.
• Ability to run simple benchmarks and interpret timing results.

Learning path

1. Start with this basics module and complete the exercises.
2. Learn format-specific conversion (ONNX/TFLite/Core ML/OpenVINO/TensorRT) for your target device.
3. Master quantization-aware training for models that lose too much accuracy with PTQ.
4. Integrate efficient pre/post-processing and memory management.
5. Harden for production: monitoring, fallback paths, and safe updates.

Next steps

• Pick one device you can access this week and deploy a tiny model end-to-end.
• Collect 5–10 real frames from that device and use them for calibration and testing.
• Repeat with a second runtime to compare results.

Mini challenge

Reduce end-to-end latency by 30% without dropping more than 1% absolute accuracy on your validation sample. Try input downscale, INT8 quantization, and faster post-processing. Document each change and its impact.

Quick Test

The quick test is available to everyone. Log in to save your progress and resume later.