Continuous Training Pipelines Basics

Why this matters

Vision models degrade as environments, cameras, and data drift. Continuous training keeps your model current without constant manual effort. As a Computer Vision Engineer, you will:

• Automate retraining when data changes or performance drops.
• Gate deployments with objective metrics to avoid regressions.
• Track datasets, code, and models so you can reproduce and roll back fast.
• Close the loop from production feedback to improved datasets and models.

Concept explained simply

Continuous training is an automated loop that watches data and performance, retrains the model when needed, tests it against quality gates, and deploys it safely if it beats the current model.

Mental model

Think of a conveyor belt with checkpoints: data flows in, gets cleaned and labeled, a model is trained and tested, a gate checks its quality, and only then it can roll onto production. Sensors on the belt (monitors) trigger the belt to run when drift or schedule hits.

Pipeline building blocks

1. Ingest: Collect new images/videos and metadata from production and storage.
2. Curate: Filter duplicates/near-duplicates, sample hard cases, and enforce privacy rules.
3. Label/Review: Auto-label where possible; route unsure cases to human review.
4. Split: Create train/val/test with leakage prevention (by scene, camera, or time).
5. Preprocess: Resize, normalize, augment (keep val/test clean).
6. Train: Reproducible training with fixed seeds and config.
7. Evaluate: Report task metrics (e.g., mAP@IoU, IoU, F1), latency, and resource usage.
8. Register: Store model artifact, metrics, and lineage (code, data, params).
9. Deploy: Safe rollout (shadow/canary) if quality gates pass.
10. Monitor: Track live metrics, drift, and errors; feed back examples for the next cycle.

Choosing triggers

• Schedule: predictable, simple; may retrain unnecessarily.
• Drift: efficient; needs good statistics and thresholds.
• Performance: aligns with business; needs reliable ground truth sampling.

Worked examples (3)

1) Retail shelf detection — weekly schedule + gates

Goal: Keep a shelf product detector fresh as packaging changes.

• Trigger: Every Sunday 02:00.
• Curate: Prioritize low-confidence detections and new SKUs.
• Metrics: mAP@0.5 on time-split validation; FPS on target device.
• Gate: mAP >= 0.55 and latency <= 25ms; otherwise reject.

{"trigger":"weekly","curation":{"sample":"low_conf + new_SKU"},"train":{"epochs":50,"seed":42},"eval":{"map_threshold":0.55,"latency_ms":25},"deploy":{"strategy":"canary","traffic":0.1}}

2) Manufacturing defect segmentation — drift trigger

Goal: Handle new camera lighting that changes texture appearance.

• Trigger: Input brightness histogram shift >15% vs. baseline.
• Label loop: Review 200 uncertain masks via human-in-the-loop.
• Gate: Mean IoU >= 0.82; false negative rate <= 6% on critical defects.
• Deploy: Shadow for 48 hours; promote if stable.

3) Traffic sign classifier — performance trigger

Goal: Seasonal signs appear (e.g., detours) and hurt precision.

• Trigger: 7-day rolling precision drops below 0.90.
• Data: Add 1,000 recent false positives/negatives; balance by class.
• Augment: Add motion blur and glare.
• Gate: F1 >= 0.92 and calibration error <= 0.03; canary 20% for 24 hours.

Quality gates and metrics

• Detection: mAP@IoU thresholds (e.g., 0.5, 0.5:0.95), class-wise precision/recall.
• Segmentation: Mean IoU, boundary IoU for small objects.
• Classification: F1, per-class recall, calibration (ECE).
• Operational: Latency (p95), throughput, memory/VRAM.

Who this is for

• Computer Vision Engineers moving from notebooks to productionized training.
• ML Engineers owning both model performance and deployment safety.

Prerequisites

• Comfort with training vision models (classification/detection/segmentation).
• Basic understanding of metrics (precision/recall, mAP, IoU).
• Familiarity with version control for code and data.

Learning path

• Before: Reproducible experiments and dataset versioning.
• Now: Continuous training pipeline basics (this lesson).
• Next: Automated evaluation suites, safe rollout strategies, advanced drift detection.

Exercises

Complete these exercises; then take the quick test. You can attempt the test without login; only logged-in users have progress saved.

Exercise 1 — Design a basic continuous training pipeline

Draft a minimal plan for a defect detection model pipeline. Include triggers, data steps, training config, evaluation gates, and deployment strategy.

• Must specify at least one schedule or drift trigger.
• Include 2–3 metrics with thresholds.
• State how you will monitor post-deployment.

Exercise 2 — Define quality gates and rollback

Write the exact gate rules that decide promote/reject, and the rollback criteria if the canary underperforms after deployment.

• Include one accuracy-like metric and one operational metric.
• Define canary duration and traffic percentage.
• Specify alert thresholds that trigger rollback.

Common mistakes and self-check

• Missing data lineage: You cannot reproduce the model. Self-check: Can you list exact dataset IDs, labeling version, and code commit for the last model?
• Leaky splits: Similar frames in train and test inflate metrics. Self-check: Are splits grouped by camera, scene, or time window?
• Metric tunnel vision: Only optimizing mAP but ignoring latency. Self-check: Report accuracy and p95 latency side by side.
• Always-on retraining: Wastes compute. Self-check: Do you have clear drift/performance triggers?
• Deploying without canary: Risky jumps. Self-check: Is there a staged rollout with automatic rollback?

Practical projects

• Home security detector: Build a pipeline that retrains weekly and triggers on night-time brightness drift. Gate with precision/recall and Raspberry Pi latency.
• Packaging OCR updater: Collect low-confidence OCR crops, relabel, retrain monthly; canary 10% traffic for 24 hours with alerting.
• Road sign classifier: Seasonal drift trigger + human review for new sign types; champion–challenger promotion.

Next steps

• Automate evaluation suites with robust, class-balanced test sets.
• Define rollout policies (shadow, canary, blue/green) per risk level.
• Add richer drift monitors (feature, embedding similarity, class mix).

Mini challenge

Your current detector is accurate but slow on a new edge device. Propose a retraining plan that preserves accuracy while enforcing a strict latency gate. Include the trigger, a training tweak (e.g., lighter backbone or quantization-aware training), and your canary plan.

Quick test

Everyone can take the test. Only logged-in users have their progress saved.