Why this matters
Labeling can be the most expensive part of an ML project. Active learning helps you choose which data to label next so your model improves faster with fewer labels.
- Reduce labeling cost while hitting accuracy targets.
- Speed up experiments: prioritize uncertain or informative samples.
- Handle long-tail and rare classes more effectively.
- Make a repeatable, auditable data acquisition plan.
Concept explained simply
Active learning is a learn-ask-repeat loop. The model learns from current labeled data, then asks humans to label the most useful unlabeled items. Repeat until performance or budget target is reached.
Mental model: Curiosity + Coverage + Cost
- Curiosity (uncertainty): label where the model is unsure.
- Coverage (diversity/representativeness): avoid labeling many near-duplicates.
- Cost: prefer cheaper items when value is similar.
You balance these three forces with a scoring rule and a batching strategy.
The core loop (step-by-step)
- Train: Fit a baseline on what you have (even a small seed set).
- Score pool: For each unlabeled item, compute uncertainty (e.g., entropy), diversity/novelty (e.g., distance to labeled set), and optional cost.
- Select batch: Rank by a combined score (e.g., 0.7×uncertainty + 0.3×novelty) and pick a batch.
- Label: Send the batch to annotators with clear instructions and quality checks.
- Retrain and evaluate: Track a fixed validation set metric (e.g., F1) and calibration.
- Decide: Stop, continue, or adjust strategy based on gains.
Tips for cold start
- Use small random seed labels (e.g., 50–100) to train a weak model.
- Or use diversity-only selection for the first batch.
- Or use embeddings from a pretrained model to estimate novelty before any labels.
Batch size guidance
- Smaller batches (10–50) adapt faster but require more coordination.
- Larger batches (100–1,000) are efficient operationally but less adaptive.
- Start small; grow as model stabilizes.
Acquisition strategies you should know
- Uncertainty sampling: pick lowest confidence predictions.
- Least confident: 1 − max class probability.
- Margin: difference between top two class probabilities (smaller is more uncertain).
- Entropy: −∑ p log p.
- Query by Committee (QBC): train multiple models; pick items with highest disagreement.
- Diversity/representativeness: cover the space with methods like k-center greedy or farthest-first traversal in embedding space.
- Hybrid: combine uncertainty and diversity (and optional cost) into one score.
When to use what
- Few labels or high redundancy: emphasize diversity.
- Model already decent: emphasize uncertainty to chase decision boundary.
- Rare classes: add class-aware sampling or cost weighting.
Worked examples
Example 1 — Text classification: uncertainty + diversity
Unlabeled items A–E have model positive probabilities and novelty scores (0–1, higher is more novel):
- A: p=0.51, novelty=0.50
- B: p=0.55, novelty=0.20
- C: p=0.80, novelty=0.60
- D: p=0.50, novelty=0.40
- E: p=0.60, novelty=0.90
Define uncertainty as u = 1 − |p − 0.5|. Score s = 0.7u + 0.3 novelty.
- A: u=0.99 → s≈0.843
- B: u=0.95 → s≈0.725
- C: u=0.70 → s≈0.670
- D: u=1.00 → s≈0.820
- E: u=0.90 → s≈0.900
Top-2 to label: E and A (then D).
Example 2 — Images: balancing uncertainty with class imbalance
Binary defect detection where positives are rare. If you only use uncertainty, you may oversample easy negatives near 0.5. Add a class-aware prior or reweight uncertainty so suspected positives (p close to 0.5 but with high anomaly score) are prioritized. Alternatively, use QBC trained on class-weighted models to increase disagreement on potential positives.
Example 3 — NER: token-level disagreement
Sequence labeling can use uncertainty at token or span level. Compute span entropy or committee vote entropy per token, then aggregate per sentence (e.g., max or mean). To avoid long-sentence bias, normalize by length and add a diversity term so you do not keep labeling the same entity types repeatedly.
Offline simulation for planning
If you have a labeled historical pool, simulate active learning by hiding labels and revealing them batch-by-batch according to your strategy. Track curves of validation F1 vs. labels used. This helps you pick batch sizes, strategy weights, and stopping rules before spending real budget.
Common mistakes and self-checks
- Mistake: Using accuracy on a shifting validation set. Fix: Keep a fixed, stratified validation set.
- Mistake: Oversampling near-duplicates. Fix: Add diversity/novelty via embeddings.
- Mistake: Ignoring label quality. Fix: Add gold checks, consensus, or spot audits.
- Mistake: Huge batches early. Fix: Start small; adjust after each round.
- Mistake: No clear stop rule. Fix: Define performance/budget thresholds and plateau criteria.
Self-check
- Can you explain why your chosen items are informative (uncertainty) and not redundant (diversity)?
- Do you have a written stopping rule before you start labeling?
- Is your validation set stable and representative?
Exercises
These exercises mirror the tasks below. The quick test is available to everyone; only logged-in users will have progress saved.
Exercise 1 — Score and pick a batch
Use u = 1 − |p − 0.5| and s = 0.7u + 0.3 novelty. Items:
- A: p=0.51, novelty=0.50
- B: p=0.55, novelty=0.20
- C: p=0.80, novelty=0.60
- D: p=0.50, novelty=0.40
- E: p=0.60, novelty=0.90
Pick the top 2 items to label and show your calculations.
Exercise 2 — Batch size and stopping
Validation F1 by round: 0.62 → 0.69 → 0.72 → 0.725. Mean prediction entropy: 0.42 → 0.39 → 0.38. Propose: (a) a concrete stopping rule, and (b) the next batch size and selection tweak if you continue.
Checklist before you submit
- Included both uncertainty and novelty in your reasoning.
- Explained a measurable stopping rule (plateau and/or budget).
- Mentioned batch size rationale and any switch to diversity.
Practical projects
- Project 1: Simulate active learning on a small labeled dataset. Compare random vs. uncertainty vs. uncertainty+diversity. Plot F1 vs. labels used.
- Project 2: Build a simple acquisition function service that outputs top-N items given model scores and embeddings.
- Project 3: Create labeling guidelines and a quality checklist. Run a tiny pilot batch and measure inter-annotator agreement.
Who this is for
- Applied Scientists and ML Engineers managing labeling budgets.
- Data Scientists improving models with limited labels.
Prerequisites
- Basic supervised learning (train/validation/test).
- Understanding of probabilities and softmax outputs.
- Ability to compute simple statistics from model outputs.
Learning path
- Start here: active learning basics and the acquisition loop.
- Next: label quality management, annotation tooling, and class imbalance strategies.
- Then: offline simulation and evaluation design; production rollout and monitoring.
Next steps
- Do the exercises above and take the quick test.
- Run a 1–2 round pilot on your data with a small batch and a clear stop rule.
- Document your acquisition function and rationale.
Mini challenge
Design a 3-round plan for your dataset: pick a batch size, acquisition function (with weights), a validation metric, and a stopping rule. Write one paragraph justifying each choice.