Regression Test Sets

Who this is for

• NLP engineers building or maintaining models in production.
• ML practitioners who need stable quality during frequent model retrains.
• QA/DS collaborators defining guardrails for model behavior.

Prerequisites

• Basic NLP task knowledge (classification, NER, QA, summarization).
• Comfort with standard metrics (Accuracy/F1, ROUGE/BLEU/BERTScore).
• Familiarity with dataset versioning and reproducible evaluation.

Learning path

1. Understand the purpose of regression test sets.
2. Learn test types: invariance, directional expectation, minimal pairs, slices.
3. Define acceptance criteria: metric deltas and pass/fail rules.
4. Version and automate your test set.
5. Iterate after failures: triage, fix, expand tests.

Why this matters

In real NLP work, you will retrain, fine-tune, or swap models. A regression test set prevents silent quality drops. Typical on-the-job tasks:

• Protect critical behaviors: e.g., sentiment must flip on negation, PII must be detected, instructions must be followed.
• Enforce non-regression on key user journeys: names recognized, dates normalized, labels stable for compliance.
• Catch drift or tokenization changes that break rules-based post-processing.

Concept explained simply

A regression test set is a small, curated dataset with expected behaviors. Every time the model changes, you run the set and compare results. If a behavior breaks or a metric drops beyond a threshold, the build fails.

Core test types

• Invariance tests: Output should not change under harmless edits (punctuation, casing, whitespace, synonyms, reorderings that keep meaning).
• Directional expectation tests (DET): Output should predictably change with meaning-altering edits (add "not", swap entities, change tense when required).
• Minimal pairs: Two near-identical inputs differing by one factor; expected outputs reveal sensitivity or invariance.
• Slice tests: Evaluate performance on focused subsets (e.g., short texts, rare entities, specific languages, domain-specific terms).
• Challenge sets: Hard, failure-prone examples (negation, coreference, sarcasm, long context).

Acceptance criteria (pass/fail rules)

• Aggregate metric guardrail: e.g., "F1 must not drop more than 0.5 points from baseline."
• Slice guardrails: e.g., "Location NER F1 ≥ 0.90; rare names F1 drop ≤ 1 point."
• Behavioral tests: e.g., "Adding 'not' flips sentiment label in ≥ 95% of minimal pairs."
• Determinism strategies: fix random seeds or average results over N runs for stability.

How to build a regression test set (step-by-step)

1. List critical behaviors: What must never break? (negation, PII redaction, date parsing)
2. Draft minimal examples: 5–20 per behavior to start; keep texts short and focused.
3. Write expected outcomes: Exact labels, spans, or metric thresholds.
4. Organize slices: Tag items (e.g., domain=medical, entity=PERSON, language=en).
5. Baseline and version: Save expected predictions/metrics with a version tag and notes.
6. Automate evaluation: Run on every model change; produce a clear pass/fail summary.
7. Iterate: When a failure is legit improvement (e.g., better tie-breaking), update expectations with justification.

Worked examples

Example 1: Sentiment classification (invariance + directional)

• Invariance pair: "Great service!" vs. "Great service." Expected: same positive label.
• Directional pair: "Great service" vs. "Not great service" Expected: positive → negative flip.
• Slice: short reviews under 10 words must keep ≥ 95% accuracy.

Example 2: Named Entity Recognition (slice + challenge)

• Slice: PERSON names with diacritics ("Łukasz", "José") F1 ≥ 0.90.
• Invariance: Extra whitespace or capitalization changes should not alter spans.
• Challenge: Overlapping entities in "Dr. Jane Miller of Miller Labs" must detect PERSON and ORG correctly.

Example 3: QA / extraction (minimal pairs)

• Question: "Who wrote 1984?" Context: mentions George Orwell once. Expected: "George Orwell".
• Minimal pair: Add distracting author names in context; expected answer unchanged.
• Guardrail: Exact-match ≥ 85% on single-hop questions; no more than 5% empty answers on answerable items.

Common mistakes and self-check

• Only tracking aggregate metrics. Self-check: Do you have at least 3 critical slices?
• Unstable tests due to randomness. Self-check: Did you fix seeds or average across runs?
• Tests too broad. Self-check: Are examples minimal and focused on one behavior?
• Letting the test set drift silently. Self-check: Are changes to expectations reviewed and logged with reasons?
• Ignoring false positives. Self-check: Do you manually review failures before reverting a model?

Practical projects

• Create a 100-item regression suite for your current NLP task: 40 invariance, 40 directional, 20 challenge items.
• Build a slice dashboard: per-slice metrics and pass/fail badges.
• Introduce automated reports that summarize deltas vs. baseline and top failing examples.

Exercises

These mirror the tasks below in the Exercises section of this page. Complete them and check your work in the collapsible solutions.

• Exercise 1: Sentiment regression micro-suite.
• Exercise 2: NER acceptance criteria and versioning notes.

Regression test set checklist

• Behaviors defined and prioritized.
• Minimal, labeled examples for each behavior.
• At least 3 slices with guardrails.
• Baseline results saved with version tag.
• Pass/fail rules documented and automated.
• Process for updating expectations after legitimate improvements.

Mini challenge

Pick one model bug you have recently seen. Write 3 minimal pairs that would have caught it, and add them to your regression set with clear expected outcomes. Re-run and verify the bug would now be flagged.

Quick Test access

You can take the quick test below to check your understanding. Available to everyone; only logged-in users get saved progress.