Why this matters
Key terms
- Blue/Green: two identical environments; switch all traffic by flipping a route.
- Canary: route a small percentage to the new version and monitor.
- Shadow: send a copy of traffic to the new version but do not serve its results to users.
- Feature flag: dynamically enable/disable a feature or model without redeploying.
- Versioned endpoints: separate URLs like /v1 and /v2 to isolate changes.
Core rollback patterns for NLP services
- Immutable artifacts: models, tokenizers, and prompts are versioned and read-only after publish.
- Backward-compatible changes: add fields to schemas; do not change meaning of existing fields.
- Blue/Green or versioned endpoints: keep v1 warm and routable at all times.
- Canary rollout with guardrails: define thresholds for p95 latency, 5xx rate, accuracy proxy, and safety metrics (toxicity/PII leakage).
- Dual-write during migrations: write to both old and new indexes/stores so you can revert without data loss.
- Config-first rollback: separate model weights from runtime configs (thresholds, prompts) so you can revert configs instantly.
Rollback triggers and guardrails
Set objective thresholds before rollout. If any breach occurs within the observation window, rollback immediately.
- Reliability: p95 latency +30% vs baseline, 5xx rate above threshold, queue backlog growth.
- Quality proxies: CTR drop for recommendations, increase in invalid extractions, lower exact match on sampled evals.
- Safety: toxicity or bias metrics exceed limits, PII redaction failure rate increases, jailbreak detection triggers.
Example guardrails for a classification API
- p95 latency <= 300 ms
- 5xx < 0.2%
- Agreement with shadowed baseline >= 98% on random sample
- Toxicity false negative rate change <= +0.5%
Worked examples
1) Sentiment model canary spikes harmful outputs
Situation: v2 introduces a new tokenizer. Canary at 10% shows a 1.5% increase in toxicity false negatives.
- Freeze canary at 10%.
- Route traffic back to v1 (100%).
- Disable v2 feature flag.
- Investigate tokenizer vocabulary mismatch and retrain calibration.
- Re-run shadow test before next attempt.
Why this works
Immediate traffic cut plus config disable avoids further exposure while retaining logs for debugging.
2) QA service migrates vector index
Situation: You plan to switch to a new embedding model and index layout.
- Dual-write embeddings to old and new indexes.
- Shadow-query the new index and compare hit quality.
- Canary route 5% of read traffic to new index.
- Trigger: recall drops > 2% on sampled questions. Rollback by routing reads to old index; keep dual-write.
- Post-rollback: analyze query analyzer differences; fix chunking or normalization.
Key insight
Dual-write keeps rollback safe because you never lose data during back-and-forth switches.
3) LLM summarizer increases latency
Situation: v2 uses a larger model with longer max tokens; p95 latency breaches SLO during peak.
- Lower max tokens via config. If not enough, disable v2 with feature flag.
- Route all traffic to v1; drain in-flight requests.
- Warm a smaller v2b model; run shadow tests.
Extra tip
Make prompt and decoding settings (temperature, top_p, max_tokens) rollbackable via config so you can fix latency without replacing the model.
Step-by-step: Build a rollback plan
- Define baselines: record stable v1 metrics for latency, error rate, accuracy proxies, safety metrics.
- Set guardrails: exact thresholds and observation windows for canary and post-switch.
- Prepare controls: versioned endpoints, feature flags, traffic router, warm standby.
- Data safety: dual-write or snapshot before migrations; schema changes must be additive.
- Runbook: short, unambiguous steps with owners, commands, and verification checks.
- Drills: practice dry-runs with shadow traffic; time your rollback.
Minimal rollback checklist
- Stable version routable and warm
- Feature flag to disable new model
- Config store with version history
- Dual-write enabled for stateful changes
- Verification queries and dashboards prebuilt
- Communication template for stakeholders
Exercises
Complete the two exercises below. They mirror the items in the Exercises section of this page. Use the checklists to verify your work.
- Exercise 1: Draft a one-page rollback runbook snippet for an NLP API.
- Exercise 2: Simulate a canary decision using provided metrics and justify rollback or proceed.
Self-check after exercises
- Does your runbook include clear triggers, owners, and a verification step?
- Did you consider tokenizer/embedding/version compatibility?
- Is the canary decision grounded in predefined guardrails, not intuition?
Common mistakes and how to self-check
- Mistake: Rolling back code but not configs. Self-check: Are prompts/thresholds/versioned and reverted too?
- Mistake: Incompatible tokenizers. Self-check: Assert tokenizer and model versions together in the request path.
- Mistake: Index migration without dual-write. Self-check: If you revert reads, do you lose writes? If yes, fix the plan.
- Mistake: No safety metrics. Self-check: Include toxicity/PII/bias guardrails alongside latency.
- Mistake: Unwarmed standby. Self-check: Is v1 pod count sufficient for immediate switch?
Practical projects
- Build a blue/green deployment demo for a text classification API with a one-click rollback using feature flags.
- Implement dual-write and read-switch for a small vector DB; include a scripted rollback.
- Create a safety metrics dashboard (toxicity/PII) and wire it to automated canary halt criteria.
Mini challenge
Your new summarizer shows a 0.8% increase in PII leaks during canary while latency improves. What is your move?
Suggested approach
- Immediate rollback: safety breach overrides latency gains.
- Investigate: strengthen redaction prompt and threshold, add upstream PII filter, re-canary.
Learning path
- Observability for ML: request tracing, structured logs, and SLOs.
- Traffic shaping: canary, shadow, and weighted routing.
- Data contracts and schema evolution for NLP pipelines.
- Vector database operations: re-indexing, dual-write, and cutover.
- Prompt/config management with versioning and audit trails.
Next steps
- Write your rollback runbook and practice a 10-minute drill.
- Configure safety guardrails and alerts for your next rollout.
- Take the Quick Test below. Anyone can take it; only logged-in users have saved progress.
Progress and test
The Quick Test is available to everyone. If you log in, your progress will be saved so you can resume later.