Monitoring And Incident Response For Prompt Failures

Who this is for

This lesson is for Prompt Engineers and ML/AI practitioners who need reliable production behavior from LLM-powered features: chat assistants, content generation, retrieval-augmented apps, and tool-using agents.

Prerequisites

• Basic prompt design and versioning
• Understanding of LLM responses, tokens, latency, and cost
• Familiarity with logging concepts and simple alert thresholds

Why this matters

In real products, LLMs fail in ways that upset users and teams: refusals after model policy changes, hallucinations, broken tool calls, and safety violations. Monitoring and incident response minimize user harm, downtime, and cost.

• Real task: Detect and stop a surge of harmful outputs after a model update.
• Real task: Triage a sudden spike in tool-call errors due to schema changes.
• Real task: Roll back a prompt version when quality/regulatory metrics dip.

Concept explained simply

Monitoring watches signals that matter (quality, safety, reliability). Incident response is the playbook you run when those signals cross thresholds. Together they form your safety net.

Mental model: D-T-M-L Loop

• Detect: Observe leading and lagging indicators.
• Triage: Understand what failed and how bad it is.
• Mitigate: Contain user impact quickly.
• Learn: Add guards, tests, and prompts to prevent repeat issues.

What counts as a prompt failure

• Safety violation: toxic content, PII leakage, disallowed advice.
• Hallucination: confident but false statements.
• Refusal/overrefusal: model declines when it should answer.
• Schema mismatch: tool/function output fails validation.
• Context misuse: ignores system rules, follows user-injected instructions.
• Latency/timeout: response too slow or fails to complete.
• Cost spike: abnormal token usage per request.

Signals and metrics to monitor

Track a balanced set of safety, quality, and reliability indicators. Start with these:

• Safety violation rate: Percent of outputs flagged by a moderation pass or rules.
• Hallucination proxy rate: Share of answers failing self-check prompts or citation checks.
• Refusal rate: Model declines when it should answer (overrefusal).
• Schema/tool error rate: Percentage of tool calls failing validation or throwing errors.
• Output validation failures: JSON schema errors, missing fields, wrong types.
• Latency p95/p99: Slow tails are user-visible.
• Token drift: Sudden increase in input or output tokens per request.
• User correction rate: Ratio of “that’s wrong” reactions or immediate re-asks.
• Guardrail triggers: Injection detected, profanity, policy risk flags.

Set actionable thresholds. Example:

• Sev1: Safety violation rate > 0.3% for 10 minutes or any critical policy breach.
• Sev2: Tool-call error rate > 5% for 15 minutes or hallucination proxy rate > 4%.
• Sev3: Refusal rate > 8% or p95 latency > 8s for 15 minutes.

Minimal monitoring setup

You can start simple and grow:

• Log per-request: timestamp, prompt_id, prompt_version, user_segment (non-PII), model, input_length, output_length, latency_ms, outcome (ok/refusal/error), safety_flags, validation_errors, tool_call_status, feedback (thumbs_up/down), session_id (hashed).
• Aggregate dashboards: rolling rates for safety violations, refusals, schema errors, latency percentiles, token usage.
• Alerts: email or chat for Sev1/Sev2 thresholds.

Safety note: Never log secrets or raw PII. If you must, use redaction and hashing. Store minimum necessary and set retention limits.

Incident response playbook

1. Confirm incident (Detect): Verify metric breach and sample affected logs.
2. Classify severity (Triage): Sev1/2/3 using pre-defined criteria.
3. Contain (Mitigate): Options:
   • Feature-flag off the affected capability.
   • Rollback prompt_version or switch to backup model.
   • Tighten system prompt: add “must-cite-or-decline” or stricter JSON schema.
   • Enable stricter moderation filters or reduce temperature.
4. Diagnose root cause (Triage): Check recent changes: prompts, models, tools, data, traffic.
5. Patch and verify (Mitigate): Ship a small fix, test on a canary segment, then expand.
6. Retrospective (Learn): Document cause, impact, changes required, and add tests/monitors.

Worked examples

Example 1 — Sudden refusals after model policy shift

• Detect: Refusal rate jumps from 2% to 15% in 10 minutes.
• Triage: A/B shows only new prompt_version affected.
• Mitigate: Rollback to previous version. Reduce temperature by 0.2.
• Fix: Rephrase system rule to explicitly allow the target use-case while staying compliant. Add test prompts to catch overrefusal.

Example 2 — Tool-call schema change breaks flows

• Detect: Schema validation failures spike to 12% after a backend release.
• Triage: Tool returns field price_cents instead of priceUSD.
• Mitigate: Temporarily disable tool use for the capability; fallback to safe text response.
• Fix: Update function-call schema in prompt and parser; add contract test and alert on schema mismatch > 1%.

Example 3 — Hallucinated pricing in product answers

• Detect: Hallucination proxy rises to 6% in ecommerce Q&A.
• Triage: Answers lack citations; RAG retrieval is occasionally empty.
• Mitigate: Enforce cite-or-decline template and increase retrieval top-k.
• Fix: Add self-check: “Verify every price matches retrieved context.” If check fails, respond with “I don’t have that information.” Add nightly evals on price questions.

Instrumentation essentials

• Request context: prompt_id, prompt_version, system_prompt_hash, model_name, temperature, tools_enabled.
• Input/output: token counts only (avoid raw PII), redaction markers if applied.
• Outcomes: safety_flags[], validation_errors[], tool_call_status, retries, chosen mitigation (if any).
• User signals: thumbs_up/down, correction provided (boolean), segment (non-PII).
• Trace ids: request_id, session_id (hashed) for correlation.

Retention: keep detailed logs short-lived; aggregate metrics longer. Limit access to sensitive logs.

Runbooks and on-call basics

• Severity and RTO targets:
  • Sev1 (user harm risk): Acknowledge < 5 min, mitigate < 15 min.
  • Sev2 (business impact): Acknowledge < 15 min, mitigate < 60 min.
  • Sev3 (quality drift): Acknowledge within day, fix in next release.
• Communication: incident channel, concise status notes, final postmortem.
• Change freeze after incident until cause is understood.

Exercises

Self-check checklist

• [ ] Each metric has a clear purpose and owner
• [ ] Thresholds are numeric and time-bounded
• [ ] There is a defined containment action for each Sev level
• [ ] Rollback criteria are explicit
• [ ] Post-incident learning steps are included

Common mistakes and how to self-check

• No clear thresholds: If alerts are vague, you will ignore them. Add numeric cutoffs and time windows.
• Logging raw sensitive data: Redact and hash; store minimum necessary.
• Only lagging indicators: Add leading signals like token drift and guardrail triggers.
• Skipping canaries: Ship fixes to a small segment first.
• No rollback plan: Write exact conditions and steps now, not during an incident.

Practical projects

• Build a canary dashboard: Compare new vs old prompt_version across 5 core metrics.
• Create a cite-or-decline guardrail and measure hallucination proxy rate change.
• Write a JSON schema for your tool outputs and log validation errors with alerts.

Learning path

• Start: Basic prompt versioning and A/B comparisons.
• Next: Add logging fields and a simple dashboard for core metrics.
• Then: Implement alert thresholds and a rollback runbook.
• Advanced: Automated self-check prompts, offline evals, and incident retros.

Next steps

• Finish Exercise 1 and refine with your team’s requirements.
• Set initial thresholds and run a simulated incident drill this week.
• Take the quick test to confirm your understanding. Everyone can take it; log in to save progress.

Mini challenge

You observe a 3x spike in token usage with stable traffic and no code deploys. In two sentences, state your top hypothesis and first mitigation step. Then list one log field you’ll add to confirm cause.