Incident Triage

Why this matters

In API engineering, incidents happen: error spikes, latency regressions, dependency outages, bad deploys. Fast, calm triage protects customers and buys time to fix. Strong triage reduces downtime, confusion, and duplicate work.

• Real tasks: classify severity, stabilize impact, route work to the right owners, communicate clear updates, and gather the minimum data to decide the next step.
• You will use runbooks, alerts, dashboards, logs, traces, rollbacks, feature flags, rate limits, and incident roles.

Who this is for

• API Engineers and on-call developers who respond to production issues.
• Platform/SRE teammates who coordinate incident response.
• Team leads who need consistent, low-drama incident handling.

Prerequisites

• Basic understanding of your service topology and dependencies (API, DB, cache, message bus).
• Access to metrics, logs, and traces in your observability stack.
• Familiarity with your alerting, paging, and rollback/feature-flag tooling.

Concept explained simply

Triage is the first 15–30 minutes of an incident. Your goal is not to find deep root cause. Your goal is to stop the bleeding, size the problem, and point the right people in the right direction with clear updates.

Mental model: STABLE

• S – Stabilize: Reduce impact quickly (rollback, disable feature, rate limit, fail open/closed).
• T – Triage: Set severity, declare roles, and open an incident channel.
• A – Assess: Collect key signals (metrics/logs/traces) to form a first hypothesis. Timebox to 10 minutes.
• B – Bound: Narrow the blast radius (is it one endpoint, one region, one rollout group?).
• L – Loop: Communicate status and ETAs at a steady cadence (e.g., every 15 minutes).
• E – Escalate/Execute: Pull in owners and run runbooks or mitigations.

Step-by-step triage flow (15–30 minutes)

1. Declare: Name incident, set severity, create a shared channel, assign roles (Incident Lead, Comms, Ops). If solo, you are all roles.
2. Stabilize now: If a recent change correlates, consider rollback/disable flag. If traffic overwhelms capacity, apply rate limits or temporary degradation (e.g., serve cached responses).
3. Timebox data gathering: 10 minutes to check dashboards, top logs, and traces. Capture 3–5 facts (metrics trend, timeline, change list, scope).
4. Bound blast radius: Identify which endpoints/regions/tenants are affected. Redirect traffic if possible.
5. Update cadence: Post a brief update: What we know, What we’re doing, Next update time.
6. Call owners: Page service owners of the suspected component. Share timeline and current mitigation.

Worked examples

Exercises

These mirror the exercises below. Try first, then open the solution.

Self-check checklist

• You can state a severity with a short, clear justification.
• You propose a mitigation that reduces impact within 10 minutes.
• You identify whether it is global or scoped (endpoint/region/tenant).
• Your status update mentions impact, action, and next update time.

Common mistakes and how to self-check

• Hunting root cause too early: Ask, "Does this reduce impact now?" If not, defer.
• Unclear ownership: Assign a lead and specific owners for each action.
• Silent investigation: Post updates on a cadence even if there’s no news.
• Over-mitigating: Prefer reversible, scoped mitigations; avoid platform-wide changes unless necessary.
• Missing the timeline: Keep a lightweight log of actions to avoid loops and aid handoffs.

Practical projects

• Create a one-page runbook template: severity, roles, first 10-minute checks, rollback steps, status update snippet.
• Build a dashboard for the golden signals of one service (error rate, p95 latency, traffic, saturation) with deploy markers.
• Set up a safe game day: trigger a controlled failure in staging, practice STABLE, and time your first mitigation.

Learning path

• Start: Incident triage (this page) – stabilize first, then investigate.
• Next: Alert design and SLOs – alert on symptoms, not just causes; define acceptable error/latency.
• Then: Post-incident reviews – capture learnings and improve runbooks, alerts, and tests.
• Later: Resilience patterns – circuit breakers, backpressure, bulkheads, and graceful degradation.

Mini challenge

In the last 4 minutes, error rate is normal, but p95 latency doubled for write endpoints only. Cache hit rate is fine; DB write IOPS maxed. No deploys in 30 minutes. What is your STABLE plan in 5 bullet points?

Next steps

• Save your runbook template where on-call can find it.
• Practice with a teammate: 10-minute drills using the examples above.
• Take the Quick Test below. Everyone can take it for free; if you’re logged in, your progress will be saved.