Circuit Breakers Basics

Why this matters

Circuit breakers keep your APIs responsive when dependencies slow down or fail. As an API Engineer, you will:

• Protect upstream services (auth, payments, search) from cascading failures.
• Degrade gracefully with timeouts and fallbacks instead of timeouts piling up.
• Recover quickly by probing a failing dependency safely.

Who this is for

• API engineers and backend developers shipping services that call other services or third‑party APIs.
• Platform/SRE folks designing reliability policies.

Prerequisites

• Basic HTTP/REST and async request handling.
• Know what timeouts and retries are (including exponential backoff).
• Comfort reading pseudocode or simple config examples.

Concept explained simply

A circuit breaker wraps a call to a dependency and watches for failures/slow responses. It has three states:

• Closed: Normal traffic flows. Errors are counted. If failures exceed a threshold in a window, it trips to Open.
• Open: Calls fail fast (no waiting). After a cool-down period, it moves to Half‑Open.
• Half‑Open: A small number of trial requests are allowed. If they succeed, the breaker closes; if not, it goes back to Open.

Key dials you choose:

• Error/slow threshold (e.g., 50% failures or p95 latency > target).
• Minimum number of calls before evaluating (e.g., 20 requests).
• Open state duration (cool‑down) before probing (e.g., 30s).
• Half‑Open probe count (e.g., 1–10 requests).
• Fallback behavior (cached data, default response, or hard fail).

Mental model

Like an electrical breaker, it protects the house (your service) from drawing too much from a failing appliance (dependency). Trip early, stop the damage, then cautiously test if it’s safe again.

Worked examples

Design choices and defaults

• When to measure failure: include timeouts and circuit-open rejections as failures; optionally include slow responses beyond SLO as failures.
• Windows: rolling window by count (e.g., last 50 calls) is simple and stable; time windows (e.g., 10s) work well with high traffic.
• Timeouts + retries + breaker: always set timeouts first; use limited retries with backoff; let the breaker trip if underlying health is bad.
• Isolation: combine with bulkheads (separate thread pools) so a slow dependency can’t consume all resources.
• Telemetry: record state transitions and reasons to help tuning.

Implementation sketch (pseudocode)

// CallWithBreaker(depCall, cfg)
if breaker.state == OPEN and now < breaker.nextProbeAt:
  return fallbackOrFastFail()

if breaker.state == OPEN and now >= breaker.nextProbeAt:
  breaker.state = HALF_OPEN
  breaker.allowedProbes = cfg.halfOpenProbes

result = depCall.withTimeout(cfg.timeout).withLimitedRetries(cfg.retries)
updateWindow(result)

if breaker.state == HALF_OPEN:
  if result.success and result.latency <= cfg.latencySLO:
    breaker.successProbes += 1
    if breaker.successProbes >= cfg.requiredProbeSuccesses:
      breaker.state = CLOSED
      resetMetrics()
  else:
    breaker.state = OPEN
    breaker.nextProbeAt = now + cfg.openDuration
    return fallbackOrFastFail()
else: // CLOSED
  if windowFailureRate() >= cfg.failureThreshold or windowP95() > cfg.latencyThreshold:
    breaker.state = OPEN
    breaker.nextProbeAt = now + cfg.openDuration
    return fallbackOrFastFail()

return result

Self-check checklist

• I set timeouts lower than my client timeout budgets.
• I defined what counts as a failure (errors, timeouts, slow responses).
• I chose thresholds and windows that match traffic volume.
• I have a clear fallback (or intentional fast-fail) per endpoint.
• I log breaker state changes with context (service, reason, window stats).
• I tested Half‑Open behavior under recovery.

Common mistakes and how to self-check

• Too aggressive trips: If your breaker opens during minor blips, increase minimum calls per window or raise thresholds. Check logs for “opened with only few samples”.
• Infinite retries: Limit retries; otherwise you overload the dependency faster.
• No fallback: Decide per endpoint: cached data, default, queued work, or fail fast. Ensure user-visible messages are clear.
• Shared pool exhaustion: Without bulkheads, threads get stuck even if breaker opens late. Use separate pools per dependency.
• Forgetting slow=fail: If users care about latency, count slow responses as failures for breaker evaluation.

Learning path

1. Set client timeouts and small, bounded retries.
2. Add a simple failure-rate breaker on one dependency.
3. Introduce latency-based trips and Half‑Open probes.
4. Add fallbacks and telemetry for state transitions.
5. Combine with bulkheads and load-shedding.
6. Tune thresholds in staging load tests; then ship gradually.

Practical projects

• Project 1: Wrap a slow dummy endpoint with a breaker. Simulate 50% failure and verify Open/Half‑Open transitions.
• Project 2: Add a cache fallback for a product detail call. Measure user-perceived latency before/after.
• Project 3: Build a dashboard panel showing breaker state, open durations, and reasons.

Exercises

Complete these before the quick test. Your answers are not auto-saved unless you are logged in.

• Write your answer as a small config snippet or bullet list.
• Sanity-check: would this trip during normal spikes? Can you explain each dial?

Mini challenge

Pick one critical dependency. Propose Closed→Open criteria, Open duration, and Half‑Open probe rules in three lines. Keep it production-practical.

Next steps

• Integrate breaker metrics into alerts (e.g., “breaker opened > X times in 10m”).
• Run a load test that forces the breaker to open and recover; adjust thresholds.
• Document fallbacks so product and support teams know user impact.

Note: The quick test is available to everyone. Only logged-in users have their progress saved.