luvv to help

Menu

Topic 5 of 8

Monitoring Downstream Dependencies

Learn Monitoring Downstream Dependencies for free with explanations, exercises, and a quick test (for API Engineer).

Published: January 21, 2026 | Updated: January 21, 2026

Table of Contents

Why this matters

Your API rarely works alone. It calls databases, caches, queues, internal microservices, and third‑party APIs. If those dependencies slow down or fail, your users feel it—even when your service code is fine. Monitoring downstream dependencies helps you:

  • Pinpoint the real cause of latency and errors (yours vs. dependency).
  • Trigger safe fallbacks (circuit breakers, cached responses) before users churn.
  • Protect your service with backpressure and timeouts.
  • Escalate to the right owner quickly with clear evidence.

Concept explained simply

Downstream dependencies are the services or systems your API calls to fulfill a request. Examples: a database query, a Redis read, an HTTP call to a payment provider, a message published to Kafka.

Monitoring them means instrumenting your client code and runtime to collect and visualize the Golden Signals per dependency:

  • Latency: P50, P95, P99 of calls.
  • Errors: error rate segmented by type (timeouts, 5xx, 4xx, connection failures).
  • Traffic: request rate (RPS/QPS), payload size if relevant.
  • Saturation: connection pool usage, queue lag, thread saturation, retry bursts.
Mental model

Imagine each dependency as a lane on a highway. Latency is the speed, traffic is the number of cars, errors are accidents, saturation is when too many cars queue for the same lane. Your job is to place speed cameras (timers), counters for accidents (error counters), and sensors for congestion (pool and queue metrics) on each lane you use. Then add signs that trigger when conditions get dangerous (alerts) and detours (fallbacks) when a lane is blocked.

What to measure for each dependency

  • Client-side latency: include status and outcome labels. Example metric names: dependency_request_duration_seconds with labels like dependency=payment_api, status=success|timeout|5xx|4xx|conn_error.
  • Client-side success/error rates: dependency_requests_total and dependency_errors_total, labeled by error cause.
  • Timeouts, retries, circuit breaker state: dependency_timeouts_total, dependency_retries_total, circuit_breaker_state (0=closed,1=open).
  • Saturation: DB connection pool utilization and wait time; queue publish/consume lag; thread/worker utilization.
  • Dependency-specific health: cache hit ratio, consumer lag, HTTP 429 rate, rate-limit headers (if exposed via logs/metrics).
  • Traces: add spans around each dependency call and propagate trace context downstream. Use span attributes for endpoint, method, and error type.

Worked examples

1) Database (Postgres) calls
  • Metrics:
    • db_client_query_duration_seconds (histogram) with labels db=orders, operation=select|insert|update.
    • db_pool_in_use and db_pool_size; db_pool_wait_seconds (time waiting for a connection).
    • db_errors_total by timeout|deadlock|conn_reset.
  • Dashboards:
    • Panel 1: P50/95/99 latency per operation.
    • Panel 2: Error rate by error type.
    • Panel 3: Pool saturation = db_pool_in_use / db_pool_size; pool wait time.
  • Alerts (example):
    • Pool saturation > 0.9 for 5 minutes.
    • Query P95 > 200 ms for 15 minutes while traffic steady (avoid alerting on load spikes without capacity context).
2) Third‑party payment API
  • Metrics:
    • http_client_request_duration_seconds with labels dependency=payment_api, route=/charge, status=2xx|4xx|5xx|timeout|conn_error.
    • dependency_retries_total with reason=timeout|5xx|rate_limit.
    • circuit_breaker_state for payment_api.
  • Fallbacks:
    • If circuit opens, queue non‑urgent payments and show "processing" status.
    • For idempotent operations, retry with bounded backoff.
  • Alerts:
    • Error budget burn: if 5% of charges fail for 10 minutes and 20% for 2 minutes (multi-window burn).
    • Rate-limit spike: 429 rate > 1% sustained.
3) Kafka (publish/consume)
  • Metrics:
    • kafka_publish_errors_total, kafka_publish_duration_seconds.
    • consumer_lag per topic/partition and end_to_end_latency_seconds (produce-to-process).
  • Dashboards: producer latency and error rate; consumer lag heatmap; end-to-end latency percentiles.
  • Alerts: consumer lag > SLO threshold (e.g., > 1 minute for 10 minutes), or publish 5xx/timeouts burst.
4) Redis cache
  • Metrics: cache_hit_ratio, cache_get_duration_seconds, conn_errors_total.
  • Fallbacks: if cache down, bypass to DB with tighter timeouts and reduced concurrency.
  • Alert: hit ratio drops >= 20% from baseline while DB latency rises (paired alert to catch thundering herd).

How to instrument dependencies (step-by-step)

  1. Wrap each dependency call with a timer and a span. Label by dependency name, operation, and outcome.
  2. Record errors by type: timeout, connection, 4xx, 5xx, rate_limit, canceled.
  3. Emit saturation metrics: pool in-use/size, queue lag, worker concurrency, retry counts.
  4. Set timeouts and bounded retries; expose retry and circuit-breaker metrics.
  5. Propagate trace context to downstream if supported; otherwise at least tag requests with a correlation ID.
  6. Build dashboards per dependency with Golden Signals panels and add SLO-based alerts.

Dashboards and alerting that reduce noise

  • Use SLOs per dependency: e.g., "99.5% of payment calls under 800 ms and success rate ≥ 99.0% during business hours."
  • Alert on error budget burn (multi-window) instead of single spikes.
  • Group alerts by dependency and surface top offenders: endpoints, regions, or tenants.
  • Add runbooks: what the alert means, common causes, immediate mitigations, owners to page.

Common mistakes and how to self-check

  • Mistake: Only server-side metrics, no client-side view. Self-check: Can you see a dependency's latency from your service perspective with P95/P99? If not, add client timers.
  • Mistake: Aggregating all dependencies into one metric. Self-check: Are metrics labeled by dependency and operation? If not, split them.
  • Mistake: Alerting on any 5xx. Self-check: Are alerts tied to user impact via SLO and sustained windows? If not, refactor alerts.
  • Mistake: Infinite retries without backoff. Self-check: Are retries bounded and instrumented? Do you see retry storms in metrics?
  • Mistake: Ignoring saturation. Self-check: Do you monitor pool wait time, queue lag, and circuit state? Add them if missing.

Exercises

Do them now. They are short and practical.

Exercise 1: Define SLIs/SLOs and alerts for a payment provider

You call payment_api for charges. Define 3 SLIs with targets and actionable alerts.

  • Deliverables: 3 SLIs, SLO targets, alert triggers.
  • Tip: tie alerts to user-impacting burn, not single spikes.
Hints
  • Golden Signals: latency, errors, saturation.
  • Consider rate limits (429) separately from 5xx.
  • Use multi-window burn (fast + slow) for alerts.

Exercise 2: Instrumentation plan for DB + Cache

Your service uses Postgres and Redis. List the exact client-side metrics you will emit, including labels and histogram buckets.

  • Deliverables: metric names, types, labels, buckets, and any saturation gauges.
  • Tip: include error types and operation names.
Hints
  • One histogram per dependency call type.
  • Label by operation and outcome.
  • Add pool and hit-ratio saturation signals.

Self-check checklist

  • I can attribute a slow user request to the exact dependency span.
  • I have P50/95/99 latency and error rate per dependency and operation.
  • Timeouts, retries, and circuit states are visible.
  • Saturation metrics exist: pool usage, queue lag, worker concurrency.
  • Alerts map to SLOs and include a short runbook note.

Mini challenge

Pick your noisiest alert from the last week and rework it into an SLO burn alert with two windows (e.g., 2% over 1 hour and 10% over 10 minutes). Add one saturation condition that suppresses alerts during maintenance windows. Write a one-paragraph runbook entry.

Practical projects

  • Build a "Dependency Health" dashboard: one row per dependency, panels for latency, error rate, and saturation.
  • Implement a circuit breaker around one critical dependency and expose its state as a metric and trace attribute.
  • Add end-to-end latency tracing for a request that touches DB, cache, and a third-party API; verify spans and timings.

Who this is for

  • API Engineers, Backend Engineers, SREs working with microservices or integrations.

Prerequisites

  • Basic HTTP, databases, and message queues knowledge.
  • Familiarity with metrics and tracing concepts.

Learning path

  • Start with Golden Signals basics.
  • Instrument client-side dependency metrics.
  • Create SLOs and SLO-based alerts.
  • Add fallbacks and validate via load tests.

Next steps

  • Finish the exercises, then take the quick test below.
  • Pair with a teammate to review one dependency dashboard for clarity and actionability.
  • Pilot SLO-based alerting for one dependency before rolling out broadly.

About the quick test

The quick test is available to everyone. If you are logged in, your progress will be saved automatically.

Practice Exercises

2 exercises to complete

Instructions

You call a third-party payment_api for charges. Define three SLIs (latency, success rate, saturation/rate limit), set SLO targets, and propose actionable alerts using multi-window error budget burn. Keep user impact in mind.

Expected Output
Three SLIs with precise definitions, SLO targets, and at least two alert conditions tied to burn rates or sustained degradation.
Distributed Tracing BasicsDashboards For Endpoint Health

Have questions about Monitoring Downstream Dependencies?

AI Assistant

Ask questions about this tool