Observability Stack Basics

Why this matters

As a Machine Learning Engineer, you ship models that must be fast, reliable, and cost-effective in the cloud. Observability lets you answer: Is my service healthy? Why is it slow? Is the model drifting? Without it, debugging becomes guesswork.

• Real tasks you will face: define SLIs/SLOs for inference latency and error rate; investigate spikes in 5xx; track GPU/CPU saturation; correlate a bad release with performance drop; prove that a fix worked.
• Outcome: faster incident resolution, safer deploys, data-driven capacity planning, and trustworthy ML systems.

Concept explained simply

Observability is your system's "flight instruments." It uses telemetry to show what is happening and why.

• Metrics: numeric time series (e.g., requests per second, 95th percentile latency). Great for trends, SLOs, and alerts.
• Logs: structured records of events (e.g., inference result, error details). Great for investigation and audit.
• Traces: a request's path across services, with spans and timing. Great for finding slow hops and bottlenecks.
• Dashboards: curated views of key metrics for humans.
• Alerting: rules that notify on symptoms (latency, error rate) rather than mere causes.

Core building blocks (ML-flavored)

{
  "timestamp": "2026-01-01T12:00:00Z",
  "level": "INFO",
  "service": "ml-infer",
  "model_name": "recommender",
  "model_version": "v23",
  "request_id": "98f7-21ab",
  "trace_id": "4fdc1a3c9d8be321",
  "latency_ms": 142,
  "status_code": 200,
  "user_region": "eu-west",
  "input_schema_version": "1.3",
  "features_used": ["age", "country", "session_length"],
  "prediction": 0.87
}

Worked examples

Hands-on exercises

Do these now. They mirror the listed exercises below and include solutions.

Common mistakes and self-check

• High-cardinality labels (e.g., user_id) explode costs and slow queries. Self-check: ensure labels are coarse (region, model_version), not unique per user.
• No correlation IDs. Self-check: every request log includes request_id and trace_id.
• Unstructured logs. Self-check: JSON with consistent keys and types.
• Noisy alerts (false positives). Self-check: include time windows and thresholds tied to user impact; use multiple windows for burn-rate where applicable.
• Unlimited retention. Self-check: set retention by data type (e.g., metrics 15–30 days, logs 7–14 days, traces sampled).
• Percentiles from averages. Self-check: use histograms for p95/p99; avoid averaging percentiles.

Mini challenge

Scenario: Your recommendation API sees occasional timeouts from one region. In 10 minutes, outline what to check and in what order using metrics, logs, and traces. Keep it to 6 bullet points. Bonus: add one alert you would create to catch this earlier.

Who this is for

• Machine Learning Engineers deploying models to cloud services
• Data/Platform Engineers supporting ML inference pipelines
• SREs collaborating with ML teams

Prerequisites

• Basic cloud deployment knowledge (containers or functions)
• Familiarity with HTTP APIs and status codes
• Basic understanding of time-series metrics and JSON

Learning path

1. This subskill: Observability Stack Basics
2. Next: Deployment and CI/CD for ML services
3. Then: Model monitoring (drift, data quality) and A/B rollout strategies
4. Later: Cost optimization and capacity planning

Practical projects

• Instrument a toy ML inference service with metrics (requests, latency) and add one SLO dashboard.
• Create a structured logging middleware that injects trace_id and model_version.
• Build a demo trace with three spans: feature_encode, predict, postprocess; include attributes for cache_hit and model_version.

Next steps

• Implement at least one SLO in your current service and wire an alert with a runbook.
• Add correlation IDs across your gateway and inference service.
• Review metric labels and remove high-cardinality offenders.

Saving your progress

The quick test below is available to everyone. If you log in, your answers and progress will be saved.