Reliability And Operations

1. Pick one service. Define its critical user journeys and SLIs (e.g., request latency < 300ms, availability).
2. Set SLOs (e.g., 99.9% monthly) and compute error budget.
3. Create alerting based on burn rate, not raw thresholds.

1. Draft escalation paths and severities (SEV1–SEV3).
2. Write an incident runbook and practice a 30-minute simulated outage.
3. Adopt a lightweight post-incident review template.

1. Measure current load, peak factors, and growth rate.
2. Forecast next 3–6 months and define headroom (e.g., maintain 30–50%).
3. Create a scaling plan (HPA settings, instance counts, storage growth).

1. Establish a baseline with a representative test (ramp-up, steady, spike).
2. Track p95/p99 latency, error rate, and resource saturation.
3. Find the knee of the curve and record safe throughput.

1. Start with small, contained failures (kill 1 pod, inject latency in staging).
2. Verify failover paths, time-to-recover, and alerts.
3. Document guardrails and abort conditions.

1. Define RPO/RTO for each data store.
2. Automate backups and test restores regularly.
3. Document the DR runbook and run a timed drill.

1. Standardize rollout strategies (canary, blue/green, feature flags).
2. Add preflight checks: health, dependencies, rollback plan.
3. Ship a release using a checklist and capture learnings.

Alert on fast and slow burn rates for a 99.9% availability SLO using request_error_ratio.

# Recording rule example (ratio of errors to total over 5m)
- record: job:http_error_ratio:5m
  expr: sum(rate(http_requests_total{status=~"5.."}[5m]))
        / sum(rate(http_requests_total[5m]))

# Fast-burn alert (~14.4x budget burn)
- alert: HighErrorBudgetBurn
  expr: job:http_error_ratio:5m > (1 - 0.999) * 14.4
  for: 2m
  labels:
    severity: page
  annotations:
    summary: Fast burn rate threatens SLO
    runbook: "Open incident, initiate rollback if recent change"

# Slow-burn alert (~6x budget burn)
- alert: ModerateErrorBudgetBurn
  expr: avg_over_time(job:http_error_ratio:5m[1h]) > (1 - 0.999) * 6
  for: 30m
  labels:
    severity: ticket
  annotations:
    summary: Slow burn rate requires attention
    runbook: "Create task, investigate top offenders"

Tip: Tie alerts to clear actions and escalation paths.

---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 4
  maxReplicas: 12
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-pdb
spec:
  minAvailable: 3
  selector:
    matchLabels:
      app: api

Why: HPA adds capacity during spikes; PDB protects quorum during maintenance or node churn.

import http from 'k6/http';
import { sleep } from 'k6';

export const options = {
  stages: [
    { duration: '2m', target: 50 },  // ramp-up
    { duration: '5m', target: 50 },  // steady
    { duration: '1m', target: 150 }, // spike
  ],
  thresholds: {
    http_req_duration: ['p(95)<300'],
    http_req_failed: ['rate<0.01'],
  },
};

export default function () {
  const res = http.get('https://staging.example/api/health');
  sleep(0.2);
}

Record p95, p99, error rate, and resource saturation to find your safe throughput.

#!/usr/bin/env bash
set -euo pipefail
TS=$(date +%Y%m%d-%H%M%S)
BACKUP_DIR=/backups
DB_URL=${DB_URL:-postgres://app:secret@db:5432/app}

# 1) Create dump
pg_dump --format=custom --file="$BACKUP_DIR/app-$TS.dump" "$DB_URL"

# 2) Verify restore into a temp database
TMP_DB="verify_$TS"
psql "$DB_URL" -c "CREATE DATABASE \"$TMP_DB\";"
pg_restore --exit-on-error --dbname="${DB_URL/app/$TMP_DB}" "$BACKUP_DIR/app-$TS.dump"

# 3) Integrity check (example: count critical tables)
psql "${DB_URL/app/$TMP_DB}" -c "SELECT count(*) FROM users;"

# 4) Cleanup temp DB
psql "$DB_URL" -c "DROP DATABASE \"$TMP_DB\";"

echo "Backup verified: $BACKUP_DIR/app-$TS.dump"

Automate and schedule this. A backup is only as good as your last successful restore test.

Severity levels: SEV1 critical outage, SEV2 partial degradation, SEV3 minor impact.

# Incident roles
- Incident Commander (IC): decision maker, timekeeper
- Communications: status updates, stakeholder sync
- Ops/Tech lead: triage, mitigation

# First 10 minutes checklist
[ ] Declare severity and open an incident channel
[ ] Assign roles (IC, Comms, Tech lead)
[ ] State impact, scope, start time, suspected area
[ ] Mitigate user impact (rollback, feature flag, scale out)
[ ] Update status every 10–15 minutes

# Status update template
Impact: <what users see>
Scope: <regions/services>
Start time: <UTC>
Current action: <mitigation/diagnosis>
Next update: <UTC time>

Fix: Use SLO burn-rate alerts and add minimum durations (for:) to avoid flapping. Tie every page to a clear action.

Fix: Schedule restore verifications. Keep immutable copies and track restore timings against your RTO.

Fix: Plan for peaks and variability. Use p95 demand, add headroom, and account for failover (N+1 or zone loss).

Fix: Prefer canary or blue/green. Automate health checks and rollback triggers.

Fix: Assign Incident Commander, Comms, and Tech lead at declaration. Timebox updates and log a timeline.

• [ ] SLOs defined; alerts routed; dashboards ready
• [ ] Capacity headroom ≥ 30% and HPA configured
• [ ] Rollout strategy and rollback plan documented
• [ ] Backups verified in last 7 days
• [ ] Runbook with triage steps and owners
• [ ] Launch comms and status page templates ready