What you will learn
How to design, run, and automate performance regression tests that catch slowdowns in APIs before they reach production. You will set meaningful baselines and thresholds, reduce noise, and wire tests into CI so that bad performance is blocked early.
Who this is for
- API Engineers who ship endpoints and microservices.
- QA/SET engineers adding performance gates to CI.
- SRE/Platform engineers enforcing SLOs and capacity plans.
Prerequisites
- Basic HTTP or gRPC knowledge.
- Familiarity with CI runs and environment variables.
- Comfort with reading charts and percentiles (p50, p95, p99).
Why this matters
- Real task: Your new pagination change increases p95 latency by 40%. A regression test fails the merge and saves a costly incident.
- Real task: A library upgrade reduces throughput by 10%. Regression tests highlight the drop before a marketing campaign drives traffic.
- Real task: A hot code path adds CPU pressure. Regression tests reveal the spike and guide you to optimize.
Concept explained simply
Performance regression tests measure key performance metrics on every change and compare them with a known-good baseline or a fixed budget. If results are slower beyond an allowed tolerance, the test fails and blocks the change.
Key terms
- Baseline: Recorded metrics from a known-good version.
- Budget: A hard limit you must not exceed (for example, p95 latency <= 220 ms).
- Threshold: A rule that uses a baseline or budget to pass/fail a run.
- p95 / p99: Percentiles indicating worst-case user experiences.
Mental model
Think of a speed trap with guardrails. Your service carries a baseline speed record. Each commit must run through the trap. If it is too slow compared to the baseline (beyond tolerance) or exceeds a budget, the gate closes.
Baselines vs budgets
Baseline rules find relative slowdowns (did we get worse than last good run?). Budget rules enforce absolute limits (are we still within our SLO?). Use both to stay fast and consistent.
Key components of a performance regression test
- Metrics: Latency (p95/p99), throughput (RPS/QPS), error rate, CPU/memory.
- Scenarios: Representative requests (read vs write, cold vs warm cache).
- Load model: Users, arrival rate, duration, and warm-up period.
- Baselines and budgets: Stored numbers and hard limits.
- Thresholds: Rules that compare the current run to baseline/budget.
- Environments: Stable infra, seeded data, isolated noise sources.
- CI integration: Short PR gate + deeper scheduled runs.
- Reporting: Clear pass/fail plus deltas and top offenders.
Noise reduction tips
- Warm up the system to stabilize caches and JIT.
- Pin test concurrency and data shape; avoid shared noisy neighbors.
- Repeat runs and use medians of multiple samples for gating.
- Measure percentiles instead of averages.
Worked examples
Example 1: REST endpoint p95 regression
Baseline: p95 latency 180 ms. Budget: 220 ms. Tolerance: +15% vs baseline.
Current run: p95 = 260 ms.
- Baseline + 15% = 207 ms. Current 260 ms > 207 ms (relative fail).
- Also 260 ms > 220 ms (budget fail). Gate should fail.
Threshold sketch
thresholds:
- metric: http_req_duration_p95
rule: current <= min(baseline * 1.15, 220)
Example 2: gRPC streaming throughput
Baseline: 1,200 QPS at 1% error (budget error <= 2%). Tolerance: -5% drop allowed.
Current run: 1,125 QPS at 0.8% error.
- Allowed floor = 1,200 * 0.95 = 1,140 QPS.
- Current 1,125 QPS < 1,140 QPS: regression (throughput drop too large).
- Error is within budget. Gate should fail due to throughput.
Example 3: DB-heavy query with warm cache
Scenario: First run warms cache, next runs are measured. Baseline p99 = 400 ms; budget = 450 ms; tolerance +10%.
Current p99 = 436 ms.
- Baseline + 10% = 440 ms. Current 436 ms ≤ 440 ms (relative pass).
- Also 436 ms ≤ 450 ms (budget pass). Gate should pass.
Step-by-step: Design your first regression test
- Pick critical user flows: Identify top endpoints by traffic or business value (e.g., GET /orders, POST /checkout).
- Choose metrics: p95 latency, error rate, and RPS. Add CPU/memory if resource pressure matters.
- Stabilize environment: Fixed data set, warm-up for 1–3 minutes, consistent concurrency.
- Set baselines and budgets: Record metrics from a clean main build. Define absolute SLO-aligned budgets.
- Write thresholds: Example: p95 must be ≤ min(baseline * 1.10, 220 ms) and errors ≤ 1%.
- Automate in CI: Short PR run (1–3 min) for key flows; nightly longer run for full coverage.
Example threshold snippet
thresholds:
- name: read_p95
when: scenario == 'read'
rule: p95_ms <= min(baseline.read_p95_ms * 1.10, 220)
- name: error_rate
rule: error_rate_pct <= 1.0
- name: throughput
rule: rps >= baseline.rps * 0.95
Exercises
Complete these before taking the quick test. They mirror the tasks you will do at work.
Exercise 1: Decide pass/fail for latency
Baseline p95: 180 ms. Budget: 250 ms. Allowed tolerance: +15% vs baseline. Current run p95: 220 ms. Compute pass/fail and write a one-line reason.
Show reminder
Tolerance bound = baseline * 1.15. Gate uses min(tolerance bound, budget).
Exercise 2: Throughput floor
Baseline throughput: 1,200 RPS. Allowed drop: 5%. Current run: 1,120 RPS. Error rate stays 0.7% with budget 1%. Compute pass/fail and recommended next step.
Show reminder
Allowed floor = baseline * (1 - drop%). If below floor on 2 consecutive runs, fail the gate.
Self-check checklist
- I computed both the relative threshold and the absolute budget.
- I considered error rate, not only latency/throughput.
- I noted a next action if the result fails (rerun, profile, or block merge).
Common mistakes and how to self-check
- Using average latency instead of p95/p99. Self-check: Are you gating on percentiles?
- No warm-up period. Self-check: Do you discard the first minute before measuring?
- Changing test data per run. Self-check: Is your data set seeded and stable?
- Updating baseline on every run. Self-check: Do you only update after intentional improvements and stable confirmation?
- Ignoring error rate. Self-check: Are latency and success thresholds both enforced?
- One-off pass/fail without repetition. Self-check: Do you rerun to reduce noise before failing the PR?
Practical projects
- Project 1: PR Gate Basics — Add a 2-minute performance gate to a critical GET endpoint with rules: p95 ≤ min(baseline*1.10, 220 ms) and error ≤ 1%.
- Project 2: Nightly Deep Run — Build a 15-minute job covering read/write flows, with throughput floor ≥ baseline*0.95 and CPU ≤ baseline*1.10.
- Project 3: Report Deltas — Generate a short report showing deltas vs baseline (p95, p99, RPS, error%), and store last-green baselines.
Learning path
- Start: Performance basics (latency percentiles, throughput, error rate).
- Next: Load modeling (arrival rate vs users, warm-up, think time).
- Then: CI integration (short PR checks vs nightly suites).
- Advanced: Profiling hot paths and capacity planning.
Next steps
- Implement one small PR gate on your most-used endpoint.
- Run it 3 times, collect medians, and set your baseline/budget.
- Add error rate and throughput thresholds, not just latency.
Mini challenge
Your baseline p99 is 400 ms. A change improves p99 to 360 ms consistently over three runs. Budgets remain the same. Do you update the baseline now? If yes, what new tolerance rule will you set to prevent future regressions?
Quick Test
Take the quick test below to check your understanding. Available to everyone. Only logged-in users will see saved progress.