luvv to help

Menu

Topic 3 of 8

Request Deduplication

Learn Request Deduplication for free with explanations, exercises, and a quick test (for API Engineer).

Published: January 21, 2026 | Updated: January 21, 2026

Table of Contents

Why this matters

Duplicate requests happen in real systems: user double-clicks, mobile retries on flaky networks, load balancers retry on 502/timeout, queues re-deliver messages. Without guardrails you may double-charge a card, create duplicate orders, or melt your origin with a thundering herd. Request deduplication prevents harm, cuts load, and improves perceived reliability.

  • Real API Engineer tasks: design idempotency for payments and orders; implement request coalescing for hot GETs; add ETag/If-None-Match; build at-least-once dedup for consumers.
  • Impact: fewer incidents, predictable costs, happier users.

Who this is for and prerequisites

  • Who: API and backend engineers building services with retries, payments, ordering, uploads, or high-read endpoints.
  • Prerequisites: basic HTTP (methods, status codes, headers), caching basics, and one server language (e.g., Go/Java/Node/Python) with a key-value store or relational DB.

Concept explained simply

Request deduplication ensures that two or more requests that represent the same logical operation are processed once, and all callers get a consistent response.

Mental model

Imagine stamping each logical operation with a unique ticket. The first request with ticket T does the work and stores the result under T. Any later request with the same T skips work and reads that stored result.

Core patterns you can combine

  • Idempotency keys (writes): client sends a unique key (e.g., Idempotency-Key) per logical operation. Server atomically stores request fingerprint and response; duplicates return the stored response.
  • Request coalescing / single-flight (reads): when many identical GETs arrive concurrently, only one hits the origin; others await its result.
  • Conditional requests (reads): use ETag/If-None-Match or Last-Modified/If-Modified-Since to avoid sending bodies when nothing changed.
  • Cache-key dedup: cache at edge or service level; align cache keys with request semantics.
  • Message dedup (queues): track processed message IDs to avoid reprocessing under at-least-once delivery.
  • Distributed dedup store: use atomic put-if-absent or unique constraints to guarantee one winner.

Worked examples

1) Payments: POST /charges
  1. Client generates Idempotency-Key per checkout attempt.
  2. Server does atomic insert into charges_dedup(idempotency_key UNIQUE, request_hash, response, status, expires_at).
  3. On first insert: process payment, store full response, return 201 Created.
  4. On duplicate with same payload: return stored response (usually 201 Created or 200 OK).
  5. On duplicate with different payload: return 409 Conflict with a helpful message.
2) Hot GET coalescing
  1. Key each in-flight request by method+path+normalized query (e.g., GET:/product?id=42).
  2. If a flight exists, wait on it; else create a flight and fetch origin.
  3. On completion, release all waiters and optionally populate cache.
  4. Always ensure timeout and cleanup to prevent leaks.
3) ETag with If-None-Match
  1. Server returns ETag: W/"abcd1234" for GET /profiles/7.
  2. Client sends If-None-Match: W/"abcd1234" on next GET.
  3. If unchanged, server returns 304 Not Modified (no body) — bandwidth and CPU saved.
4) Queue consumer dedup
  1. Each message carries a logical operation ID.
  2. Consumer attempts atomic insert into processed_ids(op_id PRIMARY KEY, processed_at, expires_at).
  3. On success: process; on conflict: skip.
  4. Pick TTL based on how long duplicates may arrive.

Design steps (safe defaults)

  1. Define the dedup scope: per user, per resource, or global.
  2. Choose a key: client-provided idempotency key (writes) or normalized request signature (reads).
  3. Pick store with atomicity: unique index (SQL), SETNX (Redis), or compare-and-set.
  4. Record enough to detect misuse: store request hash and response metadata.
  5. Select TTL: match your retry windows and business needs (e.g., 24–72 hours for payments).
  6. Decide duplicate behavior: return original response; if payload mismatch, return 409 Conflict.
  7. Monitor: count dedup hits, conflicts, and key collision rates.

Exercises and practice

Complete the two exercises. You can check hints and full solutions in the exercise cards below.

Exercise 1 — Design idempotency for POST /orders

Design a key format, storage plan, TTL, and response rules (including mismatch handling). Outline the data you will store and how you will make the insert atomic.

Exercise 2 — Build request coalescing for GET /product?id=123

Sketch code that prevents multiple in-flight calls from hitting the origin simultaneously. Include keying, timeout, error propagation, and cleanup.

Self-check checklist:

  • You defined a clear dedup scope and TTL.
  • Your plan uses an atomic uniqueness guarantee.
  • You handle duplicate-with-different-payload as a conflict.
  • Your coalescing code cannot deadlock or leak entries on timeout/errors.
  • You emit metrics for wins, conflicts, and fallbacks.

Common mistakes and how to self-check

  • Relying only on caches for writes: caches help reads; writes need idempotency keys or unique constraints.
  • No atomicity: checking then inserting allows races. Use insert-if-absent or unique index.
  • Missing payload hash: you can’t detect wrong key reuse; add a request hash.
  • Unbounded in-flight map: single-flight without timeouts leaks memory. Add deadlines and cleanup.
  • Wrong TTL: too short loses dedup protection; too long causes unexpected 409s. Align with retry windows.
  • Key includes volatile fields: timestamps/nonces make every request appear unique. Normalize inputs.

Practical projects

  • Add Idempotency-Key to a demo checkout service. Store response bodies and headers; verify 409 on mismatch.
  • Implement single-flight middleware for a service’s hottest GET endpoint with metrics and timeouts.
  • Build a small queue worker that deduplicates operations using a SQL unique index and TTL cleanup.

Learning path

  • Before this: HTTP caching, status codes, and basic DB transactions.
  • Now: Request deduplication (this lesson) with idempotency keys and coalescing.
  • Next: Rate limiting, circuit breakers, and adaptive concurrency to further protect origins.

Next steps

  • Ship one dedup change behind a feature flag and monitor metrics.
  • Document client guidance for Idempotency-Key generation.
  • Add synthetic tests that simulate retries and verify identical responses.

Mini challenge

Scenario

Your cart service occasionally creates duplicate orders during traffic spikes. You already cache GET /product but POST /orders has no protection. In under 10 minutes, outline the minimal changes to make POST idempotent and coalesce GET /inventory?id=42 under load. List the keys, TTLs, and failure behavior.

Quick Test

Everyone can take this test for free. Note: only logged-in users have their progress saved.

Practice Exercises

2 exercises to complete

Instructions

Design an idempotent POST /orders endpoint that avoids duplicate orders when clients retry. Provide:

  • Key format (header name and composition).
  • Atomic storage design (table/record structure and uniqueness guarantee).
  • What to store (request hash, response, status, headers).
  • TTL and rationale.
  • Behavior on true duplicate vs duplicate-with-different-payload.
Expected Output
A concise design document (5–10 bullets) covering key, atomic insert, stored fields, TTL, and response behavior for duplicates and mismatches.

Request Deduplication — Quick Test

Test your knowledge with 10 questions. Pass with 70% or higher.

10 questions70% to pass
Caching Headers And ETagsConnection Pooling Awareness

Have questions about Request Deduplication?

AI Assistant

Ask questions about this tool