Idempotency And Consistency

Why this matters

APIs run over unreliable networks and clients retry when they don’t get a response. Without idempotency and clear consistency guarantees, users get double charges, lost updates, or confusing data states. API Engineers design endpoints and storage interactions to be safe under retries, concurrency, and eventual consistency.

• Prevent duplicate side effects (payments, emails, order placements).
• Protect data integrity during concurrent updates.
• Offer predictable reads and writes across services.

Quick note: Everyone can take the exercises and quick test. Only logged-in users have their progress saved.

Concept explained simply

Idempotency

An operation is idempotent if performing it once or many times leads to the same final state. Network retries should not multiply the effect.

• HTTP methods intended as idempotent: GET, PUT, DELETE, HEAD, OPTIONS.
• POST is not idempotent by default; you can make a POST effectively idempotent using an Idempotency-Key or natural uniqueness constraints.

Consistency

• Strong consistency: After a write, all reads see the latest value.
• Eventual consistency: Data replicas converge over time; short windows may show stale data.
• Read-your-writes: A writer immediately sees their own latest write even if others briefly see older data.

Mental model

Imagine stamping each intent with a unique “receipt” (idempotency key). If the same receipt shows up again, you return the stored result instead of doing the work twice. For concurrent edits, think of every record carrying a version tag. Only updates with the correct tag succeed, preventing overwriting someone else’s work.

Core patterns and tools

• Idempotency keys: Client sends a unique key per logical request. Server stores the first request’s result and replays it for duplicates.
• Uniqueness constraints: Use a business or surrogate unique field to deduplicate (e.g., payment_id, request_key with unique index).
• Optimistic concurrency control (OCC): Use ETag or version fields with If-Match to stop lost updates.
• Conditional requests: If-Match/If-None-Match for HTTP, or version checks at the DB level.
• Transactions and outbox: Persist intent and side effects atomically; publish messages from an outbox to avoid duplicates or missing events.
• Inbox/dedup store: Keep processed message IDs to ignore repeats in at-least-once messaging.
• Saga orchestration: Break long transactions into steps with compensations for eventual consistency across services.

Worked examples

Response codes
201 Created — first successful creation
200 OK — replayed result for a duplicate retry
409 Conflict — same key, different body

PUT /profiles/123
If-Match: "v3"
{ "display_name": "Ali" }

Hands-on exercises

Mirror of the exercises below. Do them here, then check solutions via the toggles.

Exercise 1 — Design an idempotent POST /payments

1. Define how you store and lookup Idempotency-Key, including tenant/route scoping and TTL.
2. Decide how you compute and store a request body hash to detect conflicts.
3. Specify response codes and payloads for first-time, duplicate, and conflicting requests.

Exercise 2 — Implement optimistic concurrency with ETag

1. Add a version or updated_at field to a resource.
2. Return ETag on reads; require If-Match on updates.
3. Handle 412 Precondition Failed with a clear error message and guidance to re-fetch.

Checklist

• POST retries do not duplicate work.
• Returning clients get the same response for the same key.
• Conflicting reuses of keys are rejected clearly.
• Concurrent updates do not overwrite each other.

Common mistakes and self-check

• Storing idempotency keys without scoping: Always scope by tenant/app and route; otherwise collisions leak across endpoints.
• Accepting same key with different payloads: That hides bugs. Return 409 Conflict with a short explanation.
• Short retention windows: Too short and late retries create duplicates. Match TTL to your client retry policies.
• Ignoring body hash: Without it, you cannot detect conflicting replays.
• Skipping conditional updates: Without If-Match/OCC, you risk lost updates in concurrent writes.
• Relying on exactly-once delivery: Design handlers to be idempotent and deduplicate processed messages.

Practical projects

1. Payment-like endpoint: Build POST /charges with Idempotency-Key, conflict detection, and stored result replay. Add metrics for duplicate rate and conflict rate.
2. Profile service with OCC: Add versioned resources with ETag and If-Match. Demonstrate concurrent updates with one failing gracefully.
3. Mini saga: Place order -> reserve inventory -> capture payment. Use outbox for events and an inbox/dedup table for consumers. Add compensations for failures.

Learning path

1. Master HTTP semantics (GET/PUT/DELETE idempotency; conditional headers).
2. Implement idempotent POSTs with keys and uniqueness constraints.
3. Add OCC (ETags/versions) to prevent lost updates.
4. Adopt outbox/inbox for reliable async flows and eventual consistency.
5. Practice with a small saga and compensating actions.

Who this is for

• API engineers and backend developers designing public/internal APIs.
• Engineers integrating with payments, orders, or any side-effecting operations.
• Developers moving from monoliths to distributed systems.

Prerequisites

• Basic REST and HTTP knowledge (methods, status codes, headers).
• Familiarity with relational or document databases.
• Basic understanding of message queues is helpful.

Next steps

• Harden your POST endpoints with idempotency keys and TTL policies.
• Add ETag/If-Match to at least one resource today.
• Introduce outbox/inbox tables for any async flows you own.
• Take the Quick Test below to confirm your understanding.

Mini challenge

You run a promotion endpoint that can be applied once per user per cart. Design the API so retries don’t apply the promotion multiple times, and concurrent updates don’t overwrite the cart. In 6–8 bullet points, outline keys, conflicts, versioning, and responses.