Instrumentation QA

Why this matters

Bad instrumentation breaks funnels, corrupts cohorts, and leads to wrong decisions. Product Analysts are often the last line of defense before numbers hit dashboards. Instrumentation QA ensures events are correct, trustworthy, and useful.

• Ship features with confidence by catching broken or missing events before launch.
• Reduce rework by validating event names, properties, and identity early.
• Protect experiments: confirm exposures, assignments, and conversions are recorded reliably.
• Keep data clean: prevent duplicates, wrong types, timezone drift, and PII leaks.

Concept explained simply

Instrumentation QA is the practice of validating that each event emitted by your app matches the tracking plan and arrives in analytics as expected.

Mental model: Contract → Pipeline → Proof

• Contract: The tracking plan is a contract. It defines event names, triggers, properties, types, and identity rules.
• Pipeline: Events move from client to collector to storage. Any hop can introduce issues.
• Proof: You need evidence at three levels: local payloads, server acceptance, and analytics tables.

How to run Instrumentation QA (step-by-step)

1. Read the tracking plan
   Checklist to prepare

   • Event name, trigger condition, and platforms (web, iOS, Android).
   • Properties: required/optional, type (string, number, boolean, array), constraints (enums, ranges).
   • Identity: user_id, anonymous_id, session_id, device_id rules; login/logout transitions.
   • PII policy: which fields are allowed; hashing/omitting sensitive data.
   • Sampling or throttling rules, retries, offline queuing.
2. Create test scenarios
   Cover normal and edge cases

   • Happy path trigger exactly as described in plan.
   • Boundary values (e.g., quantity=1 and a large quantity).
   • Negative cases (missing optional property, invalid input rejected by app).
   • State transitions (anonymous → logged in, logout, new session).
   • Network conditions (offline, slow, retry) and timezones.
3. Capture the event locally
   How to inspect

   • Web: use your browser's Network inspector to view the analytics request payload.
   • Mobile: use device logs or a proxy to view payloads sent by the SDK.
   • Verify: event name, properties, types, timestamp format, identifiers, event_id uniqueness.
4. Validate ingestion
   What to confirm

   • Requests are accepted (no client or server errors).
   • Retry logic does not create duplicates; event_id remains constant across retries.
   • Consent/state respected (no events when tracking is disabled).
5. Check analytics tables
   Warehouse/dashboard checks

   • New columns exist with correct types; unexpected extra columns flagged.
   • Null rates on required properties near zero.
   • Daily event counts sensible vs. traffic; no sudden drops/spikes after release.
   • Identity joins work: user_id and anonymous_id relate properly through sessions.
6. Document and gate
   Lightweight QA gate

   • Paste payload screenshots/snippets, table samples, and a short sign-off note.
   • Record any known limitations and a follow-up ticket if needed.

Worked examples

Practical QA checklist

• ☐ Event name matches the plan exactly (case and spelling).
• ☐ All required properties present; optional properties null or omitted as defined.
• ☐ Property types correct; enums uppercase where required.
• ☐ Values within allowed ranges; arrays length sane.
• ☐ timestamp is ISO 8601 UTC; no timezone drift.
• ☐ identity fields consistent: user_id, anonymous_id, session_id.
• ☐ event_id unique per occurrence; duplicates not produced by retries.
• ☐ PII policy respected; no disallowed fields.
• ☐ Ingestion accepts requests; no error spikes.
• ☐ Warehouse columns/types correct; null rates acceptable.

Exercises

Do these to build muscle memory. Use the checklist above. Solutions are hidden below each exercise.

Exercise 1 — Validate an Add to Cart event

{
  "event": "add_to_cart",
  "required": ["product_id", "price", "currency", "quantity", "timestamp", "anonymous_id", "session_id"],
  "types": {
    "product_id": "string",
    "price": "number (>=0)",
    "currency": "string (ISO 4217, uppercase)",
    "quantity": "integer (>=1)",
    "timestamp": "ISO 8601 UTC",
    "anonymous_id": "string",
    "user_id": "string | null",
    "session_id": "string"
  }
}

1) {"event":"add_to_cart","product_id":"SKU-9","price":19.99,"currency":"usd","quantity":1,"timestamp":"2025-07-10T12:00:05Z","anonymous_id":"anon-1","user_id":null,"session_id":"s-1","event_id":"e-101"}
2) {"event":"add_to_cart","product_id":"SKU-10","price":"24.00","currency":"EUR","quantity":2,"timestamp":"2025-07-10 12:01:10","anonymous_id":"anon-1","user_id":null,"session_id":"s-1","event_id":"e-102"}
3) {"event":"add_to_cart","product_id":"SKU-10","price":24.00,"currency":"EUR","quantity":0,"timestamp":"2025-07-10T12:01:20Z","anonymous_id":"anon-1","user_id":null,"session_id":"s-1","event_id":"e-103"}
4) {"event":"add_to_cart","product_id":"SKU-10","price":24.00,"currency":"EUR","quantity":2,"timestamp":"2025-07-10T12:01:20Z","anonymous_id":"anon-1","user_id":null,"session_id":"s-1","event_id":"e-103"}
5) {"event":"add_to_cart","price":12.00,"currency":"GBP","quantity":1,"timestamp":"2025-07-10T12:02:00Z","anonymous_id":"anon-1","user_id":null,"session_id":"s-1","event_id":"e-104"}

• Task: List all violations and the likely fix for each.

Exercise 2 — Identity and session QA

Page Viewed: {"event":"page_viewed","anonymous_id":"anon-2","user_id":null,"session_id":"s-9","timestamp":"2025-09-01T10:00:00Z","event_id":"p1"}
Add to Cart: {"event":"add_to_cart","anonymous_id":"anon-2","user_id":null,"session_id":"s-9","timestamp":"2025-09-01T10:01:00Z","event_id":"a1"}
Login: {"event":"login_success","anonymous_id":"anon-2","user_id":"u-77","session_id":"s-9","timestamp":"2025-09-01T10:02:00Z","event_id":"l1"}
Purchase: {"event":"purchase_completed","anonymous_id":"anon-NEW","user_id":"u-77","session_id":"s-NEW","timestamp":"2025-09-01T10:05:00Z","event_id":"c1"}

• Task: Is identity stitching correct? What would you fix?

Common mistakes and how to self-check

• Mismatched event names: Compare against the plan programmatically or with a strict checklist.
• Wrong property types: Inspect raw payloads; spot quotes around numbers and booleans.
• Timestamp drift: Verify UTC Z suffix and compare app time vs server time.
• Identity resets: Observe anonymous_id, user_id, and session_id across login/logout flows.
• Duplicate events on retry: Confirm event_id remains constant for a single occurrence; retries should not create new IDs.
• Missing required fields: Attempt to emit when fields are null; the client should block or fill defaults.
• PII leakage: Scan payload keys for disallowed info (emails, phone numbers) if policy forbids them.

Mini challenge

Pick any event in your app and run through the QA checklist end-to-end. Write a 3-bullet sign-off: what you tested, what you found, and 1 improvement to bake into the tracking plan.

Who this is for

• Product Analysts who own dashboards, experiments, or product metrics.
• Any team member reviewing event tracking before release.

Prerequisites

• Basic understanding of event tracking plans and JSON payloads.
• Ability to use a browser network inspector or mobile logs.

Learning path

1. Review the tracking plan format and naming conventions.
2. Practice inspecting payloads locally for 2–3 common events.
3. Validate identity flows (anonymous → logged in → logged out).
4. Check warehouse tables and build a small “canary” chart of daily counts.
5. Create a reusable QA checklist for your team.

Practical projects

• Build a one-page QA template that includes plan snippet, payload screenshot, table sample, and sign-off.
• Create a canary dashboard showing daily event counts, null rates for required properties, and duplicate rates by event_id.
• Write a simple data dictionary for your top 10 events with examples of good and bad payloads.

Next steps

• Automate parts of your checklist (e.g., a script to validate types and required fields on sample payloads).
• Add identity transition tests to your release checklist.
• Pair with Engineering to enforce schema validation client-side.

Quick Test

Anyone can take the test for free. If you are logged in, your progress will be saved.