Error Taxonomy And Root Cause Analysis

Why this matters

As a Prompt Engineer, you don’t just write prompts—you ship reliable behaviors. When outputs fail, you must quickly identify what kind of error it is and why it happened. A clear error taxonomy plus root cause analysis (RCA) turns random debugging into a repeatable process. You’ll use this skill to triage production issues, improve prompts, design evaluations, and prevent regressions.

• Triage user reports (e.g., “the model ignored the schema”)
• Diagnose failures in data extraction, RAG answers, and tool-using agents
• Design targeted fixes and regression tests that stick

Concept explained simply

Error taxonomy = naming the type of mistake. Root cause analysis = discovering why it occurred so your fix actually works. Together, they help you move from patching symptoms to preventing the next incident.

Mental model

Think of it like a medical diagnosis:

• Symptom: What you observe (e.g., extra text, wrong format)
• Diagnosis: Standardized label (e.g., schema non-compliance)
• Root cause: The underlying reason (e.g., prompt didn’t show a JSON example)
• Treatment: Minimal change that addresses the cause (e.g., add explicit JSON schema + single example)

Error taxonomy for prompt engineers

Use these categories to label issues consistently. One incident can have multiple categories; pick the primary one first.

Root cause analysis (RCA) workflow

1. Triage – Is it severe, frequent, user-visible? Reproduce once.
2. Classify – Apply the taxonomy: pick a primary error type (and secondary if needed).
3. Localize – Where is it happening? System vs user prompt, examples, retrieval, tools, parameters.
4. Hypothesize – Use 5 Whys and form a minimal, testable hypothesis.
5. Experiment – Change one variable at a time; collect before/after metrics.
6. Fix – Implement the smallest, durable change.
7. Guard – Add a regression test to prevent recurrence.

Worked examples

Example 1: Summarization misses constraints

Symptom: “Provide a 3-bullet summary with a title.” Model gives 5 bullets, no title.

• Category: Instruction-following failure; Omission
• Root cause: Constraints are buried mid-paragraph; “should” not “must”
• Fix: Move constraints to end as numbered MUST list; add one ideal example
• Guard: Test that length == 3 and title exists

Example 2: JSON extraction drifts into prose

Symptom: Model outputs JSON + an explanation sentence.

• Category: Format / Schema non-compliance
• Root cause: No single JSON-only example; no explicit "Output JSON only" rule
• Fix: Provide strict schema, one example, and a DO NOT add prose rule
• Guard: Automatic JSON validator test

Example 3: Agent picks wrong tool

Symptom: Agent queries a web search tool when a local knowledge base would be more precise.

• Category: Tool-use / API orchestration error
• Root cause: Tool descriptions lack selection criteria
• Fix: Add success criteria and decision rules: “Use KB if query mentions internal product codes; else use Web.”
• Guard: Simulated queries with expected tool choices

Exercises

Do these now. They mirror the graded exercises below.

Exercise 1: Identify error type(s) and root cause

Scenario: Your extraction prompt requests exact ISO date format and fields {name, start_date, end_date}. The model returns {"name": "Event" , "dates": "Jan–Mar 2024"} and sometimes adds a note line.

• Task: 1) Label the primary and secondary error categories. 2) Write a minimal hypothesis about the cause. 3) Propose a one-change fix and a guard test.

Exercise 2: Prevent hallucinated sources

Scenario: In a RAG QA system, answers occasionally include fabricated citations when the context lacks evidence.

• Task: 1) Classify the error. 2) Draft a refusal policy line. 3) Suggest one retrieval tweak. 4) Define a metric to track improvement.

Common mistakes and self-check

• Mistake: Fixing multiple things at once. Self-check: Did I change exactly one variable per test?
• Mistake: Vague labels (“it’s bad”). Self-check: Did I assign a specific taxonomy category?
• Mistake: Overfitting to one example. Self-check: Did I validate on a set of varied cases?
• Mistake: Ignoring evaluation metrics. Self-check: Do I have a numeric success rate (e.g., JSON validity %)?
• Mistake: Missing guardrails. Self-check: Did I add a regression test that would catch this again?

Practical projects

• Build a tiny “error triage” sheet: columns for Symptom, Category, Hypothesis, Change, Metric, Result, Guard.
• Create a 10-case dataset for your prompt and track: format compliance, omission rate, hallucination rate.
• Design an agent tool-selection rubric and test with 8 synthetic queries.

Who this is for

• Prompt Engineers and Data/ML practitioners who need reliable LLM behavior
• Product folks triaging LLM features and user reports

Prerequisites

• Basic prompt engineering (system/user messages, few-shot examples)
• Familiarity with common LLM parameters (temperature, max tokens)

Learning path

1. Learn the error taxonomy and memorize 3–5 core categories
2. Practice RCA with 5 Whys and single-variable experiments
3. Add metrics and guard tests to your workflow
4. Apply to RAG, extraction, and agent tasks

Mini challenge

Given a user complaint “The bot keeps ignoring the word limit and sometimes cites blogs we never provided,” write: 1) two categories, 2) a single root cause hypothesis, 3) one minimal fix, and 4) a guard test.

Note: You can take the Quick Test below. Anyone can take it; progress is saved only for logged-in users.