Why this matters
Example 3: Summarization with LLM-as-judge
Goal: Summarize support chats. Metric: rubric-based judge score (0–1) averaging 5 dimensions (faithfulness, coverage, concision, tone, actionability).
- Benchmark set: 30 chats, with reference bullet points.
- Judge rubric: Score each dimension 0–1; final is mean.
- Results: A = 0.74, B = 0.79. Bootstrap 1,000 samples → 95% CI for delta: [0.03, 0.08].
- Gate: delta CI lower bound ≥ 0.02 → Ship B.
Why CI matters
It protects you from random gains on a small set. If even the lower bound is positive, the improvement is likely real.
Build a mini regression set now
Use this simple, repeatable structure:
- Case ID
- Input
- Expected (label, JSON fields, or rubric notes)
- Scoring rule
- Reason it was added (bug it catches)
Template you can copy
{
"id": "R-001",
"input": "Title: Wireless phone stand, fast charge",
"expected": "Electronics > Chargers",
"scoring": "Exact match category",
"reason": "Previously misclassified as Accessories"
}- Checklist to finish: covers common, edge, negative, and ambiguous cases; includes at least one previously broken case; each case has a clear scoring rule; temperature/seed documented.
Exercises (hands-on)
These mirror the exercises below so you can practice and then check the solutions.
Exercise 1: Compute accuracy and decide gate
You have 10 product titles with golden categories (provided in the exercise). Two prompts (A and B) produce predictions. Compute accuracy for each, list failing IDs, and decide if B should ship using the gate: Accuracy ≥ 0.9 and no more than 1 top-level mistake.
Data you’ll use
Golden labels (id:category) 1: Electronics > Chargers 2: Apparel > Shirts 3: Home > Kitchen 4: Electronics > Audio 5: Apparel > Shoes 6: Home > Decor 7: Electronics > Accessories 8: Home > Bedding 9: Apparel > Pants 10: Electronics > Chargers Predictions A 1 Chargers; 2 Shirts; 3 Kitchen; 4 Speakers; 5 Shoes; 6 Decor; 7 Accessories; 8 Bedding; 9 Pants; 10 Batteries Predictions B 1 Chargers; 2 Shirts; 3 Kitchen; 4 Audio; 5 Shoes; 6 Decor; 7 Accessories; 8 Bedding; 9 Pants; 10 Chargers
Exercise 2: Design a summarization rubric and score
Create a 5-criterion rubric (0–1 per criterion). Score two summaries (provided in the exercise) and compute the weighted score (weights: faithfulness 0.4, coverage 0.25, concision 0.15, tone 0.1, actionability 0.1). Gate: score ≥ 0.75. Decide which summary passes.
Summaries to score
Source points: {issue: login lockout, root cause: rate-limit spike, fix: cache rule update, status: resolved}
Summary X: "User lockouts occurred due to rate limit. We updated cache rules; users can log in again."
Summary Y: "Some users had issues. The team worked on servers and things are better now."- Self-check checklist: metric chosen before looking at results; decision gate defined; scoring repeatable by another person; you can explain any failure.
Common mistakes and self-check
- Leakage: Using training or seen examples as test. Self-check: Can a teammate verify labels without knowing your prompt?
- Tiny, biased sets: Only easy cases. Self-check: Do you have edge/negative/ambiguous examples?
- Vague scoring: "Looks okay" judgments. Self-check: Is your metric rule precise (exact match, regex, rubric)?
- Ignoring randomness: One lucky run. Self-check: Fixed seed or multi-run average/majority vote?
- No gates: Improvements ship by vibes. Self-check: Do you have numeric thresholds and critical-failure rules?
- Not updating regression set: Bugs reappear. Self-check: Did you add new failures to the set with a reason?
Practical projects
- Project 1: Build a 50-case benchmark for your app’s main task (classification or extraction). Include at least 10 edge cases. Define a gate and a change log template.
- Project 2: Create a 20-case regression set from past bugs. Wire it into your daily prompt tweak routine. Track pass/fail over a week of changes.
- Project 3: Summarization judge: Write a 5-criterion rubric and evaluate two prompts on 30 documents. Report mean score and a 95% bootstrap CI for the delta.
Learning path
- Before this: Prompt design basics; metric selection.
- Now: Benchmarking and regression sets (this lesson).
- Next: A/B testing with users, online metrics, monitoring drift, and alerting.
Assessment
Take the Quick Test below to check understanding. Available to everyone; only logged-in users will have their progress saved.
Next steps
- Turn your current bug list into a regression set with clear expected outcomes.
- Pick one primary metric and one critical-failure rule for your main task.
- Schedule a 30-minute weekly review to add new failures and prune redundant cases.
Mini challenge
Pick a recent prompt change you made. Run it against your (even tiny) regression set. If anything fails, write a one-sentence reason per failure and add the case permanently. Decide: ship, hold, or iterate.