NLP Evaluation And Error Analysis

Use macro-F1 for imbalanced classes to weight each class equally; micro-F1 for global performance when class balance matters.

# Example: news topic classification
from sklearn.metrics import confusion_matrix, classification_report, f1_score
import numpy as np

y_true = ["sports","sports","politics","tech","tech","tech","health","health","health","health"]
y_pred = ["sports","tech","politics","tech","tech","sports","health","health","tech","health"]

labels = sorted(set(y_true))
print(classification_report(y_true, y_pred, labels=labels, digits=3))
print("micro-F1:", f1_score(y_true, y_pred, average="micro"))
print("macro-F1:", f1_score(y_true, y_pred, average="macro"))
print("Confusion:\n", confusion_matrix(y_true, y_pred, labels=labels))

# Simple slice analysis by presence of a keyword
docs = [
    "Local sports team wins cup", "Trade rumors in soccer", "Debate in parliament",
    "New GPU features", "AI startup raises", "Coach resigns",
    "Healthy diet tips", "Mental health awareness", "Sleep science", "Hospital funding"
]
contains_health_word = [int("health" in d.lower()) for d in docs]

# Compute per-slice F1
slices = {"has_health_word": [i for i,v in enumerate(contains_health_word) if v==1],
          "no_health_word": [i for i,v in enumerate(contains_health_word) if v==0]}
for s, idxs in slices.items():
    yt = [y_true[i] for i in idxs]
    yp = [y_pred[i] for i in idxs]
    print(s, "macro-F1:", f1_score(yt, yp, average="macro"))

Action: Prioritize slices with the largest drop vs overall macro-F1 and read a few misclassified examples to find patterns.

Evaluate exact-match entities, not tokens, when span correctness matters.

# Gold and predicted entity spans: (start, end, label) with end exclusive
GOLD = {"Alice lives in Paris": [(0,5,"PER"),(15,20,"LOC")]}
PRED = {"Alice lives in Paris": [(0,5,"PER"),(15,20,"ORG")]}

def prf_entity(gold_spans, pred_spans):
    gold = set(gold_spans)
    pred = set(pred_spans)
    tp = len(gold & pred)
    fp = len(pred - gold)
    fn = len(gold - pred)
    prec = tp/(tp+fp) if tp+fp else 0
    rec = tp/(tp+fn) if tp+fn else 0
    f1 = 2*prec*rec/(prec+rec) if prec+rec else 0
    return prec, rec, f1

for txt in GOLD:
    p, r, f = prf_entity(GOLD[txt], PRED[txt])
    print("Entity P/R/F1:", round(p,3), round(r,3), round(f,3))

The model predicted Paris as ORG instead of LOC, causing a span-level mismatch.

Automatic metrics like ROUGE correlate imperfectly with quality. Combine with a short human rubric.

# Quick ROUGE-1 recall approximation
from collections import Counter

def rouge1_recall(ref, hyp):
    ref_unigrams = Counter(ref.lower().split())
    hyp_unigrams = Counter(hyp.lower().split())
    overlap = sum((ref_unigrams & hyp_unigrams).values())
    total = sum(ref_unigrams.values())
    return overlap / total if total else 0

ref = "The committee approved the budget after revisions."
hyp = "The budget was approved by the committee."
print("ROUGE-1 recall:", round(rouge1_recall(ref, hyp), 3))

Human rubric (rate 1–5 each):

• Faithfulness: no unsupported facts.
• Coverage: important points included.
• Clarity: easy to understand.

Report average scores and agreement between annotators.

import random

def add_typos(text, p=0.05):
    chars = list(text)
    for i in range(len(chars)-1):
        if random.random() < p and chars[i].isalpha() and chars[i+1].isalpha():
            chars[i], chars[i+1] = chars[i+1], chars[i]
    return ''.join(chars)

texts = ["I loved the movie", "This product is terrible", "Great quality"]
labels = [1, 0, 1]

# Suppose we have a predict_proba() that returns class probabilities
# and predict() that returns 0/1. Replace with your model API.

def evaluate(texts, labels):
    # Dummy: pretend every positive
    preds = [1]*len(texts)
    acc = sum(int(p==y) for p,y in zip(preds,labels))/len(labels)
    return acc

acc_clean = evaluate(texts, labels)
acc_noisy = evaluate([add_typos(t, p=0.1) for t in texts], labels)
print("Accuracy clean vs noisy:", acc_clean, acc_noisy, "delta:", acc_noisy - acc_clean)

Track the delta. If the drop is large, add normalization or data augmentation.

# Group by a simple heuristic (presence of pronouns as a proxy slice example)
from collections import defaultdict

docs = [
  ("He was satisfied with service", 1, 1),
  ("She disliked the delay", 0, 1),
  ("They are happy", 1, 1),
  ("She is not impressed", 0, 0),
]
# each tuple: (text, true_label, pred_label)

def group(text):
    t = text.lower()
    if " she " in " " + t + " ": return "she"
    if " he " in " " + t + " ": return "he"
    if " they " in " " + t + " ": return "they"
    return "other"

by_g = defaultdict(list)
for txt, y, yhat in docs:
    by_g[group(txt)].append((y, yhat))

def tpr(pairs):
    tp = sum(1 for y,yhat in pairs if y==1 and yhat==1)
    pos = sum(1 for y,_ in pairs if y==1)
    return tp/pos if pos else 0

for g, pairs in by_g.items():
    print(g, "TPR:", round(tpr(pairs),3))

Report the largest gap across groups and whether it is acceptable for your use case.

Detect: Stakeholders care about minority class, but you report accuracy. Fix: Report macro-F1 and per-class metrics; align with business goals.

Detect: Token F1 looks high but spans look wrong. Fix: Report entity-level exact-match F1 and per-entity breakdown.

Detect: Overall metrics stable but complaints increase in a subgroup. Fix: Define slices by length, domain, language, demographics proxies where appropriate; compare gaps.

Detect: Many “errors” are actually mislabeled. Fix: Sample 50–100 items and re-annotate; compute estimated label error rate.

Detect: Previously fixed bugs reappear after an update. Fix: Add curated test cases into a permanent suite and block releases when failing.

Detect: Can’t reproduce last week’s numbers. Fix: Log model version, data version, seed, metrics, and date every run.

• evaluation_report.md with metrics and key findings.
• errors.md with 5 recurring themes and example IDs.
• regression/ folder with JSONL of test cases.
• metrics.csv with at least two runs.