Annotation Guidelines

Why this matters

High-quality models start with high-quality labels. Clear annotation guidelines reduce confusion, improve inter-annotator agreement, cut rework, and make your dataset reproducible. In real projects, you will onboard annotators, review edge cases, and measure agreement. Well-written guidelines let you do all of that efficiently.

• Ship a sentiment model faster because annotators agree on sarcasm rules.
• Lower costs by reducing re-annotation cycles.
• Enable fair evaluation by documenting exactly what each label means.

Concept explained simply

Annotation guidelines are the rulebook for labeling. They answer three big questions: What is labeled, how it is labeled, and what to do when it is unclear. If every annotator follows the same rules, you get consistent labels.

Mental model

Think of your guidelines as a decision tree that any new annotator can follow to reach the same label. If two people disagree, your guideline is missing a branch or an example.

Core components of great annotation guidelines

1. Task definition: What problem are we solving? Single-label or multi-label? Classification or span labeling?
2. Label set and definitions: Every label has a plain-language definition and contrasts with others.
3. Decision rules: Step-by-step checks and precedence when rules conflict.
4. Scope and constraints: What text is in scope; allowed knowledge (text only, or context allowed).
5. Edge cases: Ambiguity, sarcasm, emojis, code-mixed text, hashtags, typos.
6. Boundary rules (for spans): Include or exclude punctuation, determiners, prepositions, hyphenation, multiword entities.
7. Examples and counterexamples: At least 3 per label, including borderline cases and common traps.
8. Quality control: Tie-breakers, when to use an uncertain tag (if allowed), escalation path, and gold checks.
9. Annotator UX: Short keyboarding tips, form completion rules, and how to report issues.
10. Review plan: How you calculate agreement and update rules after pilot runs.

Worked examples

Example 1: Binary toxic comment classification

• Labels: TOXIC, NOT_TOXIC
• Definition: TOXIC includes personal attacks, slurs, threats, or demeaning content targeted at a person or group. Mere disagreement or strong language without a target is NOT_TOXIC.
• Decision rules:
  1. Is there a targeted person or group? If no, label NOT_TOXIC.
  2. Is the language demeaning, threatening, or a slur? If yes, label TOXIC.
  3. Quoting toxic text without endorsing it is NOT_TOXIC. If endorsement is clear, TOXIC.
• Edge cases: Sarcasm that implies a slur is TOXIC. Profanity aimed at objects (e.g., this app is s***), NOT_TOXIC.
• Examples:
  • "You are worthless" → TOXIC
  • "This idea is stupid" → NOT_TOXIC
  • "They should all be kicked out" (about a group) → TOXIC

Example 2: Intent classification for support tickets

• Labels: BUG, FEATURE_REQUEST, BILLING, OTHER
• Definitions: BUG is unexpected incorrect behavior; FEATURE_REQUEST is asking for new capability; BILLING concerns charges, invoices, or payments; OTHER is none of the above.
• Decision rules:
  1. If it mentions charge, invoice, or card → BILLING.
  2. If it requests something not currently available → FEATURE_REQUEST.
  3. If something used to work and now fails → BUG.
  4. Else → OTHER.
• Examples:
  • "App crashes after login" → BUG
  • "Can you add dark mode?" → FEATURE_REQUEST
  • "I was double charged" → BILLING

Example 3: Span labeling for product feature extraction

• Entities: FEATURE, VALUE, BRAND
• Boundary rules:
  • Exclude trailing punctuation.
  • Include adjectives only if they are part of the feature name (e.g., smart battery vs battery life: FEATURE is battery life; smart is an opinion unless part of a named feature).
  • Do not overlap entities. If overlap is unavoidable, prioritize FEATURE over VALUE.
• Examples:
  • "Battery life is excellent" → FEATURE: battery life; VALUE: excellent
  • "Apple AirPods Pro" → BRAND: Apple; FEATURE: AirPods Pro is not a feature; treat as product name if such a label exists; otherwise do not label.
  • "Noise-canceling works" → FEATURE: Noise-canceling

Write your draft guideline

1. State the task: One sentence on what annotators do.
2. Define labels: One or two sentences each, with contrasts.
3. Add rules: A numbered decision list. Include tie-breakers.
4. Document edge cases: Use a bullet list.
5. Provide examples: 3 per label, including borderline cases.
6. Pilot and refine: Run a small batch, measure agreement, update rules.

Task: Classify each comment as TOXIC or NOT_TOXIC.
Labels:
- TOXIC: ...
- NOT_TOXIC: ...
Decision rules:
1) ...
2) ...
Edge cases: ...
Examples:
- Text: ... → Label: ...
Quality control: Use gold checks; escalate unclear cases in thread.

Exercises

Do these to cement your skills. The quick test is available to everyone; only logged-in users get saved progress.

• Exercise 1: Write concise labeling guidelines for a binary toxic comment dataset.
• Exercise 2: Create span labeling rules for product feature extraction.

Common mistakes and how to self-check

• Vague labels: Replace adjectives like clear or obvious with precise tests. Self-check: Can a new annotator decide in under 15 seconds?
• No counterexamples: Add at least one near-miss per label.
• Missing tie-breakers: Add default behavior and escalation path.
• Span boundary drift: State punctuation, determiners, and multiword rules.
• Scope creep: Remind annotators to use only the provided text; no web searches unless allowed.

Practical projects

• Build a 2-page guideline for sentiment on app reviews, including sarcasm rules and 12 examples.
• Create span rules for extracting ingredients and quantities from recipes; pilot 50 items and report agreement.
• Design a multi-label topic taxonomy for forum posts and write decision rules with examples.

Mini challenge

Given: "Yeah, great service... after only 3 disconnects today." Decide how your sarcasm rule would classify sentiment. Write a one-sentence rule and your label decision.

Learning path

1. Start with binary classification guidelines.
2. Move to multi-class and multi-label tasks.
3. Add span labeling with strict boundary rules.
4. Run a pilot, measure agreement, and iterate.

Who this is for

• Aspiring NLP engineers preparing datasets.
• Data annotators and QA reviewers.
• Researchers creating reproducible corpora.

Prerequisites

• Basic NLP task types (classification, NER).
• Comfort reading short text snippets.
• Willingness to write simple decision rules.

Next steps

• Complete the exercises below.
• Take the quick test to validate understanding.
• Apply your guideline to a 50-sample pilot and compute agreement.