Text Data Collection And Labeling

Why this skill matters for NLP Engineers

High-performing NLP systems start with quality datasets. As an NLP Engineer, you will define label schemas, source and sample text, write annotation guidelines, measure labeler agreement, use weak supervision to scale, and split/version datasets to train reliable models. Mastering this skill lets you ship robust models faster and with fewer surprises.

Who this is for

• Aspiring and junior NLP Engineers who need end-to-end dataset workflows.
• Data scientists transitioning to NLP projects.
• ML engineers who want repeatable, auditable dataset pipelines.

Prerequisites

• Basic Python and familiarity with NumPy/Pandas.
• Intro ML knowledge (train/validation/test concepts).
• Comfort with version control (e.g., Git concepts).

Learning path

1. Define the task and label schema: Clarify task scope, labels, definitions, and edge cases.
2. Source and sample text data: Collect from multiple sources, deduplicate, and sample representatively.
3. Write annotation guidelines and pilot: Small pilot, refine instructions, measure inter-annotator agreement (IAA).
4. Scale labeling with quality checks: Use spot-checks, gold questions, and weak supervision to boost coverage.
5. Handle imbalance and create splits: Stratify, group-aware splitting, and track dataset versions with metadata.

Worked examples

1) Define a label schema and validate labels

# Intent classification: 5 intents + OTHER
LABELS = {
    0: "greeting",
    1: "order_status",
    2: "refund_request",
    3: "product_info",
    4: "complaint",
    5: "other"  # catch-all, use sparingly with clear guidance
}

samples = [
    {"text": "Hi there", "label": 0},
    {"text": "Where is my order?", "label": 1},
    {"text": "Details about your earbuds?", "label": 3},
]

def validate_labels(rows, label_map):
    bad = []
    for i, r in enumerate(rows):
        if r["label"] not in label_map:
            bad.append((i, r))
    return bad

print(validate_labels(samples, LABELS))  # [] means all good

Tip: Avoid overlapping labels. Use mutually exclusive definitions and examples per label, including counter-examples for the “other” class.

2) Representative sampling with stratification

import numpy as np
from sklearn.model_selection import train_test_split

# Simulated labels with imbalance
y = np.array([0]*50 + [1]*20 + [2]*10 + [3]*5 + [4]*15 + [5]*30)
X = np.arange(len(y))

X_train, X_tmp, y_train, y_tmp = train_test_split(
    X, y, test_size=0.3, random_state=42, stratify=y
)
X_val, X_test, y_val, y_test = train_test_split(
    X_tmp, y_tmp, test_size=0.5, random_state=42, stratify=y_tmp
)

print({
    'train_size': len(X_train),
    'val_size': len(X_val),
    'test_size': len(X_test)
})

Stratification maintains the label distribution across splits, reducing evaluation variance.

3) Inter-annotator agreement (Cohen’s kappa)

from sklearn.metrics import cohen_kappa_score

# Two annotators labeled the same 12 items
ann_a = [0,0,1,1,2,0,3,1,2,2,4,5]
ann_b = [0,0,1,2,2,0,3,1,2,2,4,5]

kappa = cohen_kappa_score(ann_a, ann_b)
print("Cohen's kappa:", round(kappa, 3))

# Rough interpretation:
# 0.0-0.20 slight, 0.21-0.40 fair, 0.41-0.60 moderate,
# 0.61-0.80 substantial, 0.81-1.00 almost perfect

If kappa is low, refine guidelines, add examples, or merge ambiguous labels.

4) Weak supervision with labeling functions and majority vote

import re
from collections import Counter

def lf_greeting(text):
    return 0 if re.search(r"\b(hi|hello|hey)\b", text, re.I) else None

def lf_complaint(text):
    return 4 if re.search(r"\b(broken|angry|terrible|complain)\b", text, re.I) else None

def lf_order_status(text):
    return 1 if re.search(r"\border\b|\btracking\b", text, re.I) else None

LFs = [lf_greeting, lf_complaint, lf_order_status]

def weak_label(text):
    votes = [lf(text) for lf in LFs]
    votes = [v for v in votes if v is not None]
    if not votes:
        return None
    return Counter(votes).most_common(1)[0][0]

texts = [
    "Hello team!",
    "I'm angry, this is broken",
    "Order 123 needs tracking info",
    "What is your return policy?"
]
print([weak_label(t) for t in texts])  # e.g., [0, 4, 1, None]

Use weak labels to prioritize manual review or pretrain models. Always mark provenance so you can filter or reweight later.

5) Handling class imbalance with class weights

import numpy as np
from sklearn.utils.class_weight import compute_class_weight

labels = np.array([0]*50 + [1]*20 + [2]*10 + [3]*5 + [4]*15 + [5]*30)
classes = np.unique(labels)
weights = compute_class_weight('balanced', classes=classes, y=labels)
class_weight = {c:w for c,w in zip(classes, weights)}
print(class_weight)

# Example model usage (if supported):
# clf = LogisticRegression(class_weight=class_weight, max_iter=1000)

Combine with data augmentation or active learning to increase minority class coverage.

6) Group-aware splits and dataset versioning

import hashlib, json, random
from sklearn.model_selection import GroupKFold

# Example: group by user_id to avoid leakage across splits
samples = [
    {"text": "Hi", "label": 0, "user_id": 100},
    {"text": "Where's my order?", "label": 1, "user_id": 101},
    {"text": "Angry about this", "label": 4, "user_id": 100},
    {"text": "Order tracking please", "label": 1, "user_id": 102},
    {"text": "Hello there", "label": 0, "user_id": 103},
]

X = list(range(len(samples)))
y = [s["label"] for s in samples]
groups = [s["user_id"] for s in samples]

# GroupKFold for demonstration (no stratification here)
gkf = GroupKFold(n_splits=3)
splits = list(gkf.split(X, y, groups))
print("n_splits:", len(splits))

# Simple dataset fingerprint
payload = json.dumps(sorted(samples, key=lambda s: s["text"]))
version = hashlib.sha1(payload.encode("utf-8")).hexdigest()[:10]
meta = {"version": version, "n_rows": len(samples), "labels": sorted(set(y))}
print(meta)

Record dataset version, source timestamps, and preprocessing steps in a small metadata file for reproducibility.

Drills and exercises

• Write a one-sentence task statement and a label map with clear, mutually exclusive definitions.
• Collect a small text pool (100–300 rows). Deduplicate exact and near-duplicates.
• Draft annotation guidelines with at least 3 positive and 3 negative examples per label.
• Run a 30-item pilot with two annotators. Compute Cohen’s kappa and list 3 disagreements.
• Create stratified train/val/test splits; verify label proportions per split.
• Implement 3 labeling functions and compare their coverage and precision on a sample.
• Create a dataset metadata JSON containing version hash, sources, label stats, and split sizes.

Common mistakes and debugging tips

• Labels overlap: Merge or redefine; add counter-examples to guidelines.
• Using “other” too often: Tighten rules; create a new label if “other” exceeds ~10–15% without clear reason.
• Leakage across splits: Ensure group-aware splits (user/session/document). Check duplicates across splits.
• Poor IAA: Add more examples, clarify edge cases, run a second pilot before scaling.
• Imbalance ignored: Use stratified sampling, class weights, augmentation, or active learning.
• Untracked changes: Version everything: raw dump, cleaning script, split seeds, and metadata.

Practical projects

• Intent classifier for support tickets with 5–7 intents and a controlled “other”.
• Toxicity or policy violation triage with weak supervision plus manual review.
• FAQ routing using domain-specific labeling functions and active learning loops.

Mini project: Content moderation classifier

Goal: Build a small dataset for a binary classifier (allowed vs. needs review) that you can iterate on quickly.

1. Define schema: Two labels with precise rules. Add examples and counter-examples.
2. Source text: Collect 500–1,000 lines from safe, permitted sources. Deduplicate.
3. Pilot + guidelines: Label 50 items with two annotators. Compute IAA; revise guidelines.
4. Weak supervision: 3–5 labeling functions for common policy triggers. Use majority vote; mark weak labels.
5. Balance + splits: Stratify into 70/15/15. Ensure no near-duplicates across splits.
6. Version: Save a metadata JSON with version hash, counts, and process notes.

Subskills

• Defining Task And Label Schema: Craft clear task scope and mutually exclusive labels with edge cases and counter-examples.
• Data Sourcing And Sampling: Gather, clean, deduplicate, and stratify samples to reflect real-world distributions.
• Annotation Guidelines: Write concise, example-rich rules that annotators can follow consistently.
• Inter Annotator Agreement Basics: Measure agreement (e.g., Cohen’s kappa) and iterate guidelines to improve.
• Weak Supervision Basics: Use labeling functions and heuristics to generate provisional labels at scale.
• Handling Class Imbalance: Apply stratification, class weights, augmentation, or active learning.
• Train Validation Test Splits For Text: Create leakage-safe, stratified, and group-aware splits.
• Dataset Versioning Practices: Hash, document, and track data and preprocessing for reproducibility.

Next steps

• Finish the drills and mini project to solidify workflow habits.
• Take the skill exam to assess readiness. Progress is saved for logged-in users; everyone can take the exam.
• Apply these practices to your next NLP task and compare model performance before/after better labeling.