Weak Supervision Basics

Who this is for

• Applied Scientists who need labeled data quickly to prototype and iterate.
• Data Scientists/ML Engineers working with limited annotation budgets or fast-changing labels.
• Anyone building classifiers where simple rules or weak signals can approximate labels.

Prerequisites

• Basic supervised learning concepts (labels, train/val/test).
• Binary/multi-class classification understanding.
• Comfort reading simple rules/regex and interpreting confusion matrices.

Why this matters in the job

• Bootstraps labels for new problems before full annotation is funded.
• Lets you ship a credible baseline model in days, then refine with targeted labeling.
• Captures domain knowledge as code (labeling functions) that is reusable and auditable.
• Enables rapid iteration when taxonomies shift or new classes appear.

Concept explained simply

Weak supervision means you generate noisy, programmatic labels using heuristics, existing models, patterns, metadata, or distant sources instead of hand-labeling everything. You write small labeling functions (LFs) that vote for a class or abstain. Because LFs are noisy and may conflict, you combine them with a label model to estimate per-example probabilistic labels and per-LF accuracies. You then train a standard classifier on these probabilistic labels and evaluate on a small clean set.

Mental model

Imagine a committee of noisy experts. Each expert can say Class A, Class B, or pass. Some experts are more reliable; some copy others. The label model learns whom to trust and how to down-weight correlated experts. The output is a probability per class for each example, which you use to train a downstream model.

Typical pipeline

1. Collect unlabeled data and a small gold set (clean labels) for evaluation.
2. Write diverse LFs that capture different signals; allow them to abstain often.
3. Analyze LF coverage, conflicts, and estimated accuracies.
4. Fit a label model to get probabilistic labels.
5. Train a classifier on these probabilistic labels; validate on the gold set.
6. Iterate: improve LFs, handle class imbalance, prune or merge redundant LFs.

Worked examples

Example 1: Sentiment from text (binary)

Task: Positive vs Negative for short reviews.

• LF1: If text contains words like 'love', 'great', 'amazing' and no strong negation within 3 tokens, vote POS; else abstain.
• LF2: If text contains 'terrible', 'awful', 'hate', vote NEG; else abstain.
• LF3: If rating metadata >= 4 stars, vote POS; else abstain.

Outcome: LF1 and LF2 have moderate coverage; LF3 has high precision but only when rating exists. Conflicts happen for sarcastic texts. The label model will often trust LF3 more if it proves precise on the gold set.

Example 2: Spam detection (binary)

• LF1: If contains 'free', 'prize', 'win', vote SPAM.
• LF2: If contains 3+ exclamation marks or suspicious URL patterns, vote SPAM.
• LF3: If contains business terms like 'meeting', 'invoice', 'report', vote HAM.

Conflicts: 'free meeting invite' triggers both SPAM and HAM. Majority vote ties; the label model uses estimated accuracies to break ties or assign a soft probability (e.g., SPAM=0.45).

Example 3: News topic classification (multi-class)

• LF1: If mentions 'stock, earnings, market', vote BUSINESS.
• LF2: If mentions 'tournament, scored, coach', vote SPORTS.
• LF3: Use a weak external model (older topic classifier) — if confidence > 0.8, emit its predicted class; else abstain.

Downstream: The label model down-weights LF3 if it duplicates LF1/2 too often or proves overconfident on the gold set.

Designing good labeling functions

1. Start precise, not broad: prefer narrow high-precision rules that abstain often.
2. Diversify signals: text patterns, metadata, model predictions, time-of-day, source.
3. Make abstention easy: explicitly code conservative default to abstain.
4. Handle negation and context: include simple guards like 'not', 'no', question marks.
5. Limit overlap: avoid multiple LFs that are near-duplicates; or mark them and expect correlation.
6. Test on a tiny gold set: estimate precision/coverage; remove low-value LFs.
7. Document each LF: intended signal, expected failure modes.

Exercises you can do now

These mirror the graded exercises below. Do them here, then record your final answers in the exercise submissions.

Exercise 1 (mirrors ex1): Write 3 labeling functions for sentiment

1. Use the following mini-dataset: ['love the camera but not the battery', 'this is awful', 'great value!!!', 'ok quality, decent price', 'hate the update'].
2. Define 3 LFs. For each LF, write: name, trigger rule, label (POS/NEG), and an abstain condition.
3. Apply them to the 5 examples. For ties, abstain.
4. Report per-LF coverage and your expected precision (rough estimate).

Exercise 2 (mirrors ex2): Compute coverage, conflict, and MV

Task: Spam (1) vs Ham (0). Use 8 examples E1..E8 with LF votes below (A=abstain):

E1: L1=1, L2=A, L3=A   (Gold=1)
E2: L1=A, L2=1, L3=A   (Gold=?)
E3: L1=A, L2=A, L3=0   (Gold=0)
E4: L1=A, L2=A, L3=0   (Gold=?)
E5: L1=1, L2=A, L3=0   (Gold=0)
E6: L1=A, L2=A, L3=A   (Gold=?)
E7: L1=1, L2=1, L3=A   (Gold=1)
E8: L1=1, L2=A, L3=0   (Gold=?)

1. Compute coverage for L1, L2, L3 (non-abstains / 8).
2. Compute conflict rate across all 8 examples (where at least two emitted and disagreed).
3. Compute majority vote per example (tie -> abstain).
4. On the gold subset {E1,E3,E5,E7}, estimate accuracy for each LF and MV (exclude abstains from denominator).

Common mistakes and how to self-check

• Overly aggressive LFs: If an LF fires on most data, inspect precision; add stricter guards or abstain more.
• No abstains: Forces noisy labels; add uncertainty thresholds and negate cases.
• Duplicate LFs: Highly correlated rules double-count evidence; merge or keep one.
• Ignoring class imbalance: Add class priors, balance thresholds, or write LFs for minority class specifically.
• Evaluating only on weak labels: Always maintain a small gold set to detect drift and estimate true gains.

Practical projects

• Customer Support Triage: Build weakly supervised intent classifier using keywords, FAQ matches, and time-of-day.
• Job Post Moderation: Flag posts using pattern rules, user reputation metadata, and a small pre-trained toxicity model.
• App Review Sentiment: Combine star ratings (when present), sentiment lexicons, and negation-aware rules.

Learning path

1. Master LF design: write precise, diverse, abstaining rules.
2. Metrics: measure coverage, conflict, and per-LF precision on a gold slice.
3. Label model: move beyond majority vote; learn LF accuracies/correlations.
4. Train on probabilistic labels: calibrate thresholds and loss weighting.
5. Iterate with error analysis: add/modify LFs for recurring failure modes.
6. Scale: template LFs, auto-generate variants, and maintain documentation.

Mini challenge

Given texts: ['urgent: win tickets now', 'quarterly report draft', 'love this product', 'not great, actually disappointing'], propose 5 LFs with triggers, abstain logic, and expected precision. Identify two likely conflicts and how the label model could resolve them.

Next steps

• Extend to semi-supervised learning: mix weak and a small set of strong labels.
• Add data slices: write LFs that specifically target known hard cases.
• Tune thresholds and class priors to handle imbalance.

Quick test

Everyone can take the quick test; only logged-in users will see saved progress.