Text Preprocessing And Normalization

Why this matters for an NLP Engineer

Text preprocessing and normalization turn messy, real-world text into consistent, machine-readable input. This unlocks reliable tokenization, accurate embeddings, fairer model training, safer data handling (PII redaction), and stable production pipelines. Strong preprocessing often delivers bigger gains than swapping models.

What you will be able to do

• Clean and normalize multilingual text safely and reproducibly.
• Detect language, segment sentences, and standardize casing/accents.
• Handle Unicode, encoding errors, and mojibake.
• Reduce noise: HTML, emojis, URLs, boilerplate, and typos.
• Apply stopwords and lemmatization thoughtfully to preserve meaning.
• Redact PII while keeping text useful for modeling.
• Build reusable, testable preprocessing pipelines for batch and realtime.

Who this is for

• Data scientists and NLP engineers preparing datasets for training or inference.
• ML engineers shipping text models to production.
• Analysts standardizing logs, chat transcripts, forms, or reviews.

Prerequisites

• Comfortable with Python and basic string handling.
• Familiarity with regex and basic data structures (lists, dicts).
• Basic NLP concepts: tokens, sentences, vocabulary, stemming/lemmatization.

Learning path (roadmap)

1. Text cleaning essentials: trim whitespace, remove boilerplate/HTML, normalize quotes and dashes.
2. Unicode and encoding: ensure UTF-8, apply Unicode normalization (NFC/NFKC), fix mojibake.
3. Sentence segmentation and token-friendly formatting: robust sentence splitting before tokenization.
4. Normalization: casefolding, accent stripping when appropriate, consistent number/date formats.
5. Noise and typos: strip URLs, emails, code blocks; handle duplicates; apply light spell correction where safe.
6. Stopwords and lemmatization: language-aware, domain-aware decisions; avoid over-removal.
7. Language detection: route multilingual text to language-specific flows.
8. PII redaction: mask emails, phones, IDs; preserve structure; log redaction stats.
9. Reusable pipelines: compose idempotent, testable steps with timing and metrics.
10. Evaluate and debug: build small gold sets; measure impact on downstream metrics.

Tip: Idempotency

Make each step safe to run multiple times without changing the result. This simplifies retries and batch vs. streaming parity.

Worked examples

1) Unicode normalization and accent handling

import unicodedata

def normalize_unicode(text: str, form="NFC", strip_accents=False):
    t = unicodedata.normalize(form, text)
    if strip_accents:
        t = ''.join(ch for ch in unicodedata.normalize('NFD', t)
                    if unicodedata.category(ch) != 'Mn')
    return t

sample = "Café – coöperate — final"
print(normalize_unicode(sample, form="NFKC", strip_accents=False))
print(normalize_unicode(sample, form="NFKC", strip_accents=True))

Notes: NFKC can fold compatibility characters (e.g., ligatures). Accent stripping is language- and task-dependent.

2) Smart case folding with domain exceptions

def smart_casefold(text: str):
    # Preserve patterns like URLs, emails, and all-caps acronyms length >= 2
    import re
    url_re = re.compile(r"(https?://\S+)")
    email_re = re.compile(r"(\b[\w.%+-]+@[\w.-]+\.[A-Za-z]{2,}\b)")

    placeholders = {}
    def stash(pattern, t, tag):
        i = 0
        def repl(m):
            nonlocal i
            key = f"__{tag}{i}__"
            placeholders[key] = m.group(0)
            i += 1
            return key
        return pattern.sub(repl, t)

    t = text
    t = stash(url_re, t, 'URL')
    t = stash(email_re, t, 'MAIL')

    # Casefold general text
    t = t.casefold()

    # Restore placeholders
    for k, v in placeholders.items():
        t = t.replace(k, v)
    return t

print(smart_casefold("EMAIL Bob@Example.COM said SEE HTTP://EXAMPLE.COM"))

Keep semantic tokens intact before casefolding to avoid breaking them.

3) Sentence segmentation (robust to abbreviations)

import re

ABBREV = {"mr.", "mrs.", "dr.", "prof.", "inc.", "e.g.", "i.e.", "vs."}

# Simple heuristic: split on punctuation followed by space and capital, but guard abbreviations

def split_sentences(text: str):
    candidates = re.split(r"([.!?])\s+", text)
    sents = []
    cur = ""
    for i in range(0, len(candidates), 2):
        part = candidates[i]
        punct = candidates[i+1] if i+1 < len(candidates) else ''
        piece = (part + punct).strip()
        if not piece:
            continue
        lower_piece = piece.lower()
        if punct and any(lower_piece.endswith(a) for a in ABBREV):
            cur += (piece + " ")
        else:
            cur += piece
            sents.append(cur.strip())
            cur = ""
    if cur.strip():
        sents.append(cur.strip())
    return sents

print(split_sentences("Dr. Smith went home. It was late, e.g. 11 p.m. He slept."))

For production, prefer tested libraries, but start with clear heuristics and tests.

4) De-duplication and near-duplicates

import hashlib

def stable_fingerprint(text: str) -> str:
    # Normalize whitespace and case before hashing
    t = ' '.join(text.split()).casefold()
    return hashlib.sha1(t.encode('utf-8')).hexdigest()

seen = set()
rows = ["Hello   world!", "hello world!", "Hello  WORLD !"]
unique = []
for r in rows:
    fp = stable_fingerprint(r)
    if fp not in seen:
        seen.add(fp)
        unique.append(r)
print(unique)

Hash on a normalized representation to collapse trivial duplicates.

5) PII redaction with structure-preserving masks

import re

PATTERNS = {
    'email': re.compile(r"\b[\w.%+-]+@[\w.-]+\.[A-Za-z]{2,}\b"),
    'phone': re.compile(r"\b(?:\+?\d[\s-]?){7,15}\b"),
    'ip': re.compile(r"\b(?:\d{1,3}\.){3}\d{1,3}\b"),
}

REPLACERS = {
    'email': lambda m: "<EMAIL>",
    'phone': lambda m: "<PHONE>",
    'ip': lambda m: "<IP>",
}

def redact_pii(text: str):
    t = text
    for name, pat in PATTERNS.items():
        t = pat.sub(REPLACERS[name], t)
    return t

print(redact_pii("Email me at a.b@example.com or call +1-202-555-0147 from 10.0.0.1"))

Ethics and compliance matter: only keep what you need. Log redaction counts for auditing.

6) A minimal, composable preprocessing pipeline

from dataclasses import dataclass, field
from typing import Callable, List
import time

@dataclass
class Step:
    name: str
    fn: Callable[[str], str]

@dataclass
class Pipeline:
    steps: List[Step] = field(default_factory=list)

    def run(self, text: str):
        t = text
        metrics = []
        for s in self.steps:
            start = time.perf_counter()
            t = s.fn(t)
            metrics.append((s.name, len(t), time.perf_counter() - start))
        return t, metrics

import re, unicodedata

def to_utf8(t):
    return unicodedata.normalize('NFKC', t)

def normalize_ws(t):
    return re.sub(r"\s+", " ", t).strip()

def strip_urls(t):
    return re.sub(r"https?://\S+", " <URL> ", t)

pipe = Pipeline([
    Step("unicode_nfkc", to_utf8),
    Step("strip_urls", strip_urls),
    Step("normalize_ws", normalize_ws),
])

clean, metrics = pipe.run("Check https://ex.com   NOW — okay? ")
print(clean)
print(metrics)

Keep steps small, named, timed, and idempotent.

Drills and exercises

• Create a function that converts curly quotes and long dashes to straight ASCII equivalents, then normalizes whitespace.
• Build a regex to remove only tracking parameters from URLs (e.g., utm_*), leaving the base URL.
• Write a sentence splitter that keeps numbers like 3.14 or version 2.0 intact.
• Implement a language detection heuristic using Unicode script ranges and stopword hits for 2–3 languages.
• Design a PII redactor that masks but preserves string lengths for emails and phones.
• Create tests that verify idempotency for each preprocessing step.

Answer checks you can use

• Running the same step twice yields the same output.
• No loss of required tokens (e.g., URLs protected before casefolding) in golden examples.
• Metrics show no step unexpectedly increases output length.

Common mistakes and debugging tips

• Over-normalization: Stripping accents in languages where they change meaning. Fix: make it language-aware; do not strip for languages that rely on diacritics.
• Aggressive stopword removal: Removing negations or domain-specific terms. Fix: customize stopword lists per task.
• Breaking semantic tokens: Lowercasing inside URLs/emails mid-step. Fix: protect tokens with placeholders, then restore.
• Encoding drift: Mixing forms (NFC/NFKC). Fix: set one canonical form at pipeline start.
• Sentence splitter leaks: Abbreviations causing false splits. Fix: maintain abbreviation lexicon and tests.
• Unbounded regex: Catastrophic backtracking on pathological inputs. Fix: prefer explicit, linear-time patterns; test worst cases.
• Non-idempotent steps: Replacing tokens differently on second pass. Fix: write steps to be order-safe and re-entrant.

Mini project: Clean a noisy support chat dataset

Goal: Build a pipeline that ingests raw multilingual chat logs and outputs clean, language-tagged, PII-redacted, sentence-segmented text ready for modeling.

1. Sample 1,000 messages representative of the noise: emojis, links, devices, mixed languages.
2. Define a gold set of 50 messages with expected outputs (before/after pairs).
3. Implement steps: Unicode normalize → protect URLs/emails → casefold → strip URLs (replace with <URL>) → normalize whitespace → language detect → PII redact → sentence split.
4. Collect metrics: step runtimes, tokens per message, redaction counts, language distribution.
5. Evaluate: measure downstream improvement on a small intent-classification model or keyword recall.
6. Package: expose a function preprocess(text, lang_hint=None) and a CLI that reads/writes JSONL.

Extension ideas

• Add light spell correction for English only when confidence is high.
• Preserve emoji semantics by mapping to shortcodes like :smile: or category tags.
• Cache language detection for repeated users or domains.

Subskills

• Cleaning And Normalizing Text: Whitespace, punctuation, HTML/boilerplate removal, consistent casing.
• Language Detection Basics: Route text to language-aware tokenization and resources.
• Handling Unicode And Encoding: UTF-8 safety, normalization forms, mojibake repair.
• Sentence Segmentation: Reliable sentence boundaries across abbreviations and edge cases.
• Handling Noise And Typos: Remove URLs/emails/code, de-duplicate, cautious spellfix.
• Stopwords And Lemmatization Basics: Reduce sparsity while preserving meaning.
• PII Redaction Basics: Mask sensitive data with auditability.
• Building Reusable Preprocessing Pipelines: Composable, idempotent, testable steps with metrics.

Next steps

• Turn your mini project into a small, reusable library with tests and benchmarks.
• Create a 200–300 sentence gold set to regression-test your pipeline when requirements change.
• Prepare a short report: pipeline diagram, step timings, error cases, and how they were mitigated.