Handling Unicode And Encoding

Why this matters

Real-world text comes from many sources and languages. As an NLP Engineer, you will:

• Ingest multilingual data (UTF-8, UTF-16, Windows-1252) and prevent mojibake (garbled text like Café).
• Normalize punctuation, whitespace, and accent marks to keep tokenization stable.
• Handle emojis and grapheme clusters so you do not split human-perceived characters.
• Make comparisons and search robust with case folding (e.g., Straße vs STRASSE).
• Avoid breaking downstream models by ensuring consistent encoding and normalization.

Concept explained simply

Unicode is a universal list of characters (each has a code point). An encoding (like UTF-8) decides how to store those code points as bytes. Normalization makes visually similar text share the same internal representation.

• Bytes vs text: bytes are raw data; text (strings) are decoded bytes.
• Encodings: UTF-8 is dominant; UTF-16/UTF-32 use more bytes per character. Legacy encodings (e.g., Windows-1252) still appear.
• Normalization forms: NFC (composed), NFD (decomposed), NFKC/NFKD (compatibility forms; can change meaning/appearance). NFC is a safe default for general storage.
• Case folding: use casefold() for robust case-insensitive matching across languages, not just lower().
• Grapheme clusters: some visible characters are multi-code-point (e.g., 👩🏽‍💻). Avoid naive slicing.
• Invisible/odd whitespace: NBSP (\u00A0), ZWSP (\u200B), BOM (\uFEFF). Clean or replace them.

Mental model

Think of a pipeline:

1. Bytes → Decode once (choose a correct encoding, usually UTF-8; handle BOM).
2. Text → Clean whitespace and control characters (replace NBSP, remove ZWSP/BOM).
3. Normalize → Choose NFC by default; use NFKC and/or accent-stripping only when required.
4. Case strategy → For comparisons/search, casefold. For display, keep original casing.
5. Tokenize → After normalization to keep tokens consistent.
6. Output → Encode as UTF-8 for storage and exchange.

Worked examples

Bad text: "Café"
Heuristic fix (Python):
"Café".encode("latin-1").decode("utf-8")  → "Café"
Note: Works only if the specific mis-decoding happened. Verify before bulk fixing.

# Keep accents but normalize shape
unicodedata.normalize("NFC", "naie")  → "naïve"

# Strip accents to ASCII (search/indexing)
text = unicodedata.normalize("NFKD", "naïve")
stripped = "".join(ch for ch in text if unicodedata.category(ch) != "Mn")
→ "naive"

"Straße".lower()    → "straße"
"Straße".casefold() → "strasse"  # better for case-insensitive matching

# Replace NBSP with normal space, remove ZWSP and BOM
text = text.replace("\u00A0", " ")
text = text.replace("\u200B", "")
text = text.replace("\uFEFF", "")

Practical projects

• Multilingual reviews cleaner: build a function that reads mixed-encoding reviews, decodes safely, normalizes to NFC, replaces NBSP, removes zero-width characters, and outputs UTF-8.
• Search-normalizer: implement a casefold + accent-stripping routine that creates a comparable key for deduplication while retaining original text for display.
• Legacy CSV migration: convert Windows-1252 files to UTF-8, validate special punctuation, and produce a report of cleaned characters.

Exercises

Complete the exercise below. The Quick Test is at the end of this lesson. Your progress in the test is saved if you are logged in; everyone can take it for free.

raw = b"\xef\xbb\xbfCaf\xc3\xa9 \xe2\x80\x94 price\xa0$3"

Café — price $3

import unicodedata, re

def normalize_text(data):
    # 1) Decode, auto-strip BOM if present
    if isinstance(data, bytes):
        text = data.decode("utf-8-sig", errors="strict")
    else:
        text = str(data)
    # 2) Replace NBSP with normal space
    text = text.replace("\u00A0", " ")
    # 3) Remove ZWSP and BOM remnants just in case
    text = text.replace("\u200B", "").replace("\uFEFF", "")
    # 4) Normalize to NFC
    text = unicodedata.normalize("NFC", text)
    # 5) Collapse excessive whitespace
    text = re.sub(r"\s+", " ", text).strip()
    return text

print(normalize_text(b"\xef\xbb\xbfCaf\xc3\xa9 \xe2\x80\x94 price\xa0$3"))
# → Café — price $3

Self-check checklist

• I can explain the difference between Unicode and UTF-8.
• I know when to use NFC vs NFKC.
• I can safely remove or replace NBSP, ZWSP, and BOM.
• I use casefold() for case-insensitive matching.
• I avoid slicing text that may contain grapheme clusters like emojis.

Common mistakes and how to self-check

• Mixing bytes and strings: encoding twice or decoding twice causes mojibake. Self-check: assert isinstance(x, str) after decoding step.
• Assuming ASCII: files default-opened without encoding can break on special characters. Self-check: always set encoding when reading/writing.
• Using lower() instead of casefold(): misses language-specific rules. Self-check: compare both on tricky words (e.g., Straße).
• Over-aggressive normalization: NFKC or accent stripping can change meaning. Self-check: compare before/after samples; only apply where justified.
• Ignoring invisible chars: NBSP/ZWSP leak into tokens. Self-check: reveal code points with repr() to spot \u00A0, \u200B, \uFEFF.
• Naive slicing of emojis: breaks grapheme clusters. Self-check: avoid character-by-character slicing for user-facing text.

Learning path

1. Master Unicode vs encodings and NFC normalization (this lesson).
2. Whitespace and punctuation normalization strategies for token stability.
3. Case handling and locale-aware comparisons (casefold, accent policies).
4. Tokenizer selection and testing on multilingual and emoji-rich data.
5. Performance profiling of normalization in large pipelines.

Who this is for

• NLP Engineers preparing robust preprocessing pipelines.
• Data Scientists handling multilingual corpora.
• ML Engineers and Data Engineers ingesting text from diverse sources.

Prerequisites

• Basic Python knowledge (strings vs bytes, file I/O).
• Familiarity with regular expressions and text processing.
• Awareness of tokenization in NLP.

Mini challenge

Given text: "\uFEFFThis\u200B product\u00A0rocks! Price—$5"

• Describe a cleaning sequence to produce: "This product rocks! Price—$5" with NFC preserved and a single space between words.
• Explain why removing ZWSP and replacing NBSP is important for tokenization.

Next steps

• Integrate a normalization function into your preprocessing pipeline.
• Create tests with tricky cases (emojis, accents, NBSP).
• Take the Quick Test below to confirm understanding. Progress is saved if you are logged in; the test is available to everyone for free.

Quick Test — How it works

Answer the questions. Aim for 70% or higher. If you do not pass, review the exercises and retry.