Bag Of Words And Tf Idf

Why this matters

As an NLP Engineer, you often need a fast, reliable baseline to classify text (spam vs. ham, sentiment, topic routing) or to compute document similarity (search, deduplication). Bag-of-Words (BoW) and TF-IDF are the classic features that power these tasks, especially when data is small-to-medium or when deep models are overkill. They are transparent, quick to train, and easy to debug.

• Create baseline classifiers for new text problems in minutes.
• Build interpretable features for compliance-sensitive use cases.
• Enable search and ranking via cosine similarity over TF-IDF vectors.

Who this is for

Beginners to intermediate practitioners who want strong, interpretable NLP baselines and who need to understand how token-level statistics become features.

Prerequisites

• Comfort with Python basics (lists, dicts)
• Familiarity with simple math (counts, logs)
• Basic understanding of what a token/word is

Concept explained simply

BoW turns each document into a long vector of word counts. The vector has one position per vocabulary term. TF-IDF improves this by down-weighting words that appear in many documents (like "the") and up-weighting words that are distinctive (like "refund").

Core definitions and formulas

• Term Frequency (TF): number of times a term t appears in document d. Often raw count; sometimes normalized by document length.
• Document Frequency (DF): number of documents that contain term t.
• Inverse Document Frequency (IDF): a common smoothed version is idf(t) = log((N + 1) / (DF(t) + 1)) + 1, where N is the number of documents.
• TF-IDF: tfidf(t, d) = TF(t, d) * IDF(t), usually followed by L2 normalization per document.

Worked examples

Example 1: Tiny corpus by hand

Corpus (3 docs):

1. "love pizza love"
2. "hate pizza"
3. "love pasta"

Vocabulary: love, pizza, hate, pasta

Example 2: Why TF-IDF helps

In support tickets, the word "the" appears everywhere and is not predictive. TF-IDF down-weights it. The phrase "refund" or bigram "credit card" becomes comparatively strong, helping classifiers focus on signal, not noise.

Example 3: Cosine similarity for search

Represent the query and documents as L2-normalized TF-IDF vectors. Cosine similarity is then the dot product. Documents with overlapping distinctive terms rank higher, enabling simple keyword-based search with good relevance.

Practical usage (Python)

from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer

corpus = [
    "love pizza love",
    "hate pizza",
    "love pasta"
]

# Bag-of-Words counts
cv = CountVectorizer(lowercase=True, ngram_range=(1,1), min_df=1)
X_counts = cv.fit_transform(corpus)
print("BoW shape:", X_counts.shape)
print("Vocabulary:", cv.get_feature_names_out())

# TF-IDF
tfidf = TfidfVectorizer(lowercase=True, ngram_range=(1,1), min_df=1, norm='l2')
X_tfidf = tfidf.fit_transform(corpus)
print("TF-IDF shape:", X_tfidf.shape)
print("Sample TF-IDF row:", X_tfidf[0].toarray())

Common mistakes and self-check

• Data leakage: fitting vectorizers on full data. Self-check: Did you call fit only on training data?
• Uncontrolled vocabulary growth: memory blow-ups. Self-check: Did you set max_features/min_df?
• Ignoring normalization: cosine similarity becomes meaningless. Self-check: Is norm='l2' used for TF-IDF similarity tasks?
• Over-aggressive stopword removal: losing sentiment or domain terms (e.g., "not"). Self-check: Inspect top features; is "not" missing when it matters?
• Mismatched preprocessing between train and inference. Self-check: Is the same tokenizer/casing applied everywhere?

Exercises

These exercises mirror the tasks below. You can do them here and then compare with the solutions.

1. Exercise 1: Compute TF-IDF by hand on a tiny corpus and L2-normalize vectors. Verify you get the same order of importance as the worked example.
2. Exercise 2: Use a vectorizer to compare unigrams vs. bigrams on short texts. Observe which features become most informative.

Mini challenge

You have 2000 short app reviews. Build two baselines: (a) BoW + Logistic Regression, (b) TF-IDF + Logistic Regression. Use unigram+bigrams and cap vocabulary to 20k. Which setup wins on validation F1, and why? Write 3 bullet points reflecting on the most influential features.

Practical projects

• Spam filter baseline: Train a TF-IDF + linear classifier on a labeled email dataset. Report precision/recall and top positive/negative features.
• FAQ matcher: Index FAQs with TF-IDF, then match user queries via cosine similarity. Add bigrams and compare ranking improvements.
• News topic router: Fit TF-IDF with min_df and max_df, then train a linear SVM. Inspect misclassifications and adjust preprocessing.

Learning path

1. Master BoW and TF-IDF basics (this page).
2. Experiment with n-grams, stopwords, and normalization.
3. Feature selection: chi-square or mutual information to prune features.
4. Compare classifiers: Logistic Regression, Linear SVM, Naive Bayes.
5. Move to embeddings when data and task justify it, keeping BoW/TF-IDF as a strong baseline.

Next steps

• Complete the exercises and take the quick test below.
• Apply TF-IDF to a small dataset you already have and inspect top features.
• Iterate preprocessing (casing, n-grams, min_df) and note validation changes.