Every day, billions of messages, emails, reviews, and articles are written by humans. Computers can store those words — but can they truly understand them? This guide explains the first clever tricks scientists invented to teach computers how to read.

Imagine you have a robot friend who is very good at maths but has never seen a book before. You hand it a story and ask, “Is this story about cats or dogs?” The robot stares blankly — it only speaks numbers, not words! So how do we translate human language into a form computers can work with? That is precisely the puzzle that Natural Language Processing (NLP) tries to solve.

Over many decades, clever researchers came up with two foundational ideas — Bag of Words (BoW) and Term Frequency–Inverse Document Frequency (TF-IDF) — that turned paragraphs of human text into neat tables of numbers that computers could finally start to understand and compare.

All of that data above is mostly text. Without methods like BoW and TF-IDF, none of it could be processed by machine learning algorithms. These two techniques form the very alphabet of modern AI’s ability to deal with language.

Computers are fluent in numbers. Language is full of meaning, emotion, sarcasm, and context. Bridging that gap is one of the biggest challenges in Artificial Intelligence.

When you look at the word “bank”, you instantly know from context whether it means the side of a river or a place to keep money. A computer, looking at just the raw letters B-A-N-K, has no idea which meaning applies. This is called the ambiguity problem, and it is just one of many challenges in language.

Why Can’t Computers Just Read Words?

The journey to solve these problems started with the simplest possible approach: what if we just count words? That idea gave birth to the Bag of Words model — humble in design, yet surprisingly powerful in practice.

Imagine emptying every word from a sentence into a cloth bag and shaking it around. You lose track of the order, but you can still count how many times each word appears. That is the essence of the Bag of Words model.

The Bag of Words approach was one of the earliest and most intuitive ways scientists tried to represent text for computers. The core idea is beautifully simple: treat each piece of text (called a document) as just a collection of words, completely ignoring the sequence in which those words appear. What matters is not where a word is but how often it shows up.

Step-by-Step: How BoW Works

Let’s walk through a real example together. Say we have three short sentences:

Step 1 — Build a Vocabulary

Collect every unique word that appears across all your documents. This is called the vocabulary or corpus vocabulary. In our case 7 unique words make up the entire vocabulary: the, cat, sat, on, mat, hat, dog.

Step 2 — Count Each Word Per Document

For every sentence, go through the vocabulary and write down how many times each word appears. If a word doesn’t appear, write 0.

Each row in the table above is called a vector — a list of numbers that represents one document. Sentence A is now the vector [2, 1, 1, 1, 1, 0, 0]. This is something a computer can work with!

Step 3 — Use the Vectors

Now that every document is a row of numbers, we can do all sorts of clever maths — like measuring how similar two documents are by checking how close their vectors are to each other in mathematical space.

BoW is a great starting point, but it has some significant blind spots. Understanding what it gets wrong is just as important as knowing what it gets right.

Problem 1 — Common Words Dominate

Words like “the”, “a”, “is”, and “and” appear in almost every sentence in the English language. In a BoW model, these high-frequency words get very large counts — making them look extremely “important” when in reality they carry almost no meaning about the topic. These are called stop words.

Problem 2 — Order Is Completely Lost

Consider these two sentences:

• “The dog bit the man” — A dog attacked a human. Sad but not unusual.
• “The man bit the dog” — A human attacked a dog. Very unusual headline news!

In a BoW model, both sentences produce identical vectors. The counts of “the”, “dog”, “bit”, “man” are the same. The computer cannot tell these sentences apart — a huge problem for tasks needing real understanding.

Problem 3 — Huge and Sparse Vectors

Real-world vocabularies can have hundreds of thousands of words. If you are working with a collection of news articles, your vocabulary might contain 80,000 words. Every document then becomes a vector with 80,000 numbers — and most of those numbers will be zero (since any single article uses only a small fraction of all words). This is called sparsity, and it wastes a huge amount of computer memory and slows down processing.

Problem 4 — No Sense of Word Importance

Here is the biggest problem of all: BoW treats every word equally. The word “the” and the word “photosynthesis” both count as one occurrence each — yet clearly “photosynthesis” is far more informative about the topic of a biology article than “the” ever could be. BoW has no way to express that some words are more meaningful than others.

TF-IDF doesn’t just count words — it measures how important a word is to a specific document compared to all other documents. It rewards rare, meaningful words and punishes common, useless ones.

TF-IDF stands for Term Frequency – Inverse Document Frequency. It was developed by computer scientist Karen Spärck Jones in 1972 and remains one of the most widely used text analysis tools in the world today. The technique elegantly combines two separate measurements into a single importance score.

Part 1 — Term Frequency (TF)

The first component, Term Frequency, answers a simple question: “How often does this word appear in this particular document?”

If you have a 100-word document and the word “cat” appears 5 times, then the Term Frequency of “cat” in that document is 5 ÷ 100 = 0.05. Dividing by the total number of words is called normalisation — it ensures that a 1,000-word document doesn’t automatically get higher scores than a 100-word document just because it’s longer.

Part 2 — Inverse Document Frequency (IDF)

This is where TF-IDF gets clever. The second component, Inverse Document Frequency, asks: “In how many documents does this word appear at all?”

If a word appears in every single document in your collection, it’s probably not very helpful for identifying what makes any one document special — it’s likely a common word like “the” or “is”. But if a word appears in only 2 out of 1,000 documents, it must be doing something unique in those 2 documents — it’s probably a topic-specific, meaningful term.

The IDF formula takes the logarithm of how rare a word is across the whole collection. Using a logarithm prevents very rare words from getting wildly large scores, and keeps the numbers manageable.

Part 3 — Putting It Together: TF × IDF

The final TF-IDF score for any word in any document is simply the product of these two values:

This scoring system naturally gives high scores to words that are both frequent in this document and rare across the collection — the hallmarks of a truly meaningful, topic-defining word. Common words like “the” or “and” have very low IDF scores (since they appear everywhere), so their TF-IDF score stays low no matter how often they appear in a single document.

Numbers make more sense when you see them in action. Let’s calculate TF-IDF scores for a real set of documents, step by step.

We will calculate the TF-IDF score for the word “cat” in Document 1.

Calculating Term Frequency (TF)

Document 1 has 10 words: the, cat, sat, on, the, mat, the, cat, is, fat. The word “cat” appears 2 times out of 10 total words.

Calculating Inverse Document Frequency (IDF)

We have 3 documents total (N = 3). How many documents contain the word “cat”? Doc 1 has it, Doc 3 has it — so 2 documents contain “cat” (df = 2).

The Final TF-IDF Score

Now let’s compare scores for different words across our documents:

Notice how “the” gets a score of exactly 0 because it appears in every document (so log(3/3) = log(1) = 0). Meanwhile, “fat”, “jog”, and “friends” — words unique to a single document — receive the highest scores. This is TF-IDF working exactly as intended!

Before applying BoW or TF-IDF, raw text must go through a cleaning pipeline. Skipping this step leads to noisy, inaccurate results — like trying to count items in a messy room without first tidying it up.

Lowercasing

Without converting everything to lowercase, a computer would think “Cat” and “cat” are completely different words — which they clearly are not. Converting all text to lowercase ensures that identical words are counted as one, no matter how they were originally capitalised.

Removing Punctuation

Punctuation marks like commas, full stops, exclamation marks, and quotation marks carry no word meaning in BoW or TF-IDF. Removing them prevents “hello!” and “hello” from being counted as two separate vocabulary items.

Stop Word Removal

Stop words are ultra-common words — “the”, “a”, “an”, “is”, “in”, “on”, “of” — that appear so frequently they tell us nothing useful about what a document is about. Most NLP libraries (like NLTK or spaCy) include pre-built lists of stop words that you can remove in a single step.

Stemming & Lemmatisation

These two techniques reduce words to their base forms so that “running”, “runs”, and “ran” all count as the same word rather than three separate ones. This dramatically reduces vocabulary size and improves accuracy.

• Stemming is the cruder, faster method — it chops word endings off using rules, sometimes producing non-real words (e.g. “fishing” → “fish”, “studies” → “studi”).
• Lemmatisation is smarter and slower — it uses a dictionary to find the actual root form of a word (e.g. “better” → “good”, “are” → “be”).

Tokenisation

Finally, the cleaned text is split into individual tokens — usually individual words, but sometimes phrases or sub-words. This converts the text string into a list that the BoW or TF-IDF algorithm can iterate over.

Both BoW and TF-IDF are built right into the popular Python library scikit-learn. You can implement them in just a few lines of code.

Bag of Words with CountVectorizer

from sklearn.feature_extraction.text import CountVectorizer

# Our collection of documents
documents = [
    "The cat sat on the mat",
    "The cat sat on the hat",
    "The dog sat on the mat"
]

# Create the BoW model
vectorizer = CountVectorizer()
X = vectorizer.fit_transform(documents)

# See the vocabulary
print(vectorizer.vocabulary_)
# → {'cat': 0, 'dog': 1, 'hat': 2, 'mat': 3, 'on': 4, 'sat': 5, 'the': 6}

# See the count matrix
print(X.toarray())
# → [[1, 0, 0, 1, 1, 1, 2],   ← Sentence A
#     [1, 0, 1, 0, 1, 1, 2],   ← Sentence B
#     [0, 1, 0, 1, 1, 1, 2]]   ← Sentence C

TF-IDF with TfidfVectorizer

from sklearn.feature_extraction.text import TfidfVectorizer
import pandas as pd

documents = [
    "The cat sat on the mat",
    "The cat sat on the hat",
    "The dog sat on the mat"
]

# Create TF-IDF model (automatically removes stop words if asked)
tfidf = TfidfVectorizer(stop_words='english')
X_tfidf = tfidf.fit_transform(documents)

# Show as a readable dataframe
df = pd.DataFrame(
    X_tfidf.toarray(),
    columns=tfidf.get_feature_names_out(),
    index=['Sentence A', 'Sentence B', 'Sentence C']
)
print(df.round(3))
# Notice that "the", "on", "sat" disappear (stop words).
# Unique words like "dog", "hat", "cat" get the highest TF-IDF scores.

Finding the Most Important Keywords

import numpy as np

# Get the vocabulary list
feature_names = tfidf.get_feature_names_out()

# For each document, find the top 3 keywords by TF-IDF score
for i, row in enumerate(X_tfidf.toarray()):
    top_indices = np.argsort(row)[::-1][:3]
    top_keywords = [feature_names[j] for j in top_indices if row[j] > 0]
    print(f"Document {i+1} keywords: {top_keywords}")

# Output:
# Document 1 keywords: ['cat', 'mat']
# Document 2 keywords: ['hat', 'cat']
# Document 3 keywords: ['dog', 'mat']

Notice how the keywords extracted are meaningful and topic-specific — no stop words, no generic terms. This is TF-IDF doing its job.

Both methods convert text to numbers, but they do it differently and each has its own strengths and ideal use cases. Here is everything you need to know to choose between them.

Bag of Words — Pros & Cons

TF-IDF — Pros & Cons

BoW and TF-IDF are not just classroom exercises — they power real systems you interact with every day, from spam filters in your inbox to search bars on websites.

BoW and TF-IDF were revolutionary when invented, but the field of NLP has evolved dramatically. Understanding their limitations naturally leads us to the more powerful techniques that came after them.

The Key Problem That Remained

Both BoW and TF-IDF share one deep weakness: they have no understanding of meaning. To them, the words “car” and “automobile” are completely unrelated — they are different strings of letters. But humans know these words mean the same thing! This is called semantic blindness.

Why Still Learn BoW and TF-IDF?

• They’re the foundation. Every data scientist must understand these methods before moving to complex ones. BoW and TF-IDF teach you the core challenges of text representation.
• They’re still competitive. For many practical tasks — spam filtering, document categorisation, keyword extraction — TF-IDF outperforms or matches deep learning methods, especially with small datasets.
• They’re explainable. You can look at the TF-IDF scores and immediately understand why a document was classified a certain way. Neural network decisions are often impossible to explain — a major disadvantage in healthcare, law, and finance.
• They’re fast and cheap. Training a TF-IDF model takes seconds on a laptop. Training a large language model takes weeks on expensive GPU clusters costing millions of dollars.
• They appear in interviews. Almost every NLP job interview includes questions about BoW and TF-IDF. Knowing them deeply signals strong foundational knowledge.

We’ve gone from the very basics of why computers struggle with words all the way through to Python code and real-world applications. Here’s everything you need to remember.

Quick Reference Cheat Sheet