Text Becomes a Vector: My First Steps in NLP

Lecture 14. The last one in the course, and the topic closes the entire playlist with a nice twist: every model I've used so far, from linear regression back in the first post to feature selection in the previous post, expects a vector of numbers as input. A pixel is already a number. A wine's chemical measurement is already a number. But text, a movie review written by a real person, isn't a number at all. This lecture is about the first step of any NLP (Natural Language Processing) task: how to turn text into a vector without losing what matters.

Standardize and tokenize: the bare minimum

Before any counting, the professor cleans the text: everything lowercase, punctuation gone.

def standardize(text):
    text = text.lower()
    return "".join(c for c in text if c not in string.punctuation)

def tokenize(text):
    return standardize(text).split()

tokenize("I write, erase, rewrite, erase again, and then a poppy blooms!")

Output: ['i', 'write', 'erase', 'rewrite', 'erase', 'again', 'and', 'then', 'a', 'poppy', 'blooms'].

Notice "Erase" and "erase" are now the same word, and the comma right after the first "erase" is gone instead of sticking to it. Without this step, the model would treat "erase" and "erase," (comma glued on) as two completely different words, and "Write" and "write" too, inflating the vocabulary with duplicates that shouldn't exist. It's the same spirit as the normalization I already saw back in the KNN post, just rescaling text into a canonical form instead of rescaling numbers.

The homemade vectorizer: word becomes index

class Vectorizer:
    def standardize(self, text):
        ...
    def tokenize(self, text):
        ...
    def make_vocabolary(self, dataset):
        self.vocabulary = {"": 0, "[UNK]": 1}
        for text in dataset:
            text = self.standardize(text)
            tokens = self.tokenize(text)
            for token in tokens:
                if token not in self.vocabulary:
                    self.vocabulary[token] = len(self.vocabulary)

    def encode(self, text):
        text = self.standardize(text)
        tokens = self.tokenize(text)
        return [self.vocabulary.get(token, 1) for token in tokens]

The idea is literal: every word seen during training gets an integer, an index in a dictionary. "[UNK]" (unknown) is reserved at index 1 from the start, for any new word that shows up later, in a test sentence, that wasn't in the training vocabulary. The professor tests this with a sentence using the word "still", absent from the 3-sentence training dataset:

Output: the word "still" becomes index 1, the same as [UNK], while the other words in the sentence (already seen during training) become their real indices.

That's the same problem an unseen category solved back in the Titanic post with OneHotEncoder: what to do when production data brings something training never saw. There it was an embarkation category, here it's a word.

The dataset: 50 thousand real movie reviews

The professor downloads the IMDB Large Movie Review Dataset, from Stanford, one of the most cited datasets in sentiment analysis: 50 thousand real IMDB reviews, half labeled positive, half negative, already split 25 thousand train / 25 thousand test.

train_df = text_dataset_from_directory('aclImdb/train')
test_df = text_dataset_from_directory('aclImdb/test')

Output: 25000 training rows, 25000 test rows, columns text and label.

Bag-of-words: every word is a vote, no order

The first strategy for vectorizing an entire review: a vector the size of the vocabulary, with 1 in every position whose word appears in the review, 0 everywhere else. Completely ignores word order, just marks presence.

class UnigramTransformer(BaseEstimator, TransformerMixin):
    def fit(self, X, y=None):
        self.vocabulary = {"": 0, "[UNK]": 1}
        for text in X:
            for token in set(self.tokenize(text)):
                if token not in self.vocabulary and len(self.vocabulary) < self.max_features:
                    self.vocabulary[token] = len(self.vocabulary)
        return self

    def transform(self, X, y=None):
        rows, cols, data = [], [], []
        for row, text in enumerate(X):
            for token in set(self.tokenize(text)):
                rows.append(row)
                cols.append(self.vocabulary.get(token, 1))
                data.append(1)
        return csr_matrix((data, (rows, cols)), shape=(len(X), len(self.vocabulary)))

Notice the csr_matrix: with a 10-thousand-word vocabulary and each review only using a few hundred of those words, the matrix is almost entirely zero. Storing a dense vector of 10 thousand positions, almost all zero, for each of the 20 thousand reviews would waste a huge amount of memory. The sparse format only stores the positions with a nonzero value.

pipeline = Pipeline([("vectorizer", UnigramTransformer(10000)), ("classifier", RandomForestClassifier(random_state=42))])
pipeline.fit(train_texts, train_labels)

Output: 0.832 validation accuracy, just from which words show up, with no order at all.

TF-IDF: not every word carries the same weight

Bag-of-words treats "the" (shows up in almost every review) and "wonderful" (shows up mostly in the good ones) the same way: both count as 1 if present. But "the" carries no information at all about whether a review is good or bad, while "wonderful" carries plenty. TF-IDF fixes this with two multiplied quantities:

class TfidfTransformer(BaseEstimator, TransformerMixin):
    def fit(self, X, y=None):
        doc_freq = {}
        for text in X:
            for token in set(self.tokenize(text)):
                doc_freq[token] = doc_freq.get(token, 0) + 1
        ...
        self.idf = {token: math.log(len(X) / freq) for token, freq in doc_freq.items()}
        return self

    def transform(self, X, y=None):
        ...
        for token, count in token_counts.items():
            tf = count / total_tokens_in_document
            value = tf * self.idf[token]

TF (term frequency) is simple: how many times the word appears in the review, divided by the review's total word count, the "local" weight. IDF (inverse document frequency) is the "global" weight: log(total documents / documents containing the word). A word appearing in almost every document (like "the") has an IDF near zero, barely counting at all. A rare word, present in few documents, has a high IDF. Multiplying the two, a word only gets a high weight if it's frequent in that specific review and rare across the rest of the corpus, exactly the kind of word that helps tell one review apart from another.

pipeline = Pipeline([("vectorizer", TfidfTransformer(10000)), ("classifier", RandomForestClassifier(random_state=42))])

Output: 0.8316 validation accuracy, essentially tied with bag-of-words (0.832).

Interactive: type a sentence below and watch the TF-IDF weight of each word, computed against a small 16-review example corpus. A common word like "the" or "was" gets a short bar, a rare, loaded word like "wonderful", "terrible" or "boring" gets a much taller one, and a word that this tiny example corpus never saw shows up marked as [UNK], with no weight, the same problem the homemade vectorizer had above.

On the full dataset, against the official TfidfVectorizer

With both techniques tested on validation, the professor retrains on the entire training set (25 thousand reviews, not just the 20-thousand slice) and measures on the real test set:

TF-IDF wins by a small margin over bag-of-words, and more importantly: my homemade TF-IDF implementation (0.84292) lands less than half a percentage point from scikit-learn's own official implementation (0.83872), the same "matches the professional tool, down to the decimal" test I already saw back in the normal equation post and in the PCA post. The real TfidfVectorizer has extra optimizations and details (L2 normalization of the final vector, for instance), but the core idea, the same TF times IDF math, is identical.

Wrapping up

And with that, the 14 lectures in this course close out. I started fitting a line with the normal equation and end here, turning a movie review into a vector of weighted words. Along the way there were trees, ensembles, clusters, dimensionality reduction, imbalanced data, with Bishop holding down the theoretical side of almost all of it, except the last three lectures (DBSCAN, semi-supervised learning, and NLP), which came after his 2006 book and still fit right into the same recurring logic: data becomes a vector, a vector becomes a decision. Thanks, professor Boldt.

Practical application

To confirm the lecture's finding (bag-of-words, homemade TF-IDF, and the official TfidfVectorizer landing on similar results) on a dataset different from IMDB, I used scikit-learn's own 20newsgroups: real forum posts, on two clearly distinct topics, sci.space and rec.sport.baseball, 1190 training posts and 791 test posts.

from sklearn.datasets import fetch_20newsgroups
cats = ['sci.space', 'rec.sport.baseball']
train = fetch_20newsgroups(subset='train', categories=cats, remove=('headers','footers','quotes'), random_state=42)
test = fetch_20newsgroups(subset='test', categories=cats, remove=('headers','footers','quotes'), random_state=42)

I reproduced all three approaches (UnigramTransformer and TfidfTransformer exactly as the professor built them, plus the official TfidfVectorizer), with RandomForestClassifier(random_state=42) on top of a 5000-word vocabulary:

Three close numbers again, on a completely different topic and dataset from the IMDB movies, which gives confidence the lecture's pattern wasn't a coincidence of that specific dataset: any reasonable text vectorization (word presence or TF-IDF) already delivers most of the useful signal for separating two clearly distinct classes, here rocket science against baseball bats.