Five Word2Vec Models, Two Corpora, One Classifier

A, B and C appear twice, once trained on SST and once retrained on text8. Every model goes through the same evaluation. I look up each word vector, average the vectors into one sentence representation, fit logistic regression, and test on the same held-out SST-2 sentences. The classifier is never tuned for an individual model. This keeps the comparison focused on the embeddings.

Sentiment accuracy is the share of the 1,821 SST-2 test sentences classified correctly. Word similarity is the Spearman correlation between the model's cosine similarities and human ratings on WordSim-353: 1 is perfect agreement, 0 is none.

A2 is a control, not a sixth model. Its 0.664 score is the lowest in the table, but that result is useful: it makes the claim about G-ft's warm start testable. The three text8 rows are A, B and C trained on Wikipedia instead of SST. More training text covers them.

These examples show the held-out sentiment task at sentence level. For each model, the table gives the classifier's two labels with their probabilities and the top-5 closest training sentences by cosine similarity. Neighbour agreement is the fraction of those retrieved sentences whose gold label matches the test example. It gives a small, concrete view of what evidence sits nearby in the embedding space.

I selected all five by rule, not by eye. The rule appears above each example. Accuracy within this small set is A 3/5, B 3/5, C 3/5, G 4/5, G-ft 3/5.

The five queries are meant to expose differences between the models: a polysemous noun, a sentiment adjective, a domain-sensitive word, a rare film term, and the most frequent word in English.

plot

terrible

star

cinematography

the

king - man + woman = queen is the canonical test of whether an embedding space supports this kind of arithmetic. Below it are four queries built from film-review vocabulary. Every word in those four occurs in SST, so the SST-trained models cannot miss them for lack of vocabulary. The standard Google analogy benchmark follows.

king - man + woman expected queen

actor - man + woman expected actress

better - good + bad expected worse

worst - bad + good expected best

comedy - funny + scary expected horror

Why A, B and C fail

An analogy works when the offset between two words, such as man to woman, points the same way for other pairs, such as king to queen and actor to actress. A model learns a shared direction like that only after it sees each word in many different contexts. In the 6,920 training sentences, the expected answers appear between 11 and 131 times, and even the most common query word appears only 214 times. The table gives each count next to the count in the phrase rows the models were trained on.

The phrase counts overstate what the models learned from. Most phrase rows are fragments of those same sentences (see the data section), so queen appears 43 times but in only 11 places in the original text. GoogleNews was trained on about 100 billion tokens, more than 150,000 times the 633,720 tokens here. The similarity benchmarks agree: on WordSim-353, A scores +0.029, B -0.036 and C +0.005, against G's +0.700.

Retraining A, B and C on text8 confirms the explanation. With 17,005,207 tokens and the same hyperparameters, all three answer queen, and their analogy accuracy rises to 0.372, 0.314 and 0.353. See More training text.

Sentiment classification holds up anyway. A reaches 0.761 on SST-2 against G's 0.798. A classifier over averaged vectors only needs positive and negative words to separate roughly. That is a much coarser property than consistent analogy offsets, so the two evaluations measure different things.

Accuracy needs to be read alongside coverage. gensim drops any question with an out-of-vocabulary word, so the SST-trained models are scored on only 18.2% of the set against G's 79.0%. G-ft's 0.675 comes from structure inherited through the warm start, not learned from SST.

A model stores each word as 300 numbers, which can't be drawn directly. PCA and t-SNE are two standard ways to squeeze them down to two, so every word becomes a dot on a page. PCA keeps the two directions along which the words vary most. t-SNE tries to keep each word's nearest neighbours close, at the cost of distorting larger distances. If a model has learned what words mean, related words should land near each other.

To test that, I picked 32 words before looking at any plot, eight from each of four groups: positive words (good, charming), negative words (dull, tedious), filmmaking terms (screenplay, editing) and genres (horror, western). Each dot is coloured by its group.

Clusters are easy to imagine in a scatter plot, so each one also gets a silhouette score. It compares how close a word sits to its own group against how close it sits to the nearest other group, and runs from -1 to 1. Near 0 means the groups are mixed together; higher means they form separate clumps. The plots below show G-ft, the clearest case, and A trained on each corpus.

A flat picture throws most of the information away. The first two PCA axes keep between 16% and 37% of the variance, depending on the model. So the last two score columns skip the projection and measure the same thing on the full 300-dimensional vectors, using cosine distance. The last of those asks only whether positive and negative words form separate groups.

This analysis is included because sentiment accuracy alone makes the embeddings look more capable than they are.

The three benchmarks compare cosine similarity with human ratings using Spearman correlation. Coverage is the fraction of pairs where both words are in the vocabulary. It is part of the result: the SST-trained models cover only about half of WordSim-353 because many benchmark words never occur in the movie reviews.

This also clarifies what the warm start contributes. G-ft scores 0.585 on WordSim-353 against 0.029 for A, trained from scratch on the identical corpus with identical settings. The pretrained initialization does more than add 0.031 accuracy. It supplies all of Model G-ft's measurable lexical structure.

The sections above blame the size of the training data for the failures of A, B and C on similarity and analogies. To test that directly, I retrained all three on text8, the first 100 MB of cleaned English Wikipedia (mattmahoney.net/dc/text8.zip). It has 17,005,207 tokens, 27 times the SST phrase corpus. Every hyperparameter matches the SST runs. One implementation detail differs: C's text8 run uses sparse Adam, which updates only the rows a batch touches, because dense Adam is too slow at this vocabulary size (see Cost to run). C (text8) also trained on a PACE ICE A100 GPU, so its training time is not comparable with the laptop times. The sentiment test is also unchanged: the classifier is still fit on the SST phrase labels and scored on the same SST-2 test sentences, so only the word vectors differ.

Testing the handout's claims

Page 1 of the handout makes two claims about the architectures that the two corpora can test. CBOW trains several times faster. On text8, B took 177 seconds against A's 889, a 5.0× difference (963,171 against 191,213 words per second). The claim holds. Skip-gram does better on small data, and CBOW needs large data. On SST, skip-gram leads CBOW by 0.015 in sentiment accuracy averaged over three seeds, and neither has any similarity structure. On text8, the accuracy gap disappears (+0.002 for B), while A still leads on WordSim-353 by 0.028 and on analogies by 0.058. CBOW closes the gap as the data grows, which fits the claim, but skip-gram stays ahead on the finer-grained tests. The text8 figures are single runs. On SST, the hyperparameter sweep puts the seed-to-seed standard deviation of accuracy at 0.002 to 0.007, so the text8 accuracy gap is within noise. The WordSim-353 and analogy gaps are far larger than that.

Model G-ft, confidently right and confidently wrong

Two test examples Model G-ft, the fine-tuned one, gets right and two it gets wrong. All four come from the same 1,821-sentence test split. Within each outcome, these are the highest-confidence cases.

Confident failures across models

The failures follow directly from the representation. not a bad journey at all averages not and bad into negative territory. there is n't a weak or careless performance amongst them pools two strongly negative adjectives without preserving the negation that reverses them. In a great deal of sizzle and very little steak, the negative meaning comes from the contrast between clauses, which averaging also removes.

A Word2Vec model comes down to two matrices and a dot product. It has no hidden layer or nonlinearity. W_in stores one row per word and becomes the final embedding table. W_out is used for the training objective, then discarded.

The sentiment data comes from SST-2, but its two releases cannot be swapped freely. GLUE replaces every test label with -1 because the split is reserved for its leaderboard. I use the labelled SetFit test split instead. src/data.py raises an error unless both classes appear in the loaded test labels.

What the rows look like

Five training examples and four test examples, two positive and two negative. Each example was selected by a stated rule rather than picked by eye.

Five training examples

Four test examples

Second training corpus: text8

A, B and C were also trained on text8, the first 100 MB of an English Wikipedia dump cleaned by Matt Mahoney (mattmahoney.net/dc/text8.zip). It has 17,005,207 tokens and a vocabulary of 135,335 words at min_count=2. The text contains only lowercase letters and spaces. Punctuation is removed and numbers are spelled out digit by digit, so 1989 becomes one nine eight nine. text8 is used only to train word vectors; the classifier and every test stay on SST. The file is downloaded by src/run_text8.py rather than stored in the repository.

The pretrained baseline is GoogleNews-300, a skip-gram model with negative sampling trained on roughly 100 billion news tokens. It comes from code.google.com/archive/p/word2vec, is loaded through gensim-data, and is capped at the 500,000 most frequent words. I did not use GloVe as the pretrained baseline because it factorizes a co-occurrence matrix rather than training Word2Vec. That would change the question being tested.

The pretrained model

A and B intentionally change two things together: the architecture and the loss. All other settings stay fixed, so the comparison is between the two standard Word2Vec configurations. The sweep below separates the two changes.

Sweep

Each setting below was trained 3 times on the SST phrase corpus, with seeds 42, 43, 44. The tables give the mean and standard deviation across seeds. One factor changes at a time, and everything else stays at A's or B's settings. Training time is the mean wall clock on the M2 Air.

Architecture and loss

Context window

Vector dimensions

Negative samples

Wider windows help both models. Going from a window of 2 to 10 raises A from 0.738 to 0.755 and B from 0.715 to 0.741. For skip-gram it also more than doubles training time, from 11.2 to 26.1 seconds, because every extra context word is another prediction. CBOW needs more dimensions than skip-gram. B gains 0.056 going from 50 to 300 dimensions, while A gains only 0.006. The number of negatives shows no clear trend. 5, 10 and 20 negatives give 0.761, 0.752 and 0.760. The middle setting is the lowest, which fits no trend, and with three seeds a gap that small cannot be separated from seed noise. 20 negatives take 38 seconds.

Training times drift as the fanless M2 Air warms up. The 10-negative settings were the first runs of the sweep, on a cool machine, and came out faster than the 5-negative runs even though they do more work. Small time differences between settings are therefore not reliable.

Fine-tuning Word2Vec is much simpler than fine-tuning a transformer. There are no frozen layers or adapters. I initialize W_in with published vectors, then continue ordinary skip-gram training on movie reviews at a low learning rate.

1. Build the vocabulary from our corpus first intersect_word2vec_format only overwrites rows that already exist. It does not import Google's three million words, so G-ft keeps the vocabulary built from SST.
2. Set vectors_lockf to ones explicitly gensim 4.x does not create it. Without it the import either fails or silently freezes every row, leaving the imported vectors unchanged.
3. Keep the learning rate low: alpha 0.005, 8 epochs Google never published their output matrix, so W_out starts random. Early updates must learn a new decoder as well as adjust the imported vectors. A standard learning rate distorts those vectors during the first epoch.
4. Measure drift rather than assuming it Cosine between each word's warm-start vector and its final vector. The mean is 0.9609 and the median is 0.9731, confirming that the low learning rate preserved most of the pretrained structure.

Writing the training loop makes each part of Word2Vec concrete. src/sgns.py implements the vocabulary and counts, subsampling of frequent words, the unigram^0.75 noise table, a dynamic window, the two embedding matrices, and linear learning-rate decay. to_keyedvectors wraps W_in in a gensim object, which lets C use the same evaluation code as every other model.

Two component checks made the failure quick to locate. Subsampling retains 17.6% of occurrences of the. The noise table maps the from a raw frequency of 4.30% down to a sampled 1.31% while lifting execrable from effectively zero to 2e-5. Both results match the expected effect of the 0.75 exponent.

Query latency measures brute-force most_similar, which runs a dense matrix-vector product over the full vocabulary. The mean, p50 and p95 per query all come from the same 200 calls, cycling through the five query words after one untimed warm-up call each. Memory is the RSS increase in an isolated subprocess, compared with the size of the matrices the model retains.

The 2 · V · d · 4 prediction holds within 3% for A, G-ft and C. B comes in at 1.40 because hierarchical softmax also keeps Huffman-tree code and point arrays that the formula omits.

A separate benchmark determined how C trains. On this vocabulary, MPS is slower than CPU (64.3 against 44.6 ms per batch) and sparse gradients are slower than dense gradients (52.6 against 44.6). The model is small enough for GPU launch and data transfer overhead to dominate. Sparse updates also offer little benefit. A batch of 8,192 pairs with 10 negatives touches roughly 90,000 index slots across 14,309 rows, so almost every row updates on each step.

Four setup and measurement problems took most of the debugging time. In each case, the visible error pointed away from the actual cause. The symptoms and fixes are recorded here.

1. gensim has no wheels for Python 3.14 The machine's default interpreter is 3.14. Installing gensim there starts a Cython source build and fails late in the process. The virtual environment therefore uses Homebrew Python 3.11.
2. import gensim fails on a fresh unpinned install gensim 4.3.x imports scipy.linalg.triu, which scipy 1.13 removed. The ImportError points at gensim even though the incompatible scipy version is the cause. requirements.txt pins scipy<1.13 and numpy<2.
3. The SST-2 split with hidden labels The GLUE copy sets every test label to -1. An accuracy calculation against those placeholders can still produce a plausible number. src/data.py prevents that by requiring two classes in the test labels.
4. Whole-process peak RSS is not a model measurement The first memory table compared process peak RSS against the matrix prediction and returned 297.8 MB for a model with 34.3 MB of matrices. Process peak RSS also includes the interpreter, gensim, numpy, torch, and the resident corpus. src/measure_mem.py now measures an RSS delta in an isolated subprocess per model.

After pip install -r requirements.txt, these commands produce the results the page is built from, in order:

1. python -m src.data corpus statistics
2. python -m src.main trains A and B on SST and loads G
3. python -m src.run_e trains C on SST, 160 seconds on CPU
4. python -m src.finetune trains G-ft from the GoogleNews vectors, then the A2 control with the same schedule and random initialization
5. python -m src.run_eval, python -m src.measure_mem, python -m src.run_analogies, python -m src.testcases similarity, latency, memory, errors, analogies and the worked test examples
6. python -m src.examples the example tables for training data and for G-ft's right and wrong calls, read back from the data and from G-ft's mined errors
7. python -m src.run_text8 downloads text8 and retrains A and B on it, about 18 minutes
8. python -m src.run_e --text8 trains C on text8. On a GPU cluster, sbatch slurm/c_text8.sbatch runs the same command; it saves a checkpoint after every epoch, so an interrupted job resumes where it stopped.
9. python -m src.visualize PCA and t-SNE projections and the cluster scores
10. python -m src.sweep the hyperparameter sweep
11. python -m src.build_report rebuilds both report variants

src/build_report.py reads every reported figure from results/*.json when it builds the page. Re-running a model and rebuilding therefore updates the prose and tables together. gensim training with four workers is not deterministic, so rerunning a model shifts its scores by roughly the seed spread measured in the sweep.