Before a transformer can pay attention to anything, it has to turn words into numbers. An embedding is how it does that — and the trick is that the numbers are arranged so that meaning becomes geometry. Work through the ideas below.
Lesson progress0%
0
First, untangle: tokenizing ≠ embedding
These two get mixed up constantly, so let's separate them before anything else. Going from text to vectors is really two lookups back to back, and only the second one knows anything about meaning. Click a word in the sentence.
① Tokenize
text → an ID number · a plain dictionary lookup
token: cat
ID in vocabulary: 2543
The ID is just a row number. It carries zero meaning — "cat" could be #2543 or #9 and nothing about the word would change.
→
② Embed
ID → a learned vector · this is where meaning lives
look up row 2543 in the embedding table:
This vector was learned from data. Similar words get similar rows — that's the part tokenizing can't do.
The one-line version: tokenizing is a fixed dictionary — it assigns addresses. Embedding is a learned table — it stores meaning at those addresses. Everything else on this page is about that second table. (Real tokenizers also split rare words into sub-word pieces — more on that in Section 6.)
1
A word becomes a list of numbers
So the embedding table hands each token a fixed list of numbers — a vector. That's the whole object: word → vector. Tap a word and watch it turn into numbers.
cat
→
Each bar is one number. Blue = positive, orange = negative. Words from the same family (try cat then dog, or happy then sad) get similar-looking patterns — that similarity is the whole point.
2
Similar meanings sit close together
Because each word is a point in space, distance means similarity. Words about the same thing cluster; unrelated words drift apart. Hover a word for its nearest neighbours, flip to the analogy view, or measure any two. (Want to rotate this space? That's Section 4.)
Tap or hover any word to light up its closest neighbours.
Royalty & peopleAnimalsFoodFeelings
—
how close these two sit
Careful with that number: it's a stand-in computed from distance on this flattened map, so it only runs 0 → 1. Real models use cosine similarity — the angle between the two full vectors — which runs from −1 (pointing opposite) through 0 (unrelated) to +1 (identical). Small angle = similar meaning.
What do the axes mean? Usually nothing you can name. To draw hundreds of dimensions on flat paper we project them down (PCA, t‑SNE). The horizontal and vertical axes are just "the directions things spread out most" — they rarely line up with a tidy human concept. So read the picture by distance and direction between points, not by the axis numbers. (The one exception: sometimes a direction does carry meaning — that's the analogy trick.)
Why analogies work: meaning is stored as directions, not just positions. The arrow from man → woman is roughly the same arrow as king → queen — a "gender" direction. So you can literally do arithmetic: king − man + woman ≈ queen. Follow the two parallel arrows.
One honest caveat, because this example gets oversold: the arrows are exactly parallel here because we arranged them that way. In real embeddings it's approximate, and the standard way of scoring it quietly excludes the three input words from the answer — without that rule, the vector closest to "king − man + woman" is usually just king again. It's a real effect and a beautiful one, but it's a tendency, not a law: plenty of analogy types fail outright.
The same trick has a dark side — and it matters especially for us. "Gender is a direction" isn't a neutral fact about geometry. It's a summary of how people wrote. Push the identical arithmetic a little further on published embeddings and man → computer programmer comes back as woman → homemaker. The vectors hold no opinions; they hold whatever the training text took for granted, stereotypes included.
And whose text was it? Overwhelmingly English, Western, and scraped from the web. So Ghanaian names, Twi and Ga words, local places and concepts are thinly represented — they land in the wrong neighbourhood, or get shattered into sub-word pieces that carry no meaning. If you build search, matching, or recommendations on an off-the-shelf embedding, that gap silently becomes your product's behaviour. Knowing where a model is thin is part of knowing how to use it.
3
Real embeddings live in hundreds of dimensions
The map is 2D so your eyes can read it. A real embedding isn't 2 numbers — it's hundreds or thousands. GPT-style models often use 768, 1,536, or more. Each dimension is a tiny independent "knob" for some shade of meaning.
One row of the table, stretched out so you can see every number.
One word — cat — as 64 numbers:
This follows whichever word you picked back in Section 1 — go change it and come back. Note there's no pattern to see: real embedding values look like noise to the eye. The structure only shows up when you compare one word's numbers against another's, which is what the table view does.
This vector has 64 dimensions. We can only draw 2 or 3 of them, so to make the map in Section 2 we flatten the rest away (PCA, t‑SNE) — keeping the clusters, losing the fine detail. The picture is a shadow of the real thing. Next section lets you spin the 3-dimensional version.
The embedding table itself — every row is a word, every column a dimension:
−1+1each cell = one learned number
Tap or hover any cell to read its value.
This is literally the data the model stores — a grid of numbers, nothing more. This little one is 17 words × 16 dims; a real table is ~50,000 × 768: same shape, just enormous. Look down the rows: words in the same colour group have similar-looking rows — "similar words, similar vectors," visible as raw data.
The trade: more dimensions = more room to separate subtle meanings (river‑bank vs money‑bank), but more numbers to learn and store. Typical sizes: word2vec/GloVe ~50–300, BERT 768, larger LLMs 4,096+.
4
Spin it in 3D — one more axis you can actually feel
The 2D map flattened everything onto a page. Here's the same words with a third axis added — and the trick to seeing depth on a flat screen is motion. Your brain rebuilds 3D from a moving picture automatically. So grab the cloud and rotate it (or let it spin).
And this is where flattening costs you something real. Look at "happy" and "sad" on the 2D map in Section 2 — they sit almost on top of each other. That's not a mistake: opposites genuinely share contexts ("I feel ___ today"), so they land close. But they are opposites, and the flat map had no room left to say so. Rotate this cloud and watch them separate in depth. Same story for apple and bread, and for cat and lion: touching in the shadow, a good distance apart in the space.
drag to rotate · tap or hover a word for its 3-D neighbours
Royalty & peopleAnimalsFoodFeelings
Depth cues you're reading without noticing: nearer words are bigger and bolder, farther ones smaller and fainter, and near words pass in front of far ones — but the spin itself is what makes the shape solid. The third axis here happens to carry a sub-distinction the flat map merged: pleasant vs. unpleasant, pet vs. wild animal, fruit vs. prepared food, royal vs. ordinary person.
The real lesson isn't "3D is better" — it's that every projection loses something. Going 2D → 3D bought back one distinction. A real embedding has hundreds of dimensions and we just used up the last one a human can picture. There is no 4-D view button. So past this point we stop trusting our eyes and lean on the math — cosine similarity, nearest neighbours — which reads all the dimensions at once and never has to flatten anything.
5
Where do the numbers come from? They're learned
Nobody types these vectors in by hand. The embedding table starts as pure random noise, and training nudges it, one small step at a time, following a single idea:
"You shall know a word by the company it keeps." — words that show up in similar contexts should end up with similar vectors.
Watch it happen. Below, nine words start scattered at random. Each training step pulls together words that co‑occur in text and pushes apart words that don't. Press Train and watch structure emerge from noise.
epoch: 0
Each step is a cartoon of gradient descent: the model reads text, sees which words share contexts, and adjusts every vector a hair to reduce its prediction error. Repeat billions of times over a huge corpus and the random table becomes the meaningful map from Section 2.
Concretely, the table is one big matrix of shape vocabulary × dimensions — e.g. 50,000 words × 768 numbers. It's just another layer of weights, trained by the same backpropagation as the rest of the network. "Learning an embedding" = filling in that matrix.
6
Are all words embedded at once? What about new text?
A natural question: is every word embedded together, and what happens when you bring your own corpus? The key split is training (done once, expensive) versus using the model (instant lookups).
Are all words embedded at the same time?
Yes — the whole vocabulary's vectors are learned together, once, during training. The matrix from Section 5 is filled in as a unit, because a word's position depends on all the others it shares contexts with.
I have a new corpus I want to embed — do I retrain?
Usually no. If the words are already in the vocabulary, embedding your new text is just instant lookups in the existing table — no learning happens. You only retrain (or fine‑tune) if you want to move the vectors to fit a new domain (legal, medical, code).
What about a word the model has never seen?
Modern tokenizers rarely hit a truly unknown word, because they split rare words into sub‑word pieces that are in the vocabulary. "tokenization" → tokenization; a surname might become Kowalski. Each piece has a learned vector, and they're combined. So "new" text still maps to known rows.
STATIC one vector per word
word2vec, GloVe. "bank" has one fixed vector no matter the sentence. Fast, simple, but can't tell river‑bank from money‑bank.
CONTEXTUAL vector depends on the sentence
BERT, GPT, all transformers. The starting embedding is still a lookup, but attention then reshapes it using the surrounding words — so "bank" ends up different in "river bank" vs "bank account". This is what Section 7 sets up.
So the mental model: training builds the table once (all words together). Using the model is lookups — cheap and instant. New text reuses known rows (whole words or sub‑word pieces). Changing what the vectors mean is the only thing that needs more training.
7
Where embeddings sit in a transformer
Putting it together: embeddings are the doorway from language into math. Everything downstream (attention, the layers that make transformers powerful) works only on these vectors, never on raw text. Tap a stage.
One subtle addition: attention is order‑blind — "dog bites man" and "man bites dog" would look identical. So right after the word embedding, the model adds a positional embedding encoding where each token sits. Word‑meaning + position, added together, is what flows into the first attention layer.
8
What attention does: it reshapes the vectors by context
The embedding lookup gives every word one vector — "bank" is "bank" no matter the sentence. Attention is the step that fixes that. Each token looks at the other tokens, decides which ones matter, and pulls their meaning in — producing a new, context-aware vector. Pick a sentence and watch where "bank" ends up.
the sentence · bank is the token being reshaped:
How much bank attends to each word (the "attention weights"):
Where "bank" lands in meaning-space:
In one line: a token's new vector is a weighted blend of all the tokens' vectors — and those weights (how much to attend to each) are exactly what "attention" computes. The context word with the biggest weight drags "bank" toward its meaning. Stack this operation a few dozen times and you have a transformer: the embeddings are the raw clay, and attention is what sculpts them into meaning-in-context. (This is the "contextual" half of Section 6, finally shown in motion.)
9
What people actually use embeddings for
The move you've been exploring — similar things land at nearby vectors — works for anything you can embed: not just words, but images, audio, users, products, proteins, transactions. Once something is a vector, "find similar", "cluster", "spot the outlier", and "feed it to a model" all become easy. That's why embeddings are everywhere. Tap a field.
One idea, many data types: the same "meaning → geometry" trick powers semantic search, recommender systems, fraud detection, and protein models alike. Learn embeddings once and you start seeing the pattern everywhere.
✓
Check your understanding
Reading about geometry isn't the same as being able to use it. Answer these and you'll know which parts actually landed. Every answer explains itself — including the wrong ones, which are the ones worth reading.
Five questions across the key ideas.
The lesson at a glance
✓ Grounded in published work
Where these ideas come from
This page is an intuition-first explainer, not a paper — but every claim on it traces back to real work. The primary sources, so you can go and read the originals:
That's embeddings — the doorway from language into math.
Where this sits: embeddings are step one. Attention (Section 8) is what a transformer does with them, and tokenization (Section 0) is what happens just before. Those are the next two explainers we're building.
This is one of the interactive explainers we make at AgenticGHX. If it was useful, the rest — talks, learning tracks, and research out of Ghana — is one tap away.