I have two embedding spacesâbasically, two different ways a computer turns things into numbers. One is CIELAB, a 3-number representation of color. The other is BERT (Devlin et al., 2019), a 768-number representation of the meaning of a word. They have nothing in common at first glance: different sizes, different inputs, different purposes.
But I show that if you give the computer 950 examples of "this color name goes with this RGB", you can fit a smooth bridge from one space to the other. From then on it can guess the color of words it has never seen:
- royal comes out a deep purple (0.31, 0.22, 0.50)
- navy comes out a dark blue-gray (0.20, 0.22, 0.34)
- jazz comes out a dusty rose (0.66, 0.42, 0.42)
It's sometimes intuitive, sometimes surprising. It can also guess the name of colors it has never seen, picking concepts that are close to the right answer even when it gets the exact word wrong.
Then I do the same trick with images instead of colors: 55 anchor pictures of common objects, plus a vision model that has never been trained on word labels, and suddenly it starts naming new images. Not always with the right word, but always with a word that lives near the right one.
The interesting part isn't that this works. It's why it works with so few anchors. My claim is that the structure being lined up is already there in both spaces before I lined anything up, because all the data was filtered through humans first. We're not discovering some deep truth about the world. We're noticing that both models inherit the same human bias, so of course they line up.
Two stories that started this
A friend's daughter, learning to write and growing up multilingual, left a note for Bea, a toddler. She finished four lines of careful handwriting with a heart. The message, "dear Bea, you're very cute," took me a long minute to read, because not one of those words was spelled by a single set of rules.
One thought sat beneath all four spellings. The surface forms fused every writing system she had.
Around the same time Bea, who loves buses and had never seen a real train, was given a model train. She picked it up, looked at it carefully, and called it a "bus".
The shape of how we organize concepts is more shared than the labels we put on them. Bea named the train "bus" because in her concept space, buses and trains both meant "thing people ride in" to her, and the closest label she had for that neighborhood was "bus". This is a textbook thematic/functional overextensionâchildren consistently overextend along functional and thematic lines before settling into adult taxonomies.
The Platonic Representation Hypothesis vs. Human Bias
When you train a big neural network on a pile of human-curated data, it ends up with a giant cloud of vectors that has some internal geometry. Every modality gets its own cloud: images live in one cloud, words in another, colors in a third.
There's a popular hypothesis (Huh et al., 2024), the Platonic Representation Hypothesis, that all sufficiently good models, regardless of modality, eventually converge on the same internal cloud. The argument is roughly: they're all approximating the same underlying world, so they'd better look the same in the limit.
I think there's a simpler explanation: All the data we feed these models was already filtered through human cognition before it ever became training data. The clouds aren't shaped by the world, they're shaped by how humans organize the world before we hand it to the model.
If that's right, two clouds from two different modalities should already share their internal organization (which concepts sit near which) even though their coordinates are wildly different. And if they already share that organization, you should be able to bridge them with a tiny amount of glue: a few known correspondences, fit with a simple smooth or rigid map.
Testing the premise: Images and Word Vectors
To test this on a harder modality, I used vision space (DINOv2, a self-supervised image model) and word space (BERT). Same trick: a small set of anchored picture/word pairs, then a rigid rotation (orthogonal Procrustes) lining the two clouds up.
My anchors? 55 photos of common objects (apple, bus, car, chair, cow, dog, pizza, tree, etc.).
I then tested on 12 messy, out-of-distribution images that aren't in the anchor set. Some are photos, some look like CAPTCHA-style obscured shots. The model is asked to pick a word from a 3,000-word vocabulary it's never been told about.
If you just look at top-1 accuracy, it's 0%. But if you look at the top-5 lists, something interesting happens:
- Bus image: returns train, bus, boston, railroad, station
- Car image (clean): returns boat, truck, ship, ford, train
- Car image (CAPTCHA): returns boat, train, railroad, ford, truck
- Plane image: returns ship, plane, aircraft, boat, fleet
Because top-1 is zero across the board, the interesting question is whether the top-5 contains at least one word in the same semantic category as the ground truth. By that rule, 8 of the 12 test images are categorical hits.
This is the same thing Bea did with the model train, scaled up. The aligned model knows it's looking at some kind of vehicle even when it doesn't pick the right one.
How few is few? Anchor-count sweeps
Refit the held-out-cluster reconstruction on n training anchors (n = 10, 20, 40, 80, 160, 320, 534), against a shuffled-correspondence control at each n. The real run falls from 34.7 to 18.0 mean E00 as n grows; the shuffled control stays at chance (~38-41) at every n. The gap (shuffled - real) reaches 90% of its full value by n = 160.
Why this matters
When the image-to-words bridge misses, it misses to train, ship, ford. Things that move. The shuffled and identity controls don't even do that. They miss to nothing in particular.
That's the part that matters to me. If two embedding spaces converged because both approximated some external truth about the world, you'd expect failures to be unstructured: the model would either get it right or get it wrong in any direction. Instead the failures cluster, and they cluster the way humans cluster these concepts.
Less grand than Platonic, but it fits with fewer assumptions. We're not learning that the world has a hidden geometric structure that all good models discover. We're learning that the human sorting survives the embedding process.