Word Embeddings: Turning Words into Vectors
Word embeddings turn words into numerical vectors, like coordinates on a map of meaning. Words with similar meanings, like "king" and "queen," are placed close together in this vector space. This is fundamental for text analysis in machine learning, allowing models to grasp semantic relationships instead of just matching text. The footgun is assuming the vector's individual numbers are human-interpretable; they are abstract features learned from data.
The mental model
A word embedding is a way to represent words numerically. Think of it as assigning every word in a vocabulary a specific coordinate in a multi-dimensional space. In this space, the distance and direction between words capture their semantic relationships. "King" isn't just a string of letters; it's a point, and the path from "king" to "queen" is mathematically similar to the path from "man" to "woman".
How it works
Embeddings are learned, not manually assigned. A model is trained on a massive text corpus by learning to predict a word given its surrounding context words. To do this efficiently, the model creates a dense vector of real numbers (e.g., a list of 300 numbers) for each word. Words that frequently appear in similar contexts, like "coffee" and "tea," will end up with very similar vectors because the model needs to make similar predictions for both.
When to use it
• Semantic Search: Finding documents that are conceptually related, not just matching keywords. • Text Classification: Determining the sentiment or topic of a document by understanding the nuance of the words used. • Machine Translation: Mapping words from one language to another by finding equivalent points in their respective embedding spaces.
When not to use it
• When you need explainable rules: The dimensions of an embedding vector are not human-interpretable. You can't point to one number and say it represents "animal-ness." The meaning is encoded in the vector as a whole. • For simple string matching: If you just need to find exact occurrences of a word, simpler methods like keyword search are more efficient.
One canonical example
The most famous example demonstrates the linear relationships embeddings can capture: vector('king') - vector('man') + vector('woman') results in a vector that is extremely close to vector('queen'). This shows the model learned the concepts of gender and royalty as directions in the vector space.
Interview question
Which statement accurately describes the individual numerical values that constitute a word embedding vector?
- a.A) Each number directly quantifies a specific, human-understandable attribute of the word, like its part of speech or sentiment.
- b.B) They are pre-defined scores assigned by experts to reflect a word's dictionary definition.
- c.C) They are abstract features learned from the word's context, with the vector's overall pattern encoding its meaning.Correct
- d.D) They primarily indicate the word's frequency of occurrence within the training data.
Why? this is the answer
Word embeddings represent words as abstract numerical features learned from context, where the collective vector encodes meaning. The card explicitly states that individual numbers are not human-interpretable, making option A, which suggests direct human-understandable attributes, incorrect.
Just read this? Test yourself on what you have been reading.
Read the original → en.wikipedia.org
- #nlp
- #machine learning
- #llms
- #vectors
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on nlp — each one lists the topics its interview covers.
See open roles