What is a word embedding and how does it beat one-hot encoding?
Tests dense semantic vectors versus sparse one-hot representations. A good answer defines embeddings as learned real-valued vectors where similar words are close, contrasts them with orthogonal one-hot vectors lacking similarity, and names Word2Vec or GloVe.
What's really being asked
This question checks whether you understand the fundamental difference between symbolic and distributed representations in NLP. Interviewers want to see that you know one-hot encoding treats words as atomic labels with no inherent relationship, while embeddings map words into a continuous vector space where geometric relationships correspond to linguistic semantics. At the senior level, they also care whether you can articulate why dimensionality and sparsity matter for generalization, downstream model performance, and computational efficiency.
The full answer
Four things in order. First, define a word embedding as a learned real-valued vector representation, typically produced by methods like Word2Vec or GloVe, where words that appear in similar contexts occupy nearby points in vector space. Second, explain the one-hot encoding baseline: each word gets a unique binary vector with dimension equal to vocabulary size, so every pair of words is orthogonal and equidistant, meaning the model receives no prior knowledge that king and queen are related. Third, highlight that embeddings compress thousands or millions of sparse dimensions into a dense low-dimensional space, usually between fifty and three hundred dimensions, which improves memory usage and generalization. Fourth, mention that the learned geometry captures analogies and semantic relationships, enabling transfer learning because pre-trained embeddings can initialize downstream classifiers or sequence models.
The mistakes people make
The biggest red flag is describing embeddings as just a compressed or hashed version of one-hot vectors without mentioning that the values are learned from co-occurrence statistics or prediction tasks. Another weak pattern is confusing embeddings with manual feature engineering like bag-of-words or TF-IDF, which still produce sparse vectors and do not create a shared semantic space. Saying that embeddings are simply lookup tables without explaining the training objective or the geometric properties also signals shallow understanding.
What usually comes next
An interviewer might push you to explain how Word2Vec skip-gram differs from GloVe, or ask you to derive why cosine similarity is the natural distance metric for embeddings. They could also ask how you would handle out-of-vocabulary words, or whether embeddings capture polysemy, or how contextual embeddings like ELMo or BERT changed the landscape. Be ready to discuss the trade-offs between static and contextual embeddings.
A concrete example
Imagine a vocabulary of ten thousand words. In one-hot encoding, king is represented by a single one in a ten-thousand-dimensional vector, and queen by a different one; a neural network must learn from scratch that these words are related. With a two-hundred-dimensional Word2Vec embedding, king and queen might have a cosine similarity of roughly zero point seven because they share contexts like throne, crown, and royal. Moreover, the vector arithmetic king minus man plus woman yields a vector closest to queen, demonstrating that the embedding space encodes gender relationships geometrically rather than symbolically.
Interview question
What is the main reason word embeddings capture semantic relationships that one-hot encoding cannot?
- a.They are lookup tables that assign a random dense vector to each word
- b.Embeddings compress one-hot vectors into smaller spaces using deterministic hashing
- c.One-hot encoding fails because its binary values cannot exceed zero or one
- d.Embeddings learn dense vectors where contextually similar words are close togetherCorrect
Why? this is the answer
Embeddings are trained on co-occurrence patterns so words with shared contexts occupy nearby points in dense vector space, whereas one-hot vectors are orthogonal and equidistant. Distractor A is tempting but wrong because embeddings are not simply hashed or compressed one-hot vectors; their values are learned to create meaningful geometric relationships.
Just read this? Test yourself on what you have been reading.
Read the original → en.wikipedia.org
- #nlp
- #word-embeddings
- #one-hot-encoding
- #word2vec
- #glove
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on nlp — each one lists the topics its interview covers.
See open roles