What is a word embedding and how does it beat one-hot encoding?

WHAT THIS TESTS: This question checks whether you understand the fundamental difference between symbolic and distributed representations in NLP. Interviewers want to see that you know one-hot encoding treats words as atomic labels with no inherent relationship, while embeddings map words into a continuous vector space where geometric relationships correspond to linguistic semantics. At the senior level, they also care whether you can articulate why dimensionality and sparsity matter for generalization, downstream model performance, and computational efficiency.

A GOOD ANSWER COVERS: Four things in order. First, define a word embedding as a learned real-valued vector representation, typically produced by methods like Word2Vec or GloVe, where words that appear in similar contexts occupy nearby points in vector space. Second, explain the one-hot encoding baseline: each word gets a unique binary vector with dimension equal to vocabulary size, so every pair of words is orthogonal and equidistant, meaning the model receives no prior knowledge that king and queen are related. Third, highlight that embeddings compress thousands or millions of sparse dimensions into a dense low-dimensional space, usually between fifty and three hundred dimensions, which improves memory usage and generalization. Fourth, mention that the learned geometry captures analogies and semantic relationships, enabling transfer learning because pre-trained embeddings can initialize downstream classifiers or sequence models.

COMMON WRONG ANSWERS: The biggest red flag is describing embeddings as just a compressed or hashed version of one-hot vectors without mentioning that the values are learned from co-occurrence statistics or prediction tasks. Another weak pattern is confusing embeddings with manual feature engineering like bag-of-words or TF-IDF, which still produce sparse vectors and do not create a shared semantic space. Saying that embeddings are simply lookup tables without explaining the training objective or the geometric properties also signals shallow understanding.

LIKELY FOLLOW-UPS: An interviewer might push you to explain how Word2Vec skip-gram differs from GloVe, or ask you to derive why cosine similarity is the natural distance metric for embeddings. They could also ask how you would handle out-of-vocabulary words, or whether embeddings capture polysemy, or how contextual embeddings like ELMo or BERT changed the landscape. Be ready to discuss the trade-offs between static and contextual embeddings.

ONE CONCRETE EXAMPLE: Imagine a vocabulary of ten thousand words. In one-hot encoding, king is represented by a single one in a ten-thousand-dimensional vector, and queen by a different one; a neural network must learn from scratch that these words are related. With a two-hundred-dimensional Word2Vec embedding, king and queen might have a cosine similarity of roughly zero point seven because they share contexts like throne, crown, and royal. Moreover, the vector arithmetic king minus man plus woman yields a vector closest to queen, demonstrating that the embedding space encodes gender relationships geometrically rather than symbolically.

Source: Wikipedia: Word embedding