Explain word embeddings and why they beat one-hot encoding for large vocabularies

WHAT THIS TESTS: This question probes whether you see representations as more than storage formats. A senior candidate should explain that word embeddings are learned, dense, real-valued vectors that encode semantic relationships geometrically, whereas one-hot encoding is a sparse, symbolic representation that treats every word as an isolated dimension. The interviewer wants to hear that you understand generalization, the curse of dimensionality, and why similarity-aware representations matter for downstream model performance.

A GOOD ANSWER COVERS: A strong response hits four things in order. First, define word embeddings as low-dimensional vectors, typically 50 to 1024 dimensions, where words with similar meanings cluster closer together in vector space. Second, contrast this with one-hot encoding, which assigns each word a unique binary vector whose length equals the vocabulary size, often 30,000 to 500,000 entries in production systems, making it impractically sparse. Third, explain the semantic advantage: because one-hot vectors are mutually orthogonal, every pair has zero cosine similarity, so the model receives no hint that cat and kitten are related; embeddings encode this via proximity. Fourth, mention practical benefits like reduced parameter counts, better gradient flow, and the ability to perform analogies such as king minus man plus woman approximating queen.

COMMON WRONG ANSWERS: Red flags include claiming embeddings are just a compression trick to save RAM, which misses the geometric encoding of meaning. Another mistake is saying one-hot encoding fails only because it is big; size is a symptom, but the root problem is the lack of shared structure between similar words. Candidates who confuse embeddings with manual feature engineering or who cannot name a single embedding algorithm like Word2Vec, GloVe, or FastText also signal shallow knowledge.

LIKELY FOLLOW-UPS: Expect the interviewer to ask how embeddings handle out-of-vocabulary words, whether static embeddings like Word2Vec differ from contextual ones like BERT, or how you would initialize an embedding layer for a custom domain with limited data. They may also ask you to compare cosine similarity versus Euclidean distance for measuring semantic closeness.

ONE CONCRETE EXAMPLE: Imagine a sentiment classifier with a vocabulary of 100,000 words. In one-hot encoding, each input vector has 99,999 zeros and a single one, so the first hidden layer needs 100,000 times hidden-size parameters just to get started, and the network must independently learn that excellent and outstanding are positive from scratch. With 300-dimensional embeddings, excellent and outstanding occupy nearby vectors because they appear in similar contexts during pre-training. The classifier can transfer sentiment signal between them, requiring far fewer labeled examples to perform well.

Source: Wikipedia: Word embedding