Static vs contextual word embeddings

WHAT THIS TESTS This checks whether you understand the fundamental limitation of static embeddings, polysemy, and can weigh accuracy against computational cost when choosing an embedding strategy.

A GOOD ANSWER COVERS Static embeddings such as Word2Vec and GloVe learn a single fixed vector per word from global co-occurrence statistics. The vector for a word is identical no matter the sentence, so a word with multiple senses collapses its meanings into one point, unable to distinguish, for example, a river bank from a financial bank. Their virtues are that they are tiny, fast to look up, require no model at inference, and work well for simple classification or when compute is scarce. Contextual embeddings from models like ELMo and BERT generate a representation by running the whole sentence through the model, so the same word yields different vectors depending on its neighbors, resolving polysemy and capturing syntax and long-range dependencies. The cost is much higher: you run a full transformer per input, with greater latency and memory. You choose pretrained static embeddings for lightweight pipelines, edge devices, or as cheap features. You choose to fine-tune a contextual model when task accuracy depends on disambiguating meaning, the data is rich enough, and you can afford the compute.

COMMON WRONG ANSWERS Saying static embeddings adapt to context; claiming contextual embeddings are always strictly better regardless of cost; forgetting that contextual models need a forward pass per input; confusing GloVe's co-occurrence training with contextualization.

LIKELY FOLLOW-UPS How does BERT produce a contextual vector? Why can't Word2Vec handle polysemy? When is the extra cost not worth it? Can you cache contextual embeddings?

ONE CONCRETE EXAMPLE For a fast keyword tagging service on modest hardware, pretrained GloVe vectors as features are cheap and adequate. For a legal document classifier where the meaning of terms shifts with surrounding clauses, you fine-tune BERT so each occurrence gets a context-sensitive vector, accepting the higher inference cost for the accuracy gain.