Mixture of Experts architecture and routing

WHAT THIS TESTS This probes whether you understand sparse conditional computation: how MoE decouples total model capacity from the compute spent per token, and the role of the routing network.

A GOOD ANSWER COVERS In a Transformer MoE layer, the single dense feed-forward network is replaced by a set of parallel expert FFNs, for example eight experts. A small router, or gating network, takes each token's hidden state, produces a score per expert, and selects the top-k experts, commonly the top two. Only the chosen experts process that token; their outputs are combined weighted by the router's scores. The benefit at inference is that although the model has a large total parameter count, only a fraction is activated per token, so the floating-point operations per token resemble a dense model far smaller than the total size. This gives the representational capacity of a big model at the inference compute of a small one. The catch is memory: all experts must be loaded even though only a few fire, so VRAM is governed by total parameters. Training also needs a load-balancing auxiliary loss so the router does not collapse to favoring a few experts.

COMMON WRONG ANSWERS Claiming all experts run for every token; confusing active parameters with total parameters when estimating memory; ignoring the router entirely; forgetting load-balancing loss; assuming MoE reduces memory as well as compute.

LIKELY FOLLOW-UPS Why is a load-balancing loss needed? How does top-k choice affect quality and cost? Why is memory still high? How does expert parallelism work across GPUs?

ONE CONCRETE EXAMPLE In an 8-expert layer with top-2 routing, each token's hidden state goes to the router, which picks two experts; only those two FFNs compute for that token. Across a sequence, different tokens hit different expert pairs, so the layer's effective capacity is large while each token's compute stays close to a two-expert dense layer.