Why are MoE models larger but cheaper to run?

### What this tests This tests your grasp of sparse versus dense model architectures. The interviewer wants to confirm you understand the distinction between a model's total parameter count (its capacity for knowledge) and the computational cost (FLOPs) required for inference on a single input. It's a practical check on your understanding of modern LLM efficiency.

### A good answer covers * **Define the Architecture:** Start by describing a Mixture-of-Experts (MoE) model as a system composed of a 'router' network and multiple 'expert' sub-networks. * **Explain Sparse Activation:** The key is that for each input token, the router dynamically selects a small subset of experts (e.g., 2 out of 8) to process it. The other experts remain inactive and consume no computational resources for that token. * **Distinguish Parameters vs. FLOPs:** This is the core of the answer. * **Large Total Parameters:** The model is 'larger' because its total parameter count is the sum of all its experts (plus shared components). This large parameter count allows it to store more knowledge. * **Low Inference FLOPs:** The model is 'cheaper to run' because the number of floating-point operations (FLOPs) per token depends only on the *active* experts, not the total. The cost is comparable to a much smaller dense model. * **Contrast with Dense Models:** Briefly mention that in a traditional dense model, *all* parameters are used for *every* token, making inference cost scale directly with model size.

### Common wrong answers * **It's just an ensemble:** Describing it as a simple ensemble where you run all models and average the results. This is incorrect; the efficiency comes from *not* running all the experts. * **Confusing training and inference:** While training MoEs has unique challenges like load-balancing, the question is specifically about why they are 'cheaper to run' (inference). * **Ignoring the router:** Failing to mention the routing mechanism, which is the component responsible for selecting the experts and enabling sparsity.

### Follow-up the interviewer might ask * "What are some of the challenges in training an MoE model?" * "If Mixtral 8x7B uses two 7B experts per token, why is its performance often compared to a 70B model, not a 14B one?"

### One concrete example "Take Mixtral 8x7B. It has 8 distinct 'expert' networks, each around 7 billion parameters. For any given token, a router selects the top 2 experts to process it. So, while the total parameter count is ~47B, the actual computation for a single token only uses the equivalent of a ~14B parameter model. This is why it can run on hardware that supports a 13B model, but its performance on benchmarks is competitive with much larger dense models like Llama 2 70B."