Mixture of Experts: Scaling LLMs with a Team of Specialists

The mental model

Instead of one giant, dense neural network where every neuron processes every input, a Mixture of Experts (MoE) model acts like a team of specialists. A "router" or "gating network" examines each incoming task (e.g., a token in a sentence) and directs it to the one or two experts best suited to handle it. This means for any given input, most of the model remains dormant, leading to a "sparse" activation. You get the knowledge capacity of a massive model with the computational cost of a much smaller one.

How it works

In an MoE layer of a transformer, the process is: 1. An input token's representation arrives at the layer. 2. A small, trainable gating network analyzes the token and assigns a score to each available expert, indicating its relevance. 3. The gating network selects the top-k experts (e.g., k=2) with the highest scores. 4. The token's representation is sent only to these selected experts for processing. 5. The outputs from the chosen experts are combined (often as a weighted sum based on the gating scores) to produce the final output for that token.

When to use it

• To scale model size without prohibitive cost: Drastically increase a model's parameter count (and thus its knowledge) while keeping inference computation manageable. • For faster inference: An MoE model with 50 billion total parameters might infer as fast as a dense 13 billion parameter model because only a fraction of the parameters are used per token. • To specialize parts of a model: Different experts can learn to specialize in different types of data, patterns, or languages within the training set.

When not to use it

• Training complexity: MoE models are harder to train. They require careful handling of "load balancing" to ensure all experts are utilized and the gating network doesn't just send all traffic to a few favorites. • Memory requirements: While inference is computationally cheap, the full set of parameters for all experts must still be loaded into memory (VRAM), so it doesn't reduce the model's memory footprint.

One canonical example

The Mixtral 8x7B model is a well-known MoE. It contains 8 distinct "expert" networks, each with roughly 7 billion parameters. For any given token, a gating network selects the top 2 experts to process it. While the model has a total of ~47B parameters, it only uses about 13B active parameters during inference. This gives it the performance and speed of a 13B model but with the knowledge capacity derived from a much larger parameter space.