Mixture of Experts: Scaling Models by Activating Specialists

Why it exists

Training larger models generally leads to better performance, but the computational cost grows prohibitively. Mixture of Experts was developed to get the benefits of a massive parameter count—like deeper knowledge and nuance—without the crippling compute costs of training an equally large dense model. It's a strategy for more efficient scaling.

The mental model

Think of an MoE model not as a single giant brain, but as a committee of specialized consultants. For any given task (a token), a dispatcher (the router) doesn't ask the entire committee to weigh in. Instead, it intelligently selects the one or two most relevant experts to handle the job. The other experts remain silent, saving time and energy.

How it works

In a standard Transformer, every token is processed by the same dense feed-forward network (FFN) layers. In an MoE, these FFN layers are replaced with MoE layers. Each MoE layer consists of two parts: a gating network (or router) and a set of experts (e.g., 8 or 16), where each expert is its own FFN. When a token arrives, the router decides which one or two experts are best suited to process it. The token is sent only to those active experts. The outputs are then combined. This means that for any given token, most of the model's parameters in that layer are inactive, a concept called sparse activation.

When to use it

Use MoE when your goal is to pre-train the largest possible model on a fixed compute budget. MoEs achieve the same quality as a dense model much faster during pre-training. They also offer significantly faster inference speeds compared to a dense model with the same total number of parameters, because only a fraction of the model is used per token.

When not to use it

MoEs are not a good fit for memory-constrained environments. Although inference is computationally sparse, all expert parameters must be loaded into VRAM, leading to massive memory requirements. For example, Mixtral 8x7B requires enough VRAM to hold a 47B parameter model. Additionally, MoEs have historically been more difficult to fine-tune without overfitting, so a traditional dense model may be more reliable for tasks requiring heavy specialization on smaller datasets.

One canonical example

Mixtral 8x7B is a prominent open-source MoE model. The name "8x7B" signifies it has 8 distinct "expert" networks, each with roughly 7 billion parameters. For any input token, its router selects only 2 of these 8 experts to perform calculations. This results in the model using about 14B active parameters for inference, giving it the speed of a smaller model while benefiting from the knowledge stored across its full ~47B parameters.