Multi-Armed Bandits for Model Selection

Why it exists

Traditional model deployment relies on static metrics like validation error or slow, manual A/B tests. These methods are not adaptive. When a model's performance degrades in production due to data drift, a human must intervene to analyze, decide, and deploy a fix.

The mental model

Think of your candidate models as a row of slot machines ('multi-armed bandits'). Each 'arm' is a different model version. Your goal is to maximize your total winnings (e.g., conversions, accuracy, revenue) over time. A MAB algorithm automatically decides which arm to pull (which model to serve) for each incoming request to learn which one is best.

How it works

A MAB algorithm constantly balances two actions: exploration and exploitation. Exploration means sending a small amount of traffic to less-proven models to gather performance data. Exploitation means sending most of the traffic to the model that has historically performed the best. Algorithms like Thompson Sampling or Upper Confidence Bound (UCB) manage this trade-off automatically, adapting traffic splits in real-time as new performance data arrives, effectively automating the A/B/n testing process.

When to use it

Use MABs to automate the selection between several competing models in a live environment. It is especially powerful for situations with potential model drift, where a model's performance can change over time. It replaces manual A/B test analysis and deployment decisions, enabling a more adaptive and resilient system.

When not to use it

MABs can be overkill for simple, one-off deployments where you have high confidence in a single model and don't expect its performance to change. A standard A/B test is simpler if you only need to compare two models for a fixed period and have the resources to analyze the results manually.

One canonical example

An e-commerce site has three different recommendation models (A, B, C). Instead of a fixed 50/50 A/B test, a MAB controller is deployed. Initially, it sends traffic roughly equally to A, B, and C. If model B starts generating more clicks and purchases, the MAB automatically increases its traffic share while still sending a small fraction to A and C to ensure B's superiority isn't just a fluke. If a new model D is added, the system starts exploring it immediately without manual reconfiguration.