Failover: Automatic Recovery When Things Go Wrong

Why it exists

Systems fail. Hardware breaks, software has bugs, and networks go down. Failover exists to minimize or eliminate the user-facing impact of these inevitable failures by having a backup ready to go automatically. Without it, any single failure could cause a complete outage requiring manual intervention.

The mental model

Think of failover like the emergency power generator for a hospital. The main power grid is the primary system. If it fails, sensors detect the outage and automatically start the generator (the standby system) to take over the load. The goal is a seamless transition so critical operations continue uninterrupted. The key is that it's automatic; a manual switch would be a 'switchover'.

How it works

A failover system continuously monitors the health of the primary component, often using a 'heartbeat' signal. If the heartbeat stops, the monitoring system declares the primary dead and triggers the switch. This involves redirecting traffic (like updating DNS or IP addresses) to the standby server, which then becomes the new primary.

When to use it

Use failover for any stateful or critical service where availability is more important than the cost of redundancy. Common examples include primary databases in active-passive clusters, network load balancers, and critical application servers. It's for services where even a few minutes of downtime is unacceptable.

When not to use it

Avoid failover for stateless services that can be easily replaced by a load balancer spinning up a new instance. If the cost of maintaining a fully redundant, synchronized standby system outweighs the business cost of a brief outage, a manual switchover might be sufficient. If a brief, planned outage is acceptable, you may not need it.

One canonical example

A common database setup is an active-passive cluster. The active database handles all requests while constantly replicating its data to a passive, standby database. If the active database server fails, the system automatically promotes the passive server to become the new active one and redirects all application traffic to it, minimizing data loss and downtime.