Softmax Function: Turning Scores into Probabilities
The softmax function turns a list of raw scores from a model into a clean probability distribution where all values sum to 1. It's most often the final step in a neural network for multi-class classification, like deciding if an image is a 'cat', 'dog', or 'bird'. The main footgun is mistaking a high softmax probability for high model confidence; it only reflects the score's strength relative to the other scores, not its absolute certainty.
### The Mental Model The softmax function converts a vector of raw scores, called logits, into a proper probability distribution. Think of it as forcing a model to make a clear choice. It takes messy, unbounded numbers and transforms them into a set of values between 0 and 1 that all add up to exactly 1. It's a "soft" version of `argmax`, which would rigidly pick the single best option (100%) and assign 0% to all others.
### How It Works For a given vector of `K` real numbers `z = [z_1, z_2, ..., z_K]`, the softmax of the i-th element is calculated in two steps: 1. **Exponentiate:** Take the mathematical constant `e` to the power of each score (`e^z_i`). This makes all numbers positive and exaggerates the differences between them. A larger score grows exponentially faster than a smaller one. 2. **Normalize:** Divide each exponentiated score by the sum of all exponentiated scores. This scales the results so they collectively sum to 1.
The formula is: `Softmax(z_i) = e^(z_i) / Σ(e^(z_j))` for all j from 1 to K.
### When to Use It * **Multi-class Classification:** As the final activation function in a neural network to get probabilities for one of many possible classes (e.g., an ImageNet model predicting one of 1000 object categories). * **Language Models (LLMs):** In the final layer to produce a probability distribution over the entire vocabulary for predicting the next word or token.
### When NOT to Use It * **Multi-label Classification:** When an input can belong to multiple classes at once (e.g., tagging a news article with 'politics', 'finance', and 'europe'). Use a `sigmoid` function on each output neuron independently, as their probabilities don't need to sum to 1. * **Regression:** When predicting a continuous value, like a house price or temperature. A linear activation function is typically used in the output layer for regression.
### One Canonical Example A neural network classifying an image as a 'cat', 'dog', or 'fish' produces the raw logits `[2.0, 1.0, 0.1]`.
1. **Exponentiate:** * `e^2.0 ≈ 7.39` * `e^1.0 ≈ 2.72` * `e^0.1 ≈ 1.11` 2. **Normalize:** * The sum is `7.39 + 2.72 + 1.11 = 11.22`. * Cat probability: `7.39 / 11.22 ≈ 0.66` (66%) * Dog probability: `2.72 / 11.22 ≈ 0.24` (24%) * Fish probability: `1.11 / 11.22 ≈ 0.10` (10%)
The model's final output is a probability distribution `[0.66, 0.24, 0.10]`, indicating 'cat' as the most likely class.
Read the original → en.wikipedia.org
Get five bites like this every day.
Tezvyn delivers a daily feed of 60-second tech bites with quizzes to lock in what you learn.