What is the role of temperature in token sampling?

What's really being asked

This tests your understanding of the mechanism for controlling the randomness of an LLM's output. Interviewers want to see that you can connect a user-facing parameter (temperature) to the underlying math (modifying logits before the softmax function) and explain the practical trade-offs between deterministic and creative outputs.

The full answer

• Definition: Temperature is a hyperparameter, typically ranging from 0.0 to 2.0, that scales the logits before they are passed into the softmax function to calculate token probabilities. • Mechanism: The core operation is probabilities = softmax(logits / temperature). It directly manipulates the values that determine the final probability distribution for the next token. • Low Temperature (e.g., 0.2): Dividing by a number less than 1 makes the logits larger, exaggerating the differences between them. This creates a 'sharper' or 'peakier' probability distribution where the model is very likely to select the highest-probability token. A temperature of 0 is commonly handled as greedy decoding by an API, while low positive temperatures can still sample different outputs. • High Temperature (e.g., 1.5): Dividing by a number greater than 1 makes the logits smaller, shrinking the differences between them. This creates a 'flatter' distribution, giving lower-probability tokens a higher chance of being selected. This leads to more random, creative, or surprising outputs. As temperature approaches infinity, the distribution approaches uniform randomness. • Practical Use Cases: Use low temperature (~0.1-0.5) for factual, predictable tasks like code generation, summarization, or closed-domain Q&A. Use high temperature (~0.8-1.2) for creative tasks like story writing, brainstorming, or creating marketing copy.

The mistakes people make

• Vagueness: Simply stating "it makes the output more or less random" without explaining the connection to logits and the softmax function. Confusion with other methods: Incorrectly equating temperature with top-p (nucleus) sampling. Temperature reshapes the entire probability distribution, while top-p truncates* it by considering only the most probable tokens that sum to a certain probability mass. Incorrect mechanism: Claiming temperature is a parameter of* the softmax function itself, rather than a divisor for its inputs.

Follow-up the interviewer might ask

• "How does temperature sampling differ from top-k sampling? When would you use one over the other?" • "Describe a scenario where you might use a high temperature but a low top-p. What would be the expected output?"

A concrete example

Imagine a model's logits for the next three tokens are [2.0, 1.0, 0.1].

• At T=1.0: The probabilities are [0.66, 0.24, 0.10]. The most likely token is strongly favored. • At T=0.5: The scaled logits become [4.0, 2.0, 0.2]. The probabilities get much sharper: [0.86, 0.12, 0.02]. The model becomes very deterministic, almost certainly picking the first token. • At T=2.0: The scaled logits become [1.0, 0.5, 0.05]. The probabilities flatten to [0.50, 0.30, 0.20]. The other tokens now have a much more reasonable chance of being sampled, leading to more varied output.