What is the role of temperature in token sampling?

### What this tests This tests your understanding of the mechanism for controlling the randomness of an LLM's output. Interviewers want to see that you can connect a user-facing parameter (temperature) to the underlying math (modifying logits before the softmax function) and explain the practical trade-offs between deterministic and creative outputs.

### A good answer covers * **Definition:** Temperature is a hyperparameter, typically ranging from 0.0 to 2.0, that scales the logits before they are passed into the softmax function to calculate token probabilities. * **Mechanism:** The core operation is `probabilities = softmax(logits / temperature)`. It directly manipulates the values that determine the final probability distribution for the next token. * **Low Temperature (e.g., 0.2):** Dividing by a number less than 1 makes the logits larger, exaggerating the differences between them. This creates a 'sharper' or 'peakier' probability distribution where the model is very likely to select the highest-probability token. A temperature of 0 is equivalent to greedy decoding (always picking the single most likely token). * **High Temperature (e.g., 1.5):** Dividing by a number greater than 1 makes the logits smaller, shrinking the differences between them. This creates a 'flatter' distribution, giving lower-probability tokens a higher chance of being selected. This leads to more random, creative, or surprising outputs. As temperature approaches infinity, the distribution approaches uniform randomness. * **Practical Use Cases:** Use low temperature (~0.1-0.5) for factual, predictable tasks like code generation, summarization, or closed-domain Q&A. Use high temperature (~0.8-1.2) for creative tasks like story writing, brainstorming, or creating marketing copy.

### Common wrong answers * **Vagueness:** Simply stating "it makes the output more or less random" without explaining the connection to logits and the softmax function. * **Confusion with other methods:** Incorrectly equating temperature with top-p (nucleus) sampling. Temperature *reshapes* the entire probability distribution, while top-p *truncates* it by considering only the most probable tokens that sum to a certain probability mass. * **Incorrect mechanism:** Claiming temperature is a parameter *of* the softmax function itself, rather than a divisor for its inputs.

### Follow-up the interviewer might ask * "How does temperature sampling differ from top-k sampling? When would you use one over the other?" * "Describe a scenario where you might use a high temperature but a low top-p. What would be the expected output?"

### One concrete example Imagine a model's logits for the next three tokens are `[2.0, 1.0, 0.1]`.

* **At T=1.0:** The probabilities are `[0.66, 0.24, 0.10]`. The most likely token is strongly favored. * **At T=0.5:** The scaled logits become `[4.0, 2.0, 0.2]`. The probabilities get much sharper: `[0.86, 0.12, 0.02]`. The model becomes very deterministic, almost certainly picking the first token. * **At T=2.0:** The scaled logits become `[1.0, 0.5, 0.05]`. The probabilities flatten to `[0.50, 0.30, 0.20]`. The other tokens now have a much more reasonable chance of being sampled, leading to more varied output.