Least-to-Most Prompting: Solving Hard Problems Incrementally

WHY IT EXISTS Standard prompting techniques like chain-of-thought often fail when a problem is significantly harder than the examples provided in the prompt. They struggle with this 'easy-to-hard' generalization. Least-to-most prompting was designed specifically to bridge this gap, enabling models to tackle more complex reasoning tasks.

THE MENTAL MODEL Think of solving a complex, multi-step math problem. You don't jump to the final answer. Instead, you solve the first part, use that result to solve the second, and continue step-by-step. Least-to-most prompting applies this incremental logic to LLMs, first breaking a problem down and then solving the pieces in order.

HOW IT WORKS This strategy involves a two-stage process. First, you prompt the model to decompose the main problem into a series of simpler subproblems. For example, asking it to list the steps needed to answer a complex question. Second, you prompt the model to solve each subproblem sequentially, explicitly including the answers from previous steps in the context for the current one. This creates a chain of reasoning where each solution builds on the last.

WHEN TO USE IT Use this for complex reasoning tasks that can be broken into a clear sequence of smaller, dependent steps. It's highly effective for symbolic manipulation, compositional generalization (e.g., combining known commands in novel ways), and multi-step math problems. It is the right tool when the problem's complexity far exceeds that of your few-shot examples.

WHEN NOT TO USE IT Avoid this for simple, single-step problems where a direct prompt or standard chain-of-thought is sufficient; the overhead is unnecessary. It's also a poor fit for problems that cannot be cleanly decomposed into a linear sequence. If subproblems are interdependent in a complex web rather than a simple chain, this method will likely fail.

ONE CANONICAL EXAMPLE On the compositional generalization benchmark SCAN, standard chain-of-thought prompting scored only 16% accuracy. Using least-to-most prompting with just 14 examples, the same model (GPT-3 code-davinci-002) achieved over 99% accuracy, demonstrating its powerful ability to generalize from simple instructions to complex ones.