Python's `yield`: Functions That Pause and Resume

WHY IT EXISTS Processing large sequences of data presents a memory problem. Loading a gigabyte-sized file or a list with millions of numbers into memory can exhaust system resources. Generators were created to process data lazily, computing and yielding one item at a time, using a constant, small amount of memory.

THE MENTAL MODEL A function with a yield statement is not a normal function; it's a generator. Think of it as a pausable factory for a sequence of values. Calling a normal function with return runs it once and it's done. Calling a generator function creates an iterator object that is frozen in time. Each time you ask for a value, the function unfreezes, runs until it hits the next yield, hands that value back, and freezes again, remembering its exact state.

HOW IT WORKS When you call a function containing yield, Python returns a generator object. This object implements the iterator protocol. When a for loop or next() function requests an item, the generator's code executes from where it last left off. The yield statement sends a value to the caller and pauses execution. When the function runs to completion or hits a return, it raises a StopIteration exception, which signals to the for loop that the iteration is complete.

WHEN TO USE IT Use generators for memory-efficient iteration, especially with data streams that are too large for RAM. This includes reading large files line-by-line, handling continuous data from network sockets or sensors, or implementing infinite sequences (like Fibonacci numbers). They are also excellent for building clean data processing pipelines where the output of one generator feeds into the next.

WHEN NOT TO USE IT Avoid generators when you need random access to elements (e.g., my_sequence[5]), need to know the length of the sequence beforehand (len(my_sequence)), or need to iterate over the data multiple times. In these cases, a list or tuple is the correct data structure, provided it fits in memory. Converting a generator to a list with list(my_gen) is possible but negates the memory benefits.

ONE CANONICAL EXAMPLE Reading a large log file without loading it all into memory. A normal function might return f.readlines(), which creates a massive list. A generator yields one line at a time. For example: def read_logs(filepath): with open(filepath) as f: for line in f: yield line. This function can process a file of any size using only enough memory to hold a single line.