Lazy evaluation produces values on demand, so memory stays proportional to the working set
Lazy evaluation defers computation until a value is needed. Instead of building a full list, a generator produces one item at a time and suspends between items, so memory is proportional to the largest single item and the pipeline state, not the total number of items. itertools composes this idea with C-level iterators for combinations, chaining, grouping, and slicing, so a pipeline can process a stream of millions of records with constant memory. The price is that generators are single-pass: once consumed they cannot be replayed, and operations that require the whole dataset, such as sorting, forcing a full materialization. Lazy evaluation is therefore a memory optimization that trades reusability and random access for a bounded footprint.
Generators yield items one at a time; list comprehensions and list() materialize everything.
itertools offers islice, chain, groupby, accumulate, and count for composable lazy pipelines.
Generator expressions use parentheses and are more memory-efficient than list comprehensions.
Trade-off: laziness hides when work happens, which complicates debugging and makes performance harder to reason about.
Beware of materializing at the end: sorted(), list(), and len() force the full dataset into memory.
Common mistake: iterating a generator twice and silently getting nothing the second time.
Common mistake: using generator expressions inside a function that returns a list, defeating the memory benefit at the boundary.
Version note: itertools has been stable since Python 2.3; itertools.batched was added in 3.12 for chunking iterables.
0-2 years experience
2-5 years experience
5-8 years experience
8+ years experience
