How does the yield keyword turn a function into a generator, and what happens to the function's state between calls?
yield suspends the frame; the generator object preserves locals between resumes
When the compiler sees yield anywhere in a function body, it stops emitting a normal function and instead emits a generator function. Calling that function does not run any of the body: it builds a generator object that wraps a suspended frame. Each next() resumes the frame exactly where it was suspended, with local variables, the instruction pointer, and the evaluation stack intact. CPython tracks this with the generator state machine: GEN_CREATED, GEN_RUNNING, GEN_SUSPENDED, and GEN_CLOSED. The frame is not copied or replayed, so side effects before a yield happen exactly once.
- 1
Locals persist because the frame object stays alive; you can inspect it with gen.gi_frame and inspect.getgeneratorstate(gen).
- 2
PEP 342 added send(), throw(), and close(), turning generators into resumable coroutine-like objects. PEP 380 added yield from for delegation.
- 3
Trade-off: generators are lazy and constant-memory, but they are single-use and you cannot call len() or index them. If you need random access, materialize with list().
- 4
Common mistake: assuming the body runs at call time. The first next() is what executes up to the first yield.
- 5
Common mistake: treating an exhausted generator as empty rather than closed; further next() calls raise StopIteration forever, and generators cannot be restarted.
- 6
Version note: PEP 525 (3.6) added async generators, PEP 342 (2.5) added send/throw/close, and generator return values via StopIteration.value arrived in PEP 380 (3.3).
Scenario Questions
0-2 years experience
- 1You call a generator function and print the result without iterating. Why is no output produced?
- 2Why does next() on an exhausted generator raise StopIteration instead of returning None?
2-5 years experience
- 1A generator opens a database cursor and yields rows. What state is alive while it is suspended, and what is the risk?
- 2You need to push a configuration value into a running generator. Which method do you call, and what is the rule for the first call?
5-8 years experience
- 1You need a file handle to be released when a generator is abandoned early. What mechanism do you use and how do you test it?
- 2You profile a pipeline whose memory grows even though it uses generators. How do you find what is being pinned alive between yields?
8+ years experience
- 1Design a generator-based state machine that supports send(), throw(), and close() correctly and is safe against misuse.
- 2You need to persist and resume a suspended computation across process restarts. What are the hard limits of generator frames and what would you use instead?
Follow-up Questions
- What happens to a generator's local variables if the generator is garbage collected while suspended?
- Why does the first send() call have to be send(None) or next()?