A generator is a Python function that produces a sequence of values lazily, one at a time, instead of building and returning them all at once. The yield keyword is what makes a function a generator: each time execution reaches a yield, the function pauses, hands a value back to the caller, and resumes exactly where it left off on the next request. Use generators when you want to stream data, process input that does not fit in memory, or build composable data pipelines.
This guide uses modern Python (3.12 / 3.13). All examples are runnable as-is. We will cover what yield does, how generators differ from ordinary functions and lists, lazy evaluation, yield from, coroutine-style send(), async generators, and the cases where a generator is the wrong choice.
What yield does and how generators differ from normal functions
A normal function runs top to bottom and returns a single value with return. A function that contains at least one yield becomes a generator function: calling it does not run the body at all. Instead it returns a generator object (an iterator). The body only executes when you advance the iterator, and it pauses at every yield, preserving all local state between resumes.
def count_up_to(limit):
n = 1
while n <= limit:
yield n # pause here, return n, remember state
n += 1
gen = count_up_to(3)
print(type(gen)) # <class 'generator'>
for value in gen:
print(value)
# 1
# 2
# 3Notice that count_up_to(3) did nothing until the for loop started pulling values. Local variables like n survive across each yield, which is what lets a generator model an ongoing computation without you managing the state by hand.
Generator functions vs generator expressions
There are two ways to create generators:
- Generator functions use
defand one or moreyieldstatements. Best for multi-step logic, branching, or anything that needs several lines. - Generator expressions look like list comprehensions but use parentheses instead of brackets. Best for short, inline transformations.
A generator expression is the lazy sibling of a list comprehension. Swap the [] for () and you get a generator that computes values on demand instead of materialising the whole list.
# List comprehension: builds the whole list in memory immediately
squares_list = [i * i for i in range(5)]
print(squares_list) # [0, 1, 4, 9, 16]
# Generator expression: computes each value only when requested
squares_gen = (i * i for i in range(5))
print(squares_gen) # <generator object <genexpr> at 0x...>
print(list(squares_gen)) # [0, 1, 4, 9, 16]
# A generator is single-use: once exhausted, it yields nothing more
print(list(squares_gen)) # [] (already consumed above)A list can be iterated many times and indexed (squares_list[2]). A generator can be iterated once and cannot be indexed. That single-use, forward-only nature is the trade-off you accept in exchange for laziness and low memory use.
Lazy evaluation and memory efficiency
The headline benefit of generators is memory. A list of one million squares holds a million integers in RAM. The equivalent generator holds essentially nothing until you ask for the next value. The comparison below uses sys.getsizeof to make the difference concrete.
import sys
n = 1_000_000
big_list = [i * i for i in range(n)]
big_gen = (i * i for i in range(n))
print(sys.getsizeof(big_list)) # ~8,000,000+ bytes (grows with n)
print(sys.getsizeof(big_gen)) # ~200 bytes (constant, independent of n)
# Same result, vastly different footprint
print(sum(i * i for i in range(n))) # streamed, no intermediate listList vs generator at a glance
| Aspect | List [...] |
Generator (...) / yield |
|---|---|---|
| Evaluation | Eager (all values up front) | Lazy (one value at a time) |
| Memory | Holds every element | Near-constant, regardless of size |
| Reusable | Yes, iterate many times | No, single forward pass |
| Indexable / slice | Yes (xs[3], xs[1:4]) |
No |
len() |
Yes | No |
| Best for | Small/medium data reused often | Large/streaming/infinite data |
Lists are also typically faster for small collections you reuse, because there is no per-item suspend/resume overhead. Reach for generators when data is large, streamed, infinite, or only needed once.
A real use case: reading a huge file line by line
Suppose you have a 10 GB log file. Reading it into a list with f.readlines() would exhaust memory. A file object is already a lazy line iterator, and you can layer a generator on top to filter or transform without ever loading the whole file. This is the kind of streaming work we rely on for log processing and ETL across Python development projects.
def error_lines(path):
"""Yield only the ERROR lines from a large log file, one at a time."""
with open(path, "r", encoding="utf-8") as f:
for line in f: # the file object is itself lazy
if "ERROR" in line:
yield line.rstrip("\n")
# Memory stays flat even on a multi-GB file
for line in error_lines("app.log"):
print(line)next(), StopIteration, and the iterator protocol
Under the hood, a for loop calls next() on the iterator repeatedly and stops when it receives a StopIteration exception. You can drive a generator manually with the built-in next() to see this happen. Passing a default to next(gen, default) returns the default instead of raising when the generator is exhausted.
def gen():
yield "a"
yield "b"
it = gen()
print(next(it)) # 'a'
print(next(it)) # 'b'
try:
next(it) # nothing left to yield
except StopIteration:
print("exhausted") # exhausted
# Safer: supply a default instead of catching the exception
print(next(gen(), None)) # 'a'
print(next(iter([]), "empty")) # 'empty'Inside a generator, a bare return (or reaching the end of the function) ends iteration. If you return value, that value is attached to the StopIteration exception as .value rather than being yielded, which yield from uses to pass results back up the chain.
yield from: generator delegation
yield from <iterable> delegates to a sub-iterator: it yields every item the sub-iterator produces, transparently forwarding values, exceptions, and send() calls. It replaces verbose inner loops and is the clean way to flatten or chain generators.
def chain(*iterables):
for it in iterables:
yield from it # forwards each item without a manual inner loop
print(list(chain([1, 2], (3, 4), "xy")))
# [1, 2, 3, 4, 'x', 'y']
# yield from also captures a sub-generator's return value
def inner():
yield 1
yield 2
return "done" # becomes StopIteration.value
def outer():
result = yield from inner()
print("inner returned:", result) # inner returned: done
yield 3
print(list(outer())) # [1, 2, 3]Building generator pipelines
Because each generator consumes one lazy source and produces another, you can compose them into a pipeline where data flows through stage by stage and nothing is buffered in full. This style is memory-flat and reads top to bottom like a Unix pipe.
def read_numbers(lines):
for line in lines:
line = line.strip()
if line:
yield int(line)
def only_even(numbers):
for n in numbers:
if n % 2 == 0:
yield n
def doubled(numbers):
for n in numbers:
yield n * 2
raw = ["1", "2", " 3 ", "", "4", "6"]
# Stages are wired together; data is pulled through lazily by sum()
pipeline = doubled(only_even(read_numbers(raw)))
print(sum(pipeline)) # (2 + 4 + 6) * 2 = 24send(): generators as lightweight coroutines
yield is also an expression: value = yield item sends item out and waits for a value to be sent back in via gen.send(...). This turns a generator into a coroutine that can receive data, useful for accumulators and small state machines. You must prime the generator first (advance it to the first yield) with next() or send(None).
def running_average():
total = 0.0
count = 0
average = None
while True:
value = yield average # emit current average, wait for next input
total += value
count += 1
average = total / count
avg = running_average()
next(avg) # prime: run up to the first yield
print(avg.send(10)) # 10.0
print(avg.send(20)) # 15.0
print(avg.send(30)) # 20.0
avg.close() # stop the coroutineFor most concurrency work today you would reach for async/await rather than hand-rolled coroutines, but send() remains handy for streaming accumulators and parser state machines.
Async generators
Since Python 3.6 you can combine async def with yield to create an async generator: it produces values lazily while await-ing between them, ideal for streaming over the network, async database cursors, or paginated APIs. Consume it with async for inside a coroutine. This is a staple of async services we build with frameworks like FastAPI.
import asyncio
async def fetch_pages(total):
"""Pretend to page through a remote API, awaiting each request."""
for page in range(1, total + 1):
await asyncio.sleep(0.1) # simulate network I/O
yield f"page-{page} data"
async def main():
async for chunk in fetch_pages(3): # lazy, awaits between items
print(chunk)
asyncio.run(main())
# page-1 data
# page-2 data
# page-3 dataWhen NOT to use generators
Generators are not always the right tool. Prefer a list (or other concrete collection) when:
- You need to iterate the data more than once — a generator is exhausted after one pass.
- You need random access, slicing, or
len()— generators support none of these. - The dataset is small and reused often — a list is simpler and usually faster, with no suspend/resume overhead.
- You need the full result before continuing (e.g. to sort, or to return it from an API as JSON) — you will materialise it anyway, so build the list directly.
- Debugging is hard if you forget a generator is lazy: exceptions and side effects fire only when items are pulled, which can surprise you. When in doubt, wrap the call in
list(...)to force evaluation while diagnosing.
Good architecture is mostly about choosing the right data structure for the access pattern — a theme that runs through our other guides on Python coding techniques and working with Python collections.
Frequently Asked Questions
What is the difference between yield and return in Python?
return ends a function and hands back a single value, discarding all local state. yield pauses the function, hands back one value, and remembers exactly where it stopped — so the next request resumes from that point. A function with return runs once; a function with yield becomes a generator that can produce many values over time, lazily.
When should I use a generator instead of a list?
Use a generator when the data is large, streamed, infinite, or only consumed once, because it keeps memory nearly constant by computing values on demand. Use a list when you need to iterate the data multiple times, index or slice it, call len() on it, or when the collection is small and reused often (lists are simpler and typically faster for that case).
Why can I only loop over a generator once?
A generator is a forward-only iterator with no stored history: each next() advances it permanently and discards the previous state. Once it raises StopIteration, it is exhausted and will yield nothing more. If you need to iterate again, recreate the generator by calling the generator function again, or materialise the results with list(gen) and iterate that list.
What does yield from do?
yield from <iterable> delegates to a sub-iterator, yielding all of its items as if they were written inline. It also transparently forwards send(), thrown exceptions, and the sub-generator's return value (exposed via StopIteration.value). It replaces verbose inner loops and is the idiomatic way to chain, flatten, or compose generators.
How are generators different from async generators?
A regular generator (def + yield) produces values synchronously and is consumed with a normal for loop or next(). An async generator (async def + yield, available since Python 3.6) can await between yields and is consumed with async for inside a coroutine. Use async generators when each item depends on asynchronous I/O such as network requests or async database cursors.
Are generators faster than lists in Python?
Not necessarily. Generators save memory and avoid building intermediate collections, which makes them faster for large or streamed data where allocation dominates. For small collections you iterate repeatedly, a list is usually faster because it avoids the per-item suspend/resume overhead of a generator. Choose based on data size and access pattern, not on a blanket assumption that one is always quicker.
MicroPyramid has built data-intensive Python systems for startups and enterprises for 12+ years across 50+ delivered projects, using generators and streaming patterns to keep memory flat in ETL pipelines, log processors, and high-throughput async APIs with Django and FastAPI. If you are designing a Python service that needs to process more data than fits in memory, these techniques are the foundation.