Chapter 9 — Iterables, Iterators, and Generators
1. Opening problem
Consider:
numbers = [1, 2, 3]
for number in numbers:
print(number)
This looks simple, but several protocols are involved.
Python conceptually does this:
iterator = iter(numbers)
while True:
try:
number = next(iterator)
except StopIteration:
break
print(number)
Understanding this mechanism is essential for:
- lazy processing;
- streaming;
- database pagination;
- file processing;
- asynchronous systems;
- memory-efficient pipelines;
- custom containers;
- framework internals.
2. Iterable versus iterator
An iterable is an object that can produce an iterator.
Examples:
- lists;
- tuples;
- strings;
- dictionaries;
- sets;
- files;
- ranges;
- generator objects;
- custom objects implementing
__iter__.
An iterator is an object that produces one item at a time and remembers its current position.
An iterator supports:
iter(iterator) is iterator
and:
next(iterator)
until it raises StopIteration.
Example:
numbers = [10, 20, 30]
iterator = iter(numbers)
print(next(iterator))
print(next(iterator))
print(next(iterator))
A fourth call raises:
StopIteration
The list itself is reusable. The iterator is stateful and exhaustible.
3. Reusable iterables and one-shot iterators
A list can be iterated multiple times:
numbers = [1, 2, 3]
print(list(numbers))
print(list(numbers))
A generator is usually one-shot:
generator = (
number * number
for number in range(3)
)
print(list(generator))
print(list(generator))
Output:
[0, 1, 4]
[]
The generator is exhausted after the first traversal.
This distinction matters when an API accepts Iterable[T].
The caller may provide:
- a reusable collection;
- a generator;
- a database cursor;
- a file object;
- a custom one-shot stream.
Do not iterate twice unless the contract guarantees reuse or you deliberately materialize the input.
4. The iterator protocol
A custom iterator implements:
__iter__()
__next__()
Example:
class Countdown:
def __init__(self, start: int) -> None:
self._current = start
def __iter__(self) -> "Countdown":
return self
def __next__(self) -> int:
if self._current <= 0:
raise StopIteration
value = self._current
self._current -= 1
return value
Usage:
countdown = Countdown(3)
print(list(countdown))
print(list(countdown))
Output:
[3, 2, 1]
[]
Countdown is both iterable and iterator.
This is suitable for a one-shot sequence.
5. Separate iterable and iterator objects
A reusable container should normally create a fresh iterator each time.
class CountdownSequence:
def __init__(self, start: int) -> None:
if start < 0:
raise ValueError(
"start must not be negative"
)
self._start = start
def __iter__(self) -> "CountdownIterator":
return CountdownIterator(self._start)
class CountdownIterator:
def __init__(self, start: int) -> None:
self._current = start
def __iter__(self) -> "CountdownIterator":
return self
def __next__(self) -> int:
if self._current <= 0:
raise StopIteration
value = self._current
self._current -= 1
return value
Now:
sequence = CountdownSequence(3)
print(list(sequence))
print(list(sequence))
produces the same values twice.
The iterable stores configuration. Each iterator stores traversal state.
6. Generators
A generator function contains yield.
def countdown(start: int):
current = start
while current > 0:
yield current
current -= 1
Calling a generator function does not immediately execute its body:
generator = countdown(3)
Execution starts when next() is called.
print(next(generator))
The function pauses at yield, preserving:
- local variables;
- instruction position;
- exception state;
- enclosing references.
The next call resumes immediately after the previous yield.
7. Generator execution model
def example():
print("A")
yield 1
print("B")
yield 2
print("C")
Usage:
generator = example()
print("Before")
print(next(generator))
print("Middle")
print(next(generator))
print("After")
Output:
Before
A
1
Middle
B
2
After
The final "C" appears only when another next() resumes the generator and reaches completion.
At completion, the generator raises StopIteration.
8. Returning from a generator
A generator can use return:
def generate():
yield 1
return "finished"
The return value becomes the value carried by StopIteration.
generator = generate()
print(next(generator))
try:
next(generator)
except StopIteration as error:
print(error.value)
Normal for loops ignore this value.
It matters in advanced generator composition, especially yield from.
9. yield from
Instead of:
def flatten(groups):
for group in groups:
for item in group:
yield item
use:
def flatten(groups):
for group in groups:
yield from group
yield from delegates iteration to another iterable.
It also forwards advanced generator operations such as:
- values sent with
send; - exceptions sent with
throw; - closure with
close; - subgenerator return values.
For ordinary code, its most important benefit is clear delegation.
10. Generator expressions
List comprehension:
squares = [
number * number
for number in range(1_000_000)
]
This allocates the full list.
Generator expression:
squares = (
number * number
for number in range(1_000_000)
)
This computes values lazily.
Use:
total = sum(
number * number
for number in range(1_000_000)
)
No intermediate list is required.
Lazy does not automatically mean faster. It usually means:
- reduced peak memory;
- incremental processing;
- ability to stop early;
- ability to represent infinite sequences.
There is overhead per yielded item. Measure when performance matters.
11. Lazy pipelines
def read_lines(path):
with open(
path,
encoding="utf-8",
) as file:
for line in file:
yield line.rstrip("\n")
def non_empty(lines):
for line in lines:
if line.strip():
yield line
def normalize(lines):
for line in lines:
yield line.strip().lower()
Pipeline:
lines = read_lines("events.log")
filtered = non_empty(lines)
normalized = normalize(filtered)
for line in normalized:
process(line)
Each line flows through the pipeline one at a time.
Benefits:
- low memory;
- composability;
- testable stages;
- early termination.
Risk:
- the file stays open while iteration continues;
- errors occur during consumption, not pipeline construction;
- iterating only partially may delay cleanup.
Resource lifetime must be part of the design.
12. Generator resource management
This generator opens a file:
def read_lines(path):
file = open(
path,
encoding="utf-8",
)
try:
for line in file:
yield line
finally:
file.close()
The finally block runs when:
- the generator finishes;
close()is called;- the generator is garbage-collected in common implementations.
Do not rely solely on garbage collection for important resources.
A context-managed iterable API can make lifetime clearer:
from contextlib import contextmanager
@contextmanager
def open_lines(path):
with open(
path,
encoding="utf-8",
) as file:
yield (
line.rstrip("\n")
for line in file
)
Usage:
with open_lines("events.log") as lines:
for line in lines:
process(line)
13. Infinite generators
def count_from(start: int = 0):
current = start
while True:
yield current
current += 1
Consume safely:
from itertools import islice
first_ten = list(
islice(count_from(100), 10)
)
Infinite iterables are useful for:
- sequence numbers;
- retry delays;
- event simulation;
- test data;
- scheduling.
Never pass one to list() without a limiting operation.
14. itertools
The standard library provides iterator tools:
from itertools import (
chain,
count,
cycle,
islice,
repeat,
)
Examples:
combined = chain(
first_collection,
second_collection,
)
first_five = islice(
count(10),
5,
)
statuses = cycle(
["pending", "active"]
)
Iterator tools allow declarative, lazy composition.
Avoid pipelines so clever that debugging becomes difficult.
15. send
Generators can receive values.
def accumulator():
total = 0
while True:
value = yield total
total += value
Usage:
generator = accumulator()
print(next(generator))
print(generator.send(5))
print(generator.send(10))
Output:
0
5
15
The first next() primes the generator.
Then send(5) makes the suspended yield expression evaluate to 5.
This feature supports coroutine-like patterns, but modern async code usually uses async and await.
Use send only when it makes the state machine clearer.
16. throw and close
Inject an exception:
generator.throw(
RuntimeError("stop")
)
Close:
generator.close()
close() raises GeneratorExit inside the generator.
A generator should normally allow GeneratorExit to propagate.
Do not yield from a generator while handling GeneratorExit.
These mechanisms matter for cleanup and advanced control flow.
17. Type annotations
Reusable iterable input:
from collections.abc import Iterable
def total_length(
values: Iterable[str],
) -> int:
return sum(
len(value)
for value in values
)
Iterator return:
from collections.abc import Iterator
def countdown(
start: int,
) -> Iterator[int]:
...
Generator with send and return types:
from collections.abc import Generator
def accumulator() -> Generator[
int,
int,
None,
]:
...
The generic parameters represent:
- yielded type;
- sent type;
- return type.
Use Iterator[T] when callers only need normal iteration. Use Generator[Y, S, R] when advanced generator behaviour is part of the contract.
18. Iteration and mutation
Mutating a collection during iteration is risky.
numbers = [1, 2, 3, 4]
for number in numbers:
if number % 2 == 0:
numbers.remove(number)
This may skip elements because indexes shift.
Safer:
numbers = [
number
for number in numbers
if number % 2 != 0
]
For dictionaries, changing size during iteration raises an error:
for key in mapping:
del mapping[key]
Use a snapshot:
for key in list(mapping):
del mapping[key]
or build a new dictionary.
19. TypeScript comparison
JavaScript and TypeScript support iterable protocols:
const iterator = values[Symbol.iterator]();
iterator.next();
Generator:
function* countdown(
start: number
): Generator<number> {
while (start > 0) {
yield start--;
}
}
Python uses:
iter(value)
next(iterator)
Both languages support lazy generators. Python's iteration protocol is more deeply integrated into:
for;- unpacking;
- comprehensions;
sum;min;max;any;all;- constructors;
- many standard-library tools.
20. Common mistakes
Iterating twice over a generator
The second pass is empty.
Returning a list when streaming is expected
This may consume excessive memory.
Returning a generator tied to a closed resource
Bad:
def read_lines(path):
with open(path) as file:
return (
line
for line in file
)
The file is closed before consumption.
Hiding errors through laziness
Generator errors occur during iteration.
Infinite iteration without a limit
Use islice, a condition, or cancellation.
Mutating during traversal
Build a new collection or iterate a snapshot.
21. English vocabulary
| Term | Meaning |
|---|---|
| iterable | object capable of producing an iterator |
| iterator | stateful object producing one item at a time |
| exhaustion | state after no more values remain |
| lazy evaluation | computing values only when requested |
| generator | resumable function producing values |
| delegation | forwarding iteration to another iterable |
| pipeline | sequence of transformation stages |
| materialization | converting lazy values into a collection |
| infinite sequence | iterable without a natural end |
| backpressure | controlling production according to consumption |
Useful sentences:
- “The generator is exhausted after the first traversal.”
- “The iterable creates a fresh iterator for each pass.”
- “Lazy evaluation reduces peak memory but delays errors.”
- “The resource must remain open during consumption.”
- “Materializing the pipeline defeats its streaming behaviour.”
- “The API accepts any iterable, so it must not assume repeatability.”
22. Speaking task
Explain for eight minutes:
The difference between an iterable, an iterator, and a generator.
Include one production streaming example.
23. Writing task
Write a 350-word design note comparing:
- returning
list[User]; - returning
Iterator[User]; - returning a database cursor abstraction.
Discuss memory, lifetime, error timing, and repeatability.
24. Exercises
Exercise 1
Implement a reusable RangeStep iterable supporting start, stop, and step.
Exercise 2
Write a generator that reads a large CSV file line by line and yields only valid rows.
Exercise 3
Implement lazy batching:
batch(values, size=3)
Expected:
[1, 2, 3]
[4, 5, 6]
[7]
Exercise 4
Explain why this fails:
def lines(path):
with open(path) as file:
return (
line.strip()
for line in file
)
Exercise 5
Create an infinite exponential-backoff generator starting at 0.5 seconds with multiplier 2 and maximum 30 seconds.
25. Complete solutions
Solution 1
from collections.abc import Iterator
class RangeStep:
def __init__(
self,
start: int,
stop: int,
step: int = 1,
) -> None:
if step == 0:
raise ValueError(
"step must not be zero"
)
self._start = start
self._stop = stop
self._step = step
def __iter__(self) -> Iterator[int]:
current = self._start
if self._step > 0:
while current < self._stop:
yield current
current += self._step
else:
while current > self._stop:
yield current
current += self._step
Because __iter__ is a generator function, each call creates a fresh generator.
Solution 2
import csv
from collections.abc import Iterator
from pathlib import Path
def valid_rows(
path: Path,
) -> Iterator[dict[str, str]]:
with path.open(
encoding="utf-8",
newline="",
) as file:
reader = csv.DictReader(file)
for row in reader:
email = row.get("email", "").strip()
name = row.get("name", "").strip()
if not email or "@" not in email:
continue
if not name:
continue
yield {
"email": email.lower(),
"name": name,
}
The file remains open while the generator is consumed.
Solution 3
from collections.abc import Iterable, Iterator
from typing import TypeVar
T = TypeVar("T")
def batch(
values: Iterable[T],
*,
size: int,
) -> Iterator[list[T]]:
if size <= 0:
raise ValueError(
"size must be positive"
)
current: list[T] = []
for value in values:
current.append(value)
if len(current) == size:
yield current
current = []
if current:
yield current
Solution 4
The generator expression is returned after the with block ends. The file is already closed when the caller starts iteration.
Use a generator function with yield inside the with block.
Solution 5
from collections.abc import Iterator
def backoff(
*,
initial: float = 0.5,
multiplier: float = 2.0,
maximum: float = 30.0,
) -> Iterator[float]:
if initial <= 0:
raise ValueError(
"initial must be positive"
)
if multiplier < 1:
raise ValueError(
"multiplier must be at least 1"
)
if maximum < initial:
raise ValueError(
"maximum must be at least initial"
)
delay = initial
while True:
yield delay
delay = min(
delay * multiplier,
maximum,
)
26. Chapter checkpoint
You should now be able to explain:
- iterable versus iterator;
- reusable versus one-shot traversal;
- generator execution and suspension;
- exhaustion;
- generator expressions;
- resource lifetime;
- lazy pipelines;
yield from;- advanced generator communication;
- when materialization is necessary.