Chapter 10 — Comprehensions and Declarative Transformations

1. Opening problem

Imperative transformation:

result = []

for user in users:
    if user.active:
        result.append(
            user.email.lower()
        )

Comprehension:

result = [
    user.email.lower()
    for user in users
    if user.active
]

Both are correct. The important question is not which is shorter.

The question is:

Which form makes the transformation easiest to understand and change?

This chapter develops comprehensions as a declarative tool and defines their readability limits.

2. List comprehensions

General structure:

[
    expression
    for item in iterable
    if condition
]

Example:

squares = [
    number * number
    for number in range(10)
]

Filtering:

even_squares = [
    number * number
    for number in range(10)
    if number % 2 == 0
]

A comprehension creates a new list.

It should normally express one understandable transformation.

3. Evaluation order

Read a comprehension from left to right:

[
    transform(item)
    for item in items
    if valid(item)
]

Operationally:

result = []

for item in items:
    if valid(item):
        result.append(
            transform(item)
        )

The filter runs before the expression is evaluated for accepted elements.

This matters when either operation is expensive or has side effects.

Avoid side effects in comprehensions.

4. Conditional expressions

This:

[
    "even" if number % 2 == 0 else "odd"
    for number in numbers
]

transforms every element.

This:

[
    number
    for number in numbers
    if number % 2 == 0
]

filters elements.

These forms answer different questions:

conditional expression: what output should each input produce?
trailing if: should this input be included?

5. Set comprehensions

domains = {
    email.split("@", 1)[1].lower()
    for email in emails
}

A set comprehension:

removes duplicates;
does not preserve a semantic ordering contract;
requires hashable elements.

Use it when uniqueness is part of the intention.

6. Dictionary comprehensions

users_by_id = {
    user.user_id: user
    for user in users
}

Potential issue: duplicate keys silently overwrite earlier values.

users_by_email = {
    user.email.lower(): user
    for user in users
}

If duplicates indicate invalid data, validate instead of accepting last-write-wins behaviour.

Safe helper:

def index_unique(
    users,
):
    result = {}

    for user in users:
        key = user.email.lower()

        if key in result:
            raise ValueError(
                f"Duplicate email: {key}"
            )

        result[key] = user

    return result

A normal loop is clearer because collision handling is important.

7. Generator expressions

total = sum(
    order.total
    for order in orders
    if order.paid
)

This does not create an intermediate list.

Compare:

paid_totals = [
    order.total
    for order in orders
    if order.paid
]

total = sum(paid_totals)

The list version may be better if:

values are reused;
debugging requires inspection;
multiple aggregations follow;
the dataset is small;
a meaningful intermediate concept exists.

The generator version is better for single-pass consumption.

8. Nested comprehensions

Matrix flattening:

flattened = [
    value
    for row in matrix
    for value in row
]

Equivalent:

flattened = []

for row in matrix:
    for value in row:
        flattened.append(value)

The order of for clauses matches the nested loop order.

Nested comprehensions are acceptable when the structure remains obvious.

Do not compress complex business logic into one expression.

9. Cartesian products

pairs = [
    (left, right)
    for left in left_values
    for right in right_values
]

For large inputs, this can produce enormous results.

If only iteration is needed:

from itertools import product

for left, right in product(
    left_values,
    right_values,
):
    ...

Always consider output cardinality.

If each input has 10,000 values, the product has 100,000,000 pairs.

A concise comprehension can still be computationally dangerous.

10. Comprehension scope

Comprehension loop variables have their own scope:

number = 100

squares = [
    number * number
    for number in range(3)
]

print(number)

Output:

The outer binding remains unchanged.

However, names used inside the comprehension are looked up from surrounding scopes.

factor = 2

values = [
    number * factor
    for number in range(3)
]

The current factor is used during eager evaluation.

Generator expressions differ because they evaluate lazily:

factor = 2

values = (
    number * factor
    for number in range(3)
)

factor = 10

print(list(values))

The generator uses 10 during consumption.

11. Walrus operator

Assignment expressions use :=.

valid_users = [
    normalized
    for user in users
    if (
        normalized := normalize(user)
    ) is not None
]

This avoids calculating normalize(user) twice.

Without it:

valid_users = []

for user in users:
    normalized = normalize(user)

    if normalized is not None:
        valid_users.append(normalized)

The loop may be easier to read.

Use := when it reduces duplication without obscuring control flow.

Avoid clever use in deeply nested expressions.

12. Side effects are a warning sign

Bad:

[
    send_email(user)
    for user in users
]

This creates a list of return values that probably is not needed.

Use:

for user in users:
    send_email(user)

Comprehensions should create collections or lazy values. A loop communicates side effects directly.

Also avoid:

seen = set()

unique = [
    item
    for item in items
    if item not in seen
    and not seen.add(item)
]

This trick depends on set.add() returning None. It is compact but obscure.

Use an explicit helper.

13. Readability threshold

A comprehension is usually clear when it has:

one main expression;
one or two short for clauses;
one or two simple filters;
no complex exception handling;
no mutation;
no hidden I/O;
no complicated business rules.

Prefer a loop when you need:

logging;
metrics;
comments;
several intermediate names;
multiple branches;
exception handling;
early break or continue;
mutation of external state;
duplicate detection;
per-item recovery.

14. `map` and `filter`

Comprehension:

normalized = [
    normalize(email)
    for email in emails
]

Map:

normalized = list(
    map(normalize, emails)
)

map is clear when an existing named function is used.

Filter:

active = list(
    filter(is_active, users)
)

Comprehension:

active = [
    user
    for user in users
    if is_active(user)
]

Python teams often prefer comprehensions because the data flow is visible without functional nesting.

15. Chained transformations

Hard to read:

result = [
    normalize(user.email)
    for user in users
    if user.active
    if user.email is not None
    if validate(user.email)
]

Possible staged design:

active_users = (
    user
    for user in users
    if user.active
)

users_with_email = (
    user
    for user in active_users
    if user.email is not None
)

valid_emails = (
    user.email
    for user in users_with_email
    if validate(user.email)
)

result = [
    normalize(email)
    for email in valid_emails
]

Or use named generator functions.

Stages improve observability and testability when the pipeline is important.

16. Aggregation

Comprehensions and generator expressions integrate well with:

sum(...)
min(...)
max(...)
any(...)
all(...)

Example:

total_revenue = sum(
    order.total
    for order in orders
    if order.status == "paid"
)

Minimum with default:

smallest = min(
    (
        value
        for value in values
        if value > 0
    ),
    default=None,
)

Avoid materializing unless needed.

17. Dictionary inversion

inverted = {
    value: key
    for key, value in mapping.items()
}

This is safe only when values are unique and hashable.

If values repeat, earlier keys are lost.

A grouped inversion:

from collections import defaultdict

grouped = defaultdict(list)

for key, value in mapping.items():
    grouped[value].append(key)

A comprehension is not appropriate because accumulation into shared collections is the main operation.

18. TypeScript comparison

TypeScript commonly uses:

const result = users
  .filter(user => user.active)
  .map(user => user.email.toLowerCase());

Python:

result = [
    user.email.lower()
    for user in users
    if user.active
]

TypeScript method chains are expressive but may allocate intermediate arrays unless optimized by libraries.

Python comprehensions combine transformation and filtering in one construct.

For lazy pipelines, Python uses generator expressions and iterator tools.

19. Production case study: request normalization

Compact but overloaded:

commands = [
    CreateUserCommand(
        email=payload["email"].strip().lower(),
        name=payload["name"].strip(),
    )
    for payload in payloads
    if payload.get("email")
    and payload.get("name")
]

Problems:

silently skips invalid records;
untyped raw values;
repeated dictionary access;
no error reporting;
missing and blank are merged;
conversion exceptions appear inside the expression.

Better:

def parse_command(
    payload: dict[str, object],
) -> CreateUserCommand:
    email = payload.get("email")
    name = payload.get("name")

    if not isinstance(email, str):
        raise ValueError(
            "email must be a string"
        )

    if not isinstance(name, str):
        raise ValueError(
            "name must be a string"
        )

    normalized_email = email.strip().lower()
    normalized_name = name.strip()

    if not normalized_email:
        raise ValueError(
            "email must not be blank"
        )

    if not normalized_name:
        raise ValueError(
            "name must not be blank"
        )

    return CreateUserCommand(
        email=normalized_email,
        name=normalized_name,
    )


commands = [
    parse_command(payload)
    for payload in payloads
]

The comprehension remains simple because parsing complexity moved into a named function.

20. Common mistakes

Excessive nesting

Refactor to loops or helpers.

Side effects

Use a loop.

Silent duplicate overwrites

Validate dictionary keys.

Eager materialization

Use a generator expression when only one pass is needed.

Overusing functional nesting

Prefer readability.

Ignoring cardinality

A product can explode in size.

Hiding validation

Move it into a named parser.

21. English vocabulary

Term	Meaning
comprehension	compact collection-building expression
declarative	describing the result rather than control steps
transformation	converting each item
filtering	selecting items
aggregation	combining many values into one
cardinality	number of produced elements
materialization	creating the full result in memory
collision	duplicate key conflict
intermediate value	named result between stages
side effect	externally visible state change

Useful sentences:

“The comprehension expresses a single transformation clearly.”
“A normal loop is preferable because duplicate handling matters.”
“The generator expression avoids an intermediate list.”
“The Cartesian product has dangerous cardinality.”
“Validation should be extracted into a named function.”
“This expression hides side effects and should be rewritten.”

22. Speaking task

Explain for seven minutes:

When is a comprehension more readable than a loop?

Give at least three examples and two counterexamples.

23. Writing task

Write a 300-word code-review comment for a nested comprehension containing API calls, exception handling through helper functions, and mutation of a shared set.

24. Exercises

Exercise 1

Convert an imperative active-email loop into a list comprehension.

Exercise 2

Build a dictionary from product ID to product and detect duplicates instead of silently overwriting.

Exercise 3

Flatten a three-level nested collection using a generator.

Exercise 4

Calculate whether every non-empty team has at least one administrator.

Exercise 5

Refactor a side-effect comprehension that sends notifications.

25. Complete solutions

Solution 1

active_emails = [
    user.email.lower()
    for user in users
    if user.active
    and user.email is not None
]

If validation or logging becomes complex, move it to a helper or loop.

Solution 2

def index_products(products):
    result = {}

    for product in products:
        product_id = product.product_id

        if product_id in result:
            raise ValueError(
                f"Duplicate product ID: "
                f"{product_id}"
            )

        result[product_id] = product

    return result

A dictionary comprehension would hide the collision.

Solution 3

from collections.abc import Iterable, Iterator
from typing import TypeVar

T = TypeVar("T")


def flatten_three(
    groups: Iterable[
        Iterable[
            Iterable[T]
        ]
    ],
) -> Iterator[T]:
    for outer in groups:
        for inner in outer:
            yield from inner

Solution 4

every_team_valid = all(
    bool(team.members)
    and any(
        member.is_admin
        for member in team.members
    )
    for team in teams
)

If an empty teams list should be invalid:

every_team_valid = (
    bool(teams)
    and all(
        bool(team.members)
        and any(
            member.is_admin
            for member in team.members
        )
        for team in teams
    )
)

Solution 5

Bad:

[
    send_notification(user)
    for user in users
]

Good:

for user in users:
    send_notification(user)

The loop communicates that side effects are the purpose.

26. Chapter checkpoint

You should now be able to explain:

list, set, and dictionary comprehensions;
generator expressions;
filtering versus conditional transformation;
evaluation order;
comprehension scope;
readability limits;
side-effect warnings;
duplicate-key risks;
cardinality;
staged lazy pipelines.

1. Opening problem​

2. List comprehensions​

3. Evaluation order​

4. Conditional expressions​

5. Set comprehensions​

6. Dictionary comprehensions​

7. Generator expressions​

8. Nested comprehensions​

9. Cartesian products​

10. Comprehension scope​

11. Walrus operator​

12. Side effects are a warning sign​

13. Readability threshold​

14. map and filter​

15. Chained transformations​

16. Aggregation​

17. Dictionary inversion​

18. TypeScript comparison​

19. Production case study: request normalization​

20. Common mistakes​

Excessive nesting​

Side effects​

Silent duplicate overwrites​

Eager materialization​

Overusing functional nesting​

Ignoring cardinality​

Hiding validation​

21. English vocabulary​

22. Speaking task​

23. Writing task​

24. Exercises​

Exercise 1​

Exercise 2​

Exercise 3​

Exercise 4​

Exercise 5​

25. Complete solutions​

Solution 1​

Solution 2​

Solution 3​

Solution 4​

Solution 5​

26. Chapter checkpoint​

1. Opening problem

2. List comprehensions

3. Evaluation order

4. Conditional expressions

5. Set comprehensions

6. Dictionary comprehensions

7. Generator expressions

8. Nested comprehensions

9. Cartesian products

10. Comprehension scope

11. Walrus operator

12. Side effects are a warning sign

13. Readability threshold

14. `map` and `filter`

15. Chained transformations

16. Aggregation

17. Dictionary inversion

18. TypeScript comparison

19. Production case study: request normalization

20. Common mistakes

Excessive nesting

Side effects

Silent duplicate overwrites

Eager materialization

Overusing functional nesting

Ignoring cardinality

Hiding validation

21. English vocabulary

22. Speaking task

23. Writing task

24. Exercises

Exercise 1

Exercise 2

Exercise 3

Exercise 4

Exercise 5

25. Complete solutions

Solution 1

Solution 2

Solution 3

Solution 4

Solution 5

26. Chapter checkpoint