Chapter 26 — Modules, Packages, Imports, and Plugin Discovery

1. Opening problem

A module contains:

print(
    "Connecting to production database"
)

connection = connect_database()

Any import executes the connection:

import application.settings

This can happen during:

• application startup;
• test collection;
• documentation generation;
• static analysis plugins;
• command-line completion;
• worker initialization;
• migration discovery.

An import is not merely a textual include. It is a runtime operation that can locate, create, execute, cache, and bind module objects.

Import-time behavior is architecture.

---

2. Module objects

A Python source file can define a module.

# pricing.py

TAX_RATE = 0.19


def calculate(
    subtotal: float,
) -> float:
    return subtotal * (
        1 + TAX_RATE
    )

Import:

import pricing

The name pricing is bound to a module object.

print(
    type(pricing)
)

print(
    pricing.__name__
)

A module has a namespace containing functions, classes, variables, imports, and metadata.

---

3. Import has two broad jobs

import package.module

Broadly:

1. find or load the module;
2. bind a name in the importing scope.

The import machinery may also:

• import parent packages;
• consult caches;
• create module specifications;
• execute loaders;
• update sys.modules;
• initialize package attributes.

Only the import statement performs the name-binding operation described by its syntax.

The machinery can also be invoked through importlib.

---

4. sys.modules

import sys


print(
    "json" in sys.modules
)

sys.modules maps module names to loaded module objects.

Before loading, the import system checks this cache.

Consequences:

• module top-level code normally runs once per interpreter process;
• repeated imports usually return the cached object;
• modules act as process-level singleton-like namespaces;
• mutable module state is shared;
• test isolation can be affected;
• circular imports can observe partially initialized modules.

Do not use the cache as a general application registry unless the semantics are intentional.

---

5. Why modules enter the cache before execution completes

Suppose:

# a.py
import b

# b.py
import a

If the import system waited until module execution completed before caching, recursion could continue indefinitely.

A module can be inserted into sys.modules before its code has fully executed.

Therefore another module can observe a partially initialized module.

This is a central cause of circular-import failures.

---

6. Binding forms

import package.module

binds package in the local namespace.

import package.module as module

binds module.

from package import module

binds module.

from package.module import function

binds function directly to the object existing at import time.

Qualified access:

import pricing

pricing.calculate(
    subtotal
)

often makes ownership clearer than many direct imports.

---

7. Imported bindings and reload

from settings import TIMEOUT

Later reloading settings does not automatically update the already-bound local TIMEOUT.

import settings

settings.TIMEOUT

qualified access reads the current module attribute.

This is one reason hot reloading is difficult.

Existing references to:

• classes;
• functions;
• constants;
• instances

may continue pointing to old objects after module reload.

---

8. __name__ and __main__

When a file executes as the top-level program:

__name__ == "__main__"

Entry-point pattern:

def main() -> int:
    ...
    return 0


if __name__ == "__main__":
    raise SystemExit(
        main()
    )

Benefits:

• import does not run the command;
• main is testable;
• exit status is explicit;
• script behavior is separated from reusable logic.

For packages, a __main__.py supports:

python -m your_package

---

9. Running a file versus a module

python path/to/tool.py

and:

python -m package.tool

can produce different import behavior because sys.path, package context, and __package__ differ.

For package code, python -m package.tool often preserves the intended package context.

Do not repair package execution by adding arbitrary paths inside code:

sys.path.append(
    "../.."
)

Fix installation, project layout, or invocation.

---

10. Packages

A traditional package contains __init__.py.

src/
└── billing/
    ├── __init__.py
    ├── invoices.py
    └── payments.py

Import:

import billing.invoices

__init__.py executes when the package is imported.

Keep it lightweight.

Possible uses:

• package metadata;
• carefully selected public re-exports;
• simple constants;
• compatibility aliases.

Avoid expensive startup, network calls, or importing every submodule automatically.

---

11. Public package API

Internal module:

# billing/_models.py

class Invoice:
    ...

Public re-export:

# billing/__init__.py

from ._models import Invoice

__all__ = [
    "Invoice",
]

Caller:

from billing import Invoice

A package-level API can hide internal layout.

Trade-offs:

• convenient imports;
• stable public surface;
• potential circular imports;
• import-time cost;
• accidental eager loading.

Design the public API intentionally.

---

12. __all__

__all__ = [
    "Invoice",
    "create_invoice",
]

__all__ affects wildcard imports:

from billing import *

It can also communicate intended exports to tools.

It does not create strong privacy.

Avoid wildcard imports in application code because they obscure name ownership and complicate static analysis.

---

13. Absolute imports

from application.users.models import User

Advantages:

• clear origin;
• stable understanding from any module;
• easier search;
• fewer ambiguous names.

Cost:

• verbose;
• package renames affect many files.

Absolute imports are often preferred for application boundaries and cross-package dependencies.

---

14. Relative imports

from .models import User

from ..shared.clock import Clock

Relative imports are resolved from package context.

They can be appropriate inside a cohesive package.

Risks:

• deep dot chains;
• package execution confusion;
• unclear architecture when crossing many levels;
• harder extraction.

Use shallow relative imports for nearby implementation modules. Use absolute imports where ownership clarity matters.

---

15. Circular imports

Example:

# users.py
from orders import Order

# orders.py
from users import User

Potential symptoms:

• partially initialized module;
• missing attribute;
• import error;
• runtime order dependence;
• test-only failure.

Circular imports often reveal architectural cycles.

Possible fixes:

• move shared value types to a lower-level module;
• depend on protocols;
• move orchestration to a higher layer;
• defer type-only imports with TYPE_CHECKING;
• pass dependencies into functions;
• import locally as a tactical workaround.

Do not treat local imports as the universal architectural fix.

---

16. Type-only imports

from typing import (
    TYPE_CHECKING,
)


if TYPE_CHECKING:
    from application.orders import (
        Order,
    )

Use string or deferred annotations as appropriate for the project.

This can prevent a runtime cycle caused only by type annotations.

It does not fix a real runtime dependency cycle.

---

17. Local imports

def create_order():
    from application.payments import (
        charge,
    )

    return charge()

Potential reasons:

• optional dependency;
• expensive import used rarely;
• runtime cycle as a temporary workaround;
• platform-specific feature.

Costs:

• hidden dependency;
• repeated cache lookup;
• harder testing and static analysis;
• delayed failures.

Document the reason.

---

18. Dynamic imports

from importlib import (
    import_module,
)


module = import_module(
    "application.plugins.csv"
)

Dynamic imports support:

• plugin systems;
• optional features;
• configuration-selected implementations;
• command discovery.

Never dynamically import arbitrary untrusted user input.

Use an allowlist or trusted plugin metadata.

---

19. Module specifications

Modern import machinery represents loading information through module specifications.

module.__spec__

A specification can describe:

• module name;
• loader;
• origin;
• package search locations;
• parent relationship.

Application code rarely needs to construct specs manually.

Framework and plugin authors may need this layer.

---

20. Import finders and loaders

The import system can be extended through mechanisms including sys.meta_path.

A finder locates a module specification.

A loader creates or executes the module.

Custom import hooks can load code from:

• archives;
• databases;
• remote systems;
• encrypted sources;
• generated modules.

This is powerful and security-sensitive.

Prefer explicit plugin loading over custom import machinery unless the use case truly requires a new import source.

---

21. Import caches

Finders may cache directory or source information.

When creating a module file while the interpreter is running:

import importlib


importlib.invalidate_caches()

may be needed before dynamic discovery notices it.

Do not call cache invalidation on every normal import.

Use it only when the filesystem or module source changes at runtime.

---

22. Reloading

import importlib
import settings


importlib.reload(
    settings
)

Reload re-executes module code, usually reusing the module object.

Problems:

• old names can remain when new code does not redefine them;
• direct imports are not rebound;
• existing instances use old classes;
• extension modules may not support repeated initialization;
• reload is not thread-safe;
• side effects may run again;
• registries can duplicate entries.

Do not use reload as a general production deployment strategy.

Restarting the process is often safer.

---

23. Import-time side effects

Avoid:

client = ExternalClient(
    os.environ[
        "PRODUCTION_TOKEN"
    ]
)

client.connect()

at module top level.

Prefer factories:

def create_client(
    settings: Settings,
) -> ExternalClient:
    return ExternalClient(
        settings.token
    )

Application composition root:

def main() -> int:
    settings = load_settings()
    client = create_client(
        settings
    )
    ...

Imports should usually define behavior, not start the application.

---

24. Module state

Module-level constants:

DEFAULT_TIMEOUT = 5.0

are normal.

Mutable state:

_current_user = None

creates hidden process-global ownership.

Problems:

• tests affect one another;
• concurrent requests race;
• multiple application instances share state;
• reload behavior is confusing;
• dependency order is hidden.

Use explicit objects, context variables, or dependency injection where appropriate.

---

25. Namespace packages

Namespace packages allow portions of one import package to be distributed across multiple locations or distributions.

They may omit __init__.py under the native namespace-package mechanism.

Use cases:

• large organizational package namespace;
• independently distributed plugin families;
• split distributions.

Risks:

• packaging misconfiguration;
• import-path ambiguity;
• accidental mixed installations;
• test differences;
• complex ownership.

Do not use namespace packages for an ordinary single-project package without a clear reason.

---

26. Distribution package versus import package

Install command:

python -m pip install \
  beautifulsoup4

Import:

import bs4

The distribution name and import package name differ.

They are related but distinct concepts.

Another distribution may expose multiple import packages.

Do not assume a PyPI project name can always be imported directly with the same spelling.

This distinction matters for:

• dependency metadata;
• security review;
• troubleshooting;
• lock files;
• import errors;
• package indexes.

---

27. Plugin registration approaches

Explicit registry

PLUGINS = {
    "csv": CsvPlugin,
    "json": JsonPlugin,
}

Most understandable.

Decorator registration

@register_plugin(
    "csv"
)
class CsvPlugin:
    ...

Requires importing the module.

Subclass registration

class CsvPlugin(
    Plugin,
    name="csv",
):
    ...

Also import-dependent.

Distribution metadata discovery

Installed distributions can expose plugin entry points.

This supports third-party plugins without importing every possible module manually.

Choose the least magical mechanism that meets deployment requirements.

---

28. Import-dependent registries

A plugin module that is never imported never runs its registration decorator.

This common bug appears as:

The plugin class exists in the repository, but the registry is empty.

Solutions:

• explicit import list;
• packaging entry points;
• generated registry;
• controlled discovery step;
• application configuration.

Do not rely on filesystem presence alone.

---

29. Safe plugin discovery

A production plugin loader should define:

• trusted distributions;
• plugin group or namespace;
• duplicate-name policy;
• version compatibility;
• failure isolation;
• initialization timing;
• permissions;
• configuration schema;
• shutdown lifecycle;
• observability.

Importing a plugin executes its code.

Treat plugin installation as code installation, not data loading.

---

30. Import performance

Import time can dominate:

• CLI startup;
• serverless cold starts;
• test collection;
• command completion;
• worker startup.

Measure:

python -X importtime \
  -m application.cli

Possible improvements:

• remove unnecessary top-level imports;
• avoid importing optional heavy modules eagerly;
• simplify package __init__.py;
• delay truly rare features;
• reduce import-time computation;
• split CLI dispatch from heavy implementation.

Do not scatter local imports everywhere without measuring.

---

31. TypeScript comparison

ES modules also have:

• module evaluation;
• caching;
• import cycles;
• side effects;
• static and dynamic imports;
• package entry points.

Python differences include:

• sys.path;
• sys.modules;
• module objects;
• import hooks;
• namespace packages;
• python -m;
• distribution/import-name distinction;
• package __init__.py.

Both ecosystems benefit from side-effect-light modules and explicit composition roots.

---

32. Common mistakes

Database connection at import time

Move to application startup.

Editing sys.path

Fix installation or invocation.

Wildcard imports

Make ownership explicit.

Reload as deployment

Restart safely.

Circular import patched with many local imports

Fix architecture.

Registry without discovery

Ensure modules are imported or use metadata.

Dynamic import from untrusted input

Use an allowlist.

Heavy package initializer

Keep __init__.py focused.

Assuming distribution name equals import name

Verify metadata and documentation.

---

33. English vocabulary

Term	Meaning
module	runtime namespace loaded by the import system
package	module that can contain submodules
binding	association of a local name with an imported object
module cache	mapping of loaded modules in sys.modules
partial initialization	module visible before execution finishes
finder	component locating a module specification
loader	component creating or executing a module
namespace package	package spanning multiple locations
import side effect	behavior executed while importing
plugin discovery	locating installed or configured extensions

Useful sentences:

• “The import statement binds the module name after invoking the import machinery.”
• “The circular dependency observes a partially initialized module.”
• “Reloading does not update previously imported class objects.”
• “The registry remains empty because the plugin module was never imported.”
• “The distribution name differs from the import package name.”
• “This connection belongs in the composition root, not module scope.”

---

34. Speaking task

Explain for twelve minutes:

What happens when Python executes an import statement?

Include caching, module execution, binding, circular imports, and plugin discovery.

---

35. Writing task

Write a 500-word architecture review of a Python application that connects to databases, loads environment files, starts threads, and registers plugins during imports.

---

36. Exercises

Exercise 1

Demonstrate that module top-level code normally executes once.

Exercise 2

Create a circular import and refactor it using a lower-level shared module.

Exercise 3

Build an explicit plugin registry.

Exercise 4

Create a package runnable with python -m.

Exercise 5

Measure import time and identify a heavy dependency.

Exercise 6

Explain why importlib.reload() does not update existing instances.

---

37. Complete solutions

Solution 1

# counter_module.py
print(
    "counter_module executed"
)

VALUE = 42

import counter_module
import counter_module

assert (
    counter_module.VALUE
    == 42
)

The message normally appears once because the module object is cached in sys.modules.

Solution 2

Before:

users.py → imports orders.py
orders.py → imports users.py

Refactor shared identifiers:

domain/
├── identifiers.py
├── users.py
└── orders.py

# identifiers.py
from dataclasses import (
    dataclass,
)


@dataclass(
    frozen=True
)
class UserId:
    value: int

Both modules import from the lower-level identifier module instead of one another.

Solution 3

from typing import (
    Protocol,
)


class ExportPlugin(
    Protocol
):
    def export(
        self,
        rows: list[
            dict[str, object]
        ],
    ) -> bytes:
        ...


PLUGIN_FACTORIES: dict[
    str,
    type[ExportPlugin],
] = {
    "csv": CsvExportPlugin,
    "json": JsonExportPlugin,
}


def create_plugin(
    name: str,
) -> ExportPlugin:
    try:
        factory = (
            PLUGIN_FACTORIES[
                name
            ]
        )
    except KeyError as error:
        raise ValueError(
            f"Unknown plugin: "
            f"{name}"
        ) from error

    return factory()

Solution 4

src/
└── greeting/
    ├── __init__.py
    ├── __main__.py
    └── cli.py

# cli.py
def main() -> int:
    print("Hello")
    return 0

# __main__.py
from .cli import main


raise SystemExit(
    main()
)

Run after installation:

python -m greeting

Solution 5

python -X importtime \
  -m application.cli \
  2> import-times.txt

Inspect cumulative import time rather than only self time. Verify that delaying the dependency does not merely move unacceptable latency to the first request.

Solution 6

Reload re-executes the module and creates new class objects. Existing instances retain references to their original class object.

Directly imported names also remain bound to the old objects unless rebound.

---

38. Chapter checkpoint

You should now be able to explain:

1. module objects;
2. import machinery and binding;
3. sys.modules;
4. partially initialized modules;
5. import forms;
6. __main__;
7. packages and public APIs;
8. absolute and relative imports;
9. circular dependencies;
10. dynamic imports;
11. specifications, finders, and loaders;
12. cache invalidation;
13. reload limitations;
14. import-time side effects;
15. namespace packages;
16. distribution versus import packages;
17. plugin discovery;
18. import-time performance.