Chapter 27 — Project Structure, pyproject.toml, Environments, and Dependencies

1. Opening problem

A repository contains:

application.py
utils.py
utils_new.py
requirements.txt
requirements-final.txt
requirements-working.txt
setup.py
.env
venv/
tests.py

Developers run it differently:

python application.py

PYTHONPATH=. python application.py

pip install -r requirements-working.txt

sudo pip install .

The application works on one laptop and fails in CI.

This is not merely a tooling problem. Project structure and dependency policy are part of architecture.

A professional project should answer:

• What is the import package?
• What is the distribution package?
• Which Python versions are supported?
• How is the project built?
• Which dependencies are required at runtime?
• Which are optional?
• Which are only for development?
• How is an environment reproduced?
• What artifacts are deployed?
• Where do tests import code from?
• How are command-line entry points exposed?

---

2. Import package and distribution package

Import:

import company_billing

Distribution metadata:

[project]
name = "company-billing-service"

These names may differ.

The distribution package is what installers and indexes identify.

The import package is what Python imports.

Use names intentionally and document them.

Avoid creating a distribution name that could be confused with an unrelated import package.

---

3. Recommended project shape

company-billing/
├── pyproject.toml
├── README.md
├── LICENSE
├── src/
│   └── company_billing/
│       ├── __init__.py
│       ├── __main__.py
│       ├── application/
│       ├── domain/
│       ├── infrastructure/
│       └── cli.py
└── tests/
    ├── unit/
    ├── integration/
    └── acceptance/

This structure separates:

• repository root;
• importable source;
• tests;
• project metadata;
• generated artifacts.

The exact internal architecture should match the application, not a universal template.

---

4. src layout

The src layout places importable code under src/.

Benefits:

• project normally must be installed before import;
• tests are less likely to import an accidental working-tree copy;
• packaging omissions are easier to detect;
• repository support files do not appear as importable top-level modules;
• installed behavior more closely matches production.

Cost:

• editable installation is normally needed for development;
• new developers need to understand installation;
• direct execution of files under src is discouraged.

For distributable libraries and serious applications, the safety is often worthwhile.

---

5. Flat layout

project/
├── pyproject.toml
├── package/
└── tests/

Benefits:

• simpler first-run experience;
• package can be imported from repository root without installation.

Risks:

• tests may accidentally use local code rather than installed artifact;
• repository files can influence import behavior;
• packaging configuration errors may remain hidden;
• CI and local execution may differ.

A flat layout is not inherently wrong. Choose consciously.

---

6. pyproject.toml

pyproject.toml is the central configuration file for modern Python packaging and can also hold tool configuration.

Major areas:

[build-system]

[project]

[tool.some-tool]

It can describe:

• build backend;
• project metadata;
• Python requirement;
• dependencies;
• optional dependencies;
• scripts;
• package data through backend configuration;
• tool settings.

Do not copy a configuration without understanding which keys belong to the packaging standard and which belong to a particular tool.

---

7. Build system

Example using setuptools as one possible backend:

[build-system]
requires = [
    "setuptools>=77",
]
build-backend = "setuptools.build_meta"

The build frontend uses this information to create an isolated build environment and invoke the backend.

Other build backends exist.

The packaging ecosystem intentionally supports multiple backends.

Choose based on:

• pure Python versus extensions;
• project workflow;
• metadata support;
• editable installs;
• team experience;
• release automation;
• ecosystem compatibility.

Do not assume setup.py commands are the modern frontend interface.

---

8. Project metadata

[project]
name = "company-billing-service"
version = "1.2.0"
description = "Billing service"
readme = "README.md"
requires-python = ">=3.12"
dependencies = [
    "httpx>=0.27,<1",
]

Important metadata includes:

• name;
• version;
• description;
• readme;
• Python requirement;
• license information;
• authors or maintainers;
• classifiers where useful;
• dependencies;
• entry points.

Metadata is consumed by build tools, installers, indexes, and users.

---

9. Static and dynamic metadata

Some metadata can be declared dynamic when a build backend supplies it.

Example:

[project]
dynamic = [
    "version",
]

Use dynamic metadata only when the backend workflow justifies it.

Static metadata is easier for tools to inspect without building the project.

Version-source strategy should be documented.

---

10. Runtime dependencies

[project]
dependencies = [
    "httpx>=0.27,<1",
    "pydantic>=2.8,<3",
]

These should be dependencies required for ordinary installed operation.

Do not place test runners, linters, or documentation generators in runtime dependencies unless the application genuinely imports them at runtime.

For libraries, avoid overly strict upper bounds without evidence. For applications, reproducibility may be handled by an additional locking or deployment layer.

---

11. Dependency specifiers

Examples:

requests>=2.32

requests>=2.32,<3

requests==2.32.4

Each expresses a different policy.

A library often declares compatible ranges so it can coexist with other packages.

An application deployment often needs a resolved set for reproducibility.

Do not confuse declared compatibility with a complete environment snapshot.

---

12. Environment markers

dependencies = [
    "colorama>=0.4; "
    "platform_system == 'Windows'",
]

Markers allow conditional dependencies based on environment properties.

Use cases:

• operating system;
• Python version;
• implementation;
• platform.

Test every supported marker branch in CI where feasible.

Conditional dependencies can hide untested platform failures.

---

13. Optional dependencies

[project.optional-dependencies]
postgres = [
    "psycopg[binary]>=3.2,<4",
]

aws = [
    "boto3>=1.35,<2",
]

Install:

python -m pip install \
  "company-billing-service[postgres]"

Optional dependencies describe installable feature sets exposed to users.

Do not use extras as a universal replacement for development dependency management.

---

14. Dependency groups

Modern packaging specifications support dependency groups in pyproject.toml.

[dependency-groups]
test = [
    "pytest>=9,<10",
    "pytest-cov>=6,<7",
]

lint = [
    "ruff>=0.12,<1",
]

typing = [
    "mypy>=1.16,<2",
]

Dependency groups are intended for internal development workflows and are not the same as published project optional dependencies.

Tool support and installation commands depend on the selected environment or project manager.

Document the project's chosen command.

---

15. Development dependencies

A useful separation:

• runtime;
• test;
• lint;
• type checking;
• documentation;
• release;
• local infrastructure.

Benefits:

• smaller production environment;
• faster installation;
• clearer ownership;
• reduced attack surface;
• easier upgrades;
• targeted CI jobs.

Avoid one enormous development group when jobs need different tools.

---

16. Virtual environments

Create:

python -m venv .venv

Activate:

source .venv/bin/activate

A virtual environment isolates installed Python distributions for one project.

It does not isolate:

• operating-system libraries;
• Docker services;
• environment variables;
• external databases;
• globally executed tools outside the environment;
• the Python interpreter version used to create it.

Record and enforce the supported Python version separately.

---

17. Use python -m pip

python -m pip install \
  package-name

This makes the targeted interpreter explicit.

Plain:

pip install package-name

may invoke a different pip executable than expected.

In CI, also print:

python --version
python -m pip --version

This simplifies debugging environment mismatches.

---

18. Editable installation

python -m pip install \
  --editable .

An editable install makes the project importable while source changes are reflected without rebuilding a normal wheel for every edit.

Important:

• editable behavior is backend-defined through modern standards;
• it may not exactly match a regular installation;
• package-data and generated-artifact behavior can differ;
• final CI should test a built artifact too.

Use editable installs for development convenience, not as the only release validation.

---

19. Build artifacts

Install build frontend:

python -m pip install build

Build:

python -m build

Common outputs:

dist/
├── project-1.2.0.tar.gz
└── project-1.2.0-py3-none-any.whl

A wheel is an installable built distribution.

An sdist is a source distribution used to build an artifact.

Test both when publishing a reusable project.

For an internal application, deployment may use only a wheel or container, but the artifact policy should be explicit.

---

20. Test the wheel

A strong release check:

1. build the wheel;
2. create a clean environment;
3. install only the wheel and declared dependencies;
4. run smoke or acceptance tests;
5. verify entry points and package data.

This catches:

• missing modules;
• missing templates;
• incorrect package discovery;
• hidden development dependencies;
• reliance on repository files;
• import-path accidents.

Tests that run only against the source tree cannot detect every packaging failure.

---

21. Command-line entry points

[project.scripts]
billing = (
    "company_billing.cli:main"
)

After installation:

billing

Entry function:

def main() -> int:
    ...
    return 0

The generated script invokes the target.

Keep CLI parsing separate from core application logic.

Also support:

python -m company_billing

when useful through __main__.py.

---

22. Configuration files are not package data automatically

Examples:

• templates;
• SQL files;
• default configuration;
• static assets;
• schema files.

Verify that the build backend includes required files.

Use resource APIs rather than paths relative to the current working directory.

from importlib.resources import (
    files,
)


template = (
    files(
        "company_billing"
    )
    .joinpath(
        "templates/invoice.html"
    )
    .read_text(
        encoding="utf-8"
    )
)

Installed packages may not live in a normal source-tree path.

---

23. Application versus library dependency policy

Library

Usually:

• declares compatible ranges;
• avoids forcing exact transitive versions;
• supports several Python versions;
• minimizes dependencies;
• tests lower and upper supported ranges where practical.

Application

Usually:

• controls deployment environment;
• resolves a reproducible set;
• patches security issues rapidly;
• may use lock files or constraints;
• can choose one Python version;
• owns full integration testing.

Do not apply application pinning strategy blindly to a reusable library.

---

24. Lock files

A lock file records a resolved dependency graph for a particular tool and potentially one or more environments.

Benefits:

• repeatable installation;
• reviewable upgrades;
• hash or artifact recording depending on tool;
• transitive visibility.

Questions:

• Is the lock cross-platform?
• Does it include multiple Python versions?
• Are optional groups represented?
• Is it committed?
• How is it refreshed?
• How are security updates handled?
• Does production install directly from it?
• Is the file standardized or tool-specific?

Document the answers.

---

25. Constraints files

A constraints file limits versions without necessarily declaring top-level requirements.

Example:

urllib3==2.5.0
certifi==2026.7.1

Install:

python -m pip install \
  -c constraints.txt \
  .

Constraints are useful for centralized version control across several installations.

They do not replace project dependency metadata.

The project still declares what it needs.

---

26. Requirements files

A requirements file is an input to pip installation.

It may contain:

• project requirements;
• pins;
• indexes;
• options;
• included files;
• constraints references.

It is a pip-specific workflow file, not the same as standardized project metadata.

A project can legitimately use both:

• pyproject.toml for package metadata;
• requirements or constraints for a deployment workflow.

Avoid maintaining conflicting sources manually.

---

27. Reproducibility levels

Declared compatibility

httpx>=0.27,<1

Resolved versions

A lock or generated requirement set.

Artifact identity

Exact wheel files and hashes.

Runtime environment

Python build, OS libraries, environment variables, database versions, external services.

A dependency lock alone does not reproduce the complete production system.

Containers also do not guarantee reproducibility when base tags float or external services differ.

Define the required level.

---

28. Dependency review

Before adding a dependency, evaluate:

• functionality and alternatives;
• maintenance activity;
• release cadence;
• license;
• security history;
• transitive dependencies;
• binary wheels;
• platform support;
• Python support;
• free-threaded support where relevant;
• typing quality;
• import cost;
• package size;
• operational behavior;
• exit strategy.

A one-line install can create a long-term architectural commitment.

---

29. Dependency upgrades

Use a controlled process:

1. update one meaningful group;
2. inspect changelogs and migration notes;
3. regenerate resolution;
4. review transitive changes;
5. run static checks and tests;
6. build artifacts;
7. deploy gradually;
8. monitor;
9. record rollback strategy.

Avoid indefinite freezing. Old dependencies accumulate security and compatibility risk.

---

30. Private indexes and supply-chain safety

Organizations may use:

• internal indexes;
• mirrors;
• artifact repositories;
• allowlists;
• signed release workflows;
• hash verification;
• dependency scanning.

Risks include:

• dependency confusion;
• typosquatting;
• compromised maintainer accounts;
• malicious build steps;
• untrusted indexes;
• floating artifacts.

Do not embed credentials in pyproject.toml, source files, or committed URLs.

Use secure credential mechanisms provided by CI and package tools.

---

31. CI matrix

A library may test:

Python 3.12
Python 3.13
Python 3.14

An application may test only the deployed version plus an upgrade candidate.

Matrix dimensions can include:

• operating system;
• database;
• dependency range;
• free-threaded mode;
• optional extras;
• architecture.

Every dimension multiplies cost.

Test combinations that represent supported contracts and material risks.

---

32. Monorepo considerations

A Python monorepo can contain several distributions:

repo/
├── services/
│   ├── billing/
│   └── notifications/
├── libraries/
│   ├── domain-types/
│   └── observability/
└── tooling/

Questions:

• independent versions or one version?
• local editable dependencies?
• build isolation?
• release order?
• dependency cycles?
• shared tooling?
• test ownership?
• artifact boundaries?

Do not turn arbitrary folders into packages. Distribution boundaries should match ownership and release needs.

---

33. TypeScript comparison

Comparable concepts:

TypeScript/Node.js	Python
package.json	pyproject.toml project metadata and tool tables
npm package name	distribution package name
import path	import package/module name
node_modules	environment site-packages
lock file	tool-specific Python lock file
scripts/bin	[project.scripts]
workspaces	Python monorepo tooling and multiple distributions
devDependencies	dependency groups or tool workflow
optionalDependencies	optional project dependencies, with different semantics

The ecosystems differ, so direct one-to-one translation can mislead.

---

34. Common mistakes

Committing .venv

Recreate it.

Installing with sudo pip

Use isolated environments or system package policies.

One requirements file for every purpose

Separate concerns.

Exact runtime pins in a reusable library

Declare compatibility instead.

Only testing editable installation

Test the wheel.

Current working directory for package resources

Use resource APIs.

setup.py command workflows

Use modern build and install frontends.

Dependency groups confused with extras

They serve different audiences.

Distribution name assumed to be import name

Document both.

Lock file treated as complete infrastructure reproduction

Record runtime environment too.

---

35. English vocabulary

Term	Meaning
distribution package	installable project identified by package metadata
import package	module namespace imported by Python
build backend	tool implementing build hooks
build frontend	tool invoking standardized build hooks
wheel	built installable distribution
source distribution	source archive used to build a package
editable install	development installation linked to source
dependency group	named development dependency set
constraint	version limit applied during resolution
lock file	recorded dependency resolution

Useful sentences:

• “The distribution name and import package name are intentionally different.”
• “The src layout prevents accidental source-tree imports.”
• “The wheel test detects files omitted from the distribution.”
• “The library declares compatibility, while the application records a resolution.”
• “Dependency groups are not published optional features.”
• “The lock file does not reproduce operating-system dependencies.”

---

36. Speaking task

Explain for fifteen minutes:

How should an experienced TypeScript developer structure and package a professional Python project?

Include src, pyproject.toml, environments, dependencies, wheels, and reproducibility.

---

37. Writing task

Write a 600-word migration plan for a Python repository using setup.py, three conflicting requirements files, global pip installation, and direct source-tree imports.

---

38. Exercises

Exercise 1

Create a src-layout project with a CLI entry point.

Exercise 2

Write a pyproject.toml containing runtime dependencies, one optional feature, and development dependency groups.

Exercise 3

Build a wheel and test it in a clean virtual environment.

Exercise 4

Design a dependency policy for a reusable library.

Exercise 5

Design a different policy for a deployed API service.

Exercise 6

Explain the difference among a lock file, constraints file, and project metadata.

---

39. Complete solutions

Solution 1

greeting-project/
├── pyproject.toml
├── README.md
├── src/
│   └── greeting/
│       ├── __init__.py
│       ├── __main__.py
│       └── cli.py
└── tests/
    └── test_cli.py

# src/greeting/cli.py
def main() -> int:
    print("Hello")
    return 0

# src/greeting/__main__.py
from .cli import main


raise SystemExit(
    main()
)

Solution 2

[build-system]
requires = [
    "setuptools>=77",
]
build-backend = "setuptools.build_meta"

[project]
name = "greeting-service"
version = "0.1.0"
requires-python = ">=3.12"
dependencies = [
    "httpx>=0.27,<1",
]

[project.optional-dependencies]
postgres = [
    "psycopg[binary]>=3.2,<4",
]

[project.scripts]
greeting = "greeting.cli:main"

[dependency-groups]
test = [
    "pytest>=9,<10",
]

quality = [
    "mypy>=1.16,<2",
    "ruff>=0.12,<1",
]

Version selections are examples and should be reviewed for the project's current compatibility policy.

Solution 3

python -m build
python -m venv \
  .venv-wheel-test
source \
  .venv-wheel-test/bin/activate
python -m pip install \
  dist/greeting_service-0.1.0-py3-none-any.whl
python -m greeting
greeting

Run smoke tests without the repository root influencing imports when possible.

Solution 4

Library policy:

• declare broad tested compatibility;
• minimize runtime dependencies;
• avoid pinning transitive packages;
• support documented Python versions;
• test optional extras;
• build wheel and sdist;
• test lower and recent dependency ranges where valuable.

Solution 5

Application policy:

• deploy one controlled Python version;
• resolve dependencies reproducibly;
• review transitive changes;
• build one immutable artifact;
• run integration and acceptance tests;
• scan dependencies;
• roll out gradually;
• maintain rollback artifacts.

Solution 6

Project metadata declares direct dependency compatibility.

A lock file records a tool-resolved dependency graph.

A constraints file limits versions during installation but does not itself declare why the project requires packages.

They can coexist when responsibilities are clear.

---

40. Chapter checkpoint

You should now be able to explain:

1. import versus distribution package;
2. professional project layouts;
3. src layout;
4. pyproject.toml;
5. build system and project metadata;
6. runtime and optional dependencies;
7. dependency groups;
8. virtual environments;
9. editable installs;
10. wheels and sdists;
11. artifact testing;
12. package resources;
13. library versus application policies;
14. locks, requirements, and constraints;
15. dependency review;
16. CI matrices;
17. monorepo boundaries.