Day 46 of 50 Days of Python: Best Practices for Packaging and Dependency Management.
Part of Week 7: Python in Production
Proper packaging and dependency management turn one‑off scripts into reusable, installable libraries and solid services. Today, we’ll go through the best practices to implement a compact overview.
What We’ll Cover
Why pyproject.toml is the modern single‑source of truth (PEP 518/621).
Difference between sdist and wheel artifacts and why wheels rule.
Pinning vs range‑specifying dependencies; keeping security & reproducibility in balance.
Tool options: pip + venv, pip‑tools, Poetry, Hatch, Conda/Mamba.
Publishing to PyPI and setting up a private package index (e.g., Artifactory).
Key Concepts
Virtual Environment – Isolated Python interpreter + site‑packages; prevents “dependency hell”.
Build Backend – Tool that converts source → wheel (e.g., setuptools, hatchling, poetry-core).
Semantic Versioning – MAJOR.MINOR.PATCH; breaking changes bump major.
Version Specifier – requests>=2.32,<3.0: compatible upgrades within a safe range.
Lock File – Exact resolved versions (poetry.lock, requirements.txt generated by pip‑tools) for reproducible installs.
Wheel (.whl) – Built, platform‑tagged zip; installs fast and skips compilation.
Constraints File – Overrides transient deps without editing upstream requirements.
Private Index – Internal PyPI mirror for proprietary wheels; keeps supply chain under control.
Hands On: Mimimal pyproject.toml
[project]
name = "calihouse-utils"
version = "0.1.0"
description = "Helpers for the California housing project"
authors = [{ name = "You", email = "you@example.com" }]
readme = "README.md"
requires-python = ">=3.9"
dependencies = [
"pandas>=2.2,<3.0",
"numpy>=2.0,<3.0",
]
[build-system]
requires = ["setuptools>=69", "wheel"]
build-backend = "setuptools.build_meta"python -m build --wheel #To build the wheelWheel ends up in dist/ ready to publish.
Lock Dependencies with pip‑tools
Create requirements.in with unpinned top‑level deps.
Run pip-compile --generate-hashes → deterministic requirements.txt.
CI/CD installs with pip install -r requirements.txt for byte‑for‑byte reproducibility.
Tips & Tricks
Editable installs (pip install -e .) for live reload during development.
PEP 668 site‑editor support: declare venv requirement via pyproject.toml [project.scripts].
Multi‑backend projects: compile C extensions with meson-python or maturin.
Dependency scanning: use pip-audit or GitHub Dependabot for CVEs.
Docker layer caching: copy and install requirements.txt before source to speed builds.
TL;DR
Isolate with virtualenv/Conda; never pollute system Python.
Describe project metadata + deps once in pyproject.toml.
Build wheels, lock exact versions, scan for CVEs, publish to PyPI or private index.
Some Further Info to Check Out
PyPA – Packaging Python Projects guide
PEP 621: Project metadata in pyproject.toml
Poetry & Hatch documentation
Next Up: Day 47 - Introduction to Infrastructure as Code with Python (Terraform/Bicep).
It’s exactly what it says on the tin. Definitely tune in for this one… Happy Coding!
