Day 45 of 50 Days of Python: Code Optimisation and Profiling Tools

Part of Week 7: Python in Production

Welcome to Day 45! Where we’ll be touching on code optimisation practices and getting familiar with some profiling tools. Optimisation means delivering the same results in less time, memory, and compute. Which results in lower cloud bills, happier users, and room for bigger data sets. The prime directive is measure before you change, and profilers give you the hard numbers that expose true hotspots so you don’t waste effort chasing phantom slow‑downs.

---

What We’ll Cover

• The optimisation cycle: measure → identify hotspot → refactor → verify.

• Using cProfile & snakeviz for function‑level timing.

• Line‑level guidance with line_profiler; memory tracking with memory_profiler.

• Common tricks: algorithmic complexity, vectorisation, memoisation, parallelism.

• Quick wins for Pandas & NumPy heavy workloads.

---

Key Concepts

• Hotspot – region of code consuming disproportionate resources.

• Profiling – collecting performance metrics while the program runs.

• Sampling vs Instrumentation – py‑spy (sampling, low overhead) vs cProfile (instrumentation, high detail).

• Big‑O complexity – algorithm growth rate; faster code often begins with a better algorithm.

• Vectorisation – replacing Python loops with NumPy/Pandas ops executed in C.

• Memory churn – frequent allocations that trigger GC; mitigate with pre‑allocation or in‑place ops.

• Concurrency – concurrent.futures, multiprocessing, or asyncio to utilise I/O wait or multiple cores.

---

Hands On: Find and Fix a Hot Loop

A Very Silly Example

# src/stats.py
import math

def harmonic_mean(nums):
    inv_sum = 0
    for n in nums:  # HOT LOOP!
        inv_sum += 1 / n
    return len(nums) / inv_sum

Profile with cProfile/snakevis

python -m cProfile -o prof.out -m src.stats_cli large_numbers.csv
snakeviz prof.out  # opens interactive flame graph in browser

The harmonic_mean should show as 96%.

Optimise with NumPy

import numpy as np

def harmonic_mean_np(nums):
    arr = np.asarray(nums, dtype=float)
    return len(arr) / np.sum(1 / arr)

python -m timeit -s "from stats_fast import harmonic_mean_np as f; import numpy as np; nums=np.random.rand(1_000_000)+1e-3" "f(nums)"

Line-Level and Memory Profiling

# add decorator only in dev
from line_profiler import profile
from memory_profiler import profile as mprof

@profile
@mprof
def harmonic_mean_np(nums):
    arr = np.asarray(nums, dtype=float)
    return len(arr) / np.sum(1 / arr)

kernprof -l -v src/stats_fast.py
mprof run src/stats_fast.py
mprof plot

Parallelise (CPU Bound)

from concurrent.futures import ProcessPoolExecutor

def batch_hmean(batches):
    with ProcessPoolExecutor() as ex:
        return list(ex.map(harmonic_mean_np, batches))

---

TL;DR

• Profile first as guessing is a trap.

• cProfile + snakeviz for quick overview; line_profiler & memory_profiler for surgical detail.

• Vectorise, cache, and parallelise hotspots; change algorithms when possible.

• Verify gains with the same profiler to avoid hidden regressions.

---

Next Up: Day 46 – Best Practices for Python Packaging and Dependency Management.

We’ve boosted performance capabilities, so now let’s make sure your code ships safely and runs the same on every machine.

See you there and, as always… Happy coding!