Module 06: Scaling Strategies and Batch Processing

Why this matters

Module 05 taught you to find the bottleneck. This module is a toolbox of fixes that all share one theme: do less work, less often, in bigger chunks, and get non-essential work off the critical path. These are the techniques you reach for once the profiler has pointed at a real problem — and they're the ones that matter most before you spend money scaling out (more machines), because a system riddled with per-item overhead, resource leaks, and bloated payloads just wastes those extra machines too.

Four concrete disciplines: batch processing (the N+1 fix from module 05, generalized into a way of thinking about I/O), avoiding resource leaks (unclosed connections and file handles that slowly strangle a long-running process — invisible in a script, fatal in a server), reducing network overhead (payload size, compression, fewer round-trips), and offloading non-critical work off the request path. That last one is the bridge to the rest of the curriculum: the instinct "this doesn't need to happen before I respond to the user" is exactly what track 06 (background processing) is built on. This module gives you the reasoning; track 06 gives you the queue.

Concepts

Scaling up vs scaling out — and why efficiency comes first

Two ways to handle more load:

• Scale up (vertical): a bigger machine — more CPU, RAM, faster disk. Simple, no distributed-systems complexity, but bounded (there's a biggest box) and increasingly expensive per unit.
• Scale out (horizontal): more machines/replicas behind a load balancer. Effectively unbounded, but requires your app to be stateless-per-request (or externalize state — sessions in Redis, not in-process, echoing module 03's L1 caveat) and brings coordination costs.

But the theme of this module comes before either: efficiency first. A request that does 500 database round-trips (N+1), leaks a connection each time, and ships a 4MB payload doesn't need a bigger box or more replicas — it needs to stop wasting the resources it has. Scaling hardware around inefficient code just multiplies the waste and the bill. The order of operations is: profile (module 05) → remove the inefficiency (this module) → then scale the hardware if you still need to. (Track learn/03-kubernetes module 09's HPA/VPA is the "scale the hardware" half; this module is the "stop wasting it" half.)

Batch processing: amortizing per-item overhead

Module 05 named the "N+1-shaped" bottleneck: time that scales with item count because each item pays a fixed overhead — a round-trip, a syscall, a handshake, a transaction commit. Batching amortizes that fixed cost across many items by doing the work in bulk: one request carrying N items instead of N requests carrying one each.

The overhead being amortized is the whole point. A single round-trip to Postgres is ~1ms of network+protocol overhead plus the actual query work; do it 1,000 times for 1,000 rows and you pay ~1,000ms of pure overhead on top of the work. Batch it into one statement and you pay ~1ms of overhead total. The work is the same; the overhead collapses.

  N individual calls:   app ─►│ ─► app ─►│ ─► app ─►│ ─►  ... (N times)
                         [o|w] [o|w] [o|w]   o = per-call overhead (round-trip)
                         └── N×o overhead ──┘   w = the actual work

  1 batched call:       app ─►│═══════════════│─► app
                         [o|wwwwwwwwwwwwwwwww]   ONE overhead, same total work
                         └ 1×o ┘                 overhead collapsed N→1

Where batching applies (all the same idea):

• Database writes: INSERT ... VALUES (...), (...), (...) or SQLAlchemy's bulk_insert_mappings / execute(insert(T), list_of_dicts) instead of N single-row inserts each in its own round-trip (and often its own transaction).
• Database reads: WHERE id IN (:ids) or eager loading (selectinload, track 04) instead of N single-row lookups — the direct N+1 fix.
• Cache operations: MGET/MSET/pipelines instead of N GET/SET round- trips to Redis.
• External API calls: a bulk endpoint (send 100 IDs, get 100 results) instead of 100 calls; if the API has no bulk endpoint, at least concurrency (module 07) to overlap the waits.
• Processing large datasets: process in chunks (1,000 rows at a time) rather than loading everything into memory (the leak/OOM problem below) or one row at a time (the overhead problem). Chunking is batching applied to a stream — big enough chunks to amortize overhead, small enough to bound memory.

The counter-tension: bigger batches aren't unboundedly better. A batch of 1,000 is far better than 1; a batch of 10,000,000 loads everything into memory at once (defeating the point and risking OOM), holds locks/transactions open too long, and increases the blast radius if it fails. The sweet spot is "large enough to make overhead negligible, small enough to bound memory and failure impact" — typically hundreds to low thousands, tuned by measurement (module 05).

Avoiding resource leaks: the slow strangulation

A resource leak is acquiring a finite resource — a database connection, a file handle, a socket, a lock — and never releasing it. In a short script it's harmless (the process exits and the OS reclaims everything). In a long-running server it's fatal-but-slow: each leaked resource is never reclaimed, so they accumulate over hours or days until you hit a hard limit and the process falls over — the classic "it works fine, then crashes every 3 days" bug.

The usual suspects and their limits:

• Database connections. Leak one per request (a session never closed, exactly track 04 module 07's pool-exhaustion exercise) and the connection pool drains; new requests block on pool_timeout then error. The pool made the leak slower to manifest, not impossible.
• File handles / sockets. Every OS process has a limit on open file descriptors (ulimit -n, often 1024). Leak file handles (open without close) or HTTP connections (an httpx/requests client never closed) and eventually every new open()/connection fails with "Too many open files."
• Memory (via unbounded growth). Not a classic "handle" leak, but the same shape: an ever-growing in-process cache with no eviction (module 03's oversized L1), an unbounded list you keep appending to, accumulating references — memory climbs until the OOM killer strikes.

The fix is discipline about release, and Python gives you the tools:

• Context managers (with) release deterministically at block exit, even on exception — the primary defense. with open(...) as f:, with Session() as s:, with httpx.Client() as c:.
• try/finally when a with doesn't fit — release in the finally so an exception can't skip it.
• Framework scoping — FastAPI dependencies with yield open a resource before the request and close it after, automatically, for every request including failed ones (the right place to scope a DB session — track 04 module 06).
• Bounded caches (maxsize on TTLCache/LRUCache) so an in-process cache can't grow without limit (module 03).

The mental rule: every acquire needs a guaranteed release on every path, including the exception path. In a server, "I'll close it later" or "the happy path closes it" is a leak waiting for the error path.

Reducing network overhead

Between services, and between server and client, the network is often the dominant cost — and it has three levers:

• Payload size. Don't send what the caller doesn't need. Select only the columns/fields required (over-fetching a 50-column row to use 3 is waste); paginate large collections instead of returning 100,000 rows; strip null/empty fields; use field selection / sparse responses. Smaller payloads are faster to serialize, transmit, and parse — at both ends.
• Compression. Text payloads (JSON, HTML) compress dramatically (often 5–10×). Content-Encoding: gzip/br (Brotli) trades a little CPU for a large bytes-on-the-wire reduction — almost always a win for responses over a few KB. FastAPI's GZipMiddleware does this; most CDNs/proxies can too. (Don't bother compressing tiny or already-compressed payloads — images, video — the CPU isn't repaid.)
• Number of round-trips. Each round-trip pays latency (especially across a network or to a distant service). Batching (above) cuts round-trips; so does combining several small requests into one, and avoiding chatty back-and-forth protocols. This is the same "amortize the fixed per-operation overhead" idea as batching, applied to the network specifically.

A related backend-specific overhead: serialization cost. Turning objects into JSON and back is real CPU, and for large payloads it can itself be a bottleneck (module 05's decomposition sometimes fingers serialization, not the DB) — which is another reason smaller payloads help, and why formats like MessagePack or Protobuf (track 11's gRPC) exist for hot internal paths.

Offloading non-critical work off the request path

The highest-leverage scaling move is often not making work faster but moving it out of the request entirely. Ask of every step in a request: does the user's response actually depend on this completing first? If not, it doesn't belong on the critical path.

Classic examples of work that's usually not critical to the response:

• Sending a confirmation/welcome email or push notification.
• Generating a thumbnail, transcoding a video, rendering a PDF.
• Writing an analytics/audit event.
• Warming a cache, updating a search index, syncing to a third party.
• Any slow third-party call whose result the user doesn't immediately need.

  ON the request path (slow):
  request ─► create order ─► send email ─► write analytics ─► respond
            |<-- 40ms -->|<-- 800ms -->|<-- 120ms -->|   user waits ~960ms

  OFFLOADED (fast):
  request ─► create order ─► enqueue{email, analytics} ─► respond
            |<-- 40ms -->|<-1ms->|   user waits ~41ms
                             └┄► worker: email, analytics  (later, off-path)

For these, the request should do the minimal critical work (create the order, persist the record), enqueue the rest as a background job, and return immediately. The user gets a fast response; the slow work happens asynchronously by a worker. This does two things at once: it slashes response latency (the user no longer waits for the email to send) and it decouples request throughput from the slow work's throughput (a spike in orders doesn't back up behind email sending). The write-behind counter from module 01 was a special case of exactly this: acknowledge fast, do the durable/expensive part later.

The trade you accept is eventual completion and its complications — the work now happens later (the email arrives in a few seconds, not synchronously), and you need somewhere reliable to enqueue it and a worker to process it, with retries for failures. That reliable-queue-plus-worker machinery, and the correctness issues it raises (at-least-once delivery, idempotency, ordering), is precisely track 06 — background processing and realtime. This module's job is to build the instinct: on the request path, do only what the response depends on; offload the rest. Recognizing that boundary correctly is a real skill, and it's the single most effective latency win in many systems.

Command reference

Batching a write, the wrong way then the right way:

# WRONG: N round-trips (and often N transactions) — an N+1 on the write side
for row in rows:                       # 10,000 rows
    session.add(Event(**row))
    session.commit()                   # 10,000 commits/round-trips -> seconds of pure overhead

# RIGHT: one bulk statement, chunked to bound memory
from itertools import batched          # Python 3.12+
for chunk in batched(rows, 1000):      # 1,000 at a time: overhead amortized, memory bounded
    session.execute(insert(Event), list(chunk))
session.commit()                       # one commit

Pipelining Redis to collapse round-trips:

# WRONG: 1,000 round-trips
for k, v in items.items():
    r.set(k, v)

# RIGHT: one round-trip
with r.pipeline() as pipe:
    for k, v in items.items():
        pipe.set(k, v)
    pipe.execute()                     # all 1,000 SETs sent and acked together

Leak-safe resource handling and payload compression in FastAPI:

from fastapi import FastAPI, Depends
from fastapi.middleware.gzip import GZipMiddleware
from sqlalchemy.orm import Session

app = FastAPI()
app.add_middleware(GZipMiddleware, minimum_size=1000)   # gzip responses > 1KB

def get_session():                      # per-request acquire/release; closes on every path
    with Session(engine) as s:          # 'with' guarantees release even on exception
        yield s

@app.post("/orders")
def create_order(order: OrderIn, s: Session = Depends(get_session)):
    saved = persist_order(s, order)     # the ONLY critical work
    enqueue_confirmation_email(saved.id)  # offloaded: user doesn't wait for the email
    return {"id": saved.id}             # fast response; email sends in the background

Hands-on exercises

Postgres + Redis running; FastAPI app from earlier modules. pip install redis sqlalchemy httpx. A load tool for the throughput exercises.

1. Batch a write and measure the overhead collapse

Insert 10,000 rows two ways: (a) one add+commit per row, (b) chunked execute(insert(...)) of 1,000. Time both.

Expected: the per-row version takes seconds (thousands of round-trips/commits of pure overhead); the batched version is an order of magnitude faster with identical results. You've felt overhead amortization directly — the same work, the overhead collapsed.

2. Find the batch-size sweet spot

Repeat exercise 1's batched insert with chunk sizes 1, 10, 100, 1,000, 10,000, and (if you dare) all 10,000 at once. Time each and watch memory (docker stats or tracemalloc).

Expected: time drops steeply from 1→100→1,000 (overhead amortizing), then flattens — and the very largest chunks stop helping while memory use climbs. You've located the sweet spot empirically (module 05's "tune by measurement") and seen why "bigger is always better" is false for batches.

3. Pipeline Redis

Set 5,000 keys with a loop of r.set, then with a pipeline. Time both.

Expected: the pipelined version is dramatically faster — it collapsed 5,000 network round-trips into one. Same lesson as exercise 1, on the cache layer.

4. Reproduce a connection-handle leak in a server

Write a FastAPI endpoint that opens a Session (or raw connection) without closing it — no with, no dependency yield. Hit it repeatedly under load (hey -n 100) with a small pool (pool_size=5, max_overflow=0, pool_timeout=3).

Expected: after ~5 requests, subsequent requests block for pool_timeout then error with QueuePool limit ... timed out — the pool drained by leaked connections (track 04 module 07's exercise, now framed as a leak). Confirm the process doesn't recover: the leak is permanent until restart.

5. Fix the leak with a scoped dependency

Convert exercise 4's endpoint to use the get_session dependency with with ... yield. Re-run the load, including forcing some requests to raise an exception mid-handler.

Expected: connections are returned to the pool on every path (success and exception), the pool never drains, and load runs indefinitely without timeouts. You've seen with/yield guarantee release even on the error path — the core leak defense.

6. Measure a payload/compression win

Return a large JSON response (say 10,000 rows). Measure bytes-on-the-wire and latency with and without GZipMiddleware, and then with field-selection (returning only the 3 fields the client needs instead of all 50).

Expected: gzip cuts transfer size several-fold (a little CPU for a lot of bytes); field-selection cuts it again and reduces serialization CPU. Both improve end-to-end latency for a large payload — often more than you'd guess (serialization can be a real chunk of the time, per module 05).

7. Offload non-critical work off the request path

Build a POST /signup that (a) synchronously creates the user and sleeps 800ms to simulate sending a welcome email, then (b) a version that creates the user and enqueues the email work (for now, a threading/asyncio background task or a simple in-memory queue), returning immediately. Measure response latency of each.

Expected: version (a) responds in ~810ms (the user waits for the email); version (b) responds in ~10ms (the email sends after the response). Same user-visible outcome (account created, email arrives), radically different latency — and under load, (b)'s throughput no longer bottlenecks on email speed. Note what you gave up: the email is now eventual, and a robust version needs a real queue with retries — which is track 06.

8. Diagnose and fix: the service that crashes every few days

A report-generation service runs fine, then reliably crashes with "Too many open files" after 3–4 days of uptime, and sometimes "QueuePool limit ... timed out" before that. The hot code path, per report, opens a template file and a DB session, and calls a third-party API with a fresh httpx.Client():

def generate_report(report_id: int):
    f = open(f"templates/{template}.html")      # opened...
    conn = engine.connect()                       # opened...
    client = httpx.Client()                       # opened...
    data = conn.execute(query).fetchall()
    extra = client.get(THIRD_PARTY_URL).json()
    return render(f.read(), data, extra)          # ...and none are ever closed

Diagnose and fix, and explain why it took days and why dev/tests never caught it.

Independent challenge

No code given. Take a data-heavy operation in your app (a bulk import, a report, a fan-out to a third party) and make it efficient without adding hardware. First, find and eliminate an N+1-shaped cost by batching — either DB writes/reads or external calls — and measure the overhead collapse, then find the batch-size sweet spot empirically rather than guessing. Second, audit the code path for resource leaks (connections, file handles, HTTP clients) and convert every acquire to a guaranteed-release form (with/yield/try-finally), proving the fix by exhausting a small pool under load before and not after. Third, reduce network overhead on your largest response with compression and field-selection, measuring bytes-on-the-wire before and after. Fourth, identify one piece of work currently on the request path that the response does not depend on, offload it (for now to a background task), and show the response-latency drop — then write two sentences on what you'd need from track 06 to make that offload reliable (what happens if the worker crashes mid-job). Keep every before/after number, per module 05's discipline.

Common mistakes & troubleshooting

• Scaling hardware around inefficient code. More machines multiply the waste and the bill. Profile and remove the inefficiency (batching, leaks, payloads) before scaling up/out.
• Per-item round-trips (N+1 on reads and writes). One commit/query/call per item pays fixed overhead N times. Batch into bulk statements / IN queries / MGET/pipelines / bulk endpoints.
• Unbounded batch sizes. A batch of everything loads it all into memory (OOM risk), holds locks/transactions too long, and widens failure blast radius. Chunk to hundreds/low-thousands, tuned by measurement.
• Resource leaks in long-running servers. An acquire without a guaranteed release accumulates until a hard limit (fd limit, pool) crashes the process — slowly, so dev misses it. Use with/yield/try-finally on every path.
• A new HTTP client per call. Constructing (and leaking) an httpx/requests client per request wastes connections and leaks sockets. Reuse a persistent client.
• Over-fetching and uncompressed large payloads. Returning 50 columns to use 3, un-paginated collections, and uncompressed JSON waste bandwidth, serialization CPU, and latency at both ends. Select fields, paginate, gzip.
• Compressing tiny or already-compressed payloads. gzip on a 200-byte response or a JPEG spends CPU for no benefit. Set a minimum_size and skip binary/compressed content.
• Doing non-critical work on the request path. Sending email, transcoding, indexing synchronously makes the user wait and couples request throughput to slow work. Offload it (→ track 06) — but know you've traded synchronous completion for eventual completion and now need a reliable queue and retries.

Checkpoint quiz

Write down your answer to each question before expanding it — checking without attempting first is the single easiest way to fool yourself into thinking you've learned this.

1. What fixed cost does batching amortize, and why is a batch of 1,000 far better than 1 but a batch of 10,000,000 potentially worse than 1,000?
2. Why is a resource leak harmless in a short script but fatal in a long-running server, and what's the general rule that prevents it?
3. Name the three levers for reducing network overhead, and give the one-line tradeoff of using compression.
4. What single question decides whether a piece of work belongs on the request path, and what two benefits does offloading non-critical work provide at once?
5. Why should efficiency (batching, leak-fixing, payload reduction) come before scaling hardware up or out?
6. Explain why the exercise-8 "crashes every few days" service took days to fail and why tests never caught it — in terms of resource lifetime.

Cumulative review

Closed-book. These mix modules 00–06 (roughly two-thirds of the track). Write full answers before expanding; if one stumps you, redo that module's exercises.

1. A GET /feed endpoint is slow. Walk the module 05 procedure to find the bottleneck, then, supposing the decomposition shows 70% of the time in "load each post's author and comment count in a loop" and 25% in serializing a 50-field payload, prescribe the specific fixes from modules 04/06 for each and predict the approximate speedup ceiling for fixing only the serialization (invoke the relevant law by name).
2. You cache /feed's result in Redis with a 30s TTL. Under load you see the DB spike every 30s. Name the phenomenon (module 04), give two defenses, and explain why an L1 in front (module 03) would not be an appropriate addition here if the feed is per-user.
3. Contrast three ways data can fail to be where you expect it, one from each of modules 01, 02, and 06: a write-behind loss, a volatile-lru-with-no-TTL rejection, and a connection-pool exhaustion from a leak — what triggers each and what you'd see.
4. For a value read 5,000×/min, recomputed by a 300ms aggregate, changing hourly: choose a caching strategy (module 01), eviction policy + TTL (module 02), whether to add an L1 (module 03), and a stampede defense (module 04) — a complete design, each choice justified in one clause.
5. A signup endpoint takes 900ms: 40ms to create the user, 60ms for a query that turns out to be an N+1, and 800ms sending a welcome email synchronously. Give the module-06 fix for each of the two non-user-creation costs, state the resulting latency, and name what track-06 machinery the email fix will eventually need to be reliable.
6. Explain the through-line connecting module 01's write-behind, module 04's stale-while-revalidate, and module 06's request-path offloading — what single idea do all three share?

Further reading & sources

• SQLAlchemy: ORM bulk INSERT - batching writes into one round-trip instead of N single-row commits.
• Redis: pipelining - collapsing many Redis round-trips into one to amortize network overhead.
• FastAPI: dependencies with yield - per-request acquire/release scoping that guarantees resource cleanup on every path.
• FastAPI: GZip middleware - compressing responses to cut bytes-on-the-wire.
• Python: itertools.batched - chunking a stream into bounded batches (3.12+).
• FastAPI: background tasks - the in-process way to offload non-critical work off the request path (and its durability limits).

Next

07-concurrency-and-parallelism — offloading work and overlapping I/O both raise a question this module kept deferring: how do you actually do more than one thing at once? Next is the real difference between concurrency and parallelism, why async/await helps I/O-bound work and multiprocessing helps CPU-bound work, what Python's GIL does in practice, and the race conditions concurrency introduces (which you glimpsed in the caching races of modules 01 and 04).