10 - Distributed Systems Patterns
This track is about what changes the moment your system stops being a single process talking to a single database and becomes many processes, services, and datastores that must cooperate over an unreliable network. Once state lives in more than one place — a primary and a replica, a cache and a database, three services each owning their own store — failure and duplication stop being edge cases and become the default operating condition: requests get retried, messages get redelivered, nodes crash mid-operation, and the network partitions when you least expect it. Each module here is a specific, battle-tested pattern for staying correct under exactly those conditions — consistency choices, idempotency, locking, transactions, sagas, CQRS, event sourcing, and consensus — and, just as important, the judgment about when each pattern earns its complexity and when it's overkill.
How this track works
- It assumes you've finished track 04 (Databases and Data Layer) — you're
comfortable with transactions, ACID, isolation levels, lost updates,
SELECT ... FOR UPDATE, replication, and primary/replica setups, all of which this track builds directly on — and track 06 (Background Processing and Realtime), whose at-least-once delivery, retries, webhooks, and the transactional outbox recur in almost every module here (idempotency, sagas, and CQRS projections all lean on them). - Every module builds on the ones before it, and they tell one continuous story: each pattern is a response to a failure the previous module exposed. The reliability disciplines established early — per-operation consistency, idempotent operations, fencing tokens — recur throughout and converge in the capstone. Go in order.
- Each standard module (01–08) has the same shape: why it matters, concepts, a command reference with real Python/FastAPI code, progressive hands-on exercises (including a "diagnose and fix" scenario each), an independent challenge with no code, common mistakes, and a checkpoint quiz. Modules 04 and 07 also carry a closed-book cumulative review that stitches the preceding modules together.
- All exercises run locally against Postgres, Redis, and (for the coordination module) etcd in Docker, plus (for the workflow-orchestration module) a local Temporal dev server — no cloud account required.
Modules
First, consistency: this track builds on CAP, PACELC and per-operation consistency, taught in System Design Track 01: CAP Theorem, PACELC Theorem and Consistency Models. Later modules cite them as SD 13, SD 14 and SD 15.
| # | Module | What you'll be able to do | Time |
|---|---|---|---|
| 01 | Idempotency in practice | Make non-idempotent operations safe to retry with idempotency keys, natural idempotency, and idempotent consumers | 75-100 min |
| 02 | Distributed locking | Build a correct Redis lock, understand lock expiry and fencing tokens, and tell efficiency locks from correctness locks | 75-100 min |
| 03 | Distributed transactions and two-phase commit | Understand 2PC, why the coordinator blocks and doesn't scale, and when to avoid distributed transactions entirely | 75-100 min |
| 04 | The saga pattern | Coordinate multi-service transactions with choreography/orchestration sagas and compensating transactions | 90-120 min |
| 05 | CQRS | Separate read and write models, judge when the split earns its complexity, and handle the eventual consistency it introduces | 75-100 min |
| 06 | Event sourcing | Store state as an immutable event log, rebuild state by replaying, use snapshots, and pair event sourcing with CQRS | 90-120 min |
| 07 | Consensus and coordination | Reason about consensus (Raft), quorums, and leader election, and use etcd/ZooKeeper/Postgres instead of building your own | 90-120 min |
| 08 | Durable workflow orchestration | Use a durable execution engine (Temporal) for crash-proof workflows, automatic retries, and long-lived signal-driven steps | 90-120 min |
| 09 | Capstone project | Design and partially build a distributed order-processing system integrating idempotency, a saga, and event sourcing | 4-6 hrs |
Start here → CAP Theorem, then 01-idempotency-in-practice/README.md
Back to the master index: ../README.md
Once you've completed this track, you'll carry these patterns into every system design that follows — you'll recognize the saga hiding in every multi-step workflow, the idempotency key missing from every retry path, and the consensus problem lurking behind every "just have one of them do it."