Module 00: Caching Fundamentals

Why this matters

Track 04 ended on a preview of it: the cheapest query is the one you never run. Caching is that idea generalized to the entire stack. Almost every performance problem you will meet as a backend engineer is really a problem of doing expensive work repeatedly when the answer hasn't changed — recomputing the same aggregate, re-reading the same row, re-rendering the same page, re-fetching the same asset. A cache is a small, fast store that sits in front of a slow or expensive resource and remembers recent answers so the expensive path runs far less often.

Done well, a single well-placed cache can cut a p99 latency from 800ms to 8ms and drop database load by 90% — with a few dozen lines of code. Done badly, it serves a customer someone else's data, shows a "deleted" item that's still there, or falls over harder than the thing it was protecting. The difference is almost entirely conceptual: knowing what a cache actually is, where it can live, what belongs in it, and what its failure modes are — before you write a single r.set(...). That's this module. The strategies, eviction policies, and multi-level architectures in the rest of the track all build on the mental model you form here.

Concepts

What a cache is, precisely

A cache is a key-value store of previously-computed answers, placed closer (in time or space) to the consumer than the authoritative source, whose entire value proposition is that reading from it is dramatically cheaper than recomputing or re-fetching the real thing. Three properties define one:

• It is a copy, never the source of truth. The authoritative data lives somewhere else (a database, an upstream API, a filesystem, a CPU-expensive computation). The cache holds a duplicate that is allowed to be thrown away at any moment without losing information. This single property is what makes caching safe and what makes it dangerous — because a copy can disagree with its source.
• It trades space (and freshness) for time. You spend memory to store answers, and you accept that an answer might be slightly out of date, in exchange for serving it far faster.
• It is allowed to miss. Asking a cache for a key that isn't there (a miss) is normal and must always fall back to the real source. A cache that you depend on having the data isn't a cache — it's a primary datastore with no durability guarantees, which is a bug waiting to happen.

The core operation is the cache lookup: check the cache first; on a hit, return the cached answer and skip the expensive work; on a miss, do the expensive work, store the result, and return it. The fraction of lookups that hit is the hit ratio, and it is the single number that tells you whether a cache is earning its keep (module 03 goes deep on measuring and improving it).

Why caching exists: the latency and cost hierarchy

Caching exists because not all data access is equally expensive, and the differences are enormous — orders of magnitude, not percentages. A rough, much-simplified ladder of how long it takes to get a piece of data, from a backend's point of view:

The numbers are approximate and hardware-dependent, but the shape is the whole point: each rung down is roughly 10–1000× slower than the one above. A cache works by moving an answer up this ladder — from a 200ms aggregate query to a 0.8ms Redis read, or from a 0.8ms Redis read to a 100ns in-process dictionary lookup. That's why the exact same idea appears at every layer of a system: any time there's a big latency gap between where data is and where it's needed, a cache in the gap pays off.

  FAST ▲  L1 cache      ~1 ns     ┐
       │  RAM           ~100 ns   │ a cache MOVES an answer
       │  in-proc dict  ~100 ns   │ UP this ladder:
       │  SSD           ~100 µs   │
       │  Redis (LAN)   ~1 ms   ◄─┘ 200ms query ──► 1ms Redis  (~200×)
       │  PG indexed    ~1-10 ms       1ms Redis ──► 100ns dict (~10,000×)
       │  PG aggregate  ~50-500 ms
  SLOW ▼  slow upstream ~0.1-2 s
        each rung ~10-1000× slower than the one above it

Two things are being reduced at once, and it's worth separating them:

• Latency — the individual request gets faster (the user waits less).
• Load — the expensive resource does less total work (it survives more traffic, or costs less to run). A query result cached with a 60s TTL and requested 10,000 times a minute turns 10,000 database queries into ~1. The database is the real beneficiary there, not any single request.

You cache for one or both of these, and being clear about which you're buying changes the design (a cache that only helps latency but not load is still useful; one bought "for load" that doesn't actually raise the hit ratio is useless — module 03).

Where caches live: the layers

"Caching" is not one thing in one place. The same lookup-first-fall-back-on-miss pattern is implemented independently at many layers between the user and the data. Knowing the layers keeps you from solving a problem at the wrong one.

• Client-side / browser caching. The user's browser stores responses on the user's own disk and reuses them without any network request at all. Governed by HTTP headers (Cache-Control, ETag, Last-Modified — you met these in track 01, and module 04 revisits them from the caching angle). This is the cheapest possible cache because a hit means zero work for your backend — the request never leaves the user's machine.
• CDN / edge caching. A content-delivery network keeps copies of responses (especially static assets, but increasingly API responses too) in data centers physically near users. A hit is served from the edge without ever reaching your origin servers — cutting both latency (geography) and origin load.
• Server-side application caching. Inside your own infrastructure. This splits further into two sub-layers that module 03 treats as L1 and L2:
  • In-process / in-memory cache — a plain data structure (a dict, an LRU cache) living inside your application process's own memory. Fastest possible server-side hit (~100ns, no network), but local to one process: not shared between replicas, and gone when the process restarts.
  • Distributed cache — a separate service like Redis or Memcached that all your app instances share over the network (~1ms). Slower than in-process but shared (every replica sees the same cached value) and survives app restarts. This is the workhorse of backend caching and the main tool of this track.
• Database caching. The database itself caches. Postgres keeps recently-read pages in its shared buffer pool in RAM (the shared hit vs read you saw in EXPLAIN (ANALYZE, BUFFERS) in track 04 module 07), and the OS keeps a page cache underneath that. You don't manage these directly, but they're why a "warm" query is faster than a "cold" one, and why benchmarking a query twice gives different numbers.

A single user request can hit several of these in sequence: browser cache → CDN → app in-process cache → Redis → Postgres buffer pool → disk. Each layer it stops at is a layer of work everything below it didn't have to do.

  user ──► [browser] ──► [CDN/edge] ──► [ your servers ] ──► [Postgres]
           cache          cache          L1 in-proc dict      buffer pool
             │              │            L2 Redis (shared)      + disk
             │              │                  │                   │
        hit = 0 work    hit = origin      hit = no DB         the source
        for backend     never touched     query               of truth
        └──────────────── stop as early as you can ───────────────┘

Client-side vs server-side caching: a real distinction, not a synonym

These are often confused, so be precise:

• Client-side caching stores copies on the consumer's machine (browser disk, mobile app storage) and is controlled indirectly by your backend through HTTP response headers. You don't hold the data; you send instructions ("you may reuse this for 300 seconds"). Its huge advantage is that a hit costs your backend nothing. Its limitations: you can't forcibly invalidate it (the copy is on a machine you don't control — once you've said "cache for an hour," you can't take it back), and it's per-user (a value cached in one user's browser does nothing for the next user).
• Server-side caching stores copies on your infrastructure (in-process, or in Redis) and is controlled directly by your code. You can read, write, and invalidate entries whenever you want, and one cached value serves all users who need it. The cost is that the request still had to reach your servers to get the hit — you saved the database, not the network round-trip to you.

They are complementary, not alternatives. A well-tuned system uses client-side caching to eliminate requests entirely where possible, and server-side caching to make the requests that do arrive cheap. This track focuses on server-side caching (especially Redis), because that's the layer you write application code for; but never forget the cheapest cache is the one on the user's own machine.

What belongs in a cache — and what emphatically does not

A value is a good caching candidate when all of these hold:

• It's expensive to produce. A complex aggregate, a fan-out of API calls, an N+1-shaped read, a rendered template. Caching a SELECT 1 saves nothing; caching a 300ms report saves a lot.
• It's read far more often than it changes. The whole economy of a cache is amortizing one expensive computation over many cheap reads. A value read 10,000×/min and updated once/hour is ideal. A value read once and never again should not be cached (you paid the store cost for nothing).
• Slight staleness is acceptable. Because a cache is a copy, it will at times disagree with the source. If serving a value that's a few seconds or minutes old is fine (a "top products" list, a follower count, a rendered blog post), it's cacheable. This is the property people get wrong most often.

A value is a bad caching candidate — or must be cached only with extreme care — when:

• Staleness is a correctness bug, not a cosmetic one. An account balance, a remaining-inventory count you decrement on purchase, an authorization decision, a one-time token. Serving an old copy here doesn't look slightly dated — it double-spends, over-sells, or grants access that was revoked. If in doubt, ask: "what's the worst thing that happens if a user sees a value N seconds out of date?" If the answer is "money moves incorrectly" or "someone sees data they shouldn't," don't TTL-cache it.
• It's unique per request and never reused. Caching something with a cardinality as high as the request rate just fills memory with entries that never get a second hit — pure overhead.
• It's already cheap. Adding a network hop to Redis to avoid a 0.2ms indexed primary-key lookup can make things slower, not faster.
• It's sensitive and the cache isn't as protected as the source. A cache is another copy of the data in another system; if it holds PII or secrets, it's another thing to secure, expire, and reason about under a breach.

The recurring discipline of the whole track, first stated here: cache read-heavy, change-rarely, expensive, staleness-tolerant data — and nothing else, until you've measured that you should.

The two hard problems (named now, solved later)

There's a famous industry joke: "There are only two hard things in computer science: cache invalidation and naming things." It's a joke, but the first half is deadly serious and it's the reason caching is a skill and not a library call. Because a cache is a copy:

1. Invalidation — knowing when a cached copy no longer matches its source, and getting rid of it, is genuinely hard. Do it too eagerly and your hit ratio collapses (you're barely caching). Do it too late and you serve stale data. Miss an update path entirely and you serve stale data forever.
2. Consistency — even with invalidation, there are windows where the cache and source disagree, and races (two requests, one reading while another writes) that can leave the wrong value cached durably.

You don't need to solve these yet — module 01 (strategies) and module 02 (eviction & invalidation) are entirely about them. But name them now, because every design decision in this track is ultimately a position on how much staleness you'll tolerate to get how much speed. Caching is easy; invalidating is the hard part, and pretending otherwise is how stale-data bugs are born.

Command reference

There's little to run in a fundamentals module — the real tool arrives in module 01 — but here is the minimal shape of a manual cache lookup in Python, and the Redis commands you'll lean on all track. Stand up Redis first:

docker run --name redis -p 6379:6379 -d redis:7
docker exec -it redis redis-cli PING   # -> PONG

The canonical cache-aside lookup (module 01 formalizes this as a pattern) — the single most important shape in the whole track:

import json
import redis

r = redis.Redis(host="localhost", port=6379, decode_responses=True)

def expensive_report(limit: int) -> list[dict]:
    # ...imagine a 300ms aggregate query against Postgres...
    raise NotImplementedError

def get_report(limit: int) -> list[dict]:
    key = f"report:top:{limit}"          # a stable, parameterized cache key
    cached = r.get(key)
    if cached is not None:               # HIT: skip the expensive path entirely
        return json.loads(cached)
    result = expensive_report(limit)     # MISS: do the real work
    r.set(key, json.dumps(result), ex=60)  # store as a copy, TTL 60s bounds staleness
    return result                        # ...and return it

Everything in this track is a variation on, or a defense of, those ten lines: the key design, the serialization, the TTL, the hit/miss branch, and what happens when two requests miss at the same instant.

Hands-on exercises

Do these in a terminal with Redis running (docker run --name redis -p 6379:6379 -d redis:7). A couple use a tiny FastAPI app; install with pip install "fastapi[standard]" redis.

1. Feel the latency ladder yourself

Time a trivial in-memory lookup, a Redis round-trip, and a deliberately slow "computation," to internalize the orders of magnitude from Concepts.

import time, redis
r = redis.Redis(decode_responses=True)
r.set("k", "v")

def timeit(label, fn, n=1000):
    start = time.perf_counter()
    for _ in range(n):
        fn()
    per = (time.perf_counter() - start) / n * 1e6  # microseconds per op
    print(f"{label:20} {per:8.2f} µs/op")

d = {"k": "v"}
timeit("dict lookup",  lambda: d["k"])
timeit("redis GET",    lambda: r.get("k"))
timeit("sleep(0.05)",  lambda: time.sleep(0.05), n=20)  # a 'slow query'

Expected: the dict lookup is well under a microsecond, the Redis GET is hundreds of microseconds to ~1ms (network + serialization), and the "slow query" is ~50,000 µs. You've just measured the gaps a cache exploits — moving the slow-query answer up to the Redis rung is a ~50–100× win; up to the dict rung is more still (and why module 03 exists).

2. Build the cache-aside lookup by hand

Implement get_report from the Command reference against a fake slow function that time.sleep(0.3)s and prints "COMPUTING" each time it actually runs. Call get_report(10) three times in a row and observe.

Expected: only the first call prints "COMPUTING" and takes ~300ms; the next two are near-instant and silent — they hit the cache. Now docker exec redis redis-cli DEL report:top:10 and call it again: "COMPUTING" reappears. You've just seen a hit, a miss, and a manual invalidation.

3. Watch a TTL bound staleness

SET report:top:10 "..." EX 5 from redis-cli, then run TTL report:top:10 repeatedly and finally GET it after 5+ seconds.

Expected: TTL counts down from 5 to 0, then GET returns (nil) and TTL returns -2. This is time-based invalidation — the cache forgets on its own, so a bug that forgets to invalidate manually still can't serve infinitely-stale data. Note the tradeoff you just set: a 5s TTL means data can be up to 5s stale.

4. Measure a hit ratio

Reset stats context by noting the current counters, then generate traffic:

docker exec redis redis-cli INFO stats | grep keyspace

Run exercise 2's three calls again, then re-check. Compute hits / (hits + misses).

Expected: keyspace_hits and keyspace_misses climbed; with 1 miss and 2 hits your ratio is ~0.67. This single number (module 03) is how you'll later judge whether any cache is worth keeping — internalize that a low ratio means you're paying cache overhead without getting the amortization payoff.

5. Classify real values as cacheable or not

For each of the following, write down: cache it or not, and if yes, roughly what TTL and why staleness is acceptable. Then check your reasoning against Concepts.

1. The homepage "trending articles" list.
2. A user's current account balance on a banking screen.
3. The result of a 400ms "sales by region this quarter" dashboard query.
4. A per-request, per-user personalized recommendation with a unique key each time.
5. A product's name and description on a catalog page.
6. The number of items left in stock, shown on a checkout page that decrements on purchase.

Expected reasoning: (1) yes, short TTL (30–120s), trending is inherently approximate; (2) no — staleness is a correctness/trust bug; (3) yes, longer TTL (minutes), quarter data barely moves; (4) no — cardinality equals request rate, never a second hit; (5) yes, long TTL, descriptions change rarely; (6) careful — caching the display is fine but the decision to sell must read the source, or you oversell (this exact tension returns in module 02 and the capstone).

6. See a database's own cache warm up

Against any Postgres table with a few thousand rows (reuse events from track 04 module 03 if you have it), run the same query twice under EXPLAIN (ANALYZE, BUFFERS):

EXPLAIN (ANALYZE, BUFFERS) SELECT count(*) FROM events WHERE kind = 'buy';
-- run it a second time immediately
EXPLAIN (ANALYZE, BUFFERS) SELECT count(*) FROM events WHERE kind = 'buy';

Expected: the first run shows some Buffers: shared read=... (pages fetched from disk/OS); the second shows those turned into shared hit=... (served from Postgres's buffer pool in RAM) and a lower Execution Time. You just watched a cache you don't manage do its job — and learned why "run the benchmark twice" matters.

7. Diagnose and fix: the cache that never hits

A colleague added caching to a per-user profile endpoint but reports "the cache isn't doing anything — the database load didn't drop." Their code:

import time, json, redis
r = redis.Redis(decode_responses=True)

def get_profile(user_id: int) -> dict:
    key = f"profile:{user_id}:{time.time()}"   # <-- look closely
    cached = r.get(key)
    if cached:
        return json.loads(cached)
    profile = load_profile_from_db(user_id)     # the expensive call
    r.set(key, json.dumps(profile), ex=300)
    return profile

Reproduce it (stub load_profile_from_db to print "DB!" and sleep 100ms), call get_profile(1) five times, and watch every call print "DB!". Diagnose and fix.

Independent challenge

No code given. Take the "customers ranked by spend" report you built or saw in 04-databases-and-data-layer (module 07's caching exercise) and, before writing any caching code, produce a written caching design for it: name the authoritative source, decide whether it belongs in a client-side cache, a server-side distributed cache, or both, and justify the choice using the "what belongs in a cache" checklist. Pick a TTL and defend it by stating the maximum staleness a user could see and why that's acceptable for this specific report. Then name one other read in that same app for which caching would be a correctness bug, and say exactly what would go wrong. Only after the written design, implement the cache-aside lookup and confirm (via a hit-ratio measurement like exercise 4) that a realistic access pattern actually produces a high hit ratio — if it doesn't, your value wasn't a good candidate and you should be able to say why.

Common mistakes & troubleshooting

• Treating the cache as a source of truth. Depending on data being in the cache (no fallback path, or writing only to the cache) turns an eviction or a restart into data loss. Always be able to reconstruct any cached value from the authoritative source on a miss.
• Unstable cache keys. Keys built from time.time(), random values, or fields that don't determine the answer produce a permanent 100% miss rate (exercise 7). A key must be a pure function of the inputs that decide the value.
• Caching things that must not be stale. Balances, inventory counts you sell against, auth decisions, one-time tokens. Staleness there is a correctness bug, not a cosmetic one — no TTL is safe.
• Caching things that are never re-read. Per-request unique values just burn memory and cache-write time for a hit that never comes. High key cardinality relative to traffic = bad candidate.
• Caching something already cheap. Wrapping a 0.2ms primary-key lookup in a 1ms Redis round-trip is a slowdown. Cache the expensive thing, not everything.
• Forgetting the cache is another copy of the data to secure. PII/secrets in a cache are PII/secrets in one more place, with their own expiry and access rules to get right.
• Confusing client-side and server-side caching. You instruct client caches via headers and can't forcibly invalidate them; you control server-side caches directly. Reaching for one when you needed the other is a category error.
• Optimizing the hit case while ignoring the miss case. What happens when 10,000 requests all miss the same key at once? (A stampede — module 04.) The miss path and its concurrency are as important as the hit path.

Checkpoint quiz

Write down your answer to each question before expanding it — checking without attempting first is the single easiest way to fool yourself into thinking you've learned this.

1. Give the one-sentence definition of a cache, and name the single property (that it's a copy) that makes it both useful and dangerous.
2. What two distinct things does a cache reduce, and why might a cache help one but not the other?
3. Name the layers a single user request's data could be cached at, from the user's machine down to the database, and say which layer a hit costs your backend the least.
4. State the three conditions that together make a value a good caching candidate, and give one example of a value that fails the "staleness is acceptable" condition.
5. What is a cache hit ratio, and why is it the number that tells you whether a cache is worth keeping?
6. Name the "two hard things in computer science" cache-invalidation joke's serious point: why is invalidation genuinely hard, in terms of the copy property?

Further reading & sources

• Latency Numbers Every Programmer Should Know - Jeff Dean's canonical latency ladder; the source of the orders-of-magnitude intuition this module is built on.
• Redis documentation - the official docs for the distributed cache (L2) used throughout this track.
• redis-py documentation - the Python client whose get/set/ex you use in the cache-aside lookup here.
• MDN: HTTP caching - how Cache-Control, ETag, and Last-Modified drive client-side caching (revisited in module 04).
• Martin Fowler: TwoHardThings - the "cache invalidation and naming things" joke and why the first half is deadly serious.
• PostgreSQL: EXPLAIN - reading shared hit vs read to watch Postgres's own buffer-pool cache warm up.

Next

01-caching-strategies — you now know what a cache is, where it can live, and what belongs in one. Next is how your code and the cache actually coordinate reads and writes: cache-aside, read-through, write-through, and write-behind — the four strategies, their consistency-vs- latency-vs-complexity tradeoffs, and how to choose between them.