Module 04: Retention, Compaction and Topics as State

Why this matters

Every topic silently answers a question you may never have asked: how long does this data live? Get it wrong in one direction and you delete the events a new consumer needed to bootstrap. Get it wrong in the other and you pay to store terabytes nobody reads.

But the more interesting idea in this module is the second one. A compacted topic retains the latest value per key indefinitely — which turns a Kafka topic into a durable, replayable key-value store. That single property is what makes event sourcing, CQRS read models, and stream-table joins possible, and it's why "Kafka as a database" is a defensible claim in narrow circumstances rather than pure hype.

Concepts

Two cleanup policies

cleanup.policy=delete    (default)  — drop whole segments once they age out
cleanup.policy=compact              — keep the latest record per key, forever
cleanup.policy=compact,delete       — compact, and also drop very old segments

DELETE                           COMPACT
offset  key    value             offset  key    value
0       A      1                 (0 A=1 removed — superseded)
1       B      2                 (1 B=2 removed — superseded)
2       A      3                 3       C      4
3       C      4                 5       A      5
4       B      6         ──▶     4       B      6
5       A      5
                                 latest value per key survives; offsets
older segments deleted           are NOT renumbered — gaps are normal
by time or size

Compaction never renumbers offsets. Consumers must tolerate gaps — which they do naturally, since they always read forward from wherever they are.

Retention settings

The trap: retention is enforced per segment, and only on closed segments. The active segment is never deleted, so a topic with retention.ms=60000 and a 1 GB segment.bytes will happily hold data for days — because the segment hasn't rolled yet. If you want fine-grained retention, you must lower segment.ms/segment.bytes too. This is the single most common "retention isn't working" support question.

Also note retention.bytes is per partition, not per topic. Setting 10 GB on a 50-partition topic budgets 500 GB.

Compaction mechanics

Compaction runs in the background on closed segments only, so the most recent records always have duplicates by key. Relevant knobs:

Consequences worth internalising: compaction is eventual, never immediate; you cannot rely on "one record per key" being true at any given moment; and a low dirty ratio (e.g. 0.1) compacts more aggressively at the cost of more I/O.

Tombstones: deletion in a compacted topic

To delete a key, produce a record with that key and a null value:

producer.produce("user-profiles", key=b"user-42", value=None)   # tombstone

The tombstone marks the key deleted. Consumers see it and remove the key from their local state. After delete.retention.ms, compaction removes both the tombstone and the key's history.

That window exists for a reason: a consumer that was offline must still see the tombstone when it comes back, or it will keep a key that everyone else has deleted. Set delete.retention.ms longer than your worst realistic consumer downtime — the default 24 hours is often too short for a service that can be down over a weekend.

A null value is never "just an empty message" on a compacted topic. If your serializer emits null for an empty object, you will delete keys by accident.

Compacted topics as state

This is the pattern that matters:

user-profiles (compacted, keyed by user_id)
    ↓ consumer reads from offset 0 on startup
    ↓ builds an in-memory dict / local RocksDB
    ↓ then follows the tail for live updates
in-process materialised view — no database required

Properties you get for free:

• Rebuildable. Lose the local state, replay the topic, get it back.
• Bounded. Size is proportional to the number of keys, not events.
• Shareable. Any number of services can build their own view.
• Ordered per key. The last write per key wins, deterministically.

This is exactly the "changelog" mechanism Kafka Streams uses internally for state stores (module 05), and the log side of CQRS (track 10).

The limits, which matter just as much: you can only look up by key (no queries, no secondary indexes, no aggregation without doing it yourself), bootstrapping is O(number of keys) so a huge topic means slow startup, and there are no transactions across keys.

Choosing a policy

The question that decides it: would a new consumer need every record, or only the latest per key? Events → delete. State → compact.

Infinite retention and tiered storage

retention.ms=-1 keeps data forever. That's legitimate for event sourcing and audit, but expensive on broker disks. Tiered storage (KIP-405, GA in Kafka 3.9+) offloads older segments to object storage, so brokers keep only recent data locally while historical reads still work — transparently, at higher latency. It's the feature that makes "keep everything" affordable.

Command reference

Hands-on exercises

Use the broker from module 00.

1. Prove retention doesn't apply to the active segment

kt --create --topic ret-test --partitions 1 \
   --config retention.ms=10000            # 10 seconds

Produce 100 records, wait 60 seconds, then count them.

Expected: all 100 are still there. Retention only removes closed segments, and the active one hasn't rolled. Now:

kc --alter --entity-type topics --entity-name ret-test \
   --add-config segment.ms=5000

Produce a few more records to force a roll, wait, and re-count.

Expected: the old records finally disappear. Write down why "set retention.ms and nothing happened" is one of the most common Kafka questions.

2. Build a compacted topic and watch it compact

kt --create --topic profiles --partitions 1 \
   --config cleanup.policy=compact \
   --config segment.ms=1000 \
   --config min.cleanable.dirty.ratio=0.01 \
   --config delete.retention.ms=10000

Produce many updates for a few keys:

for i in range(500):
    p.produce("profiles", key=f"user-{i % 5}".encode(), value=f"v{i}".encode())
p.flush()

Count records immediately, then again after ~45 seconds of idle time (produce one extra record to force a segment roll).

Expected — and read this carefully, because it's the point of the exercise:

log end offset before compaction:  500
records still in log:              303      <-- NOT 5
distinct keys:                     5
latest per key: {'SKU-0': 1, 'SKU-1': 504, 'SKU-2': 503, ...}

Compaction reduced the log, but 303 records remain, not 5. That is correct behaviour, not a misconfiguration: compaction only processes closed segments, runs in the background, and never touches the active segment. What you are guaranteed is that reading the whole topic yields the correct latest value for every key — which is exactly what a materialised view needs.

The lesson: never write code that assumes a compacted topic contains one record per key. Assume instead that replaying it converges to one value per key, with the last write winning.

3. Confirm offsets are not renumbered

Read the compacted topic with print.offset=true.

Expected: offsets have large gaps (e.g. 495, 496, 497, 498, 499) rather than 0-4. Explain in one sentence why this is harmless for consumers.

4. Delete a key with a tombstone

p.produce("profiles", key=b"user-3", value=None)
p.flush()

Consume from the beginning.

Expected: user-3 appears with a null value. After compaction plus delete.retention.ms, both the tombstone and its history vanish. Confirm by re-reading later.

5. Build a materialised view

Write a consumer that reads profiles from offset 0 into a dict, deleting keys on null values, then follows the tail:

state = {}
while True:
    msg = c.poll(1.0)
    if msg is None or msg.error():
        continue
    k = msg.key().decode()
    if msg.value() is None:
        state.pop(k, None)          # tombstone
    else:
        state[k] = msg.value().decode()

Kill and restart it.

Expected: the state rebuilds identically from the log, with no database involved. That's the whole "topic as state" idea in fifteen lines.

6. Prove an empty value deletes a key by accident

Produce value=b"" (empty bytes) and then value=None for two different keys.

Expected: only the None one is a tombstone; the empty-bytes one is a normal record with an empty payload. Then check what your serializer does with an empty object — if it produces None, you have an accidental-deletion bug waiting.

7. Diagnose and fix: the disk that filled up

A team's events topic has retention.ms=86400000 (1 day) and 200 partitions. Disk usage grows without bound and hits 90%. kafka-configs.sh --describe confirms the retention setting is applied. Some partitions hold 30 days of data; others hold hours.

8. Clean up

kt --delete --topic ret-test; kt --delete --topic profiles

Independent challenge

No solution given. Design the topic configuration for a system with four topics: orders (business events, must be replayable for 2 years for audit), inventory-levels (current stock per SKU, ~500k SKUs, updated constantly), user-sessions (session state, expires after 30 days of inactivity), and debug-traces (very high volume, only useful for 6 hours).

Specify cleanup.policy, retention, segment settings, and estimated storage for each, stating your assumptions about event size and rate. Then solve the hard one: user-sessions needs both latest-value-per-key semantics and automatic expiry after 30 days of inactivity — explain why cleanup.policy=compact,delete alone doesn't quite give you that, and what you'd do instead.

Common mistakes & troubleshooting

• Expecting retention to apply immediately. It only removes closed segments; the active segment is never deleted, so segment.ms gates everything.
• Forgetting retention.bytes is per partition. Multiply by the partition count before sizing disks.
• Assuming a compacted topic has exactly one record per key. Compaction is a background process and the head of the log always has duplicates.
• Serializers that emit null for empty objects. On a compacted topic, that's an accidental tombstone deleting real keys.
• delete.retention.ms shorter than your consumer downtime. Offline consumers miss the tombstone and keep deleted keys forever.
• Compacting a topic of business events. You lose history that consumers needed; events want delete with long retention.
• Using a compacted topic as a general database. Key lookups only; no queries, no secondary indexes, no cross-key transactions.
• Very large compacted topics with slow bootstrap. Startup is O(number of keys); plan for it or use a local checkpoint.

Checkpoint quiz

Further reading & sources

• Kafka: Log compaction - the authoritative description of the cleaner, dirty ratio and tombstones.
• Kafka: Topic-level configs - every retention, segment and compaction setting.
• Jay Kreps: The Log - the original argument for the log as a unifying data abstraction.
• KIP-405: Tiered storage - offloading old segments to object storage.
• Track 10: Event sourcing - the pattern compacted topics and long retention support.

Next

05-stream-processing-windows-joins-and-state — you can now store state in a topic. Module 05 computes over it: windows, joins, and what a stream processor gives you beyond a consumer loop.