Module 06: Operating Kafka — Lag, Sizing and Reliability

Why this matters

Kafka fails quietly. Brokers stay green, dashboards stay flat, and the only symptom is that events are arriving later and later until someone notices the business has been running on stale data for six hours. There is no request returning 500 to alert you.

The signal that actually matters is consumer lag, and almost everyone measures it wrong — as a record count, which tells you nothing actionable. This module covers the metrics that predict incidents, how to size partitions before you're stuck with them, what replication does and doesn't guarantee, and how to stop one malformed record from halting a partition forever.

Concepts

Consumer lag, and why the record count misleads

partition 0:  [....................................]
              ^committed offset 8,400      ^log end 12,900
                        lag = 4,500 records

4,500 records is meaningless on its own. Is that 2 seconds of work or 3 hours? Depends entirely on throughput. Convert lag to time:

lag_seconds = lag_records / records_processed_per_second

That's the number to alert on, because it's the one your business understands: "the fraud service is 40 minutes behind" is actionable; "the fraud service is 4.5 million records behind" is not.

Better still, alert on the derivative. Steady lag at 500 records is healthy. Lag growing linearly means consumption is slower than production and will never recover on its own — that's the incident, and it's visible long before lag becomes large.

# Prometheus-style
kafka_consumergroup_lag                                   # raw
kafka_consumergroup_lag / rate(records_consumed_total[5m]) # seconds behind
deriv(kafka_consumergroup_lag[10m]) > 0                    # falling behind

Always measure lag per partition, not summed per group. A single stuck partition is invisible in a group total dominated by 49 healthy ones — and a single stuck partition is exactly what a poison pill produces.

Replication and ISR

partition 0, replication.factor=3
  leader   broker-1  [====================]  LEO 1000
  follower broker-2  [====================]  LEO 1000   in ISR
  follower broker-3  [==============      ]  LEO  700   lagging

  ISR = {1, 2}          high watermark = 1000 (min LEO across ISR)
  consumers can read up to the high watermark only

A follower stays in the ISR while it fetches within replica.lag.time.max.ms (default 30s). Fall behind and it's ejected — which silently shrinks your durability guarantee, because acks=all only waits for the ISR (module 00).

The settings that make this safe:

Setting	Recommended	Why
`replication.factor`	3	Survive one broker loss with room to spare
`min.insync.replicas`	2	Reject writes when durability degrades
`unclean.leader.election.enable`	`false`	Never promote an out-of-sync replica

unclean.leader.election.enable=true trades data loss for availability: an out-of-sync replica becomes leader and everything it hadn't replicated is gone, permanently. It defaults to false and should stay there unless you have an explicit, written decision that availability matters more than correctness for that topic.

Alert on ISR shrink. It's the leading indicator: durability degrades before data is lost, and that gap is your chance to intervene.

Partition sizing

Partitions are the unit of parallelism, and module 00 established that changing the count on a keyed topic breaks ordering — so this is a decision you largely make once.

partitions = max(
    target_throughput / per_partition_throughput,
    target_throughput / per_consumer_throughput,
    peak_consumer_instances
)

Then round up for headroom. Practical guidance:

Aim for 10-30 MB/s per partition as a working assumption; measure yours.
Partition count should comfortably exceed your maximum consumer count (module 01: extras idle).
More partitions cost broker file handles, memory, replication traffic, and leader-election time — a broker failure with 4,000 partitions takes far longer to recover than with 400.
A rough ceiling of a few thousand partitions per broker is sane; tens of thousands is a problem.

Over-provision modestly (12 partitions when you need 3, so you can scale to 12) — not wildly.

Poison pills and dead-letter topics

One unparseable record halts a partition forever if your consumer retries indefinitely: the offset never advances, lag climbs, and no other record on that partition is ever processed.

MAX_RETRIES = 3

def handle(msg):
    for attempt in range(MAX_RETRIES):
        try:
            process(msg)
            return True
        except TransientError:
            time.sleep(2 ** attempt)
        except PermanentError:
            break                      # don't retry what can't succeed
    to_dlq(msg)                        # park it, keep the partition moving
    return True

def to_dlq(msg):
    producer.produce(
        "orders.DLQ",
        key=msg.key(),
        value=msg.value(),
        headers=[
            ("original_topic", msg.topic().encode()),
            ("original_partition", str(msg.partition()).encode()),
            ("original_offset", str(msg.offset()).encode()),
            ("error", traceback.format_exc()[:900].encode()),
            ("failed_at", datetime.now(timezone.utc).isoformat().encode()),
        ],
    )
    producer.flush()

Two disciplines that make DLQs useful rather than a graveyard:

Distinguish transient from permanent failures. Retry a database timeout; do not retry a schema violation. Retrying permanent errors is just a slower way to block the partition.
Alert on DLQ arrivals and have a replay path. An unmonitored DLQ is a place where data goes to be forgotten. Record enough headers to replay the record after fixing the bug.

Note the ordering tradeoff: sending a record to a DLQ and continuing means you have processed later records for that key before this one. If strict per-key ordering matters more than availability, you must stop the partition instead — a real decision, not an oversight.

The metrics that matter

Broker: UnderReplicatedPartitions (should be 0), OfflinePartitionsCount (should be 0), ActiveControllerCount (exactly 1 across the cluster), request handler idle ratio, disk usage per log dir.

Producer: record-error-rate, record-retry-rate, request-latency-avg, buffer-available-bytes.

Consumer: lag per partition (in time), records-lag-max, commit-latency-avg, rebalance-rate-per-hour — that last one is how you detect module 01's rebalance storms.

Alert on: lag seconds over threshold, lag derivative positive and sustained, UnderReplicatedPartitions > 0, ActiveControllerCount != 1, rebalance rate elevated, and any DLQ traffic.

Capacity and cost

storage_per_day = events_per_day × avg_event_bytes × replication_factor
retained_total  = storage_per_day × retention_days × 1.3   # overhead headroom

At 100M events/day × 1 KB × RF 3 = ~300 GB/day; at 7 days retention that's ~2.7 TB before overhead. Compression (module 00) typically cuts this 3-5×, which is why compression.type is a cost decision as much as a throughput one.

Command reference

Concern	Command
Group lag per partition	`kafka-consumer-groups.sh --describe --group g --bootstrap-server localhost:9092`
All groups	`kafka-consumer-groups.sh --list ...`
Under-replicated partitions	`kafka-topics.sh --describe --under-replicated-partitions ...`
Unavailable partitions	`kafka-topics.sh --describe --unavailable-partitions ...`
Topic detail (ISR)	`kafka-topics.sh --describe --topic t ...`
Reassign partitions	`kafka-reassign-partitions.sh --execute --reassignment-json-file plan.json ...`
Disable unclean election	`--add-config unclean.leader.election.enable=false`
Skip a poison pill (manual)	`kafka-consumer-groups.sh --reset-offsets --shift-by 1 --group g --topic t:0 --execute ...`
Reset to a timestamp	`--reset-offsets --to-datetime 2026-08-01T00:00:00.000 --execute`
Throughput test	`kafka-producer-perf-test.sh --topic t --num-records 1000000 --record-size 1024 --throughput -1 --producer-props bootstrap.servers=localhost:9092`
Log dir sizes	`kafka-log-dirs.sh --describe --bootstrap-server localhost:9092`

Hands-on exercises

Use the broker from module 00.

1. Create lag and convert it to time

Produce 100,000 records, then start a consumer that sleeps 5 ms per record.

kg --describe --group slow

Expected: a LAG column with a large number. Compute lag / (1/0.005) = lag / 200 to get seconds behind, and confirm it matches how long the consumer actually takes to catch up. That conversion is the whole exercise.

2. Distinguish steady from growing lag

Run a consumer slightly slower than the producer, then slightly faster. Sample lag every 10 seconds for two minutes and plot both.

Expected: one line flat, one climbing linearly. Write down which alert you'd page on and why raw lag magnitude alone would have fired on both or neither.

3. Show per-partition lag hides in the total

Run 3 consumers on a 6-partition topic, then SIGSTOP one of them.

Expected: group total lag rises slowly while two specific partitions grow without bound. Confirm with --describe that the per-partition view makes the problem obvious and the summed view doesn't.

4. Build a poison pill and watch it block

Produce 100 valid JSON records with one not-json in the middle. Write a consumer that retries on exception forever.

Expected: the partition stops at that offset permanently; lag grows; the remaining valid records are never processed. This is the failure the DLQ prevents.

5. Add a DLQ and recover

Implement to_dlq() with headers. Re-run.

Expected: the bad record lands in orders.DLQ with diagnostic headers, processing continues, lag recovers. Then read the DLQ and print the headers — confirm you have enough information to replay after a fix.

6. Observe ISR shrink

(Needs a multi-broker cluster — use a 3-broker compose file.) Create a topic with RF 3, then stop one broker.

kt --describe --topic t

Expected: Isr: shrinks from 3 entries to 2 while Replicas: stays at 3. Then set min.insync.replicas=3 and confirm producers with acks=all now fail with NOT_ENOUGH_REPLICAS — durability is refusing to silently degrade, which is the behaviour you want.

7. Measure per-partition throughput

docker exec kafka /opt/kafka/bin/kafka-producer-perf-test.sh \
  --topic perf --num-records 500000 --record-size 1024 --throughput -1 \
  --producer-props bootstrap.servers=localhost:9092

Run against topics with 1, 3 and 6 partitions.

Expected: throughput scales with partitions up to a point, then plateaus as the broker (or your client) saturates. Use the number to sanity-check the sizing formula rather than trusting a rule of thumb.

8. Diagnose and fix: the silent six-hour delay

A fraud-detection consumer group reports healthy: all brokers green, no errors, CPU normal, and the team's dashboard shows "total group lag" hovering around 2,000 — the same as always. But the business reports fraud alerts arriving six hours late for a subset of customers.

Solution

A single partition is stuck, hidden by a group-level total. With, say, 48 partitions, one partition stalled at six hours of lag while 47 sit near zero produces a group total that looks normal-ish and barely moves — especially if the dashboard averages or the healthy partitions absorb the visual range. Because fraud events are keyed by customer (module 00), the affected customers are exactly those hashing to that partition — which is why it's "a subset" rather than everyone.

The likely cause is a poison pill or a permanently-failing record on that partition, with a consumer retrying forever: the offset never advances, so lag on that one partition grows linearly while everything else is fine. No error surfaces because the consumer is functioning correctly by its own lights — it's retrying, as designed.

Fixes:

Alert on per-partition lag (max across partitions), never the group sum — this is the monitoring bug that allowed six hours to pass.
Express lag in seconds behind, not records, so "6 hours" is legible.
Add retry limits plus a DLQ so a single bad record parks itself instead of blocking the partition.
Alert on lag derivative, which would have fired within minutes of the partition stalling rather than when someone noticed downstream.

9. Clean up

docker rm -f kafka

Independent challenge

No solution given. You're on call for a Kafka platform serving 40 teams: ~800 topics, 12 brokers, ~9,000 partitions. Design the alerting.

Specify every alert with its threshold, severity and rationale, deliberately keeping the pageable set small enough that on-call is sustainable — then justify what you consciously chose not to alert on. Cover at minimum: lag, under-replication, controller health, rebalance rate, DLQ traffic and disk. Then solve the multi-tenant problem: with 800 topics you cannot hand-tune thresholds per consumer group, so design an approach that produces sensible alerts for both a group processing 100k records/sec and one processing 5.

Stuck? One hint

The multi-tenant threshold problem is the interesting half, and the answer is to alert on relative rather than absolute quantities: lag expressed in seconds-behind (which self-normalises across throughput), and lag derivative or lag-versus-its-own-baseline rather than a fixed record count. That way one rule covers both groups without per-team configuration. For keeping the pageable set small, the honest split is that only "data is being lost or will be soon" pages (offline partitions, controller count wrong, disk near full, sustained lag growth on a tier-1 topic), while everything else is a ticket or a dashboard — and saying explicitly that under-replication on a non-critical topic is a ticket, not a page, is exactly the kind of judgement the exercise is testing.

Common mistakes & troubleshooting

Alerting on lag in records instead of seconds. Meaningless across services with different throughput.
Alerting on group-total lag. One stuck partition disappears into the aggregate.
Not alerting on lag derivative. Growing lag is an incident long before it's a large number.
Retrying permanent errors forever. Blocks the partition indefinitely; classify errors and use a DLQ.
An unmonitored DLQ. Data goes there to be forgotten; alert on arrivals and keep a replay path.
unclean.leader.election.enable=true by default. Trades permanent data loss for availability, usually without anyone deciding to.
acks=all with min.insync.replicas=1. ISR can shrink to the leader and durability silently degrades (module 00).
Ignoring ISR shrink. It's the leading indicator before actual loss.
Massive partition counts "for headroom". They slow leader election and recovery, and consume broker memory and file handles.
Not monitoring ActiveControllerCount. Anything other than exactly 1 across the cluster is a serious problem.

Checkpoint quiz

Show questions

Why is consumer lag in records a poor alerting signal, and what should you use instead?
Why must lag be measured per partition rather than summed per group?
What is the ISR, and what does it mean for acks=all when it shrinks?
What does unclean.leader.election.enable=true trade away?
Why is partition count effectively a one-time decision on a keyed topic, and what does over-provisioning cost?
What is a poison pill, and why does an unbounded retry loop make it worse?
What must you record when writing to a DLQ, and why?
Which three broker metrics would you alert on, and what does each indicate?

Show answers

Because a record count means nothing without throughput — 4,500 records could be two seconds or three hours of work. Convert to time (lag / records_per_second) so the alert reflects how stale the data actually is, and alert additionally on the derivative, since sustained growth indicates consumption is slower than production and will never recover unaided.
Because a single stalled partition is invisible in a group total dominated by healthy ones — the sum barely moves while one partition falls hours behind. Since records are keyed to partitions, that also means a specific subset of keys (customers, accounts) is affected while everything else looks fine.
The in-sync replica set: the replicas currently caught up with the leader within replica.lag.time.max.ms. When it shrinks, acks=all waits only for the remaining members — so durability silently degrades, and in the worst case (min.insync.replicas=1) becomes equivalent to acks=1.
Data durability, in exchange for availability. It allows an out-of-sync replica to become leader, which permanently discards every record the new leader hadn't replicated. It should stay false unless there's an explicit decision that availability outweighs correctness for that topic.
Because partition assignment is hash(key) % partitions, so changing the count re-maps existing keys and breaks per-key ordering across the change (module 00) — making it a data migration rather than a scaling knob. Over-provisioning costs broker file handles, memory and replication traffic, and materially slows leader election and recovery after a broker failure.
A record that can never be processed successfully — malformed, schema- violating, or triggering a deterministic bug. An unbounded retry loop means the consumer never advances past its offset, so the entire partition stops: lag grows without limit and every subsequent valid record on that partition goes unprocessed, while the consumer reports no error because it is retrying as designed.
The original topic, partition and offset, the error/traceback, and a timestamp — enough to locate the record in its source, understand why it failed, and replay it deterministically once the bug is fixed. Without that context a DLQ is just a bucket of undiagnosable failures.
UnderReplicatedPartitions (should be 0 — durability is degraded above that), OfflinePartitionsCount (should be 0 — those partitions are unavailable for reads and writes), and ActiveControllerCount (must be exactly 1 across the cluster — 0 means no controller, more than 1 means split brain).