05 - Identifying Bottlenecks

Why this matters

Module 04 ended on its third outcome: the app scaled to maxReplicas and latency still didn't recover. When adding replicas doesn't help, the bottleneck is somewhere the autoscaler can't reach — a CPU/memory limit throttling each pod, the database connection pool from track 14 exhausting, or a downstream dependency's rate limit shedding your requests. A load test that only reports "p95 got bad" is a smoke alarm with no map. This module is about reading the shape of a failure to locate where the system actually breaks — because you can't fix a bottleneck you've misdiagnosed, and "add more replicas" is the wrong fix for most real bottlenecks.

Concepts

A bottleneck is the first resource to saturate, and it moves

Every system has exactly one bottleneck at a time: the single resource that saturates first as load rises. Everything downstream of it is starved; everything upstream backs up behind it. Fix that one, and the bottleneck moves — it doesn't disappear, it relocates to the next-tightest resource. This is why performance work is iterative: you find the current limit, remove it, re-test, and find the new limit, which was invisible before because the first one was hiding it. The load-test skill is reading which resource is currently the wall. The tell is almost always a knee in the graphs: up to some RPS latency is flat and throughput scales linearly; past the knee, latency climbs steeply while throughput flattens (or drops). The knee is the bottleneck announcing itself; the job is identifying which resource caused it.

Latency vs. throughput curves — reading the knee

Plot latency (p95) and throughput (successful RPS) against offered load. Three signatures tell you three different things:

• Latency climbs, throughput flat, errors low — a queue is forming behind a saturated resource (CPU, a pool, a slow dependency). Requests aren't failing, they're waiting. Classic connection-pool or CPU saturation.
• Throughput flat, errors climbing — something is actively rejecting load: a rate limit, a full queue, connection refusals. The system chose to shed rather than queue.
• Throughput drops under increasing load (a "retrograde" curve) — the system is spending resources on overhead (thrashing, GC, retry storms, connection churn) instead of work. The worst signature: pushing harder makes it do less. Often a sign of a missing backpressure mechanism.

You already have the tools to draw these: k6's summary gives you throughput and percentiles per run, and running a ramping-arrival-rate test (module 03) sweeps offered load so a single run traces the curve.

CPU and memory limits: the pod-level ceiling

The first suspect, because it's the one autoscaling should have handled. Two distinct failures. CPU throttling: a pod with a CPU limit (track 03 module 02) gets throttled by the kernel (CFS) when it exceeds the limit — latency rises sharply, but the pod's CPU graph plateaus at the limit, not at the node's capacity. The tell is container_cpu_cfs_throttled_periods_total climbing while CPU sits pinned at the limit. Memory: a pod exceeding its memory limit is OOM-killed — you'll see restarts (kubectl get pods RESTARTS climbing, OOMKilled in kubectl describe pod), and latency spikes as requests in flight die with the pod. Crucially, if the limit is the wall, more replicas help (spread the load) but bigger requests/limits per pod may be the real fix — this is the HPA-vs-VPA distinction from track 03 module 09 showing up as a diagnostic question: are you compute-bound per-pod, or just need more pods?

Database connection pool exhaustion (the track-14 bottleneck)

The most common bottleneck that looks like the app and is the database boundary. Your app holds a connection pool — a fixed number of reusable DB connections (track 14 module 07). Each in-flight request that touches the DB borrows one; when all are in use, further requests wait for one to free up. Under load, the pool saturates: request latency climbs not because the DB is slow but because requests are queued waiting for a connection. The signature is diagnostic gold — app latency high, but the database's own CPU and query latency are calm. The requests aren't slow at the DB; they're slow getting to it. And here's the cruel twist that ties back to module 04: adding app replicas makes it worse — more replicas means more pools means more total connections hammering a database that has its own max_connections limit, so you scale the app and either exhaust the DB's connection limit or just move the queue. The fixes are pool tuning, a connection proxy/pooler (PgBouncer), or addressing why requests hold connections so long — never "more app pods."

Downstream dependency limits: rate limits and quotas

Your service is rarely the whole system. It calls a payment provider, a third-party API, an internal service fronted by APIM with a rate-limit policy, or an Azure service with a documented throughput quota. Under load you hit their ceiling, and it manifests as a wall you don't control: a burst of 429 Too Many Requests or 503s, latency spikes from their throttling, or timeouts. The signature: errors concentrated on requests that touch one specific downstream, while requests that don't touch it stay healthy — which is exactly why the module-03 traffic mix matters, because a uniform test that never hits the checkout path would never find the payment provider's limit. Scaling your app does nothing here (you'll just hit their limit faster); the fixes are caching, request coalescing, backpressure, a higher quota, or a circuit breaker so their limit degrades you gracefully instead of cascading.

The false bottleneck: when the test is the limit

Before you trust any ceiling, rule out that you found the generator's limit, not the system's — the module-00 warning, now a formal diagnostic step. If the box running k6 is CPU-pinned, or its network uplink is saturated, latency you attribute to the server is really queueing in your own client. The signatures of a false bottleneck: the "ceiling" RPS is suspiciously round and stable regardless of what you change server-side; server-side metrics (pod CPU, DB, dependencies) are all calm at the "limit"; and the k6 client's own host shows high CPU or network saturation. This is why serious tests run from Azure Load Testing (module 02) with enough engines that the generator has headroom — and why the diagnostic discipline is always: before blaming the server, prove the client had room to push harder.

The method: change one thing, re-test, watch the bottleneck move

Diagnosis isn't guessing — it's controlled experiment. When the knee appears, you form a hypothesis from the signature (queueing vs. rejecting vs. retrograde; which resource's graph is pinned), change one thing to test it (raise the pool size; raise the CPU limit; add a cache in front of the dependency), and re-run the identical load test. Two outcomes, both informative: latency knee moves to a higher RPS (you found and moved the real bottleneck — now find the next one), or nothing changes (your hypothesis was wrong; the real bottleneck is elsewhere — revert and try the next suspect). Changing several things at once destroys this — you learn nothing about which change mattered. This is the same discipline as track 03/06's diagnose-and-fix exercises, applied to performance: reproduce, isolate, verify.

Command reference

Hands-on exercises

Reuse the kind cluster and k6 from earlier modules. For the DB exercise, a small Postgres from track 14 plus any app that queries it works; httpbin stands in where you just need latency shapes.

1. Trace the latency/throughput curve and find the knee

Run a single ramping-arrival-rate test that sweeps offered load wide:

// sweep.js
import http from 'k6/http';
export const options = {
  scenarios: { sweep: {
    executor: 'ramping-arrival-rate', startRate: 10, timeUnit: '1s',
    preAllocatedVUs: 50, maxVUs: 500,
    stages: [
      { target: 50,  duration: '1m' },
      { target: 150, duration: '1m' },
      { target: 400, duration: '1m' },
      { target: 800, duration: '1m' },
    ],
  }},
};
export default function () { http.get(`${__ENV.BASE_URL}/`); }

k6 run -e BASE_URL=http://localhost:8080 --out json=sweep.json sweep.js

Expected: watch the live p95 — at some offered rate it stops being flat and climbs steeply. That inflection is the knee; note the RPS. The rest of the module is identifying what caused it.

2. Reproduce a CPU-limit knee

Deploy a CPU-bound app with a tight CPU limit and load it past the knee:

kubectl create deployment cpubound --image=hashicorp/http-echo -n demo -- -listen=:5678 -text=ok
kubectl set resources deployment cpubound -n demo --requests=cpu=50m --limits=cpu=100m
kubectl expose deployment cpubound --port=80 --target-port=5678 -n demo
kubectl port-forward -n demo svc/cpubound 8080:80 &
k6 run -e BASE_URL=http://localhost:8080 sweep.js

Alongside: kubectl top pods -l app=cpubound -n demo. Expected: as latency knees up, pod CPU plateaus at ~100m (the limit), not higher — the tell of CPU throttling, not node exhaustion. On Grafana, container_cpu_cfs_throttled_periods_total climbs. Diagnosis: per-pod CPU limit is the wall.

3. Change one thing, watch the knee move

Raise only the CPU limit and re-run the identical test:

kubectl set resources deployment cpubound -n demo --limits=cpu=500m
k6 run -e BASE_URL=http://localhost:8080 sweep.js

Expected: the knee moves to a higher RPS — you confirmed CPU limit was the bottleneck by moving it. Note you changed exactly one variable; that's what makes the conclusion valid.

4. Reproduce and read connection-pool exhaustion

Point an app at a Postgres (track 14) with a small pool (e.g. app pool size 5), and load it with a DB-touching endpoint. Expected: app p95 climbs steeply, but on the DB side SELECT count(*) FROM pg_stat_activity; shows connections pinned at the pool size and the DB's own CPU/query latency stays low.

Diagnose: the requests aren't slow at the database — they're queued waiting for a connection. The signature is "app latency high, DB calm." Prove it: kubectl logs for pool-timeout / "waiting for connection" messages.

5. Diagnose and fix: scaling app replicas makes the pool worse

With the pool bottleneck from exercise 4 active, attach an HPA and let it scale the app under load (the naive fix from module 04):

kubectl autoscale deployment <app> -n demo --min=2 --max=8 --cpu-percent=50
k6 run -e BASE_URL=http://localhost:8080 sweep.js

Expected — the lesson: latency does not improve, and may worsen — each new replica opens its own pool, so total connections to the DB rise, and you either hit the DB's max_connections (new errors appear) or just multiply the queues. Fix: don't scale app replicas; instead raise the pool sensibly and/or put a connection pooler (PgBouncer) between app and DB (track 14), then re-run. Confirm latency recovers without more app pods. Lesson: a connection-pool bottleneck is immune to app-replica scaling — sometimes made worse by it.

6. Reproduce a downstream rate-limit wall

Front a dependency with a rate limit (an APIM policy from track 19, or simulate with an app that returns 429 above N RPS), and drive a mixed load where only one endpoint calls it. Expected: errors concentrate on the requests that touch the limited dependency (visible via per-URL k6 tags), while other endpoints stay healthy — throughput flattens and http_req_failed climbs with 429s. Diagnosis: a downstream limit, not your app. Note that a uniform test not hitting that path would have missed it entirely — module 03 vindicated.

7. Diagnose and fix the false bottleneck (the test is the limit)

Deliberately cripple the generator: run k6 with far more VUs than a tiny box can drive, or cpulimit/a busy loop stealing CPU on the k6 host, and watch it "plateau."

# run k6 while something else pegs the client CPU
yes > /dev/null &   # burn a client core
k6 run -e BASE_URL=http://localhost:8080 sweep.js
top   # observe the k6 host CPU pinned
kill %1  # stop the CPU burner

Expected: the "ceiling" RPS appears, but all server-side metrics are calm (pod CPU low, DB idle) while the k6 host is CPU-pinned. Diagnose: the limit is the generator, not the system. Fix: free the client (kill the burner), or move to Azure Load Testing with more engines (module 02), and re-run — the "ceiling" rises. Rule locked in: prove the client had headroom before blaming the server.

8. Localize with per-endpoint tags

Add name tags so k6 reports metrics per logical endpoint:

http.get(`${__ENV.BASE_URL}/cheap`,  { tags: { name: 'cheap' } });
http.get(`${__ENV.BASE_URL}/expensive`, { tags: { name: 'expensive' } });

Run a mixed test. Expected: the summary breaks latency/errors down by name, so you can point at which endpoint carries the knee — the difference between "the app is slow" and "the /expensive path is slow," which is the difference between a useless and an actionable finding.

9. Clean up

kubectl delete deployment cpubound <app> -n demo 2>/dev/null
kubectl delete svc cpubound <app> -n demo 2>/dev/null
kubectl delete hpa <app> -n demo 2>/dev/null
# stop port-forwards and any background CPU burner; delete the track-14 Postgres if you spun one up

Expected: scratch workloads removed.

Independent challenge

Take a real app that talks to a database (a track-14 setup, or your track-07 capstone app) and run a ramping load test that drives it past its knee. Without guessing, diagnose which resource is the bottleneck by reading the signatures in this module — is it per-pod CPU/memory limits, the connection pool, or a downstream dependency? — and prove your diagnosis by changing exactly one thing and showing the knee move (or not). Then demonstrate the trap from module 04's third outcome: show that scaling app replicas does not fix a connection- pool bottleneck (and explain, from the DB's max_connections, why it can make it worse), and propose the correct fix. Finally, before trusting any ceiling you report, show you ruled out the false bottleneck by confirming the k6 host had CPU/network headroom. This draws on track 14 (pools), track 03 (limits), track 19 (rate limits), and modules 00-04 of this track.

Common mistakes & troubleshooting

• Reporting "it's slow" without locating where. A p95 number with no resource attribution is a smoke alarm with no map. Read which graph is pinned.
• "Add more replicas" as a reflex. For CPU-per-pot limits it may help; for a connection pool it makes things worse; for a downstream rate limit it does nothing. Match the fix to the bottleneck.
• Confusing a CPU limit wall with node exhaustion. CPU pinned at the limit with CFS throttling is a per-pod limit problem (raise limit/VPA), not a "need more nodes" problem.
• Missing pool exhaustion because you only watched the app. The signature is app-latency-high / DB-calm — you must look at the database side to see it.
• Uniform test that never hits the limited path. A downstream rate limit or an expensive endpoint's bottleneck is invisible if your test doesn't model the traffic mix (module 03).
• Trusting a ceiling without checking the generator. The false bottleneck: all server metrics calm, k6 host pinned. Always rule out the client first.
• Changing multiple things between runs. You lose the ability to attribute the change. One variable per re-test.

Checkpoint quiz

Write down your answer to each question before expanding it — checking without attempting first is the single easiest way to fool yourself into thinking you've learned this.

1. What is a bottleneck, and what happens to it when you fix it?
2. Name the three latency/throughput-curve signatures and what each implies (queueing vs. rejecting vs. retrograde).
3. What's the diagnostic signature of database connection-pool exhaustion, and why is "app latency high, DB calm" the giveaway?
4. Why can scaling app replicas make a connection-pool bottleneck worse, and what's the correct fix?
5. How do you tell a per-pod CPU limit wall apart from simply needing more nodes?
6. A downstream dependency's rate limit is the bottleneck. Why would a uniform load test miss it, and what does the failure look like in the results?
7. List the three signatures of a false bottleneck (the test itself being the limit) and the one thing you always check before trusting a ceiling.
8. What is the disciplined method for confirming a bottleneck diagnosis, and why must you change only one thing per re-test?

Next

06-profiling-and-application-performance — when the bottleneck is inside the code (a hot path, an N+1 query) rather than an infra limit, load tests tell you that it's slow; profiling tells you which line.