06 - Profiling and Application-Level Performance

Why this matters

Module 05 taught you to find which resource saturates — CPU, a pool, a dependency. But when the answer is "the app pods are CPU-bound," you still don't know why: which function, which query, which line is burning the CPU. Autoscaling and bigger limits paper over inefficient code by throwing money at it — a request that does 100 database queries when it needs 2 will scale, but you'll pay for four idle-but-billed replicas to serve traffic one efficient replica could handle. Profiling is how you find the hot code path so you can fix the cause instead of renting around it. This module is a survey, not a deep course in any one language's profiler — the goal is to know what profiling answers, recognize the classic application- level performance bugs, and decide when to scale infra versus fix code.

Concepts

Load testing finds that it's slow; profiling finds why

These are complementary tools at different altitudes. A load test (k6, modules 01-04) is a black-box view from outside: it tells you the system's p95 is 800ms at 400 RPS and that pods are CPU-bound — symptoms, from the user's side. A profiler is a white-box view from inside the running process: it tells you that 60% of that CPU time is spent in serializeResponse(), or that one endpoint issues 340 SQL queries per request. You need both, in order: the load test tells you where to point the profiler (which endpoint, at what load), and the profiler tells you what to change. Profiling without a load test to reproduce the pressure often shows you a process at rest — you must profile under representative load, which is exactly what your k6 scripts now give you.

CPU profiles and flame graphs — where the time goes

A CPU profiler samples the call stack many times per second while the program runs; aggregating those samples shows which functions were on-CPU most often. The standard visualization is a flame graph: each box is a function, its width is the share of total CPU time spent in it (and its children), and the stack goes bottom-up (callers below, callees above). You read it by scanning for wide boxes near the top — a wide box high in the stack is a leaf function eating CPU directly; a wide box that's wide because of one child points you down into that child. The skill is resisting the urge to optimize a function just because it looks important; the flame graph's width is the only thing that says where the time actually is. Most languages have a sampling profiler (Go's pprof, Python's py-spy, Java's async-profiler, Node's --prof/clinic) that can emit a flame graph.

The N+1 query problem — the most common app-level performance bug

If there's one code-level bug this module wants you to recognize on sight, it's N+1 queries. You fetch a list of N items with one query, then loop over them and issue one more query per item to fetch a related record — N+1 total queries where 2 would do (one for the list, one to fetch all the related records at once). It's insidious because it works perfectly in development with 3 rows and falls over in production with 3,000: the endpoint's latency grows linearly with the data, and — critically — it's the exact bug that produces the connection-pool exhaustion from module 05 and track 14, because each request holds a connection far longer while it runs its hundreds of little queries. The tell in a profile or a query log: the same query shape executed hundreds of times per request, with different IDs. The fix is a code change — eager loading / a JOIN / a batched WHERE id IN (...) — not more replicas and not a bigger pool.

Memory profiles, allocation, and leaks

The soak test (module 00) finds that memory grows unbounded over hours; a memory profiler (heap profile) finds what is retained. Two distinct things to separate: allocation rate (how much garbage you create — high allocation means the garbage collector runs constantly, and GC pauses show up as latency spikes and stolen CPU) versus a true leak (memory that's allocated and never released because something still references it — a growing cache with no eviction, event listeners never removed, a growing global map). A heap profile taken at two points in a soak test, diffed, shows what grew between them — that diff is your leak suspect. This is why module 00 insisted soak tests are a separate question: a ten-minute load test never runs long enough for a slow leak or GC pressure to show.

Latency profiling: where a single request spends its time

CPU profiles aggregate across many requests; sometimes you need to know where one slow request's time went — and much of a web request's wall-clock time is spent not on CPU but waiting (on the database, a downstream API, a lock). This is where distributed tracing from track 12 module 05 becomes a performance tool: a trace breaks one request into spans — 5ms in the handler, 400ms waiting on a DB query, 200ms calling a downstream — showing you the wait between services that a CPU profile (which only sees on-CPU time) completely misses. The rule: use a CPU profiler for compute-bound hot paths (the app is burning CPU), and tracing for I/O-bound latency (the app is waiting) — which maps precisely onto the module-04 distinction between a CPU-HPA that fires and one that never does.

Infra scaling vs. code-level fixes — when to do which

The central judgment of this module. Scaling infrastructure (more replicas, bigger limits, a faster DB tier) and fixing code (killing an N+1, caching a hot computation, fixing a leak) both improve performance, but they trade differently. Scale infra when: the code is reasonably efficient and you genuinely have more work to do (real traffic growth) — throwing hardware at linear, honest load is correct and fast. Fix the code when: the work itself is wasteful — an N+1 doing 100× the queries it needs, an O(n²) loop, a missing cache recomputing the same thing per request, a leak. The tell that you have a code problem: efficiency-per-request is bad and gets worse with data size, so scaling only delays the wall and multiplies the cost. A useful frame from track 21 (FinOps): scaling infra to serve inefficient code is a recurring bill; the code fix is a one-time cost that lowers that bill permanently. You often do both — scale to survive now, fix the code so you can scale down later.

Profiling in production: continuous profiling and overhead

You can't always reproduce a production hot path in a test. Continuous profiling (Pyroscope, Parca, Grafana Phlare, or a cloud APM) runs a low-overhead sampling profiler in production all the time, so when an endpoint gets slow you have a flame graph from the moment it happened rather than trying to reproduce it. The key property is low overhead — sampling a few hundred times a second costs single-digit-percent CPU, cheap enough to leave on. This pairs naturally with track 12's observability stack (same Grafana pane): metrics tell you when it got slow, traces tell you which request path, and a continuous profile tells you which function — the three altitudes of "why is it slow" in one place.

Command reference

Profiling tools are language-specific; this is a survey of the common ones and what each produces. You don't need all of them — recognize the category.

Reading a flame graph — the rules:

Hands-on exercises

You need an app you can profile. Any of your earlier real apps works; where a specific bug is needed, the exercises describe how to induce it. Profiling is language-specific, so adapt the tool to your app's language — the method is what transfers.

1. Point the profiler where the load test told you

Run a k6 load test (module 03) against one endpoint of a real app until it's CPU-bound (kubectl top pods shows CPU near the limit). While the load runs, capture a CPU profile of one pod (e.g. kubectl exec + py-spy/pprof appropriate to your app). Expected: a profile taken under load, not at rest — confirm the process is actually busy. This is the module's core discipline: profile under representative load, aimed by the load test.

2. Read a flame graph and find the widest box

Render the profile as a flame graph (py-spy record -o flame.svg, go tool pprof -http, etc.). Expected: identify the single widest box near the top and name the function. Write down: is it a leaf (optimize directly) or wide because of one child (follow the child)? Practice resisting the box that looks important but is narrow.

3. Induce and detect an N+1 query

Take (or write) an endpoint that lists items and fetches a related record per item in a loop, backed by a Postgres (track 14). Enable query logging or pg_stat_statements, then hit the endpoint once:

-- in the DB
SELECT query, calls FROM pg_stat_statements ORDER BY calls DESC LIMIT 5;

Expected: one query shape with a calls count in the hundreds for a single request — the same SELECT ... WHERE id = $1 executed once per row. That repeated shape is the N+1. Note how latency for this endpoint grows as you add rows to the table.

4. Fix the N+1 and re-measure

Change the code to eager-load / JOIN / batch (WHERE id IN (...)), so the endpoint issues ~2 queries instead of N+1. Re-run the same k6 test and the same pg_stat_statements check. Expected: calls drops to a small constant, endpoint p95 falls sharply, and — connecting to module 05 — each request now holds a DB connection far briefly, relieving pool pressure. You fixed the cause; contrast with the alternative of scaling replicas, which would have kept all N+1 queries and multiplied the connection load.

5. Compare the two fixes head to head

For the pre-fix N+1 endpoint, measure the RPS you can serve within SLO with 1 replica. Then, without the code fix, scale to enough replicas to hit the same RPS within SLO and note the replica count. Then apply the code fix and re-measure the single-replica RPS. Expected: the code fix serves far more RPS per replica — quantify it (e.g. "the fix let 1 replica do what 4 replicas did before"). That ratio is the recurring bill (track 21) you avoid by fixing code instead of scaling around it.

6. Use a trace for a waiting bottleneck

For an I/O-bound endpoint (one that waits on a DB or downstream), a CPU profile will look nearly idle — the time is spent waiting, not computing. Capture an OpenTelemetry trace (track 12 module 05) of one slow request instead. Expected: the trace's spans show most of the wall-clock time in a waiting span (DB query, downstream call), which the CPU profile missed entirely. Lesson: CPU profiler for compute-bound, tracing for wait-bound — the same split as module 04's HPA-fires-or-not.

7. Catch a leak with a soak + heap diff

Run a soak test (module 00) — a long, moderate hold — against an app with a deliberately unbounded cache (or any app you suspect). Capture a heap profile early and again near the end, and diff them. Expected: the diff highlights a structure that grew over the run (the unbounded cache) — the leak, invisible in a short test. Note that memory climbing steadily over hours, not load, is the signature.

8. Diagnose and fix: "scaling didn't help" was a code problem all along

Take the N+1 endpoint from exercise 3, attach an HPA (module 04), and load it. Expected: it scales replicas, cost rises, and latency improves only modestly because every replica still does N+1 queries and they collectively pressure the DB/pool (module 05). Diagnose: the profile/query log shows the wasteful work is per request, so more replicas multiply the waste rather than removing it. Fix: apply the exercise-4 code fix, then re-run — latency recovers at a fraction of the replicas. Lesson: when efficiency-per-request is the problem, infra scaling delays and multiplies the cost; the code fix removes it.

9. Clean up

# remove any profiling sidecars/exec sessions; delete induced test tables;
# scale test deployments back to 1 and delete scratch HPAs
kubectl delete hpa <app> -n demo 2>/dev/null

Expected: profiling artifacts and scratch scaling removed; keep long-lived apps if later work reuses them.

Independent challenge

Take a real endpoint on one of your apps (track 07 capstone, a track-14 DB-backed service) and run the full "why is it slow" loop with no recipe given: use a k6 load test (modules 01/03) to reproduce the pressure and confirm it's compute-bound or wait-bound; point the right tool at it accordingly (a CPU profiler + flame graph for compute-bound, an OpenTelemetry trace from track 12 for wait-bound); identify one concrete application-level cause (a hot function, an N+1, a leak); make the code-level fix; and prove with a re-run that you serve materially more RPS-per-replica than before. Then argue explicitly, for this case, whether the right production response was to scale infra, fix the code, or both — grounding the argument in the recurring-cost framing from track 21. This pulls on modules 00-05 here plus track 12's tracing and track 14's databases.

Common mistakes & troubleshooting

• Profiling at rest. A profile of an idle process shows nothing useful. Always profile under representative load driven by your k6 test.
• Optimizing the box that looks important instead of the wide one. Flame- graph width is the only signal for where time actually goes; a scary-named narrow box is a distraction.
• Using a CPU profiler on a wait-bound problem. If the app is waiting on I/O, the CPU profile looks idle and misleads you — use tracing for wait time.
• Missing an N+1 because dev data is tiny. N+1 is invisible with 3 rows and fatal with 3,000; check query counts per request (pg_stat_statements), not just dev-time feel.
• Scaling around wasteful code. Adding replicas to an N+1 endpoint multiplies the wasted queries and the cost; it delays the wall instead of removing it.
• Confusing high allocation with a leak. High GC churn (allocation rate) and a true leak (unreleased retained memory) need different fixes; a heap diff over a soak test distinguishes them.
• Trying to reproduce a prod-only hot path locally. Sometimes you can't — that's what low-overhead continuous profiling in production is for.

Checkpoint quiz

Write down your answer to each question before expanding it — checking without attempting first is the single easiest way to fool yourself into thinking you've learned this.

1. In one sentence each, what does a load test tell you versus what a profiler tells you, and why do you need them in that order?
2. How do you read a flame graph — what does the width of a box mean, and what mistake does focusing on a scary-named narrow box represent?
3. Describe the N+1 query problem, how you'd confirm it in Postgres, and why it's invisible in development but fatal in production.
4. Why won't adding replicas fix an N+1 endpoint, and how does N+1 connect to the connection-pool bottleneck from module 05?
5. When is a CPU profiler the wrong tool, and what do you use instead — and how does that map onto whether a CPU-based HPA fires?
6. Distinguish high allocation rate from a true memory leak, and what test plus technique separates them.
7. Give the tell that a performance problem is a code problem (fix the code) rather than an honest capacity problem (scale the infra), with the cost framing from track 21.

Cumulative review

Closed book. Cover the answers and write each out before checking. These mix concepts from modules 00-06.

1. Put the tools in order for a "the app is slow under load" investigation: load test, profiler/trace, autoscaler config, bottleneck signatures. What does each contribute and why that order?
2. An endpoint's latency is fine with test data but terrible in production, and it grows with the number of rows returned. Name the most likely bug, how you'd confirm it, and why more replicas won't fix it.
3. You have a CPU-bound hot path and an I/O-bound slow path. Which diagnostic tool for each, and how does this map onto whether a CPU-based HPA will fire?
4. Translate this SLO-plus-finding into a decision: "SLO is p95<300ms; at peak we're at 900ms; the pods are CPU-bound; the profile shows 70% of CPU in a function that recomputes the same result every request." Scale, fix, or both, and why?
5. A load test shows a clean throughput knee at 500 RPS with all server-side metrics calm and the k6 host CPU pinned. What's your diagnosis and next step?
6. Why does an N+1 query bug connect the code-level material of this module to the connection-pool bottleneck of module 05 and the databases of track 14?
7. You need to prove an autoscaler handles peak load and that the app isn't wasting resources per request. Sketch the two-part test plan, naming the executor type and the profiling step.
8. Distinguish, with the signature of each, three ways a service can be "unhealthy" while a naive latency threshold stays green (drawing on modules 01, 03, and 05).
9. For a queue-driven worker (track 06 KEDA), you observe it scales but the queue still grows. Is this a scaling failure, a code-efficiency problem, or a downstream limit — and how would you tell which?
10. Give the FinOps framing (track 21) for choosing a one-time code fix over permanently scaled infrastructure, and one case where scaling infra is nonetheless the right call.

Next

07-performance-testing-in-cicd — everything so far has been run by hand; now automate a lightweight load test as a pipeline gate so a performance regression is caught before it ships.