07 - Performance Testing in CI/CD

Why this matters

Every load test so far, you ran by hand — which means performance is only checked when someone remembers to check it, and a regression sails to production between those moments. In track 10 you built pipelines that gate merges on unit tests, image scans, and deployments; a performance test belongs in that same gate. The goal is narrow and specific: catch a change that makes p95 materially worse before it ships, using a test small enough to run on every pipeline without slowing developers down or — the failure mode that kills these efforts — flaking so often that the team learns to ignore it. A perf gate everyone bypasses is worse than none, because it costs pipeline time and trains people to click "merge anyway."

Concepts

The CI perf gate is a different test from your big load test

The load tests in modules 03-05 ramp to real peak load, run for many minutes, and often need Azure Load Testing's engines. That test does not belong in a per-commit pipeline — it's too slow and too expensive to run on every push. The CI gate is a small, fast, deterministic test: modest load (tens of VUs, not thousands), a short duration (a minute or two), against a freshly-deployed build, whose only job is to answer "did this change regress performance relative to what we accept?" You keep both: the big periodic/pre-release load test that proves capacity and autoscaling (modules 04-05, often on a schedule or a release branch), and the lightweight per-commit gate that catches regressions early. Confusing the two — trying to run the full peak test on every commit — is how teams end up with a 40-minute pipeline nobody wants to trigger.

It's the same k6 script and the same thresholds — that's the payoff

Everything that makes k6 CI-friendly, you already built. The script is a plain file in your repo (module 01). Thresholds set the exit code — pass is 0, breach is non-zero (module 01) — which is exactly the pass/fail signal a CI step reads. -e BASE_URL=... (module 01 ex 7) points the same script at the ephemeral environment the pipeline just deployed. So wiring k6 into CI is barely more than adding a step that runs k6 run test.js and lets its exit code fail the job — no new tooling, no rewrite. This is why module 01 insisted on threshold-based gating and parameterized URLs from the start: it was building toward this exact step. The pipeline discipline from track 10 (a failing step blocks the merge) does the rest.

Where in the pipeline it runs: deploy to ephemeral, then test

A perf gate needs something running to test, so it slots after the build/deploy of a temporary environment and before promotion. The shape, building on track 10's stages: build the image → deploy it to an ephemeral namespace or a Container Apps revision (a throwaway, not production) → run the k6 gate against that deployment → tear the environment down → the k6 exit code decides whether the pipeline proceeds. This mirrors how track 10 ran integration tests against a deployed build rather than a mock. The ephemeral environment matters: you're testing the actual built artifact on infrastructure resembling production, not a developer's guess — and tearing it down keeps the perf gate from leaking cost (the track-21 discipline, and this track's cost warning).

Regression thresholds: absolute SLO vs. relative-to-baseline

Two ways to set the gate's pass/fail bar, and the choice matters. Absolute threshold: fail if p95 > 300ms — the SLO-derived bar from module 00. Simple, stable, and it's what you want for the contract ("we promise 300ms"). Relative / baseline threshold: fail if p95 is more than X% worse than the last known good run — catches a regression (a change that made things 40% slower) even while still technically inside the SLO, which is the early warning an absolute bar misses. The subtle trap: relative thresholds are more sensitive but also more flaky, because CI runners have noisy, variable performance (shared, throttled, cold). A common robust setup is a loose absolute gate on every commit (catch gross regressions cheaply and stably) plus a baseline comparison on a more controlled environment (a schedule, or a dedicated runner) where the noise is lower. Naming which you're using, and why, is half of making the gate trustworthy.

The real enemy: flakiness, and how to not create it

A perf gate that fails randomly — passing on retry with no code change — is poison: developers stop believing red means "you broke performance" and start reflexively re-running or bypassing, so a real regression slides through unnoticed. CI runners are the flakiness source: they're shared, CPU-throttled, have variable network, and start cold. Defenses, in order of importance: (1) gate on percentiles with headroom, not a razor's edge — if the SLO is 300ms and CI noise is ±50ms, gate at 400ms so noise alone can't fail it; (2) keep the gate's load modest and its duration long enough to average out noise (a 90s run is far less noisy than a 10s one); (3) include a brief warm-up and evaluate steady state (module 03) so cold-start latency doesn't fail the gate; (4) for baseline comparisons, use a less noisy environment than a generic shared runner. And — the module-00/module-01 rule that saves you from a different kind of false pass — always include an http_req_failed threshold, so a deploy that half-broke doesn't sail through on great latency for the few requests that worked. The target is a gate that goes red when and only when performance actually regressed.

What a failed gate should tell you

A red perf gate must be actionable, or people will bypass it out of frustration. The pipeline should surface which threshold failed and by how much (k6's summary already prints ✓/✗ per threshold), ideally publish the --summary-export JSON (module 01) as a build artifact so the numbers are inspectable, and point at the offending endpoint if you tagged them (module 05). A failure that just says "perf gate failed, exit 1" sends a developer digging; one that says "p95 on /checkout was 620ms vs. the 400ms gate, error rate fine" sends them straight to the cause. This is the CI equivalent of module 05's "don't just say it's slow, say where."

Command reference

k6 flags and patterns that matter specifically in CI:

A GitHub Actions step (building on track 10's workflows):

      - name: Deploy to ephemeral namespace
        run: ./scripts/deploy-ephemeral.sh "$GITHUB_SHA"   # your track-10 deploy, to a temp ns

      - name: Performance gate
        run: |
          k6 run --quiet --no-color \
            --summary-export summary.json \
            -e BASE_URL="https://pr-${GITHUB_SHA}.ephemeral.example" \
            tests/perf/gate.js
        # non-zero exit from a breached threshold fails this step -> blocks the merge

      - name: Publish k6 summary
        if: always()                     # publish even when the gate failed
        uses: actions/upload-artifact@v4
        with: { name: k6-summary, path: summary.json }

      - name: Tear down ephemeral namespace
        if: always()                     # always clean up, pass or fail (cost + hygiene)
        run: ./scripts/teardown-ephemeral.sh "$GITHUB_SHA"

Hands-on exercises

You need a repo with a pipeline (track 10) and a small app you can deploy to an ephemeral namespace on kind or AKS. A GitHub Actions workflow from track 10 module 01 is the base.

1. Write the lightweight gate script

Make a small test, distinct from your big load test — tens of VUs, ~60-90s, both thresholds, with headroom over the SLO:

// tests/perf/gate.js
import http from 'k6/http';
import { check, sleep } from 'k6';
export const options = {
  vus: 20,
  duration: '90s',
  thresholds: {
    // SLO is p95<300ms; gate at 400ms to absorb CI noise (see Concepts)
    http_req_duration: ['p(95)<400'],
    http_req_failed:   ['rate<0.02'],
  },
};
export default function () {
  const res = http.get(`${__ENV.BASE_URL}/`);
  check(res, { 'status 200': (r) => r.status === 200 });
  sleep(1);
}

k6 run -e BASE_URL=http://localhost:8080 tests/perf/gate.js ; echo "exit: $?"

Expected: runs in ~90s, both thresholds ✓, exit 0. This is your gate; note it deliberately runs modest load with a headroom bar.

2. Confirm the exit code gates

Point it at a slow endpoint (/delay/1) so p95 blows the 400ms bar:

k6 run -e BASE_URL=http://localhost:8080 tests/perf/gate.js ; echo "exit: $?"

Expected: http_req_duration ✗, exit non-zero. In CI, that non-zero exit is what blocks the merge — verify the number yourself before trusting a pipeline to read it.

3. Wire it into the pipeline after an ephemeral deploy

Add the perf-gate step to your track-10 workflow after the step that deploys the build to a temporary namespace/revision and before any promotion, using the YAML from the command reference. Push a branch and open a PR. Expected: the Actions run deploys the build, runs k6, and the job's status reflects the k6 exit code. Confirm a normal build passes the gate.

4. Catch a real regression before it ships

On a branch, introduce a deliberate performance regression in the app — add a sleep(300ms) to the hot endpoint, or reintroduce an N+1 (module 06) — commit, and open a PR. Expected: the pipeline deploys the regressed build, runs the gate, and fails the check, blocking the merge — the regression is caught in CI instead of production. This is the entire value proposition, demonstrated end to end. Revert and confirm the gate goes green again.

5. Make the failure actionable

Confirm your workflow uploads summary.json with if: always() and add per-endpoint tags (module 05) to the gate script. Re-trigger the failing run from exercise 4. Expected: the build artifacts include the k6 summary, and the failing threshold names which endpoint and by how much — a developer can act without re-running anything locally. Compare to a bare "exit 1": one sends them to the cause, the other sends them digging.

6. Deliberately create — then fix — flakiness

Set a razor-thin, no-headroom bar and a very short run to induce flakiness:

export const options = { vus: 20, duration: '10s',
  thresholds: { http_req_duration: ['p(95)<205'] } };  // SLO is 200ms, ~no margin, 10s run

Run it in CI (or on a loaded laptop) several times. Expected: it fails sometimes with no code change — pure CI/host noise crossing a hairline bar over too-short a window. Diagnose: the gate has no headroom over the SLO and too short a run to average noise. Fix: widen the bar to absorb the measured noise (e.g. p(95)<300), lengthen the run to ~90s, and add a brief warm-up. Re-run several times and confirm it's now stably green on unchanged code. Lesson: a flaky gate trains people to bypass it — headroom and duration are what make red mean "you regressed," not "unlucky runner."

7. Diagnose and fix: the gate passes a half-broken deploy

Configure the gate with only a latency threshold (delete the http_req_failed line) and deploy a build where the endpoint returns fast 500s (module 01 ex 9 / module 06). Expected: the gate passes — the errors are fast, latency is great, exit 0 — and a broken build ships. Diagnose: a latency-only gate blesses fast failures; there's no error-rate check. Fix: restore http_req_failed: ['rate<0.02'] and re-run — now the broken deploy fails the gate. Lesson carried from modules 01 and 06: a perf gate without an error-rate threshold has a hole a broken deploy walks straight through.

8. Separate the gate from the big load test

Keep exercise 1's gate on every PR, and set up the full peak load test (modules 04-05, via Azure Load Testing) to run on a schedule (nightly) or on the release branch only — not per commit. Expected: two distinct jobs — a fast per-commit gate and a slow periodic capacity test. Confirm the per-commit pipeline stays fast (the gate adds ~90s, not 40 minutes). This is the structural answer to "why not just run the big test every time."

9. Clean up

# ensure the pipeline's teardown step ran; manually remove any leaked ephemeral ns:
kubectl get ns | grep ephemeral   # should be empty after a clean run
# delete any scheduled ALT resource if you created one for exercise 8

Expected: no ephemeral namespaces or load-testing resources left billing. Verify the teardown step ran if: always() even on the failed runs from exercises 4 and 6.

Independent challenge

Add a performance gate to a real pipeline from track 10: write a lightweight k6 gate script (distinct from your big load test — modest load, ~90s, both thresholds with SLO-derived numbers plus noise headroom), wire it into the workflow so it runs against an ephemeral deployment of the build and its exit code blocks the merge, publish the k6 summary as a build artifact so a failure is actionable, and ensure the ephemeral environment is always torn down. Then prove it works in both directions: push a change that regresses the hot path and show the gate red-blocks the merge, and push a normal change and show it passes stably over several runs (not flakily). State explicitly which threshold style you used (absolute SLO vs. baseline) and how you chose the headroom to avoid flakiness. This draws on track 10 (the pipeline), modules 00-01 (thresholds/exit codes), module 03 (warm-up/steady state), and module 06 (the regression you introduce).

Common mistakes & troubleshooting

• Running the full peak load test on every commit. Too slow and too expensive; developers stop triggering the pipeline. The per-commit gate is a small, fast test; the big test runs on a schedule/release branch.
• A razor-thin threshold with no headroom. CI runners are noisy; a bar at the exact SLO fails randomly. Gate above the SLO by the measured noise margin.
• Too-short a gate run. A 10s run is dominated by noise and cold start; ~90s averages it out. Include a brief warm-up and evaluate steady state.
• A flaky gate that people bypass. The worst outcome — it trains the team to ignore red, so a real regression slips through. Fix the flakiness; don't disable the gate.
• No error-rate threshold. A latency-only gate passes a deploy returning fast errors. Always pair http_req_duration with http_req_failed.
• Testing production, or leaking the ephemeral env. Gate against a throwaway deployment of the build, and tear it down if: always() — a leaked env is a cost and a hygiene problem.
• An unactionable failure. "exit 1" with no numbers sends developers digging. Publish the summary and tag endpoints so red says what and where.
• Confusing absolute and baseline thresholds. Know which you're using: absolute checks the SLO contract; baseline catches regressions still inside the SLO but is more flake-prone — often use a stable absolute gate per commit and baseline comparison in a controlled environment.

Checkpoint quiz

Write down your answer to each question before expanding it — checking without attempting first is the single easiest way to fool yourself into thinking you've learned this.

1. How is the CI perf gate different from the big load test from modules 04-05, and why don't you run the big one on every commit?
2. What single property of k6 (built in module 01) makes wiring it into a pipeline almost trivial?
3. Where in the pipeline does the gate run relative to build/deploy/promote, and why does it need an ephemeral environment?
4. Contrast an absolute (SLO) threshold with a baseline (relative) threshold — what does each catch, and which is more prone to flakiness?
5. Name three concrete defenses against a flaky perf gate, and why flakiness is specifically dangerous (not just annoying).
6. Why must a CI perf gate include an error-rate threshold, not just a latency one?
7. What makes a failed gate actionable, and why does that matter for whether the gate survives?
8. Give the two-tier structure that lets you both catch regressions per-commit and validate real capacity — without a 40-minute pipeline.

Next

08-capstone-project — put it all together: a realistic load test on a real cluster, proof that autoscaling handles it via Grafana, a real bottleneck found and fixed, and a performance gate in CI.