Incident Response for a Supply-Chain Compromise

Why this matters

This track built prevention (signing, provenance, hygiene) and enforcement (admission control). Track 11 module 07 taught general incident response — isolate, preserve, rotate, root-cause. This module is the intersection: what you do the day a signed, verified, admitted image turns out to be built from a compromised dependency. That's the nightmare case, because every control you built said "yes" — the image was authentic, the provenance was valid, admission let it run — and it was malicious anyway, because the compromise was upstream of all of it (module 00's compromised-dependency attack). The good news: the exact artifacts this track produced — the SBOM, the signatures, the provenance, Rekor's log — are precisely what turn this from a blind, fleet-wide panic into a scoped, fast, evidence-driven response. This is the payoff the whole track was setting up.

Concepts

Why prevention passing doesn't mean you're safe

Every control in this track answers a question honestly, and none of them answers "is this dependency trustworthy?":

• The signature (02) proved the image is the authentic output of your CI — and it was. Your CI faithfully built and signed an image containing the poisoned dependency.
• The provenance (03) proved it was built by your workflow from your commit — and it was. The commit legitimately pulled the compromised package.
• Admission (05) verified all of that and admitted it — correctly.

The compromise entered before your trust boundary, so your integrity controls faithfully certified a malicious artifact. This is not a failure of signing; it's signing working as designed on a poisoned input. The lesson from module 00, now lived: integrity ≠ safety. Which means you also need a response plan for when a validly-signed thing is bad — that's this module.

The SBOM is your blast-radius map

The single most valuable incident artifact this track produced is the SBOM store (module 01), because the first question in any supply-chain incident is "where are we exposed?" — and the SBOM answers it as a query, not a fleet-wide investigation:

• Which images contain the compromised package, at which version? A search over stored SBOMs (pkg:npm/compromised-lib@1.2.3) returns the exact list — in minutes, across hundreds of images, including ones no longer in a running registry. Without SBOMs, this is a rebuild-and-grep of your entire fleet under time pressure.
• Which of those are actually deployed, and where? Cross-reference the affected-image list with what's running (the admission/deployment records).
• What's the transitive reach? The SBOM's dependency relationships show whether the bad package is a direct or transitive dependency, informing how many services inherit it.

This is exactly the payoff module 01 promised and module 04 reinforced: pre-computing the answer to the worst question you'll be asked under pressure. Your blast-radius assessment is a query because you did the SBOM work in advance.

The response sequence, specialized for supply chain

Track 11 module 07's ordering still governs — detect → preserve → isolate → eradicate → recover → root-cause — but each step has a supply-chain-specific form:

1. Detect — the trigger is usually external: an upstream advisory (the package is announced malicious), a KEV listing (module 04), or your continuous re-scan flagging it. You rarely find it yourself first.
2. Preserve evidence — snapshot affected Pods/images and their attestations and Rekor entries before you delete anything. The signatures and provenance are evidence of what was deployed and when, and Rekor is an immutable timeline (below). Track 11's "don't destroy the crime scene" rule applies to attestations too.
3. Isolate / contain — stop the bleeding: cordon/quarantine affected workloads (NetworkPolicy egress-deny from track 11, scale to zero), and — the supply-chain-specific move — tighten admission (module 05) to block the compromised digests so the bad image can't be re-scheduled anywhere.
4. Eradicate — remove the compromised dependency: pin to a safe version, bump via the module-06 automation, rebuild.
5. Recover — rebuild → SBOM → scan → re-sign → re-attest → deploy the clean image through the normal gated pipeline (this track, end to end).
6. Root-cause & prevent — how did it get in, and what stops the next one (below).

The through-line: your response uses this track's artifacts at every step — SBOM to scope, attestations/Rekor to preserve, admission to contain, the pipeline to recover.

Revocation and re-signing: what "revoke a signature" really means

A natural instinct is "revoke the signature on the bad image." Understand what that does and doesn't mean in the Sigstore model:

• There's no CRL-style revocation of a keyless signature — the signature legitimately attests the image was built by you, which remains true. You don't un-say a true statement.
• Instead you revoke at the policy layer: update admission (module 05) to deny the specific compromised digests (a denylist / an updated ClusterImagePolicy), and remove/repoint the tags so nothing pulls them. Trust is withdrawn by the verifier's policy, not by deleting the signature.
• Re-signing applies to the fixed image: the rebuilt, clean image gets a fresh signature and provenance through the normal pipeline. You don't "re-sign the bad one" — you sign its replacement and shift traffic.
• If a signing key (key-based mode, module 02) was compromised — a different, worse incident — you rotate the key and must re-sign every image that key signed, and update every verifier's trusted key. This is precisely the pain keyless avoids (no standing key to compromise), reinforcing module 02's argument in an incident.

So "revocation" in practice = policy-layer denial of specific digests + re-signing the fixed replacement, not cryptographic un-signing.

Rekor and provenance as the forensic timeline

The transparency log and attestations you built for prevention double as forensics:

• Rekor (module 02) is an immutable, append-only record of every signature and attestation, with timestamps. During an incident it answers "when was this image signed, and by which identity/workflow run?" — an auditable timeline you can't get from mutable registry tags.
• Provenance (module 03) pinpoints the exact build: which commit, which CI run, which workflow pulled the compromised dependency. That turns "sometime in the last three months something went wrong" into "the compromise entered at commit abc123, build run #4471, on this date" — scoping which images (by provenance) are affected and when the bad dependency first entered.
• Combined with the SBOM, you get the full picture: what is affected (SBOM), how it got built (provenance), and when it was signed/deployed (Rekor) — a complete, evidence-backed incident timeline built from artifacts you already had.

Root-cause and prevention then close the loop: add the compromised package to a denylist, tighten dependency review (module 06), and if the entry point was a build-step compromise rather than a dependency, that's a SLSA-level gap (module 00) — raise the build's isolation.

Command reference

Flag/field breakdown for a containment ClusterImagePolicy denylist (from exercise 5):

• A dedicated policy whose authorities cannot be satisfied for the named compromised digests (or a Kyverno rule matching those digests with validationFailureAction: Enforce) — the module-05 mechanism used in reverse, to block specific known-bad content rather than require good content.
• Scope it by digest, not tag — the compromised artifact is defined by its immutable content (module 00), and denying the tag alone is bypassable by pulling the digest directly.
• Keep it in enforce from the moment of containment (this is not the audit-first case — you have a confirmed bad artifact), but scope it narrowly to the specific digests so you don't halt unrelated deploys.

Hands-on exercises

Local and free — you'll run a tabletop-style response against real SBOMs and a local registry, exercising the scoping and containment muscles. (A full end-to-end incident is the capstone's implicit stress test.)

1. (WSL2) Build a small SBOM "store" to query. Generate SBOMs for a few images so you have a fleet to search:

   mkdir -p sboms
   for i in python:3.11-slim node:20-slim alpine:3.20; do
     syft "$i" -o spdx-json="sboms/$(echo $i | tr ':/' '__').json"
   done
   ls sboms/
   

   Expect several SBOM files. This is your pre-computed blast-radius index.

2. (WSL2) Scope a blast radius by query, not rebuild. Pretend zlib is announced compromised. Find every affected image without touching any image:

   grep -rl '"name":"zlib"' sboms/ || grep -rl 'zlib' sboms/
   grep -rh '"name":"zlib"' sboms/ | head
   

   Expect the exact list of SBOMs (images) that contain zlib, with versions. Note the speed: this is minutes over stored SBOMs vs a fleet-wide rebuild-and-grep. This is the module-01 payoff, in an incident.

3. (WSL2) Confirm exploitability, don't just match the name. Re-scan an affected SBOM to see whether the announced CVE actually applies at that version (module 04's triage, under incident pressure):

   grype db update
   grype sbom:sboms/python_3.11-slim.json | grep -i zlib || echo "no matching CVE at this version — triage as not-affected"
   

   Write one line: why does "contains the package" not automatically mean "affected"? (Version ranges, and whether the vulnerable code path is present — module 04.)

4. (WSL2) Preserve evidence before touching anything. Simulate capturing the attestations/timeline for an affected image before remediation:

   docker run -d -p 5000:5000 --name registry registry:2 2>/dev/null; true
   docker pull alpine:3.20 && docker tag alpine:3.20 localhost:5000/app:bad && docker push localhost:5000/app:bad
   COSIGN_PASSWORD="" cosign generate-key-pair
   cosign sign --key cosign.key --yes localhost:5000/app:bad
   syft localhost:5000/app:bad -o spdx-json=evidence-sbom.json
   cosign tree localhost:5000/app:bad     # record signatures/attestations as evidence
   

   The point: snapshot the SBOM, signatures, and (in real life) Rekor entries first — track 11 module 07's preserve-before-you-delete rule, applied to supply-chain artifacts.

5. (WSL2) Contain by denying the compromised digest at admission. Get the bad image's digest and write a containment policy (module 05 in reverse):

   BAD=$(crane digest localhost:5000/app:bad)
   echo "compromised digest: $BAD"
   cat > deny-digest.yaml <<EOF
   # Conceptual containment: an admission policy (Policy Controller/Kyverno)
   # that DENIES this specific digest cluster-wide, so it can't be re-scheduled:
   #   match image digest == ${BAD}  ->  reject
   # Scoped by DIGEST (immutable), not tag (bypassable). Enforce immediately —
   # this is a confirmed-bad artifact, not an audit-first rollout.
   EOF
   cat deny-digest.yaml
   

   Reason through why the denial is by digest, not tag: the compromised artifact is its content; denying :bad alone is bypassed by pulling the digest directly (module 00).

6. (WSL2) Recover: rebuild clean and re-sign the replacement. You don't un-sign the bad image — you sign its fixed successor through the normal pipeline:

   docker tag alpine:3.20 localhost:5000/app:fixed && docker push localhost:5000/app:fixed
   FIXED=$(crane digest localhost:5000/app:fixed)
   cosign sign --key cosign.key --yes localhost:5000/app@${FIXED#*@}
   syft localhost:5000/app:fixed -o spdx-json=fixed-sbom.json
   echo "shift traffic to the fixed digest; keep the deny on ${BAD}"
   

   Note the shape of recovery: the fixed image gets a fresh signature/SBOM; the bad digest stays denied. "Revocation" was a policy-layer denial plus re-signing the replacement — never cryptographic un-signing.

7. Diagnose and fix: a signed, admitted image that's malicious anyway. The scenario that names the module. Your continuous re-scan (module 04) flags a running, validly signed and admitted image because a dependency it contains was just disclosed as a backdoor. A teammate says "but it passed signature verification, so it must be fine." Diagnose the flaw in that reasoning and lay out the correct response in order, using this track's artifacts:

   # Reasoning check: signature/provenance prove AUTHENTICITY & BUILD, not SAFETY.
   # The compromise entered upstream of your trust boundary (module 00), so your
   # controls correctly certified a malicious artifact. Signing worked; it's the
   # wrong question for this failure.
   #
   # Correct order (track 11 module 07, supply-chain form):
   #   1. Detect  : re-scan/advisory already flagged it.
   #   2. Preserve: snapshot SBOM + attestations + Rekor entries (exercise 4).
   #   3. Scope   : grep stored SBOMs for the bad package -> affected image list (exercise 2).
   #   4. Isolate : cordon/scale-to-zero affected workloads; deny bad digests at admission (exercise 5).
   #   5. Eradicate: pin/bump the dependency to a safe version (module 06).
   #   6. Recover : rebuild -> SBOM -> scan -> re-sign -> re-attest -> deploy (exercise 6).
   #   7. Root-cause: how did the dep get in; add denylist + tighten review; raise SLSA level if the entry was a build compromise.
   echo "signature proves authenticity, not safety — respond, don't dismiss"
   

   The fix is both a mindset correction (a valid signature is not a clean bill of health) and the ordered response above, every step powered by an artifact this track produced.

8. (WSL2) Clean up.

   docker rm -f registry 2>/dev/null; true
   rm -rf sboms
   rm -f cosign.key cosign.pub evidence-sbom.json fixed-sbom.json deny-digest.yaml
   docker rmi localhost:5000/app:bad localhost:5000/app:fixed alpine:3.20 2>/dev/null; true
   

Independent challenge

No commands given — build it using this module plus track 11 module 07 (the response ordering and preserve-before-delete rule), module 01 (SBOM store), module 03 (provenance), and module 05 (admission). Write a supply-chain incident-response runbook for this exact scenario: "an upstream advisory announces that a package many of our images depend on was backdoored in versions 1.4.0–1.6.2; some affected images are signed, provenance-attested, and currently running in prod." The runbook must give the ordered steps (detect → preserve → scope/blast-radius → isolate/contain → eradicate → recover → root-cause), the exact artifact or command used at each step (which store you grep, what you snapshot, how you deny the digests at admission, how you re-sign the fix), who is responsible, and the check that tells you each step is done. Include a paragraph on what "revocation" concretely means here (policy-layer digest denial

• re-signing the replacement, not un-signing), and end by naming the single detection signal you'd add so the next compromised-dependency disclosure pages you automatically.

Common mistakes & troubleshooting

• Treating a valid signature as a clean bill of health. Signing/provenance prove authenticity and build integrity, not that a dependency is trustworthy. A signed, admitted image can be malicious if the compromise was upstream — respond, don't dismiss.
• Rebuilding the fleet to find the blast radius. That's the slow panic the SBOM store exists to prevent. Query stored SBOMs for the bad package/version first; rebuild only the images that actually contain it.
• Deleting the evidence first. Destroying Pods/images/attestations before snapshotting loses the forensic timeline (Rekor, provenance, SBOM). Preserve before you eradicate — track 11 module 07's rule, extended to attestations.
• "Contains the package" = "affected." Version ranges and reachability matter (module 04). Confirm the disclosed range and exploitability before treating an image as compromised, or you'll over-scope and over-react.
• Thinking you can "revoke" a keyless signature. You can't un-say a true attestation. Revocation = deny the compromised digests at the policy layer and re-sign the fixed replacement. (A compromised signing key is the worse case that forces re-signing everything — the pain keyless avoids.)
• Containing by tag instead of digest. Denying :bad is bypassable by pulling the digest. Contain the immutable digest (module 00).
• Skipping root-cause and prevention. If you don't determine how the dependency got in (and whether it was a build-step compromise → a SLSA-level gap, module 00), you'll respond to the same class of incident again. Close the loop: denylist, tighten review (module 06), raise the build's integrity level.

Checkpoint quiz

Write down your answer to each question before expanding it — checking without attempting first is the single easiest way to fool yourself into thinking you've learned this.

Next

Continue to 08-capstone-project — assemble the entire track into one working pipeline: build, SBOM, keyless sign, provenance attestation, and a cluster that admits the signed image and rejects the unsigned one, proving both paths for real.