# Vulnerability Management as an Ongoing Practice ## Why this matters Track 11 module 01 made scanning a *gate*: a one-time check between build and push that stops a known-vulnerable image from shipping. That's necessary and still not enough, because the world doesn't hold still. An image you scanned clean, signed (module 02), and attested (module 03) on Monday can be *critically vulnerable* by Friday when a new CVE lands in a package it already contains — no rebuild, no change, and yet now exploitable. Vulnerability management is the *ongoing* discipline that catches that: continuous re-scanning, understanding where CVE data comes from, and — the hardest and most valuable skill — triaging findings by whether they're actually *exploitable in your context* rather than by CVSS number alone. This is what keeps a signed, provenance-rich pipeline from being confidently, verifiably shipping something that's now dangerous. ## Concepts ### The one-time-scan fallacy The gate from track 11 answers "is this image free of *known* CVEs *right now*?" But "known" is a moving target: the vulnerability was always in the code; it just wasn't *published* yet when you scanned. Three consequences: - **A passing gate has a shelf life.** Yesterday's clean scan says nothing about today's CVE feed. The image didn't change; the *knowledge* did. - **Already-deployed images are the real exposure.** The images running in prod right now were scanned when they shipped, possibly months ago. New CVEs land against *them*, and nothing re-checks them unless you build that in. - **You can't rebuild your way out fast enough.** With hundreds of images, "just rebuild everything when a CVE drops" isn't a plan — you need to know *which* images are affected before you can act (module 07's blast-radius question). The fix is to make scanning **continuous**, and to make it cheap by scanning the **stored SBOMs** (module 01) rather than re-pulling and re-analyzing every image. ### Continuous scanning against stored SBOMs Module 01 promised this payoff; here's the mechanism. Because an SBOM is a durable inventory bound to a digest, you can re-scan it any time the CVE feed updates, *without the image*: - **Scan the SBOM, not the image.** `grype sbom:app-sbom.json` re-evaluates a stored inventory against today's database. Fast, works offline, works for images no longer pulled anywhere. - **Schedule it.** A nightly job re-scans every stored SBOM against a freshly updated vulnerability DB and flags newly-affected images — this is the continuous half of the practice, the exact complement to track 11's build-time gate. - **Registry- and cluster-level scanners** do the managed version of this: Trivy Operator in-cluster, Microsoft Defender for Containers (track 11 module 01) re-scanning ACR on a schedule, GitHub Dependabot alerts on your source repos. Same idea — someone re-checks known artifacts as the feed changes. The stored SBOM is what makes "re-scan the whole fleet nightly" tractable instead of a fleet-wide rebuild. ### Where CVE data comes from (and why scanners disagree) A scanner is only as good as the databases it joins your SBOM against. Knowing the sources explains why two scanners give different answers for the same image: - **NVD (National Vulnerability Database)** — the U.S. government feed, the canonical source of CVE IDs and CVSS base scores. Authoritative but often *slow* to enrich and sometimes lacks precise affected-version ranges. - **Distro/ecosystem advisories** — Debian, Alpine, Red Hat, GitHub Advisory Database (GHSA), the Go vuln DB, etc. These are usually *more accurate* about which exact package versions are fixed, because the maintainers publish them. - **OSV (Open Source Vulnerabilities)** — an aggregated, machine-readable feed spanning many ecosystems (what Grype and others increasingly lean on). Scanners differ because they pull different combinations of these, update on different schedules, and match versions with different precision. That's why a finding on one tool and not another isn't necessarily a bug — it's a data-source difference, and it's a reason to understand triage rather than trust a single number. ### CVSS is a starting point, not a verdict — triage by exploitability The single most valuable skill in this module: **a CVSS score measures severity in the abstract, not risk in your context.** A CVSS 9.8 CRITICAL in a code path your image never executes is lower *real* risk than a CVSS 6.5 in your internet-facing auth flow. Triage means asking, per finding: - **Is the vulnerable code reachable/executed** in how you run the image? A library present but never called is far lower risk. - **Is there a fix available?** (`--ignore-unfixed` from track 11 — an unfixable CVE you can't remediate by rebuilding is triaged differently from a fixable one.) - **Is it actually being exploited in the wild?** This is where **KEV** and **EPSS** come in (next section) — known-exploited or high-probability CVEs jump the queue regardless of CVSS. - **What's the exposure?** Internet-facing vs internal-only, privileged vs sandboxed, reachable vs dead code. The output of triage is a *decision* per finding: fix now, fix on the next cycle, or accept-and-document. "Sort by CVSS desc and panic at the top" is how teams drown in noise and start ignoring the scanner entirely — the same "gate that can't be satisfied gets disabled" failure track 11 warned about, in triage form. ### KEV, EPSS, and VEX: triage tooling Three artifacts turn triage from vibes into signal: - **CISA KEV (Known Exploited Vulnerabilities)** — a curated list of CVEs *observed being exploited in the wild*. A KEV-listed CVE is a drop-everything signal even at modest CVSS; it means attackers are actually using it now. - **EPSS (Exploit Prediction Scoring System)** — a probability (0–1) that a CVE will be exploited in the next 30 days. It ranks the vast middle of "high CVSS, unclear if anyone cares" so you spend effort where exploitation is likely. - **VEX (Vulnerability Exploitability eXchange)** — a machine-readable statement from a vendor/producer saying "CVE-X is present in our product but **not exploitable** because [reason]" (or "affected, fix here"). VEX is how you *record and share* a triage decision so downstream consumers and your own future scans don't re-litigate the same false positive. It's the structured form of "we looked, it's not reachable." Together: KEV/EPSS tell you *what to prioritize*; VEX lets you *record what you already decided*, including justified false positives — so triage compounds instead of repeating. ### Handling false positives without turning off the alarm Scanners over-report: a package version matched too coarsely, a CVE that doesn't apply to how the package is compiled, a component present but unused. The wrong responses are (a) disabling the scanner and (b) blanket-ignoring by severity. The right response mirrors track 11's exemption discipline (module 04's narrow, reviewable exemptions): - **Suppress specific findings, explicitly and with a reason** — Trivy's `.trivyignore` / VEX, Grype's ignore rules — never a blanket "ignore all HIGH." - **Record *why*** (not reachable, false match, accepted risk with owner and expiry) so the suppression is auditable and revisited, not permanent amnesia. - **Keep the gate meaningful.** A gate drowning in un-triaged noise gets ignored; a gate that blocks only on triaged, actionable, exploitable findings stays trusted. Sustainability is a security property. ## Command reference | Command | What it does | Example | |---|---|---| | `grype ` | Scans an image for CVEs (Syft's sibling scanner) | `grype myapp:1.0` | | `grype sbom:` | Scans a **stored SBOM** — continuous scanning without the image | `grype sbom:app-sbom.json` | | `grype db update` | Refreshes the local vulnerability database | `grype db update` | | `grype --only-fixed` | Shows only findings with an available fix | `grype myapp:1.0 --only-fixed` | | `grype -o json` | Machine-readable output for triage pipelines | `grype myapp:1.0 -o json` | | `trivy image --vex ` | Applies a VEX document to suppress non-exploitable findings | see exercise 7 | | `trivy image --ignorefile .trivyignore ` | Applies explicit, reasoned suppressions | `trivy image --ignorefile .trivyignore myapp:1.0` | | `grype --fail-on critical` | Non-zero exit on findings at/above a severity (the CI gate) | `grype myapp:1.0 --fail-on critical` | Flag breakdown for `grype sbom:app-sbom.json --only-fixed --fail-on high -o json`: - `sbom:app-sbom.json` — scan a **stored SBOM** rather than an image; the source form is `sbom:`, `dir:`, or an image ref. This is the continuous-scan mode — cheap, offline, image-not-required. - `--only-fixed` — report only findings that have an upstream fix, so the output is *actionable* (the `--ignore-unfixed` idea from track 11, Grype's spelling). - `--fail-on high` — exit non-zero if any finding is HIGH or above, turning the re-scan into a gate a scheduled job can alert on. - `-o json` — structured output you can post-process (enrich with EPSS, filter by KEV, feed a ticketing system). Flag breakdown for `trivy image --vex vex.json --ignore-unfixed --severity HIGH,CRITICAL myapp:1.0`: - `--vex vex.json` — apply a VEX document so findings marked *not exploitable* are suppressed *with a recorded justification* (not blindly hidden). - `--ignore-unfixed` — drop findings with no available fix, keeping the gate achievable (track 11 module 01's sustainability lesson). - `--severity HIGH,CRITICAL` — the triage threshold you can justify; combine with VEX/ignore so what remains is exploitable *and* fixable. ## Hands-on exercises Grype and Trivy run locally, like track 11. You'll simulate the *passage of time* by scanning the same artifact against the database on different days' logic. 1. **(WSL2) Ensure Grype + a stored SBOM exist.** (Syft/Grype installed in module 01.) ```bash grype db update syft python:3.11-slim -o spdx-json=py-sbom.json grype sbom:py-sbom.json | head ``` Expect a CVE table produced from the SBOM alone — no image re-pull. This is continuous scanning's core move. 2. **(WSL2) See the image-vs-SBOM equivalence, and why SBOM scanning scales.** ```bash time grype python:3.11-slim >/dev/null time grype sbom:py-sbom.json >/dev/null ``` Expect the SBOM scan to be faster (no image fetch/analyze). Now imagine 500 images: re-scanning 500 stored SBOMs nightly is tractable; re-pulling 500 images is not. 3. **(WSL2) Filter to the actionable set — triage step one.** ```bash grype sbom:py-sbom.json --only-fixed | head grype sbom:py-sbom.json --only-fixed --fail-on critical; echo "gate exit: $?" ``` Note how many findings drop when you keep only fixable ones. The exit code is what a scheduled re-scan would alert on — the gate, applied continuously rather than once. 4. **(WSL2) Compare two scanners on the same image — data sources diverge.** ```bash grype python:3.11-slim -o json | grep -o '"id":"CVE-[0-9-]*"' | sort -u > grype-cves.txt trivy image --format json python:3.11-slim 2>/dev/null | grep -o 'CVE-[0-9-]*' | sort -u > trivy-cves.txt comm -3 grype-cves.txt trivy-cves.txt | head ``` Expect some CVEs found by one and not the other. This isn't a bug — it's the NVD/OSV/distro-advisory difference from Concepts. It's *why* you triage rather than trust a single tool's number. 5. **(WSL2) Triage by exploitability, not CVSS — reason it through.** Pick one HIGH/CRITICAL finding from exercise 1 and answer on paper: is the vulnerable package's code actually *reachable* in how you'd run this image? Is there a fix? Would you look it up in **CISA KEV** (exploited in the wild) or check its **EPSS** score before acting? Write the *decision* — fix now / next cycle / accept-and-document — and the justification. This reasoning *is* the module's core skill; do it by hand once. 6. **(WSL2) Prove the shelf-life problem.** Take a *specific, unchanged* older base and scan it, then note that the identical image scanned months apart can gain CRITICALs purely from new CVE publications: ```bash grype python:3.9-slim --only-fixed | grep -c -i critical ``` The image is frozen; the number rises over time as the feed grows. Write one sentence on why this means a build-time gate (track 11) *must* be paired with continuous re-scanning of already-shipped images. 7. **Diagnose and fix: a scanner drowning the gate in noise, resolved with VEX, not by disabling it.** A teammate set `--fail-on medium` with no suppression, the nightly gate now fails on 40 findings — including several the team has already confirmed *not exploitable* (present but unreachable) — and someone is about to just turn the gate off. Reproduce the noisy gate: ```bash grype sbom:py-sbom.json --fail-on medium; echo "noisy gate exit: $?" ``` Wrong fix: disable the scan or blanket-ignore all MEDIUM (removes the signal for everything). Right fix: triage, then *record* the decisions so they persist. Create a reasoned suppression and re-run: ```bash # A VEX/ignore document records WHY each is suppressed, with an owner: cat > .trivyignore <<'EOF' # CVE-XXXX-YYYY: vulnerable function not called in our usage; reviewed 2026-07, owner: platform, revisit: 2026-10 # (list the specific, triaged CVE IDs here — never a blanket severity) EOF trivy image --ignorefile .trivyignore --severity HIGH,CRITICAL --ignore-unfixed python:3.11-slim; echo "triaged gate exit: $?" ``` Diagnose: the gate wasn't *wrong to fire*, it was un-triaged and un-tuned — so it produced noise instead of signal, and an untrusted gate gets disabled (track 11's exact failure mode). The fix keeps the gate on but makes it block only on **triaged, exploitable, fixable** findings, with every suppression explicit, justified, owned, and set to be revisited. 8. **(WSL2) Clean up.** ```bash rm -f py-sbom.json grype-cves.txt trivy-cves.txt .trivyignore ``` ## Independent challenge No commands given — build it using this module plus module 01 (stored SBOMs), track 11 module 01 (the build-time gate), and track 10 (scheduling in CI). Design a *two-layer* vulnerability-management setup for a fleet of images: (1) the build-time gate (from track 11) that stops new vulnerable images shipping, and (2) a *scheduled continuous re-scan* of stored SBOMs that catches CVEs landing in already-signed, already-deployed images. Specify the threshold and how you keep the gate sustainable (unfixable handling, explicit VEX/ignore with owners and expiry), how you'd *prioritize* the re-scan output beyond CVSS (name the specific signals you'd use and why), and how a newly-affected already-deployed image flows from "the nightly re-scan flagged it" to "it's remediated." Finish by explaining, in two sentences, how this practice sets up module 07: when a re-scan reveals an image is affected, what does your stored SBOM let you answer instantly that you otherwise couldn't?
Stuck? One hint Layer 1 is track 11's `--exit-code`/`--fail-on` gate between build and push. Layer 2 is a nightly job that runs `grype sbom:` against a freshly-updated DB and alerts on new HIGH/CRITICAL — cheap because it scans SBOMs, not images. Prioritize with **CISA KEV** (exploited-in-the-wild → drop everything) and **EPSS** (probability of exploitation → rank the middle), not raw CVSS. Keep it sustainable with `--ignore-unfixed` plus a VEX/ignore file where every suppression has a reason, owner, and revisit date. For module 07: the stored SBOM answers "which of our images contain the affected package, at what version?" as a query, so the re-scan's finding immediately becomes a scoped blast-radius list instead of a fleet-wide rebuild.
## Common mistakes & troubleshooting - **Scanning once at build and calling it done.** A clean gate has a shelf life; new CVEs land against unchanged, already-deployed images. Pair the build-time gate with scheduled re-scans of stored SBOMs. - **Sorting by CVSS and panicking at the top.** CVSS is abstract severity, not contextual risk. A reachable CVSS 6.5 on your auth path can outrank an unreachable CVSS 9.8. Triage by exploitability, using KEV/EPSS. - **Trusting a single scanner's number as truth.** Scanners pull different feeds (NVD/OSV/distro) and match versions differently, so they legitimately disagree. Understand the data sources; don't treat one tool as ground truth. - **Disabling the scanner to escape the noise.** The track 11 failure mode: an un-triaged, un-tuned gate gets turned off. Suppress *specific* findings with a recorded reason (VEX/ignore), keep the gate blocking on triaged/exploitable findings only. - **Blanket-ignoring by severity.** "Ignore all MEDIUM" hides tomorrow's KEV MEDIUM too. Suppress named CVEs with justification, owner, and an expiry, not whole severities. - **Re-litigating the same false positive every scan.** Record triage decisions as VEX so downstream consumers and future scans inherit them — triage should compound, not repeat. - **Forgetting to update the vuln DB.** A re-scan against a stale database finds stale results. `grype db update` / Trivy's auto-update must run before the scan, or "continuous" is a fiction. ## Checkpoint quiz Write down your answer to each question before expanding it — checking without attempting first is the single easiest way to fool yourself into thinking you've learned this.
Show questions 1. Why does a passing build-time scan gate (track 11) have a "shelf life," and what practice addresses it? 2. How does scanning a stored SBOM (module 01) make continuous, fleet-wide re-scanning tractable? 3. Name three sources of CVE data and explain why two scanners can legitimately report different findings for the same image. 4. Why is a CVSS score a starting point rather than a verdict? Give a concrete case where a lower-CVSS finding is higher real risk. 5. What do KEV, EPSS, and VEX each contribute to triage? 6. Describe the *right* way to handle a false positive versus the two wrong ways. 7. When a nightly re-scan reveals an already-deployed image is newly affected, what does the stored SBOM let you answer immediately (previewing module 07)?
Show answers 1. Because "known CVEs" grows over time — a vulnerability was always in the code but gets *published* later, so an unchanged image scanned clean today can be critical tomorrow. Continuous re-scanning of stored SBOMs / deployed images (not just a one-time build gate) addresses it. 2. Because an SBOM is a durable inventory bound to a digest, you can re-scan it against today's DB *without the image* — fast, offline, and even for images no longer pulled anywhere. Re-scanning hundreds of SBOMs nightly is tractable where re-pulling hundreds of images is not. 3. **NVD** (canonical CVE IDs + CVSS, often slow/imprecise on versions), **distro/ecosystem advisories** (Debian/Alpine/RedHat/GHSA/Go — precise fix versions), and **OSV** (aggregated machine-readable feed). Scanners pull different combinations, update on different schedules, and match versions with different precision — so they disagree legitimately. 4. CVSS measures abstract severity, not risk in *your* context (reachability, exposure, exploitation-in-the-wild). Example: a reachable CVSS 6.5 in your internet-facing auth flow is higher real risk than a CVSS 9.8 in a code path your image never executes. 5. **KEV** lists CVEs *observed exploited in the wild* — a drop-everything signal regardless of CVSS. **EPSS** gives a probability of exploitation in the next 30 days — ranks the ambiguous middle. **VEX** is a machine-readable statement recording that a present CVE is (not) exploitable and why — so triage decisions persist and are shareable. 6. Right: suppress the *specific* finding with a recorded justification (VEX / ignore file), owner, and revisit date, keeping the gate on for everything else. Wrong: (a) disabling the scanner, and (b) blanket-ignoring by severity — both remove signal you need. 7. "Which of our images contain the affected package, and at what version?" — a query over stored SBOMs turns the finding into a scoped blast-radius list in minutes, instead of a fleet-wide rebuild-and-check (module 07's core move).
## Next Continue to [05-admission-control-enforcement-of-supply-chain-policy](../05-admission-control-enforcement-of-supply-chain-policy/README.md) — you've built and maintained the proofs (SBOM, signature, provenance); now make the cluster *refuse to run* anything that can't present them, extending track 11's Gatekeeper to verify signatures and attestations at admission.