Storage and Data Cost Optimization

Why this matters

Compute waste is loud — a big VM shows up at the top of every cost chart. Storage and data waste is quiet: an orphaned disk from a deleted cluster billing $40 a month for a year, a terabyte of logs sitting in premium storage that nobody has queried since Q1, a Log Analytics workspace ingesting and retaining data at full-price for two years because nobody changed the default. None of it screams; all of it adds up. This module is the cleanup and tiering half of the Optimize phase — including the exact orphaned-disk trap you were warned about in track 14 and the Log Analytics retention cost flagged in track 12, now met head-on with the sweep-and-tier discipline that catches them systematically instead of by accident.

Concepts

Orphaned resources: paying for things nothing uses (track 14)

An orphaned resource is one that still exists and still bills but is attached to nothing and serves no purpose. The canonical example — flagged directly in track 14 modules 01-02 — is an unattached managed disk: a StatefulSet's PVC used a disk with reclaimPolicy: Retain, the PVC/pod was deleted, and the disk was left behind, still provisioned, still billing, disconnected from any VM or node. Because it's not attached to anything, it never appears in a compute-utilization view; it just quietly accrues. The family of orphans is larger than disks: unattached public IPs (a static IP reserved after its load balancer was deleted), unused NAT gateways / load balancers, empty App Service plans, old snapshots, disks/NICs from deleted VMs, and idle databases. The defining trait is the same: provisioned but unattached/unused, so it evades utilization-based tools (module 03) entirely. Advisor catches some orphans, but the reliable way to find them is a deliberate sweep — an Azure Resource Graph query (track 17 module 06's estate-wide query tool) for resources in an unattached state. Finding and cleaning at least one real orphan is a hard capstone requirement precisely because everyone has one and nobody notices.

Blob storage tiers: hot, cool, cold, archive

Azure Blob Storage offers access tiers that trade retrieval cost/latency for storage cost, and using the wrong tier for the access pattern is pervasive waste:

• Hot — highest storage cost, lowest access cost. For data read/written frequently.
• Cool — lower storage cost, higher access cost, 30-day minimum. For data accessed occasionally (backups from the last month, recent logs).
• Cold — lower still, 90-day minimum, higher access cost. Infrequent access.
• Archive — cheapest storage by far, but offline: retrieval takes hours (rehydration) and costs the most. For compliance/retention data you're legally required to keep but expect never to read.

The lever is matching tier to access pattern, and doing it automatically via lifecycle management policies — rules like "move blobs to Cool after 30 days of no access, Archive after 90, delete after 365." Manually re-tiering doesn't scale; the lifecycle policy is the real tool. The trap runs both ways: leaving never-read data in Hot wastes storage cost, but archiving data you do read occasionally can cost more (early-deletion penalties + steep retrieval), so the tier has to match the actual pattern — the same "don't optimize on the average, look at the real behavior" discipline as rightsizing (module 03).

Managed disk tiers, and the "provisioned, not used" model

Managed disks (the VM/AKS block storage from tracks 07/14) also have performance tiers — Premium SSD (v1/v2), Standard SSD, Standard HDD, and Ultra — and here's the cost model that surprises people: for most disk types you pay for the provisioned size and tier, not for what you actually store or how much IOPS you use. A 1 TiB Premium SSD attached to a VM using 40 GiB bills for 1 TiB of Premium — full stop. So disk optimization has three moves: right-tier (a dev disk doesn't need Premium; Standard SSD/HDD is often fine), right-size (don't provision 1 TiB for a 40 GiB workload — though shrinking a disk is hard/risky, so size correctly up front), and — the biggest, cheapest win — delete orphans (the unattached disks above). Snapshots follow a similar "you pay for it sitting there" model and pile up from backup jobs (track 14 module 04) that never prune. The mental model that ties storage together: provisioned capacity bills whether or not you use it — the same trap as an over-provisioned VM, just quieter because storage rarely tops the chart.

Log Analytics and data-ingestion cost (track 12)

Observability has a cost tail that track 12 (module 04 logging, and the Container Insights/Log Analytics work in track 07 module 06) flagged and this module confronts. A Log Analytics workspace bills on two axes: ingestion (per GB of data you send in — the dominant cost, driven by how chatty your logs are) and retention (how long you keep it — an interactive retention window, often a default like 30-90 days included, then per-GB-per-month beyond it, plus cheaper long-term archive retention for compliance). The classic silent waste: a workspace left at a long default retention, ingesting verbose debug logs at full price, from a Container Insights integration nobody tuned. The levers: reduce ingestion (drop noisy log categories, sample, use Basic/Auxiliary table plans for high-volume low-value logs rather than the full Analytics plan), tune retention to what you actually query interactively vs. what you archive cheaply for compliance, and set a daily cap as a guardrail against a runaway log-flood blowing the bill. This is where observability (track 12) and cost meet: more telemetry is not free, and "log everything forever" is a budget decision disguised as an engineering default.

Sweep systematically: cleanup as a recurring practice, not a one-off

Every category here — orphans, wrong tiers, stale snapshots, over-retained logs — shares a failure mode: it accumulates silently and continuously, so a one-time cleanup fixes today and the waste is back in three months. The Operate-phase answer (module 00) is to make the sweep recurring: a scheduled Resource Graph query for orphans, lifecycle policies that tier/delete automatically, retention set once as policy, and — ideally — governance policy (track 17) that prevents some of it (e.g. deny disks above a size in dev, require lifecycle rules on new storage accounts). Cleanup you do by hand once is a chore; cleanup encoded as a recurring query + automatic policies is the discipline. Tie it back to tagging (track 17 module 06): an Owner tag on a resource is what lets a sweep say "this orphan belongs to team-payments, ask them before deleting" instead of the scary "is anyone using this?" — which is exactly why the taxonomy matters for cleanup, not just allocation.

Command reference

Resource Graph queries, tier reads, and Advisor are free. The disk/storage exercises create tiny resources — clean them up. Deleting an orphan stops a bill, so these commands save money.

Flag breakdown — estate-wide orphaned-disk sweep with Resource Graph (track 17 module 06's tool):

az graph query -q "Resources
| where type =~ 'microsoft.compute/disks'
| where isnull(properties.managedBy) or properties.diskState == 'Unattached'
| project name, resourceGroup, subscriptionId, sizeGb=properties.diskSizeGB, tags" -o table

• type =~ 'microsoft.compute/disks' — case-insensitive match on the disk resource type across every subscription you can see.
• isnull(properties.managedBy) or properties.diskState == 'Unattached' — the orphan test: a disk owned by no VM / in the Unattached state — the track-14 trap.
• project ... sizeGb, tags — pull the size (to estimate the ongoing bill) and the tags so you can see the Owner (track 17) before deleting — "whose is this?"

Flag breakdown — a Blob lifecycle policy that tiers then deletes automatically:

az storage account management-policy create --account-name <sa> -g <rg> --policy '{
  "rules": [{
    "name": "tier-then-expire",
    "enabled": true,
    "type": "Lifecycle",
    "definition": {
      "filters": {"blobTypes": ["blockBlob"]},
      "actions": {"baseBlob": {
        "tierToCool":    {"daysAfterModificationGreaterThan": 30},
        "tierToArchive": {"daysAfterModificationGreaterThan": 90},
        "delete":        {"daysAfterModificationGreaterThan": 365}
      }}
    }
  }]
}'

• tierToCool: 30 / tierToArchive: 90 — auto-demote blobs as they age, matching tier to a time-based access assumption (validate it against real access).
• delete: 365 — auto-expire after a year (respect any legal retention first).
• The policy runs automatically — the point is you set the pattern once and stop hand-tiering, the scalable version of az storage blob set-tier.

Hands-on exercises

Sweeps, tier reads, and retention changes are free. The disk/storage exercises create tiny billable resources and then delete them — deleting an orphan is the lesson. Remember the 8-24h cost lag when confirming a bill stopped.

1. Sweep your estate for orphaned disks (the track-14 trap, for real). Run the Resource Graph orphan query from the command reference (add the extension with az extension add --name resource-graph if needed). Every established Azure user has at least one. Write down each orphan's size and estimate its monthly cost — this is money you're spending on nothing.

2. Deliberately create an orphan, then find and kill it. Reproduce the track-14 trap end to end:

   az group create -n rg-storage-lab -l eastus
   az disk create -g rg-storage-lab -n orphan-demo --size-gb 32 --sku Standard_LRS
   # it's attached to no VM — it's born orphaned and already billing:
   az disk list -g rg-storage-lab --query "[?managedBy==null].{name:name, gb:diskSizeGb, state:diskState}" -o table
   az disk delete -g rg-storage-lab -n orphan-demo --yes   # stops the bill
   

   You just felt the whole lifecycle: a disk billing while attached to nothing, and the one command that stops it. This is what track 14's warning was about.

3. Sweep for unattached public IPs and other orphans. Run az network public-ip list --query "[?ipConfiguration==null].{name:name, rg:resourceGroup}" -o table. Extend the idea: write a Resource Graph query for another orphan class (old snapshots, or NICs with no VM). Note that each class needs its own "unattached" predicate — there's no single "show me waste" button, which is why the sweep is a practice.

4. Read and change Blob tiers. Create a storage account and a blob, inspect its tier, and demote it:

   SA=coststore$RANDOM
   az storage account create -n "$SA" -g rg-storage-lab --sku Standard_LRS
   az storage container create --account-name "$SA" -n data
   echo "cold data" > sample.txt
   az storage blob upload --account-name "$SA" -c data -f sample.txt -n sample.txt
   az storage blob show --account-name "$SA" -c data -n sample.txt --query "properties.blobTier"
   az storage blob set-tier --account-name "$SA" -c data -n sample.txt --tier Cool
   

   Compare Hot vs Cool vs Archive per-GB pricing (portal). Note Archive's retrieval is hours — write down when that's acceptable and when it's a trap.

5. Apply a lifecycle policy (automatic tiering). Attach the lifecycle policy from the command reference to $SA. This is the scalable version of exercise 4 — you set the aging rules once and stop hand-tiering. Confirm with az storage account management-policy show --account-name "$SA" -g rg-storage-lab.

6. Right-tier a disk. Note that a dev disk rarely needs Premium. Compare the monthly price of Premium_LRS vs StandardSSD_LRS vs Standard_LRS for a given size (portal or az disk list-skus if available). Write down the dev-disk default you'd standardize on — and how a governance policy (track 17) could deny Premium disks in dev automatically.

7. Tune Log Analytics retention and read ingestion (track 12). For a workspace you have (from track 07 module 06 / track 12), inspect and reduce retention:

   az monitor log-analytics workspace show -g <rg> -n <ws> --query "{retentionDays:retentionInDays, cap:workspaceCapping}" -o json
   az monitor log-analytics workspace update -g <rg> -n <ws> --retention-time 30
   

   Note the retention you started at (often a long default) and what dropping it to 30 days saves. Then reason about ingestion: which log source is chattiest, and whether a daily cap (--query workspaceCapping) is a sensible guardrail. This is track 12's "telemetry isn't free" made into a bill.

8. Estimate the ongoing cost of what you found. Combine exercises 1, 3, and 7: tally the monthly cost of every orphan, un-tiered blob set, and over-retained workspace you found. Write one number — "silent waste found: $X/month." That number is the elevator pitch for why the sweep is a recurring practice.

9. Diagnose and fix: the disk that's been quietly billing for weeks. The flagship storage incident. Scenario: Cost Analysis (module 01) shows a steady ~$X/month "Storage" line in an RG that has no running VMs. Investigate:

   # the RG bills for storage but has no VMs — where's the money going?
   az vm list -g <rg> -o table                       # empty or few
   az disk list -g <rg> --query "[?managedBy==null].{name:name, gb:diskSizeGb, created:timeCreated}" -o table
   

   Diagnose: an unattached disk (or several) left behind when a StatefulSet/VM was deleted with a Retain reclaim policy (track 14) — it's been billing every day since, invisible to every compute view because it's attached to nothing. Fix: confirm it's truly orphaned (check its timeCreated/Owner tag, ask the owning team via the track-17 Owner tag — don't delete a disk that's a detached data volume someone's about to reattach), snapshot it if unsure, then az disk delete --ids <id> --yes. Prevent recurrence: set the reclaim policy to Delete where appropriate (track 14) and add the orphan sweep to a schedule. Lesson: storage waste hides because it's decoupled from compute — only a deliberate sweep (or a governance policy) finds it.

10. Clean up.

    az group delete -n rg-storage-lab --yes --no-wait
    

    Keep the retention change on your real workspace if it's a genuine improvement. Confirm the lab RG is gone with az group show -n rg-storage-lab -o table.

Independent challenge

No commands given. Drawing on this module, the orphaned-disk lesson from track 14, the Log Analytics/observability cost from track 12, the Resource Graph query tool and Owner tag from track 17 module 06, and Cost Analysis from module 01, run a real storage-and-data waste audit on your subscription and produce a remediation plan. It must: sweep for at least three orphan classes (disks, public IPs, and one more) and total their monthly cost; identify at least one mis-tiered blob set or over-provisioned disk and propose the right tier with justification; inspect your Log Analytics retention and ingestion and propose a change with an estimated saving; and — the Operate-phase part — describe how you'd make this audit recurring (a scheduled Resource Graph query, lifecycle policies, retention-as-policy, and one governance policy from track 17 that would prevent a class of this waste). Actually clean up at least one real orphan you find (safely — check the Owner tag first). The deliverable is the audit findings, the dollar total, the remediation actions taken, and the recurring-prevention design.

Common mistakes & troubleshooting

• Assuming a compute view will surface storage waste. Orphans are attached to nothing, so utilization tools (module 03) and compute charts never show them. Only a deliberate sweep (Resource Graph) or Advisor finds them.
• Deleting a disk that isn't actually orphaned. A detached disk might be a data volume someone's about to reattach. Check timeCreated and the Owner tag (track 17), ask the owner, and snapshot if unsure — before az disk delete.
• Archiving data you actually read. Archive is cheapest to store but slow and expensive to retrieve, with early-deletion penalties. Tier to match the real access pattern; the wrong tier can cost more than Hot.
• Hand-tiering blobs instead of using lifecycle policies. Manual re-tiering doesn't scale and drifts. Encode the aging rules once as a lifecycle management policy and let it run.
• Forgetting disks bill on provisioned size/tier, not usage. A 1 TiB Premium disk storing 40 GiB bills for 1 TiB of Premium. Right-tier and right-size up front (shrinking is hard), and kill orphans.
• Leaving Log Analytics at default retention and full-price ingestion. Long default retention on verbose logs is silent money. Tune retention to what you interactively query, cut noisy ingestion, and set a daily cap (track 12).
• Treating cleanup as a one-time project. Every waste class re-accumulates. Make the sweep recurring and add governance policy (track 17) that prevents the common cases — cleanup encoded, not cleanup performed.

Checkpoint quiz

Write down your answer to each question before expanding it — checking without attempting first is the single easiest way to fool yourself into thinking you've learned this.

Next

Continue to 05-reservations-savings-plans-and-spot — you've cut waste by rightsizing (module 03) and sweeping storage (this module). The last big Optimize lever is pricing the compute you've decided you genuinely need: committing to it for a discount (Reservations, Savings Plans) or running interruptible workloads dirt-cheap (Spot) — and knowing which of the three fits a given workload.