06 - Storage: PersistentVolumes and PersistentVolumeClaims

Why this matters

You've used emptyDir volumes already (module 02's sidecar exercise) to share files between containers within a Pod — but that storage dies with the Pod. Databases, uploaded files, and anything that must survive a Pod being deleted and recreated need storage that outlives any single Pod. Kubernetes' PV/PVC system is how you request and bind that durable storage without hardcoding where it physically lives into your application manifests.

Concepts

emptyDir, recap and its limit: an emptyDir volume is created fresh when a Pod starts and is deleted permanently when that Pod is removed from a node — great for scratch space or sharing files between sidecars, useless for anything that needs to survive a Pod restart or rescheduling.

A PersistentVolume (PV) is a piece of actual storage in the cluster — a chunk of disk, a network share, whatever the underlying platform provides — represented as a Kubernetes object independent of any Pod. Think of it as a physical hard drive that exists whether or not anything is currently using it.

A PersistentVolumeClaim (PVC) is a request for storage made by a user/Pod: "I need 5Gi, with these access characteristics." It's not the storage itself — it's a claim ticket that gets matched ("bound") to an actual PV that satisfies it. This split exists so application authors write PVCs (portable, describing needs) without knowing or caring which specific disk/backend actually provides it (cluster operators or StorageClasses handle that).

A StorageClass describes how to dynamically provision a PV on demand, instead of a cluster operator manually pre-creating PVs for every possible request. When a PVC references a StorageClass, a provisioner creates a matching PV automatically the moment the PVC is created — this is "dynamic provisioning," and it's the normal mode on every real cloud cluster (AKS's default StorageClass provisions real Azure Disks this way). Local kind clusters ship a default StorageClass (usually named standard, backed by the rancher.local-path-provisioner project) that provisions storage as a directory on the kind node container's filesystem — functionally durable across Pod restarts, but tied to that one node and gone if you delete the whole kind cluster.

   dynamic provisioning chain (bottom drives the one above it):

   Pod  ──references──► PVC  ──bound to──► PV  ──backed by──► real disk
   spec.volumes         "I need           "here is           (local dir,
   .persistentVolume     5Gi RWO"          5Gi RWO"           Azure Disk…)
   Claim.claimName          │                 ▲
                            │   StorageClass  │
                            └──► provisioner ─┘  creates the PV on demand

Access modes describe how many nodes can mount a volume at once and how:

• ReadWriteOnce (RWO): read-write by a single node at a time (most common — a typical database volume).
• ReadOnlyMany (ROX): read-only by many nodes simultaneously.
• ReadWriteMany (RWX): read-write by many nodes simultaneously (requires a backend that supports it — local-path-provisioner on kind does not, so RWX PVCs will stay Pending locally).

Reclaim policy decides what happens to the underlying storage once its PVC is deleted: Delete (the PV and its underlying storage are destroyed — the default for dynamically-provisioned volumes) or Retain (the PV and its data survive the PVC's deletion, for manual recovery/inspection — a safety choice for data you can't afford to lose by accident).

How a PVC actually gets used: you don't attach a PV directly to a Pod. You reference a PVC by name in the Pod's spec.volumes, and mount it into a container like any other volume. StatefulSets (not covered in depth in this track, but worth knowing the name) automate creating a distinct PVC per replica for stateful apps like databases run with multiple replicas — out of scope here, but a natural next topic once you're comfortable with the plain PVC pattern below.

Command reference

Hands-on exercises

Continue in namespace demo.

1. Check what your cluster provides by default

kubectl get storageclass

Expected: a StorageClass (commonly standard, marked (default)) — this is what kind installs out of the box so dynamic provisioning works with zero extra setup.

2. Request storage with a PVC (dynamic provisioning)

# pvc-data.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-claim
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

kubectl apply -f pvc-data.yaml
kubectl get pvc data-claim

Expected: STATUS becomes Bound (may briefly show Pending first). Note you never specified storageClassName — it used the cluster's default StorageClass.

3. See the PV that got created for you

kubectl get pv
kubectl describe pv $(kubectl get pvc data-claim -o jsonpath='{.spec.volumeName}')

Expected: a PV whose Claim field references demo/data-claim, capacity 1Gi, and Reclaim Policy: Delete. Notice the PV was created automatically — you never wrote a PV manifest.

4. Mount the PVC in a Pod and write data

# pod-writer.yaml
apiVersion: v1
kind: Pod
metadata:
  name: writer
spec:
  containers:
    - name: app
      image: busybox:1.36
      command: ["sh", "-c", "echo \"written at $(date)\" >> /data/log.txt; sleep 3600"]
      volumeMounts:
        - name: storage
          mountPath: /data
  volumes:
    - name: storage
      persistentVolumeClaim:
        claimName: data-claim

kubectl apply -f pod-writer.yaml
kubectl exec writer -- cat /data/log.txt

Expected: one line with a timestamp.

5. Prove the data survives the Pod being deleted and recreated

kubectl delete pod writer
kubectl apply -f pod-writer.yaml
kubectl exec writer -- cat /data/log.txt

Expected: two lines now — the original from step 4 plus a new one from this recreation's startup command, proving the PVC's data outlived the deleted Pod (unlike emptyDir, which would have started empty).

6. Share the same PVC from a second Pod (read)

# pod-reader.yaml
apiVersion: v1
kind: Pod
metadata:
  name: reader
spec:
  containers:
    - name: app
      image: busybox:1.36
      command: ["sh", "-c", "sleep 3600"]
      volumeMounts:
        - name: storage
          mountPath: /data
          readOnly: true
  volumes:
    - name: storage
      persistentVolumeClaim:
        claimName: data-claim

kubectl apply -f pod-reader.yaml
kubectl exec reader -- cat /data/log.txt

Expected: since this PVC's access mode is ReadWriteOnce, this works only as long as both Pods land on the same node — on this single-node kind cluster that's guaranteed, but it's worth noting RWO's real constraint: on a multi-node cluster, a second Pod scheduled to a different node would fail to mount it simultaneously.

7. Explicit StorageClass and reclaim policy on a PVC

# pvc-explicit.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-claim-explicit
spec:
  storageClassName: standard
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi

kubectl apply -f pvc-explicit.yaml
kubectl get pvc data-claim-explicit
kubectl get pv

Expected: bound the same way, this time with the StorageClass named explicitly instead of relying on the default.

8. Delete a PVC and observe reclaim policy in action

kubectl delete pod writer reader
kubectl get pv
kubectl delete pvc data-claim
kubectl get pv

Expected: after deleting the PVC, the PV that was bound to it also disappears (or moves to Released/is cleaned up shortly after) — because the default reclaim policy for dynamically-provisioned volumes here is Delete. The underlying data on the node's local-path directory is gone.

9. Diagnose and fix: PVC stuck Pending (unsatisfiable request)

# pvc-toobig.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: too-big-claim
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 10Gi

kubectl apply -f pvc-toobig.yaml
kubectl get pvc too-big-claim

Expected: STATUS: Pending, staying that way. Diagnose:

kubectl describe pvc too-big-claim

Expected: an event like waiting for a volume to be created, either by external provisioner "rancher.io/local-path" or manually created by system administrator combined with the access mode being the real blocker — local-path-provisioner only supports ReadWriteOnce, not ReadWriteMany. Fix by changing the access mode:

spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

kubectl delete pvc too-big-claim
kubectl apply -f pvc-toobig.yaml    # after fixing accessModes
kubectl get pvc too-big-claim

Expected: Bound. This is a good example of a limitation specific to the local storage provisioner — the same manifest with ReadWriteMany might work fine against a cloud backend that supports it (e.g. Azure Files on AKS).

10. Diagnose and fix: Pod stuck Pending because of an unbound PVC

# pod-stuck.yaml
apiVersion: v1
kind: Pod
metadata:
  name: stuck-pod
spec:
  containers:
    - name: app
      image: busybox:1.36
      command: ["sh", "-c", "sleep 3600"]
      volumeMounts:
        - name: storage
          mountPath: /data
  volumes:
    - name: storage
      persistentVolumeClaim:
        claimName: does-not-exist-claim

kubectl apply -f pod-stuck.yaml
kubectl get pod stuck-pod

Expected: STATUS: Pending, indefinitely. Diagnose:

kubectl describe pod stuck-pod

Expected: an event like persistentvolumeclaim "does-not-exist-claim" not found. Fix by creating the referenced PVC (or correcting the name to data-claim-explicit, which you created in exercise 7):

kubectl delete pod stuck-pod

Edit claimName to data-claim-explicit, then:

kubectl apply -f pod-stuck.yaml
kubectl get pod stuck-pod

Expected: Running. Clean up everything from this module:

kubectl delete pod stuck-pod
kubectl delete pvc data-claim-explicit too-big-claim

Independent challenge

No YAML or commands given here — figure it out yourself using what you know from this module and earlier ones.

Task: Stand up a tiny "notes" service backed by durable storage: a Deployment whose container appends a startup timestamp to a file on a mounted PersistentVolumeClaim every time it starts, fronted by a Service. Write at least one line, then prove the data is genuinely persistent by forcing the Pod to be replaced (not just restarted in place) and reading the file back to confirm the earlier line is still there alongside the new one. Finally, explain — before you test it — what you'd expect to happen to that data if you deleted the PVC given the default reclaim policy your kind cluster uses. This combines this module's PVC mechanics with Deployments (module 03) and Services (module 04).

Common mistakes & troubleshooting

• Confusing a PVC with the storage itself: a PVC is a request/claim; the PV is the actual storage. A Pending PVC means no PV has been bound yet — usually because dynamic provisioning is still running, or because the request (size, access mode) can't be satisfied.
• Requesting ReadWriteMany against a backend that doesn't support it: local-path-provisioner (kind's default) only supports ReadWriteOnce — a PVC requesting RWX will sit Pending forever locally, which is exactly what you reproduced in exercise 9. Cloud-backed StorageClasses (Azure Files, NFS-backed classes) do support RWX.
• Forgetting the reclaim policy before deleting a PVC: with the default Delete policy, deleting a PVC destroys its data permanently and immediately — there's no recycle bin. For anything you can't afford to lose, use/patch a StorageClass or PV with persistentVolumeReclaimPolicy: Retain.
• Referencing a PVC name that doesn't exist (or is in another namespace): identical failure mode to module 05's ConfigMap namespace mistake — PVCs are namespaced, and a typo or wrong namespace leaves the Pod Pending with an event pointing at the missing claim.
• Assuming storage follows the Pod across nodes automatically: an RWO volume is only mountable by one node at a time; on a multi-node cluster a Pod using it can get stuck Pending if the scheduler can't place it on the node currently holding the volume (or if that node is down) — a real operational constraint you'll meet again with StatefulSets.

Checkpoint quiz

Write down your answer to each question before expanding it — checking without attempting first is the single easiest way to fool yourself into thinking you've learned this.

1. What is the difference between a PersistentVolume and a PersistentVolumeClaim?
2. What does a StorageClass actually do, and why does dynamic provisioning matter compared to manually pre-creating PVs?
3. What happened to your first PVC's data when you deleted the Pod using it in exercise 5? What would have happened if it had used emptyDir instead?
4. What does ReadWriteOnce actually restrict, precisely?
5. What's the practical difference between reclaim policies Delete and Retain, and when would you prefer Retain?
6. In exercise 9, why did the PVC stay Pending even though there was plenty of disk space free on the node?

Further reading & sources

• Persistent Volumes - the authoritative reference for PVs, PVCs, access modes, and reclaim policies.
• Storage Classes - how dynamic provisioning is configured and defaulted.
• Configure a Pod to Use a PersistentVolume for Storage - a hands-on task mirroring this module's exercises.
• local-path-provisioner (Rancher) - the default provisioner kind ships, including its RWO-only limitation.
• StatefulSets - the natural next topic for per-replica PVCs with stateful workloads.

Next

07-helm-package-manager — you've now hand-written a lot of YAML across five object types; Helm lets you template and package all of it into a single reusable, versioned unit.