04 - mTLS and Zero-Trust Networking

Why this matters

In track 03, NetworkPolicy could say "frontend may reach backend," but the traffic on that allowed path was still plaintext, and "frontend" was identified only by a spoofable Pod label/IP. A service mesh closes both gaps at once: every call is encrypted and both ends are cryptographically identified, with certificates issued and rotated for you. This module makes the mTLS that's already been running under your exercises explicit, teaches you to enforce it STRICTly, and — critically — shows the failure mode where turning on STRICT mTLS breaks calls from a service that isn't in the mesh.

Concepts

mTLS, and what it adds over NetworkPolicy

Plain TLS (like HTTPS) encrypts traffic and proves the server's identity to the client. Mutual TLS (mTLS) additionally proves the client's identity to the server — both ends present certificates. In a mesh, the sidecars do this on every hop automatically, so you get two things NetworkPolicy never provided:

• Encryption in transit — an attacker who sniffs the Pod network sees ciphertext, not your requests.
• Strong workload identity — each workload's certificate encodes a verifiable identity (in Istio, a SPIFFE identity derived from its ServiceAccount), so "this call is from the checkout service" is cryptographically proven, not inferred from an easily-spoofed label or IP.

This is the "same mTLS idea" you saw ACA's Dapr provide between apps (track 06 module 05) — except here Istio issues and rotates the certificates for every meshed workload with no per-app setup, instead of you wiring TLS by hand.

It's already on — Istio's default

Here's the part that surprises people: you've been using mTLS since module 01. Every frontend→backend call in modules 01-03 was already encrypted and mutually authenticated by the sidecars, because Istio enables mTLS in PERMISSIVE mode by default. You didn't configure it; it just happened. This module is about making that explicit and tightening it — the mesh's "secure by default" posture is the opposite of NetworkPolicy's "open by default until you restrict it" (track 03 module 11).

PERMISSIVE vs. STRICT

PeerAuthentication controls the mTLS mode for a workload or namespace:

• PERMISSIVE (the default): meshed workloads accept both mTLS and plaintext. This is what lets a not-yet-meshed or external service still talk to your meshed services during a migration — encryption where possible, compatibility where necessary. The catch: because plaintext is still accepted, you don't yet have a guarantee of encryption.
• STRICT: the workload accepts only mTLS; any plaintext connection is rejected. This is the real "zero-trust" posture — nothing on the network is trusted unless it presents a valid mesh certificate. STRICT is the goal, but flipping it on will break any caller that isn't presenting mesh mTLS (i.e. any non-meshed Pod), which is the central hazard of this module.

The migration pattern this enables: mesh everything under PERMISSIVE (nothing breaks), verify all real traffic is already mTLS, then flip to STRICT to forbid the plaintext you've confirmed nobody needs.

PeerAuthentication scope: mesh, namespace, workload

A PeerAuthentication applies at one of three scopes, most-specific-wins:

• Mesh-wide — in the Istio root namespace (istio-system), no selector: the default for everything.
• Namespace-wide — in a namespace, no selector: overrides mesh default for that namespace.
• Workload-specific — in a namespace with a selector: overrides the namespace default for matching Pods.

This mirrors the layering you already know from RBAC (Role vs. ClusterRole, track 11) and from NetworkPolicy's per-Pod selectors — coarse defaults, refined by more specific rules. It lets you keep one legacy service PERMISSIVE while the rest of the namespace is STRICT.

How mTLS layers with NetworkPolicy (not replaces it)

These are different layers and you want both. NetworkPolicy (L3/4) decides whether a connection is allowed to exist at all — it can drop a packet before it's even a TLS handshake. mTLS (L7-ish, in the proxies) decides whether an allowed connection is encrypted and who's on each end. A robust setup uses NetworkPolicy to keep unrelated Pods from even reaching a service's port, and STRICT mTLS so that whatever does reach it is encrypted and identified. One is the outer fence, the other is the locked door — defence in depth, and neither makes the other redundant. (Authorization of which identity may call which is a third layer — AuthorizationPolicy, module 05.)

Certificate issuance and rotation, conceptually

You never touched a certificate in modules 01-03, yet every call was mTLS. That's because istiod runs a certificate authority: when a sidecar starts, it proves its identity (via its ServiceAccount token) and istiod issues it a short-lived workload certificate, then automatically rotates it (re-issues a fresh one) long before it expires — typically every 24 hours, with no restart and no human involved. Compare this to the hand-managed TLS of track 03/04: certificates that expire and page you at 2am because someone forgot to renew. The mesh turning cert lifecycle into an invisible, self-healing background process is one of its highest-value features — you get short-lived, frequently-rotated certs (a security best practice that's painful to do by hand) essentially for free.

Command reference

Hands-on exercises

Continue in mesh-demo with the meshed frontend/backend app. You'll also create a deliberately non-meshed namespace to trigger the STRICT-breaks- legacy scenario.

1. Prove mTLS is already on (PERMISSIVE default)

BACKEND_POD=$(kubectl get pod -n mesh-demo -l app=backend -o jsonpath='{.items[0].metadata.name}')
istioctl authn tls-check deploy/frontend.mesh-demo backend.mesh-demo.svc.cluster.local

Expected: the check reports the effective policy as PERMISSIVE and that the connection uses mTLS — confirming the calls you've been making since module 01 were already encrypted and mutually authenticated without any config from you.

2. Look at a workload's certificate

istioctl proxy-config secret deploy/backend -n mesh-demo -o json | \
  jq -r '.dynamicActiveSecrets[0].secret.tlsCertificate.certificateChain.inlineBytes' 2>/dev/null | head -c 40; echo
istioctl proxy-config secret deploy/backend -n mesh-demo

Expected: a certificate exists with a short validity window — the one istiod issued to this workload and rotates automatically. You never created it. Note the short lifetime: that's the frequent-rotation property from Concepts.

3. Enforce STRICT mTLS namespace-wide

# strict-mtls.yaml
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: mesh-demo
spec:
  mtls:
    mode: STRICT

kubectl apply -f strict-mtls.yaml
kubectl exec -n mesh-demo deploy/frontend -- curl -s http://backend

Expected: still works — both frontend and backend are meshed, so both ends present mesh certificates and STRICT is satisfied. Nothing looks different from the client's side, but plaintext is now forbidden, which the next exercises prove.

4. Confirm the mode really changed

istioctl authn tls-check deploy/frontend.mesh-demo backend.mesh-demo.svc.cluster.local
kubectl get peerauthentication -n mesh-demo

Expected: tls-check now reports STRICT for backend; the PeerAuthentication is listed. You've moved from "encrypted but plaintext still accepted" to "encrypted, plaintext rejected."

5. Create a non-meshed (legacy) caller

Make a namespace without the injection label and put a client in it:

kubectl create namespace legacy
# deliberately NOT labelling it for injection
kubectl -n legacy run legacy-client --image=curlimages/curl:8.10.1 --command -- sleep 3600
kubectl get pod legacy-client -n legacy

Expected: legacy-client is 1/1 — no sidecar, so it will speak plaintext when it calls into mesh-demo. This is the migration reality: not everything is meshed at once.

6. Diagnose and fix: STRICT mTLS breaks the legacy service's calls

kubectl exec -n legacy legacy-client -- curl -s -m 5 http://backend.mesh-demo || echo "FAILED (connection reset)"

Expected: it fails — backend is STRICT and legacy-client has no sidecar, so it can only offer plaintext, which STRICT rejects. This is the canonical "mTLS broke a legacy non-meshed service" incident: nothing in mesh-demo changed, the legacy caller just can't speak mTLS. Diagnose — prove the meshed path still works so you know it's specifically the non-meshed caller:

kubectl exec -n mesh-demo deploy/frontend -- curl -s http://backend    # still works
kubectl get pod legacy-client -n legacy                                # 1/1 = not meshed
istioctl authn tls-check deploy/frontend.mesh-demo backend.mesh-demo.svc.cluster.local  # STRICT

Expected: meshed caller succeeds, legacy caller is 1/1, backend is STRICT — the diagnosis is complete: STRICT plus a non-meshed caller. You have three legitimate fixes; pick based on intent:

Fix A (right long-term): mesh the legacy caller.

kubectl label namespace legacy istio-injection=enabled
kubectl delete pod legacy-client -n legacy
kubectl -n legacy run legacy-client --image=curlimages/curl:8.10.1 --command -- sleep 3600
kubectl get pod legacy-client -n legacy   # now 2/2
kubectl exec -n legacy legacy-client -- curl -s http://backend.mesh-demo

Expected: 2/2, and the call now succeeds — the caller presents a mesh cert. This is the migration-done-right outcome: mesh the stragglers, keep STRICT.

Fix B (bridge during migration): relax just backend to PERMISSIVE. If you can't mesh the caller yet, scope a PERMISSIVE override to backend so it accepts both while you migrate — deliberately less secure, temporary:

# backend-permissive.yaml
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: backend-permissive
  namespace: mesh-demo
spec:
  selector:
    matchLabels: {app: backend}
  mtls:
    mode: PERMISSIVE

This is the workload-scoped override from Concepts (most-specific-wins). Don't apply it now — Fix A is what you want here. The point is knowing this bridge exists so a migration doesn't force an all-or-nothing cutover.

7. Verify STRICT still holds for everyone else

With the legacy namespace now meshed (Fix A), confirm STRICT is still enforced and there's no plaintext path left:

istioctl authn tls-check deploy/backend.mesh-demo backend.mesh-demo.svc.cluster.local
kubectl get peerauthentication -n mesh-demo

Expected: still STRICT, no PERMISSIVE override — the ideal end state: every caller meshed, plaintext forbidden mesh-wide.

8. See mTLS and NetworkPolicy as different layers

Reason about (and, if your kind cluster has a policy-enforcing CNI like the Calico setup from track 03 module 11, test): STRICT mTLS encrypts and identifies allowed traffic, but does it stop an unauthorized meshed Pod from connecting to backend's port? No — that's NetworkPolicy's job (L3/4), and authz of which identity may call is module 05's job. Write down which of the three layers (NetworkPolicy / mTLS / AuthorizationPolicy) would stop each of: a packet from an unrelated Pod reaching backend's port at all; an eavesdropper reading backend traffic; a meshed-but-unauthorized service successfully calling backend.

Expected: NetworkPolicy stops the first (drops the connection), mTLS stops the second (encryption), AuthorizationPolicy (module 05) stops the third — three distinct layers, all wanted.

9. Clean up the legacy namespace

kubectl delete namespace legacy

Leave mesh-demo STRICT — module 05 adds authorization on top of it.

Independent challenge

No YAML given — draw on this module, track 03 NetworkPolicy, and the injection mechanics from module 01.

Task: Simulate a realistic "tighten security without an outage" migration. Start with two meshed services (web→orders) plus one non-meshed batch job that also calls orders over plaintext, all working under PERMISSIVE. Now reach a fully zero-trust end state — orders STRICT, every real caller on mTLS — without ever breaking the batch job's traffic mid-migration. Prove at each step that both the meshed and (until you mesh it) the non-meshed caller keep working, then flip STRICT only once nothing needs plaintext, and confirm a fresh non-meshed Pod is now rejected. Finally, explain in three sentences why you'd keep a NetworkPolicy on orders even after STRICT mTLS is fully enforced.

Common mistakes & troubleshooting

• Flipping STRICT before everything is meshed. Any non-meshed caller speaks plaintext and gets rejected the instant backend goes STRICT — the exact incident in exercise 6. Mesh first (PERMISSIVE tolerates stragglers), verify all real traffic is mTLS, then STRICT.
• Assuming mTLS replaces NetworkPolicy. It doesn't — mTLS encrypts and identifies allowed traffic but doesn't decide whether a connection may exist. Keep NetworkPolicy (L3/4) and mTLS (encryption/identity) as separate layers.
• Thinking "no config" means "no mTLS." Istio is mTLS-PERMISSIVE by default, so meshed traffic is already encrypted before you write any PeerAuthentication — the opposite default from NetworkPolicy's open-by- default.
• Debugging cert errors as app bugs. A connection reset/RST on a meshed call after enabling STRICT is almost always a non-meshed caller or a DISABLE/STRICT mismatch, not the application — check the caller's READY count and istioctl authn tls-check first.
• Forgetting most-specific-wins for PeerAuthentication. A workload-scoped PERMISSIVE override silently beats a namespace STRICT for those Pods — great for a migration bridge, dangerous if you forget it's there. kubectl get peerauthentication -A shows the full picture.
• Worrying about certificate expiry. You don't manage mesh certs — istiod issues and rotates short-lived ones automatically. If you find yourself hand-renewing a mesh workload cert, you've misunderstood who owns them.

Checkpoint quiz

Write down your answer to each question before expanding it — checking without attempting first is the single easiest way to fool yourself into thinking you've learned this.

1. What does mTLS add over plain TLS, and what does it add over a NetworkPolicy that already allows the traffic?
2. Were your module 01-03 calls encrypted? Why or why not, given you never wrote a PeerAuthentication?
3. Explain PERMISSIVE vs. STRICT and why PERMISSIVE is the safe default during a migration.
4. You flip a namespace to STRICT and a batch job's calls immediately start failing with connection resets. What's the cause and the two ways to fix it?
5. mTLS and NetworkPolicy — do you keep both, and what does each stop that the other doesn't?
6. Who issues and rotates a meshed workload's certificate, and how does that compare to the hand-managed TLS of earlier tracks?
7. A workload-scoped PERMISSIVE PeerAuthentication exists in a STRICT namespace. Which wins for the selected Pods, and why might that be both useful and dangerous?

Next

05-authorization-policies — mTLS proves who a caller is; now use that proven identity to decide which services are allowed to call which, with AuthorizationPolicy.