00 - Service Mesh Concepts

Why this matters

By the end of track 03 you could route traffic with Services and Ingress, restrict it with NetworkPolicy, and secure the API server with RBAC — all real, all sufficient for a lot of systems. A service mesh exists because those tools stop scaling well once you have dozens of services that all need encryption, retries, fine-grained routing, and a live picture of who calls whom. This module builds the mental model — the sidecar pattern, the data plane / control plane split, and the honest question of when a mesh is worth its weight — before you install anything in module 01.

Concepts

The problem: cross-cutting concerns duplicated in every service

Think about what every service in a real system needs: TLS to its neighbours, retries on transient failures, timeouts so one slow dependency doesn't hang the caller, metrics on every call, and a way to say "only the checkout service may call the payments service." In track 03 you solved pieces of this per-object: a Service for discovery, a NetworkPolicy for coarse allow/deny, hand-configured TLS if you bothered. The trouble is these are cross-cutting concerns — every service needs them, so without a mesh every team re-implements them in application code, in a different library, in a different language, slightly wrong each time. A mesh's core idea is to lift all of that out of the application and into infrastructure.

The sidecar proxy pattern

A service mesh does this by putting a proxy container (Istio uses Envoy) into every application Pod, alongside your app container — a sidecar. This is the exact same Pod-with-two-containers shape you first saw as an "init/sidecar" idea in track 03, and it's the same shape ACA's Dapr sidecar took in track 06's revisions module — one process for your code, one alongside it for the platform's concerns. All traffic in and out of the app container is transparently redirected through its sidecar proxy (via iptables rules the mesh installs), so:

• your app makes a plain http://payments call, unaware anything changed;
• its sidecar intercepts the outbound call, encrypts it (mTLS), applies retry/timeout/routing rules, and records metrics;
• the payments Pod's sidecar receives it, decrypts it, checks authorization, records metrics, and hands plaintext to the payments app container.

Your application code is untouched. That transparency is the whole selling point — and also the whole cost, because now there are twice as many containers to run and one more network hop each way.

Data plane vs. control plane

The set of all those sidecar proxies — the things actually carrying and acting on your traffic — is the data plane. But the proxies don't decide policy on their own; something has to tell every proxy "route 10% to v2," "require mTLS here," "allow checkout→payments." That something is the control plane (in Istio, a component called istiod). You write Kubernetes objects (CRDs like VirtualService, PeerAuthentication); the control plane watches them, computes the right configuration for each proxy, and pushes it out. This is the same declared-desired-state / a-controller- reconciles-it loop behind every controller you've met since track 03 — a Deployment controller reconciles ReplicaSets, istiod reconciles Envoy configs. The data plane is on the request path; the control plane is not, so if istiod is briefly down, existing traffic keeps flowing on the last config it pushed.

What a mesh solves that plain Kubernetes doesn't

Line the mesh up against the track 03 tools you already know:

• Encryption in transit. NetworkPolicy controls whether Pod A may reach Pod B, but the traffic itself is plaintext on the wire. A mesh gives you mTLS everywhere automatically (module 04) — every call encrypted and both ends cryptographically identified, with certificates issued and rotated for you.
• Traffic routing by request content. A Service load-balances across all matching Pods equally by label selector — it cannot send 5% of traffic to a canary, or route requests with header x-user: beta to v2. A mesh's VirtualService can (modules 02-03).
• Identity-based authorization. RBAC governs who can call the Kubernetes API; it says nothing about which service may call which at runtime. A mesh's AuthorizationPolicy does that, keyed on the cryptographic identity mTLS establishes (module 05).
• Uniform L7 observability. In track 12 you instrumented apps for metrics and traces. A mesh gives you golden-signal metrics (request rate, error rate, latency) and a service dependency graph for every call with zero app changes, because the proxy sees every request (module 06).
• Resilience without code. Retries, timeouts, and circuit breaking become configuration instead of client-library code (module 07).

Where a mesh is overkill

A mesh is not free and not always worth it. The honest counsel:

• A handful of services get most of these benefits more cheaply from Ingress + NetworkPolicy + a decent HTTP client library; the mesh's fixed cost (a proxy per Pod, a control plane, a steeper operational learning curve, extra latency and CPU) dominates when you have three services, not thirty.
• Latency-critical paths pay for the extra hop through two proxies on every call — usually small, but real.
• Small teams take on a genuinely complex new system to operate; an outage in the mesh is an outage in everything.

The rule of thumb: reach for a mesh when the number of services and the need for uniform mTLS/authz/observability across all of them is what hurts — not as a default first move. This is exactly why the curriculum put tracks 03 and 12 first: you should feel the pain the mesh removes before adopting it.

Istio vs. Linkerd (one paragraph, deliberately)

Istio is the most feature-complete mesh (rich traffic management, Envoy- based, huge surface area) and is what this track teaches in depth. Linkerd is the notable lighter-weight alternative: a purpose-built Rust "micro-proxy" instead of Envoy, dramatically simpler to operate and lower overhead, at the cost of fewer knobs (less elaborate traffic-routing and extensibility than Istio). The tradeoff in one line: Istio buys you power and pays in complexity; Linkerd buys you simplicity and pays in flexibility. Everything you learn here about the concepts — sidecars, data/control plane, mTLS, traffic splitting, authz — applies to Linkerd too; only the CRDs and tooling differ. We go deep on Istio because the mechanics it exposes make the concepts concrete.

Command reference

This module is conceptual — you install nothing yet. These are the terms and the objects you'll meet, so the vocabulary is set before module 01.

Hands-on exercises

No cluster changes yet — these exercises make the concepts concrete by inspecting what you already have and reasoning about the gaps. Work in a notebook; several answers are things you'll verify hands-on in later modules.

1. Inventory what track 03 already gives you

On any existing kind cluster (or on paper), list, for a two-service frontend→backend app, exactly which track-03 object provides each of: service discovery, load balancing, coarse allow/deny between Pods, and external exposure.

Expected: Service (discovery + L4 load balancing), NetworkPolicy (coarse allow/deny), Ingress (external exposure). Note that encryption, percentage routing, and service-to-service authz are on nobody's list — those are the mesh's gaps to fill.

2. Find the gaps a Service can't cover

For that same app, write down what you would have to do today, without a mesh, to (a) encrypt frontend→backend traffic, and (b) send 5% of frontend traffic to a second version of backend.

Expected: (a) terminate TLS in each app or run your own sidecar/cert tooling — non-trivial and per-app; (b) approximate it with replica ratios (coarse, as track 10 module 04 warned) because a Service splits evenly by label. Both are exactly what modules 02-04 make one-liners.

3. Draw the request path, twice

Sketch a single frontend→backend HTTP request as it flows today (app → Service/kube-proxy → app), then sketch it again with a mesh (app → its sidecar → network → backend's sidecar → backend app). Count the hops and mark where encryption, retries, and metrics get applied in the meshed version.

Expected: the meshed path has two extra proxy hops; encryption/retries/metrics all live in the sidecars, not the app — this is the transparency and the overhead in one picture.

4. Separate the planes

Given the statement "we deploy a new routing rule and 30 seconds later traffic shifts," identify which plane the rule is written to and which plane acts on traffic, and answer: if the control plane crashed the moment after the rule was accepted, would already-flowing traffic stop?

Expected: you write to the control plane (a CRD istiod reads); the data plane (sidecars) acts on traffic. Existing traffic keeps flowing on the last pushed config even if istiod is down — only new config changes stall.

5. Map each mesh feature to the track-03 tool it surpasses

Make a two-column table: left, the mesh feature (mTLS, weighted routing, service authz, L7 metrics); right, the closest track-03/12 tool and the one thing it can't do that the mesh can.

Expected: mTLS vs. NetworkPolicy (policy allows/denies but doesn't encrypt); weighted routing vs. Service (even split only); service authz vs. RBAC (RBAC is API-server access, not runtime call authz); L7 metrics vs. app instrumentation (mesh needs no app changes).

6. Decide: mesh or not?

For each system, decide "mesh" or "not yet," and justify in one sentence: (a) three services behind one Ingress, one team; (b) forty microservices, six teams, a compliance requirement that all internal traffic be encrypted; (c) two latency-critical trading services where every millisecond counts.

Expected: (a) not yet — Ingress + NetworkPolicy + a good HTTP client is cheaper than a mesh; (b) mesh — uniform mTLS + authz + observability across forty services is exactly the pain a mesh removes; (c) probably not, or Linkerd if anything — the per-hop latency tax is hardest to justify where milliseconds are the product.

7. Istio or Linkerd?

For system (b) above, argue in three sentences which mesh you'd pick and why, naming one concrete factor that would flip your choice.

Expected: a defensible answer either way — Istio for rich traffic-routing and extensibility if you'll use those knobs; Linkerd for lower overhead and operational simplicity if you mostly want mTLS + basic splitting. A flipping factor: needing header-based routing / elaborate VirtualService rules pushes toward Istio; a tiny platform team pushes toward Linkerd.

Independent challenge

No commands here — this is a reasoning task drawing on track 03 Services, track 03 NetworkPolicy/RBAC, and the canary discussions in track 10 module 06 and track 06 module 05.

Task: Write a one-page "should we adopt a service mesh?" recommendation for a hypothetical platform team running twelve services on AKS. Cover: which specific problems they have today that a mesh solves (be concrete — name the track-03 tool that currently half-solves each), what new operational cost and failure modes they take on, whether their canary needs are already met by the mechanisms they know (Argo Rollouts from track 10, ACA revisions from track 06) or genuinely need mesh-based weighting, and a final go/no-go with the one condition that would change your answer. No YAML — this is the judgment call the rest of the track assumes you can make.

Common mistakes & troubleshooting

• Treating the mesh as a router replacement. It doesn't replace Services, Ingress, or DNS — it layers on top of them. Your Service objects still exist and still do discovery; the mesh adds routing rules over that.
• Thinking NetworkPolicy and mTLS are the same layer. NetworkPolicy is L3/4 allow/deny (can Pod A reach Pod B at all); mTLS is encryption + identity on the traffic that's allowed. They compose — you'll use both (module 04).
• Assuming a mesh is a security feature you can bolt on for free. It's a distributed system you now operate. The proxies, the control plane, and the certificate machinery are all things that can break and page you.
• Believing "sidecarless" or "ambient" modes make the concepts moot. Newer Istio "ambient" mode moves some functions out of per-Pod sidecars, but the data-plane/control-plane split, mTLS identity, and routing concepts are identical — learn them with sidecars first; they transfer.
• Adopting a mesh to get one feature. If all you need is percentage canaries, Argo Rollouts (track 10) does that without a mesh. Adopt the mesh when several of its features are needed uniformly across many services.

Checkpoint quiz

Write down your answer to each question before expanding it — checking without attempting first is the single easiest way to fool yourself into thinking you've learned this.

1. In one sentence, what does the sidecar proxy pattern let a mesh do that editing the application couldn't do as cleanly?
2. Which plane is on the request path — data or control — and what does that mean if the control plane briefly goes down?
3. NetworkPolicy already controls Pod-to-Pod traffic. Name two things a mesh adds that a NetworkPolicy fundamentally can't.
4. Why can't a plain Service send 5% of traffic to a canary version?
5. Give two concrete situations where adopting a mesh would be overkill.
6. State the Istio-vs-Linkerd tradeoff in one sentence each.
7. Which mesh feature relates to Kubernetes RBAC, and why is it a different layer rather than a replacement?

Next

01-installing-istio-and-sidecar-injection — put the concepts on a real cluster: install Istio, label a namespace for injection, and prove the sidecars actually attached.