03 - Canary and Blue/Green Traffic Splitting

Why this matters

You've now met the same problem — ship a new version safely by shifting a controlled slice of traffic to it — in three different worlds: ACA revisions (track 06), Argo Rollouts (track 10), and now Istio. This module makes you fluent in the Istio mechanism (weighted VirtualService routing) and, just as importantly, able to say why you'd reach for each of the three. Canary and blue/green are the highest-value everyday use of the traffic-management you learned in module 02.

Concepts

Canary vs. blue/green, in Istio terms

Both are just weight schedules over the v1/v2 subsets from module 02:

• Canary: gradually shift weight — 100/0 → 95/5 → 80/20 → 50/50 → 0/100, pausing at each step to watch metrics, so only a small blast radius is exposed to a bad version at any moment. In Istio this is editing the weight fields of a VirtualService between steps.
• Blue/green: keep both versions fully deployed but send 100% to one ("blue"), then flip 100% to the other ("green") in a single step, with an instant flip back if it misbehaves. In Istio this is 100/0 then 0/100 on the same VirtualService — no redeploy, because both subsets are already running (exactly the "old revision is still there" instant-rollback property you saw with ACA revisions in track 06 module 05).

The key Istio property: because weight is a routing decision made by the sidecars, changing it is a config edit that takes effect in seconds, and rollback is just restoring the previous weights — no image pull, no rescheduling.

Mesh weighting vs. replica-ratio weighting (the track 10 tie-in)

Track 10 module 06 built canaries with Argo Rollouts and, without a traffic provider, used replica-based weighting — approximating "10%" with pod counts, which it explicitly warned "lies to you" because pod count isn't traffic share. It also said the fix was a traffic provider such as a service mesh, and that track 13 would upgrade it to true mesh-based weighting. This module is that upgrade. With Istio, weight: 10 means 10% of requests regardless of how many Pods back each subset — one v2 Pod can take exactly 10% while ten v1 Pods share the other 90%. That precision is the entire reason Rollouts integrates with a mesh.

Crucially, the two compose rather than compete: Argo Rollouts can drive the Istio VirtualService weights for you, adding automated, metric-gated promotion on top of the precise weighting Istio provides. Istio gives the accurate knob; Rollouts (track 10) turns the knob automatically based on analysis. This module turns the knob by hand so you understand what Rollouts would be automating.

Three mechanisms, one problem — when to use which

You now know three ways to do a weighted rollout. They are not interchangeable; they live at different layers:

The honest guidance: if you already run a mesh, its weighting is the natural choice and it's precise; if you want the rollout to promote itself based on metrics, that's Argo Rollouts' job (which can drive the mesh); if you're on ACA, the platform already does this and you don't operate a mesh at all. The skill this module builds is recognising which world you're in.

Metric-gating: what the mesh gives you for free

A canary is only useful if you watch the canary's health at each step. Track 10's Argo Rollouts automated this with AnalysisTemplate queries. Even by hand, Istio helps: because every request flows through the sidecars, you get per-subset request rate, error rate, and latency without instrumenting the app (module 06 is the full treatment). So the manual canary loop is: bump weight → watch the v2 subset's error rate and latency in the mesh metrics → promote or roll back. This is the same decision Rollouts' analysis automates; doing it by hand here teaches you what "good" and "bad" look like.

Command reference

Hands-on exercises

Continue in mesh-demo. You should have backend-v1 and backend-v2 Deployments, the backend Service, and the backend DestinationRule with v1/v2 subsets from module 02.

1. Start from a clean 100/0 (all on v1)

# canary-step0.yaml
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: backend
  namespace: mesh-demo
spec:
  hosts: [backend]
  http:
    - route:
        - destination: {host: backend, subset: v1}
          weight: 100
        - destination: {host: backend, subset: v2}
          weight: 0

kubectl apply -f canary-step0.yaml
for i in $(seq 1 20); do kubectl exec -n mesh-demo deploy/frontend -- curl -s http://backend; echo; done | sort | uniq -c

Expected: 20 backend v1, 0 v2 — the stable starting point of a canary.

2. Step the canary to 5%

kubectl patch virtualservice backend -n mesh-demo --type=merge -p \
  '{"spec":{"http":[{"route":[{"destination":{"host":"backend","subset":"v1"},"weight":95},{"destination":{"host":"backend","subset":"v2"},"weight":5}]}]}}'
for i in $(seq 1 40); do kubectl exec -n mesh-demo deploy/frontend -- curl -s http://backend; echo; done | sort | uniq -c

Expected: roughly 38 v1 / 2 v2 — about 5% on the canary. Note this is a precise 5%, not "one pod out of twenty," even though both subsets have one Pod. This is the difference from replica-ratio weighting.

3. Progress through the canary

Repeat the patch with weights 80/20, then 50/50, sampling 40 requests after each and confirming the ratio tracks the weights. At each step you would, in production, be watching the v2 subset's error rate/latency (module 06) before advancing.

# 20%:
kubectl patch virtualservice backend -n mesh-demo --type=merge -p \
  '{"spec":{"http":[{"route":[{"destination":{"host":"backend","subset":"v1"},"weight":80},{"destination":{"host":"backend","subset":"v2"},"weight":20}]}]}}'

Expected: sampled ratios follow 80/20 then 50/50 as you step. This is the manual version of the setWeight steps a Rollout walks automatically.

4. Complete the canary: 0/100

kubectl patch virtualservice backend -n mesh-demo --type=merge -p \
  '{"spec":{"http":[{"route":[{"destination":{"host":"backend","subset":"v1"},"weight":0},{"destination":{"host":"backend","subset":"v2"},"weight":100}]}]}}'
for i in $(seq 1 20); do kubectl exec -n mesh-demo deploy/frontend -- curl -s http://backend; echo; done | sort | uniq -c

Expected: 20 v2, 0 v1 — v2 is now stable. The canary is complete; v1 can be retired.

5. Instant rollback (the payoff)

Imagine v2 misbehaves after full promotion. Roll back with a single weight restore — no redeploy, because v1 is still running:

kubectl apply -f canary-step0.yaml
for i in $(seq 1 20); do kubectl exec -n mesh-demo deploy/frontend -- curl -s http://backend; echo; done | sort | uniq -c

Expected: back to 100% v1 in seconds. This is the same instant-rollback property as ACA revisions (track 06 module 05) — the old version never left, so rollback is a routing change, not a deployment.

6. Blue/green flip

Blue/green is the degenerate canary: no intermediate steps, just flip. With canary-step0.yaml (blue=v1 at 100%) applied, flip green in one shot:

kubectl patch virtualservice backend -n mesh-demo --type=merge -p \
  '{"spec":{"http":[{"route":[{"destination":{"host":"backend","subset":"v1"},"weight":0},{"destination":{"host":"backend","subset":"v2"},"weight":100}]}]}}'

Expected: 100% cut to v2 instantly, with an instant flip back available. The difference from canary is purely the schedule (one big step vs. many small ones), not the mechanism.

7. Header-pinned "canary for me only"

A subtler pattern: keep 100% of public traffic on v1, but let you reach v2 by header — a canary you can dogfood before exposing anyone. Combine module 02's header match with the weighted default:

# canary-dogfood.yaml
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: backend
  namespace: mesh-demo
spec:
  hosts: [backend]
  http:
    - match:
        - headers:
            x-canary: {exact: "true"}
      route:
        - destination: {host: backend, subset: v2}
    - route:
        - destination: {host: backend, subset: v1}
          weight: 100
        - destination: {host: backend, subset: v2}
          weight: 0

kubectl apply -f canary-dogfood.yaml
kubectl exec -n mesh-demo deploy/frontend -- curl -s http://backend
kubectl exec -n mesh-demo deploy/frontend -- curl -s -H "x-canary: true" http://backend

Expected: default request → v1; x-canary: true → v2. You can validate v2 in production with zero public exposure — something replica-ratio canaries can't do at all. (Remember module 02's lesson: the header match must come before the weighted default.)

8. Diagnose and fix: weights that send everyone to the wrong version

A colleague "adjusted the canary to 10%" but everyone's on v2. Reproduce:

# canary-broken.yaml
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: backend
  namespace: mesh-demo
spec:
  hosts: [backend]
  http:
    - route:
        - destination: {host: backend, subset: v1}
          weight: 10
        - destination: {host: backend, subset: v2}
          weight: 90

kubectl apply -f canary-broken.yaml
for i in $(seq 1 20); do kubectl exec -n mesh-demo deploy/frontend -- curl -s http://backend; echo; done | sort | uniq -c

Expected: ~18 v2 / 2 v1 — the weights are inverted. The intent was "10% to the new v2," but the weights read "10% to v1, 90% to v2," so the canary is actually a near-full rollout. This is the weighted-routing analogue of module 02's ordering typo — it applies cleanly and fails silently by routing the wrong ratio. Diagnose by reading the actual weights:

kubectl get virtualservice backend -n mesh-demo -o yaml | grep -A2 weight

Expected: you see subset: v1 weight: 10 and subset: v2 weight: 90 — the numbers are on the wrong subsets. Fix by restoring the correct 90/10 split:

kubectl patch virtualservice backend -n mesh-demo --type=merge -p \
  '{"spec":{"http":[{"route":[{"destination":{"host":"backend","subset":"v1"},"weight":90},{"destination":{"host":"backend","subset":"v2"},"weight":10}]}]}}'
for i in $(seq 1 20); do kubectl exec -n mesh-demo deploy/frontend -- curl -s http://backend; echo; done | sort | uniq -c

Expected: ~18 v1 / 2 v2 — a real 10% canary. Lesson: always confirm which subset each weight is attached to, not just that the numbers sum to 100.

9. Reset for the next module

kubectl apply -f canary-step0.yaml

Expected: 100% v1. Leave the app and DestinationRule in place — module 04 adds mTLS on top.

Independent challenge

No step-by-step given — draw on this module, track 10 module 06, and track 06 module 05.

Task: Run a full five-step canary (100/0 → 95/5 → 75/25 → 50/50 → 0/100) of a payments service from v1 to v2, sampling the live split at each step to confirm it tracks the weights, and practise a mid-canary abort: partway through (say at 50/50), pretend v2's error rate spiked and roll all the way back to 100/0 in one action, then explain why that rollback needed no redeploy. Finally, write three sentences contrasting how you'd have done this same rollout with Argo Rollouts (track 10) and with ACA revisions (track 06), naming the one thing each mechanism does that the other two don't.

Common mistakes & troubleshooting

• Inverting the weights. Attaching the small weight to the wrong subset turns a 10% canary into a 90% rollout (exercise 8). Always check which subset each weight sits on, not just that they sum to 100.
• Thinking rollback needs a redeploy. With both subsets running, rollback is restoring the previous VirtualService weights — seconds, no image pull. Reaching for kubectl rollout undo here is the wrong tool.
• Canary weight without health-watching. Stepping weights without watching the canary subset's error rate/latency (module 06) is just a slow blind rollout — the point of a canary is the gate, which by hand you supply.
• Header canary rule ordered after the weighted default. Same first-match hazard as module 02 — the header match must come before the catch-all or it never runs.
• Confusing the three mechanisms' layers. Istio weighting, Argo Rollouts, and ACA revisions solve the same problem at different layers; using Istio's weights doesn't give you Rollouts' automated gating, and vice versa — combine them if you want both.
• Assuming precise weight means precise for tiny sample sizes. weight: 5 is 5% over many requests; a handful of curls can look off just from randomness — sample enough to see the ratio.

Checkpoint quiz

Write down your answer to each question before expanding it — checking without attempting first is the single easiest way to fool yourself into thinking you've learned this.

1. In Istio terms, what is the only difference between a canary and a blue/green rollout?
2. Why is rolling back an Istio canary instant and redeploy-free?
3. Track 10's Argo Rollouts used replica-based weighting by default. What does Istio weighting give you that replica ratios can't, and how do the two tools combine rather than compete?
4. You have three ways to do a weighted rollout (ACA revisions, Argo Rollouts, Istio). Give the one deciding factor that would send you to each.
5. What does a mesh give you "for free" that makes metric-gating a canary possible without touching the app?
6. In exercise 8, the weights summed to 100 and the YAML applied cleanly, yet the canary was wrong. What was the bug, and why didn't anything error?
7. What is the header-pinned "dogfood" canary pattern, and what can it do that a purely weighted canary cannot?

Cumulative review

Closed-book. Don't reopen earlier modules while attempting these — the point is to find out what actually stuck, across this track and the tracks it builds on.

1. A meshed Pod is 2/2 and appears in istioctl proxy-status, but a VirtualService routing 100% to subset: v2 sends everyone to v1 anyway. Give the two most likely causes (one from module 01, one from module 02) and how you'd tell them apart.
2. Explain, using the data-plane/control-plane split from module 00, why editing a VirtualService weight takes effect in seconds and why traffic keeps flowing even if istiod is briefly down.
3. A Service selector typo (track 03 module 04), a NetworkPolicy label typo (track 03 module 11), a missing injection label (module 01), and an inverted canary weight (module 03) are all the "same class" of bug. State the class in one sentence and what all four share at the moment of failure.
4. You need "5% of traffic to v2." Describe how you'd achieve it with (a) a plain Service (track 03), (b) Argo Rollouts without a mesh (track 10), and (c) Istio — and rank the three by how truthfully each delivers exactly 5%.
5. A DestinationRule defines subsets v1/v2, but a Pod meant to be v2 is labelled version: V2 (capital V). What breaks, what doesn't error, and which command surfaces it?
6. Blue/green in ACA revisions (track 06) and blue/green in Istio both give "instant rollback." Explain the shared reason rollback is instant in both, despite the completely different underlying platforms.
7. Why does a canary without watching per-subset metrics reduce to "a slow blind rollout," and which module gives you the metrics that make it a real canary — with no app code change?
8. You want only your own requests to hit the new version while 100% of public traffic stays on the old one. Which two module-02/03 primitives do you combine, in which order, and why does the order matter?
9. A teammate says "just use kubectl rollout undo to roll back the canary." Explain why that's the wrong tool for an Istio weighted canary and what the correct rollback actually is.
10. Map each of these to the right layer: Service, VirtualService, DestinationRule, NetworkPolicy, injection label. For each, one sentence on what stops working if it's misconfigured.

Next

04-mtls-and-zero-trust — everything you've routed so far has already been encrypted with mTLS by default; now make that explicit, enforce it STRICTly, and see how it layers with the NetworkPolicy you know from track 03.