Monitoring AKS: Azure Monitor & Container Insights

Why this matters

So far you've debugged everything with kubectl describe, kubectl logs, and kubectl top — fine for one cluster you're staring at directly, but it doesn't scale to "what happened at 3am" or "which pod is slowly leaking memory over days." Container Insights (part of Azure Monitor) collects logs and metrics from your cluster into a queryable, persistent store, so you can investigate after the fact and set up alerts instead of watching a terminal.

Concepts

Azure Monitor is Azure's umbrella observability service across all resource types; Container Insights is the AKS/Kubernetes-specific feature built on top of it. Enabling it deploys an agent (the Azure Monitor Metrics/Logs agent, running as a DaemonSet — one pod per node) to your cluster, which ships container logs, node/pod metrics, and inventory data to a Log Analytics workspace.

Log Analytics workspace is where the data actually lives — a queryable store you search with KQL (Kusto Query Language), similar in spirit to kubectl logs but across every pod, over any historical time range you retain, with real filtering/aggregation instead of grep.

This is additive, not a replacement for kubectl. You'll still use kubectl describe/kubectl logs for "what's wrong with this pod right now." Container Insights is for "what happened across the whole cluster over the last day," "show me every pod that OOM-killed this week," or setting up an alert that pages you without you watching anything.

Retention and cost. Log Analytics bills for data ingested and for retention beyond the free default window. A busy cluster logging verbosely can generate meaningful log volume — this is a real, easy-to-overlook cost knob distinct from node/LB/disk costs you've already met.

What AKS manages vs. what you own: enabling Container Insights is a one-command, Azure-managed agent deployment and pipeline — you don't write log-shipping config yourself. You still own: which Log Analytics workspace you point at (and its retention/cost settings), what queries and alerts you build on top of the collected data, and remembering to disable/detach monitoring (and clean up the workspace) when you're done with a learning cluster, since the workspace persists independently of the AKS cluster's own lifecycle in some setups.

Command reference

Hands-on exercises

1. Enable Container Insights on your cluster. Run az aks enable-addons --resource-group rg-aks-learn --name aks-learn --addons monitoring. This creates (or reuses) a default Log Analytics workspace if you don't specify one. Verify: az aks show --resource-group rg-aks-learn --name aks-learn --query addonProfiles.omsagent.enabled returns true.

2. Confirm the agent is running. Run kubectl get pods -n kube-system -l component=oms-agent -o wide. Verify: one agent pod per node, all Running.

3. Find your Log Analytics workspace. Run az monitor log-analytics workspace list --resource-group rg-aks-learn --output table (it may live in a different resource group depending on how it was auto-created — check az aks show ... --query addonProfiles.omsagent.config for the exact workspace resource ID if it's not where you expect). Note the workspace name for the portal exercises below.

4. Generate some real log data. Deploy a small app (reuse one from earlier modules) that logs something identifiable, e.g. echo "hello from $(hostname)" in a loop. Let it run for a few minutes so data has time to ship (ingestion typically lags live by a few minutes, unlike kubectl logs which is immediate).

5. Query it in the portal. In the Azure Portal, open your Log Analytics workspace → Logs, and run a KQL query against the ContainerLogV2 table filtered to your namespace/pod, e.g.: ContainerLogV2 | where PodNamespace == "demo" | order by TimeGenerated desc | take 50. Verify: you see your app's log lines, with timestamps, queryable in a way kubectl logs alone doesn't give you (e.g. across pod restarts, or across multiple pods at once).

6. Look at the Container Insights dashboards. In the portal, open your AKS cluster resource → Insights (or Monitoring → Insights). Verify: you can see node CPU/memory over time, a list of pods with their status, and container-level resource usage — the same kind of data kubectl top gives you live, now with history.

7. Diagnose using Container Insights instead of kubectl. Deliberately cause a pod failure you can investigate after the fact: deploy a Deployment with a container that exits after a short sleep (causing CrashLoopBackOff), let it crash a few times, then investigate it two ways: first with kubectl describe pod / kubectl logs --previous (what you already know), then find the same failure in the portal's Insights view or via a KQL query against KubePodInventory/ContainerLogV2 filtered to that pod name — confirm you can see the crash history there too, which is useful once a pod has been deleted and kubectl logs --previous no longer has anything to show.

8. Set up a basic alert (optional but recommended). In the portal, under your Log Analytics workspace or the AKS cluster's Alerts blade, create a simple metric alert (e.g. node CPU percentage above a threshold for 5 minutes) so you experience the alerting side, not just dashboards. Verify: the alert rule is listed as enabled; you don't need to actually trigger it.

9. Clean up. Delete the crash-looping test Deployment: kubectl delete deployment <name>. Container Insights and its Log Analytics workspace are relatively low-cost at small scale, but they do accrue ingestion/retention charges over time — if you're done with monitoring for now, disable the add-on (az aks disable-addons --resource-group rg-aks-learn --name aks-learn --addons monitoring) and consider deleting the workspace if you created a dedicated one just for this exercise (az monitor log-analytics workspace delete --resource-group rg-aks-learn --workspace-name <name> --yes). Note that disabling the add-on stops new data collection but does not by itself delete already-ingested data or the workspace — delete the workspace explicitly if you want that gone too.

Independent challenge

No commands given here — figure it out yourself using what you know from this module and earlier ones.

Task: Prove to yourself that persistent monitoring answers a question live kubectl cannot. Enable Container Insights on your cluster, then deploy a workload that fails in a way that destroys its own live evidence — for example a container that crash-loops and is eventually replaced, so that once the old pod is gone kubectl logs --previous has nothing left to show (this reuses the crash/diagnosis instinct from module 02's Pending-vs-CrashLoop material, conceptually building on that module). After the pod is gone, reconstruct what happened purely from the collected data: find the crash history and the container's log lines for that specific pod in the Log Analytics workspace. Separately, locate exactly which workspace your cluster is shipping to and which resource group it lives in. When done, disable the add-on and decide whether the workspace should be deleted too — remember it is a separate billable resource that a plain az group delete on the cluster's group will not necessarily remove.

Common mistakes & troubleshooting

• Expecting portal data instantly. Ingestion lag means log/metric data can take a few minutes to appear after being generated — don't conclude monitoring is broken after checking 10 seconds later.
• Not knowing which Log Analytics workspace got created. If you enable monitoring without specifying --workspace-resource-id, Azure auto-creates or reuses a default one, sometimes in a different resource group than expected — always confirm with az aks show --query addonProfiles.omsagent.config rather than guessing.
• Treating Container Insights as a replacement for kubectl describe. It's complementary — for "why is this pod behaving oddly right now," kubectl is still faster; for historical/aggregate questions, Container Insights is the right tool.
• Verbose application logging left on indefinitely. Every log line your app writes is a candidate for ingestion cost once Container Insights is enabled. Debug-level logging that was harmless with kubectl logs (ephemeral, free) has a real, ongoing cost once shipped to Log Analytics continuously.
• Cost pitfall: an orphaned Log Analytics workspace outliving the cluster. Because the workspace is a separate Azure resource from the AKS cluster, deleting the cluster's resource group does not automatically delete a workspace that lives in a different resource group. Explicitly check az monitor log-analytics workspace list --output table across your subscription when doing final cleanup.

Checkpoint quiz

Write down your answer to each question before expanding it — checking without attempting first is the single easiest way to fool yourself into thinking you've learned this.

1. What's the relationship between Azure Monitor, Container Insights, and a Log Analytics workspace?
2. What does Container Insights give you that kubectl logs/kubectl top alone don't?
3. Why might data not appear in the portal the instant you generate it?
4. Name one situation where you'd still prefer kubectl describe pod over a KQL query, and one where you'd prefer the KQL query.
5. Why can a Log Analytics workspace outlive the AKS cluster it was monitoring, cost-wise?
6. What table would you query in Log Analytics to see shipped container log lines?

Next

07-security-aks-aad-rbac-and-keyvault — lock down who can do what on your cluster and stop putting secrets directly in Kubernetes Secret objects.