Lesson 7: Azure AKS Internals, Cilium CNI Overlay & Azure Dependent PaaS Ecosystem

🧠 The Concept (Explain Like I’m 5)

Running Kubernetes on Microsoft Azure (AKS) alongside Azure’s managed PaaS services is like running a high-speed bullet train station:

• AKS Node Pools & Ephemeral OS Disks: Instead of booting worker nodes off slow network hard drives (which choke when 50 containers pull images at once), Ephemeral OS Disks run the node’s operating system directly inside the server’s ultra-fast local NVMe cache—at $0 extra storage cost!
• Azure CNI Overlay with Cilium: Instead of giving every temporary pod a scarce corporate VNet IP address, pods get private internal apartment numbers (100.64.x.x) routed at kernel speed by Cilium eBPF, while nodes sit on the corporate VNet street.
• Entra Workload Identity + Key Vault CSI: Pods show a short-lived cryptographic passport (OIDC token) to Microsoft Entra ID to fetch database passwords directly from Azure Key Vault into RAM—zero static secrets stored in Git or etcd.
• Azure Dependent PaaS Services: Around AKS sits Azure’s enterprise ecosystem: Azure Front Door (global CDN/WAF shield), Application Gateway for Containers (L7 ingress), Cosmos DB (planet-scale database with 5 tunable consistency levels), and Azure Service Bus (transactional message queues that trigger KEDA to autoscale pods).

🗺️ Visual Architecture: Production AKS & Azure Dependent Services Reference Stack

flowchart TD
    Users["🌍 Global Users & Mobile Apps"] -->|"Anycast HTTPS"| AFD["🛡️ <b>Azure Front Door Premium</b><br/>Global WAF + DDoS + Edge Caching"]

    AFD -->|"Private Link Origin<br/>(Zero Public IPs)"| AGC["⚖️ <b>App Gateway for Containers</b><br/>(Kubernetes Gateway API / ALB for AKS)"]

    subgraph AKSVNet ["☸️ Azure AKS Production VNet (Azure CNI Overlay + Cilium eBPF)"]
        AGC --> SysPool["<b>System Node Pool</b><br/>CriticalAddonsOnly=true:NoSchedule<br/>CoreDNS, Cilium, Konnectivity"]
        AGC --> UserPool["<b>User Node Pool (NAP / Karpenter)</b><br/>Ephemeral NVMe OS Disks + Spot VMs"]

        UserPool --> KEDA["⚡ <b>KEDA Autoscaler</b><br/>Scales Pods 0 to 500 on Queue Depth"]
        UserPool --> CSI["🔐 <b>Key Vault CSI Driver</b><br/>Entra Workload Identity OIDC"]
    end

    subgraph DependentPaaS ["🗄️ Azure Dependent Enterprise PaaS Layer (Via Private Endpoints)"]
        KV["🔑 <b>Azure Key Vault</b><br/>HSM Keys, Certs & Secrets"]
        SB["📨 <b>Azure Service Bus Premium</b><br/>FIFO Sessions & Dead-Letter Queues"]
        EH["🌊 <b>Azure Event Hubs</b><br/>Kafka-Compatible Telemetry Stream"]
        Cosmos[("🪐 <b>Azure Cosmos DB</b><br/>Multi-Region Active-Active<br/>5 Consistency Levels")]
        PgFlex[("🐘 <b>Azure PostgreSQL Flexible</b><br/>Zone-Redundant HA + PgBouncer")]
        Storage["💾 <b>Azure Disk ZRS & Azure Files NFSv4.1</b><br/>RWO & RWX Persistent Volumes"]
    end

    CSI -.->|"OIDC Federated Token"| KV
    SB -.->|"Queue Depth Metric"| KEDA
    UserPool -->|"Private Link"| Cosmos
    UserPool -->|"Private Link"| PgFlex
    UserPool -->|"Private Link"| EH
    UserPool --> Storage

🔬 Deep-Dive 1: AKS Internals, CNI Networking Models & Node Architecture

1. The 4 AKS Networking Models (Major Staff Interview Topic)

Choosing the wrong CNI plugin during AKS cluster creation requires a full cluster rebuild or complex node pool migration:

2. System vs. User Node Pools & Ephemeral OS Disks

• System Node Pools (mode: System): Must run at least 3 nodes across Availability Zones (1, 2, 3) tainted with CriticalAddonsOnly=true:NoSchedule. This guarantees application pods can never starve coredns, metrics-server, or konnectivity-agent (the secure tunnel between the Azure-managed API server and worker nodes).
• Ephemeral OS Disks (os_disk_type = "Ephemeral"):
  • The Problem: By default, Azure attaches a network-backed Managed Disk (StandardSSD_LRS or Premium_LRS) for the Linux OS. During a deployment spike, pulling 10 GB container images and writing containerd logs saturates the OS disk’s IOPS burst bucket, causing the node to freeze with PLEG is not healthy (NodeNotReady)!
  • The Architectural Fix: Always use Ephemeral OS Disks, which place the root filesystem directly on the VM host’s local NVMe/Temp SSD cache. Read/write latency drops from 5ms to < 0.2ms, node boot/scaling times drop by 50%, and OS disk storage is 100% free.

3. AKS Node Autoprovisioning (NAP — Managed Karpenter)

AKS offers Node Autoprovisioning (NAP), which runs a Microsoft-managed Karpenter controller inside the AKS control plane:

• Evaluates unschedulable pods (Pending) and provisions the exact optimal Azure VM SKU (e.g., Standard_D8ds_v5, Standard_D16ps_v5 ARM64 Cobalt/Ampere) in seconds without pre-creating tens of static VM Scale Set (VMSS) node pools.

🔐 Deep-Dive 2: AKS Entra Workload Identity & Azure Key Vault CSI

[!WARNING] Deprecation Trap: Never propose aad-pod-identity (v1) in an interview! It was deprecated because it used an iptables rule on every node to intercept calls to 169.254.169.254 (IMDS), which added latency, failed under high concurrency, and allowed pod-level race conditions.

How Modern Entra Workload Identity Works (OIDC Federation):

1. AKS OIDC Issuer: AKS exposes a public OpenID Connect discovery endpoint hosting its JSON Web Key Set (JWKS).
2. Mutating Webhook Projection: When a pod has the label azure.workload.identity/use: "true", the Workload Identity mutating webhook injects:
   • Environment variables: AZURE_CLIENT_ID, AZURE_TENANT_ID, AZURE_FEDERATED_TOKEN_FILE=/var/run/secrets/azure/tokens/azure-identity-token.
   • A projected volume containing a short-lived, cryptographically signed Kubernetes ServiceAccount JWT.
3. Federated Credential Exchange: The pod (or the Azure Key Vault Secret Store CSI Driver) sends that JWT to Microsoft Entra ID, which verifies the signature against the AKS OIDC Issuer URL (subject: system:serviceaccount:<namespace>:<sa-name>) and returns an Entra ID Access Token—zero client secrets ever exist!

🗄️ Deep-Dive 3: Azure Dependent Enterprise Services Architecture

In Indian enterprise & GCC interviews, System Design rounds always test how AKS interacts with its surrounding Azure PaaS services:

1. Ingress Tier: Azure Front Door + Application Gateway for Containers

• Legacy Pattern (AGIC — Application Gateway Ingress Controller): Used a single shared Application Gateway v2 instance updated via slow ARM control-plane calls (PUT requests taking 30–90 seconds whenever pods scaled up or down!).
• Modern Enterprise Standard (Application Gateway for Containers / ALB for AKS):
  • Built specifically for Kubernetes Gateway API (Gateway, HTTPRoute).
  • Uses a dedicated fast data-plane proxy API that registers new pod IPs in under 1 second (instead of 60-second ARM updates) and supports native mTLS, header rewriting, and canary traffic splitting.
  • Fronted globally by Azure Front Door Premium, which connects into the internal load balancer via Azure Private Link so zero public IPs exist on the AKS cluster!

2. Database Tier: Azure Cosmos DB (The 5 Consistency Levels & Hot Partition Trap)

Almost every Azure Architect interview includes a deep drill on Azure Cosmos DB:

flowchart LR
    C1["<b>1. Strong</b><br/>Linearizable Quorum<br/>RPO = 0, Highest Latency<br/>(Single-region or sync multi-AZ)"] --> C2["<b>2. Bounded Staleness</b><br/>Lags by at most K versions<br/>or T seconds (e.g. 5s)"]
    C2 --> C3["<b>3. Session (Default)</b><br/>Read-your-own-writes<br/>Monotonic reads per client"]
    C3 --> C4["<b>4. Consistent Prefix</b><br/>Updates appear in order<br/>(No out-of-order reads)"]
    C4 --> C5["<b>5. Eventual</b><br/>Lowest RU/s cost & latency<br/>No ordering guarantee"]

• The 5 Consistency Levels:
  1. Strong: Linearizable reads guaranteed to return the most recent committed write globally; doubles read RU cost compared to Eventual/Session and suffers speed-of-light cross-region write latency.
  2. Bounded Staleness: Reads may lag behind writes by at most K prefixes (e.g., 100 updates) or T time interval (e.g., 5 seconds). Ideal for multi-region DR where Strong latency is too high.
  3. Session (Default — 90% of Microservices): Guarantees Read-Your-Own-Writes and monotonic reads within a single client session token (x-ms-session-token) at half the RU cost of Strong!
  4. Consistent Prefix: Guarantees readers never see out-of-order writes (if writes are A, B, C, a reader sees A, A,B, or A,B,C, never A,C).
  5. Eventual: Lowest latency and cost, used for non-critical telemetry, view counters, or product reviews.
• The Infamous Cosmos DB Physical Partition & Hot Key Trap:
  • Every Cosmos DB Physical Partition has a hard ceiling of 50 GB storage and 10,000 RU/s throughput, while every Logical Partition Key has a hard ceiling of 20 GB.
  • The Trap: You provision a Cosmos DB collection with 100,000 RU/s, which Azure splits across 10 Physical Partitions (100,000 / 10 = 10,000 RU/s per physical partition maximum!). If a developer picks a poor partition key (e.g., /tenantId where one giant enterprise tenant generates 80% of traffic, or /date where 100% of today’s writes hit 2026-10-20), all writes hit Partition #1, triggering HTTP 429 RequestRateTooLarge at just 10,001 RU/s—even though 90,000 RU/s across the other 9 partitions sit completely idle!
  • The Fix: Use Hierarchical Partition Keys (Subpartitioning up to 3 levels: [/tenantId, /userId, /sessionId]) or Synthetic Suffixes so load spreads evenly across all physical partitions.

3. Messaging Tier: Azure Service Bus vs. Event Hubs vs. Event Grid

Interviewers love testing which Azure messaging service to pick in System Design:

4. Storage Tier: Azure Disk (ZRS) vs. Azure Files (NFSv4.1) vs. Azure Blob

• Azure Disk CSI (managed-csi-premium / UltraSSD_LRS): Block storage (ReadWriteOnce). Always use Premium_ZRS (Zone-Redundant Storage) for stateful pods in production so if AZ-1 goes down, Kubernetes can immediately reschedule the pod in AZ-2 and attach the exact same disk without waiting for cross-zone snapshots!
• Azure Files CSI (azurefile-csi-nfs): Shared file system (ReadWriteMany - RWX) over NFS v4.1 on Premium File Shares, connected via Private Endpoint.
• Azure Blob CSI (azureblob-fuse): Object storage mounted via BlobFuse2 or NFS v3—ideal for reading massive 100 GB LLM model weights across 50 GPU pods simultaneously.

💻 Production Manifests: AKS Workload Identity, Key Vault CSI & KEDA Service Bus Autoscaling

# 1. ServiceAccount annotated for Entra Workload Identity (OIDC Federation)
apiVersion: v1
kind: ServiceAccount
metadata:
  name: payments-workload-sa
  namespace: payments-prod
  annotations:
    azure.workload.identity/client-id: "11112222-3333-4444-5555-666677778888" # User-Assigned Managed Identity Client ID
    azure.workload.identity/tenant-id: "99998888-7777-6666-5555-444433332222"
---
# 2. SecretProviderClass mounting Azure Key Vault secrets via CSI Driver
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-kv-payments-secrets
  namespace: payments-prod
spec:
  provider: azure
  parameters:
    usePodIdentity: "false"
    clientID: "11112222-3333-4444-5555-666677778888" # Uses Entra Workload Identity OIDC token!
    keyvaultName: "kv-payments-prod-ind"
    tenantId: "99998888-7777-6666-5555-444433332222"
    objects: |
      array:
        - |
          objectName: cosmos-db-primary-connection
          objectType: secret
---
# 3. KEDA TriggerAuthentication + ScaledObject scaling pods on Azure Service Bus Queue Depth
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: azure-servicebus-auth
  namespace: payments-prod
spec:
  podIdentity:
    provider: azure-workload
    identityId: "11112222-3333-4444-5555-666677778888"
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: payment-settlement-worker-scaler
  namespace: payments-prod
spec:
  scaleTargetRef:
    name: payment-settlement-worker
  minReplicaCount: 2
  maxReplicaCount: 150
  triggers:
    - type: azure-servicebus
      metadata:
        queueName: settlement-orders-queue
        namespace: sb-payments-prod-ind
        messageCount: "25" # Spawn 1 new pod for every 25 queued messages
      authenticationRef:
        name: azure-servicebus-auth

🧭 Step-by-Step Operational Playbook: Deploying Private AKS with Cilium Overlay & Workload Identity

Follow this step-by-step operational sequence to provision an enterprise-grade AKS cluster integrated with Azure Key Vault and Managed Prometheus:

Step 1: Provision Private AKS with Cilium CNI Overlay & Ephemeral OS Disks

1. Plain-English Goal: Create a private AKS cluster using network-plugin-mode overlay with ebpf-dataplane cilium, OIDC Issuer enabled, and Ephemeral NVMe OS disks.
2. Action:

   az aks create \
     --resource-group rg-payments-prod \
     --name aks-payments-prod \
     --enable-private-cluster \
     --network-plugin azure \
     --network-plugin-mode overlay \
     --ebpf-dataplane cilium \
     --pod-cidr 100.64.0.0/16 \
     --node-osdisk-type Ephemeral \
     --enable-oidc-issuer \
     --enable-workload-identity \
     --enable-addons azure-keyvault-secrets-provider \
     --zones 1 2 3

Step 2: Establish the Entra ID Federated Identity Credential

1. Plain-English Goal: Link your Azure User-Assigned Managed Identity to the Kubernetes ServiceAccount (system:serviceaccount:payments-prod:payments-workload-sa) using the AKS OIDC Issuer URL.
2. Action:

   AKS_OIDC_ISSUER=$(az aks show -g rg-payments-prod -n aks-payments-prod --query "oidcIssuerProfile.issuerUrl" -o tsv)
   
   az identity federated-credential create \
     --name "fed-payments-workload-sa" \
     --identity-name "id-payments-prod" \
     --resource-group rg-payments-prod \
     --issuer "$AKS_OIDC_ISSUER" \
     --subject "system:serviceaccount:payments-prod:payments-workload-sa" \
     --audience "api://AzureADTokenExchange"

Step 3: Verify Pod Token Projection & Key Vault CSI Mount

1. Plain-English Goal: Confirm the pod received its federated OIDC token file and successfully mounted secrets from Azure Key Vault without any static credentials.
2. Action:

   kubectl describe pod -n payments-prod -l app=payment-settlement-worker | grep -A 6 "AZURE_FEDERATED_TOKEN_FILE"
   kubectl exec -n payments-prod deploy/payment-settlement-worker -- ls -la /mnt/secrets-store

Step 4: Triage Live Logs with Kusto Query Language (KQL) in Azure Monitor

1. Plain-English Goal: Query container logs and Cosmos DB throttling events across the subscription using KQL.
2. Action:

   // KQL Query: Find Cosmos DB HTTP 429 Throttled Partitions in the last 1 hour
   AzureDiagnostics
   | where ResourceProvider == "MICROSOFT.DOCUMENTDB" and Category == "DataPlaneRequests"
   | where statusCode_s == "429"
   | summarize ThrottledCount = count(), MaxRUs = max(todouble(requestCharge_s)) by partitionKeyRangeId_s, OperationName
   | order by ThrottledCount desc

🛠️ Real-World Troubleshooting & Failure Modes

Scenario 1: AKS Workload Identity Fails with AADSTS70021: No matching federated identity record found

• Symptom: Pods crash at startup when attempting to read secrets from Azure Key Vault or authenticate to Azure Service Bus, logging: AADSTS70021: No matching federated identity record found for presented assertion.
• Under-the-Hood Root Cause: Entra ID strictly validates three fields inside the Kubernetes JWT against the az identity federated-credential:
  1. iss (Issuer URL — often has a trailing / mismatch if copied manually!).
  2. sub (Subject — must exactly match system:serviceaccount:<namespace>:<serviceaccount-name>—case-sensitive!).
  3. aud (Audience — must be api://AzureADTokenExchange). Additionally, if the Pod itself is missing the label azure.workload.identity/use: "true" on spec.template.metadata.labels, the webhook never injects the token file in the first place!
• Immediate Triage & Fix:

  # 1. Inspect the exact issuer and subject configured on the Managed Identity:
  az identity federated-credential list -g rg-payments-prod --identity-name id-payments-prod -o yaml
  
  # 2. Verify the Pod has the webhook label:
  kubectl get pod <pod-name> -n payments-prod --show-labels | grep "azure.workload.identity/use=true"

Scenario 2: Stateful Pod Stuck in ContainerCreating After AZ Failure (AzureDisk Multi-Zone Attach Failure)

• Symptom: Availability Zone 1 experiences an impairment. A StatefulSet pod running in centralindia-1 is rescheduled to a healthy node in centralindia-2, but hangs indefinitely in ContainerCreating with: AttachVolume.Attach failed for volume "pvc-123": disk is in zone centralindia-1 and cannot be attached to VM in zone centralindia-2.
• Under-the-Hood Root Cause: The StorageClass was provisioned with StandardSSD_LRS or Premium_LRS (Locally Redundant Storage), which pins the physical disk to a single Availability Zone!
• Permanent Fix: Use a Zone-Redundant Storage (Premium_ZRS) StorageClass (skuName: Premium_ZRS) for stateful workloads requiring cross-AZ failover, which synchronously replicates blocks across all 3 Availability Zones and can be attached to nodes in any zone immediately.

📝 Knowledge Check & Real-World Interview Scenarios