⚡
Zero to Platform Architect2026
War-Room Triage & 43 Traps
War-Room Triage & 43 Traps⏱️ 11 min read⚡ Staff SRE Standard
🎯 Foundations Test Bank →
Quick Jump:🧭 CLI Playbook📝 Q&A Drill

☸️ Kubernetes & Stateful Storage Triage Guide

This guide details the most common, tricky, and catastrophic production Kubernetes failure modes encountered by Staff and Principal Platform Architects, along with under-the-hood root causes, diagnostic command sequences, and production-tested remediations.

🧭 Beginner-Friendly First-5-Minutes War-Room Playbook (Step-by-Step SOP)

Mental Model (Think Like an ER Doctor): When a Kubernetes pod is failing, never blindly run kubectl delete pod (that destroys forensic evidence!). Instead, check Scheduling (Where?) → Storage/Network Attachment (How?) → Container Runtime Exit Code (Why?).

Step 1: Check Cluster-Wide Warning Events (Minute 0–1)

kubectl get events -A --field-selector type=Warning --sort-by=".lastTimestamp" | tail -n 25
  • What to look for: FailedScheduling (CPU/RAM/AZ mismatch), FailedAttachVolume (EBS stuck on old node), or FailedCreatePodSandBox (CNI IP exhaustion).

Step 2: Inspect Pod Phase & Last Termination Exit Code (Minute 1–2)

kubectl get pods -n <namespace> -o wide
kubectl describe pod <pod-name> -n <namespace> | grep -A 12 "Last State:"
  • What to look for: Exit Code 137 (SIGKILL / Linux Kernel OOMKiller), Exit Code 143 (SIGTERM graceful shutdown), or Exit Code 1 (App config/exception error).

Step 3: Pull Previous Container Logs Before Crash (Minute 2–3)

kubectl logs <pod-name> -n <namespace> --previous --tail=100

Step 4: Verify Stateful CSI Volume & AZ Topology Alignment (Minute 3–4)

kubectl get pvc,pv -n <namespace>
kubectl get volumeattachment | grep <pv-name>

Step 5: Stabilize & Mitigate Blast Radius (Minute 4–5)

  • If a bad rollout caused the crash, immediately roll back: kubectl rollout undo deployment/<name> -n <namespace>.

🎯 High-Impact Real-World Scenarios

Scenario 1: EBS CSI Multi-AZ Scheduling Deadlock (VolumeZoneConflict)

The Symptom

A stateful service (e.g. PostgreSQL, Kafka, or Milvus) is updated or rescheduled. The pod remains in Pending state indefinitely with the following event:

Warning  FailedScheduling  pod/kafka-0  0/12 nodes are available: 
12 node(s) had volume node affinity conflict (VolumeZoneConflict).

Under-the-Hood Root Cause

The StorageClass was configured with volumeBindingMode: Immediate. When the PersistentVolumeClaim was created, the EBS CSI driver immediately provisioned an AWS EBS volume in an arbitrary Availability Zone (e.g., us-east-1a). Later, when the Kubernetes scheduler evaluated node placement, it placed the Pod in us-east-1b (due to node capacity, anti-affinity, or spot availability). Because AWS EBS volumes cannot cross Availability Zone boundaries, the volume cannot attach, deadlocking the pod forever.

sequenceDiagram
    autonumber
    participant K8s as Kubernetes Scheduler
    participant CSI as EBS CSI Driver
    participant AWS as AWS EC2 (EBS API)
    participant Node as Node in us-east-1b

    Note over CSI,AWS: Immediate Binding: Disk created in us-east-1a before pod scheduling!
    CSI->>AWS: CreateVolume in us-east-1a
    AWS-->>CSI: vol-012345 (AZ: us-east-1a)
    K8s->>Node: Assigns Pod to Node in us-east-1b (Resource availability)
    Node->>AWS: AttachVolume vol-012345 to Node in us-east-1b
    AWS-->>Node: ❌ ERROR: VolumeZoneConflict! EBS cannot cross AZs!
    Note over K8s,Node: 🚨 Pod stuck in Pending forever!

CLI Triage Commands

# 1. Inspect the volume binding mode of the StorageClass
kubectl get storageclass ebs-sc -o jsonpath='{.volumeBindingMode}'

# 2. Inspect the Availability Zone of the provisioned PersistentVolume
kubectl get pv $(kubectl get pvc kafka-data-kafka-0 -o jsonpath='{.spec.volumeName}') \
  -o jsonpath='{.spec.nodeAffinity.required.nodeSelectorTerms[*].matchExpressions[*].values}'

# 3. Inspect where Karpenter / K8s placed the node
kubectl get node -l node.kubernetes.io/instance-type -o wide

Remediation & Architectural Fix

  • Permanent Fix: Always set volumeBindingMode: WaitForFirstConsumer on all production block-storage StorageClass definitions. This delays EBS volume creation until the Pod is placed on a specific node, ensuring the volume is provisioned in the exact same AZ.
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-gp3-sc
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer # Enforces AZ alignment
allowVolumeExpansion: true
parameters:
  type: gp3

Scenario 2: Karpenter Spot Disruption Avalanche & Headroom Starvation

The Symptom

During an AWS EC2 Spot market interruption event, 40 worker nodes receive a 2-minute termination notice simultaneously. Workloads crash, API error rates spike to 100%, and Karpenter cannot provision replacement instances fast enough, reporting:

ERROR  controller.nodeclaim.lifecycle  failed to launch instance: 
InsufficientInstanceCapacity: We currently do not have sufficient capacity in the requested instance types.

Under-the-Hood Root Cause

The platform team constrained Karpenter’s NodePool to a narrow set of instance types (e.g. only m5.2xlarge) within a single Availability Zone. When AWS reclaimed spot capacity in that pool, all instances were terminated at once. Karpenter had no fallback instance types or alternate AZs, entering an exponential backoff loop while hundreds of evicted pods remained unschedulable.

CLI Triage Commands

# 1. Stream Karpenter controller logs for provisioning errors
kubectl logs -n karpenter -l app.kubernetes.io/name=karpenter -c controller --tail=100 -f | grep -E "insufficient|failed|reclaim"

# 2. Inspect active NodeClaims and their current state
kubectl get nodeclaims -o wide

# 3. Check unschedulable pods count
kubectl get pods --all-namespaces --field-selector=status.phase=Pending

Remediation & Architectural Fix

  • Immediate Break-Glass: Edit the Karpenter EC2NodeClass and NodePool to allow On-Demand fallback and a broad range of instance types:
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: general-compute
spec:
  template:
    spec:
      requirements:
        - key: "karpenter.sh/capacity-type"
          operator: In
          values: ["spot", "on-demand"] # Allows fallback to On-Demand
        - key: "karpenter.k8s.aws/instance-category"
          operator: In
          values: ["c", "m", "r"]       # Diversify across compute, general, memory
        - key: "karpenter.k8s.aws/instance-generation"
          operator: Gt
          values: ["4"]                 # Use gen 5, 6, 7
      disruption:
        consolidationPolicy: WhenUnderutilized
        expireAfter: 720h
  • Architectural Hardening: Enable AWS SQS Interruption Queuing in Karpenter to consume EC2 Spot Rebalance Recommendations, triggering proactive graceful draining 2 minutes before termination.

Scenario 3: CoreDNS conntrack Race & 5-Second Latency Spikes

The Symptom

Microservices experience intermittent 5004ms (5.004 seconds) latency spikes when invoking internal service endpoints or external cloud APIs. The application logs report connection timeouts, yet network bandwidth and CPU utilization are below 30%.

Under-the-Hood Root Cause

By default, Linux glibc evaluates DNS queries using ndots:5 (from /etc/resolv.conf). For any hostname with fewer than 5 dots (e.g., api.stripe.com), glibc appends internal search domains (my-namespace.svc.cluster.local, svc.cluster.local, cluster.local). This triggers 4 sequential DNS queries over UDP.

Because glibc sends the A and AAAA queries over the same socket simultaneously, Linux kernel netfilter encounters a race condition when inserting conntrack entries for both UDP packets on the same tuple. One packet is silently dropped by the kernel, forcing glibc to wait for its default timeout of 5.0 seconds before retrying.

CLI Triage Commands

# 1. Verify conntrack drop counters in kernel dmesg
sudo dmesg -T | grep -E "nf_conntrack: table full|drop"

# 2. Inspect DNS query latency using dig from inside a troubled pod
kubectl exec -it <app-pod> -- time dig api.stripe.com

# 3. Check CoreDNS metrics in Prometheus
# sum(rate(coredns_dns_request_duration_seconds_bucket{le="5.0"}[5m]))

Remediation & Architectural Fix

  1. Deploy NodeLocal DNSCache: Runs a DNS caching agent on every node as a DaemonSet (169.254.20.10). It terminates UDP queries locally on the loopback interface, eliminating iptables DNAT and kernel conntrack state for DNS.
  2. Tune Pod DNS Config: For services with high outbound traffic, configure ndots: 2 in the pod’s dnsConfig:
spec:
  dnsConfig:
    options:
      - name: ndots
        value: "2"
      - name: single-request-reopen

Scenario 4: PDB Deadlock Freezing EKS Cluster Node Drains

The Symptom

During an automated EKS cluster AMI upgrade or node refresh, kubectl drain <node> hangs indefinitely:

evicting pod default/payment-service-7f89b-x9z2p
error when evicting pod "payment-service-7f89b-x9z2p": Cannot evict pod as it would violate the pod's disruption budget.
eviction throttled, waiting 10s...

The node never drains, blocking the CI/CD pipeline or AWS Auto Scaling Group refresh.

Under-the-Hood Root Cause

The PodDisruptionBudget was configured with minAvailable: 2 on a deployment with replicas: 2. When the drain controller attempts to evict the first pod, the number of available replicas would drop to 1, which violates the minAvailable: 2 policy. The Kubernetes Eviction API returns HTTP 429 Too Many Requests, and the drain controller retries forever.

CLI Triage Commands

# 1. List all PDBs and identify which are blocking evictions
kubectl get pdb --all-namespaces

# 2. Inspect the current status of the blocking PDB
kubectl get pdb payment-service-pdb -o yaml | grep -A 8 status

Remediation & Architectural Fix

  • Break-Glass Emergency Command: Scale the deployment up to 3 replicas or temporarily bypass eviction:
kubectl scale deployment payment-service --replicas=3
# Or break-glass delete the PDB during maintenance:
kubectl delete pdb payment-service-pdb
  • Architectural Hardening: Never configure minAvailable == replicas. Always specify maxUnavailable: 1 or maxUnavailable: 25% to guarantee at least one pod can be safely evicted during rolling maintenance:
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: payment-service-pdb
spec:
  maxUnavailable: 1 # Allows 1 pod to be evicted safely
  selector:
    matchLabels:
      app: payment-service

Scenario 5: Admission Webhook Crash Freezing the Entire Cluster

The Symptom

Engineers cannot deploy, scale, or delete any pods across any namespace in the cluster. Every kubectl apply returns:

Error from server (InternalError): Internal error occurred: failed calling webhook 
"validate.kyverno.io": Post "https://kyverno-svc.kyverno.svc:443/validate": 
context deadline exceeded

Under-the-Hood Root Cause

A validating or mutating admission webhook (e.g. Kyverno, OPA Gatekeeper, or Istio) was registered with failurePolicy: Fail and a selector matching all namespaces, including kube-system and the webhook’s own namespace. When the webhook pod crashed or its node was rebooted, the Kubernetes API server could not reach the webhook. Because failurePolicy: Fail mandates rejection on webhook failure, the API server rejected all new pod creations—preventing the webhook pod itself from ever restarting!

CLI Triage Commands

# 1. List all validating webhook configurations
kubectl get validatingwebhookconfigurations

# 2. Inspect failurePolicy and timeoutSeconds
kubectl get validatingwebhookconfiguration kyverno-policy-webhook -o yaml | grep failurePolicy

Remediation & Architectural Fix

  • Break-Glass Emergency Command: Patch the webhook to failurePolicy: Ignore or delete the webhook configuration immediately:
kubectl delete validatingwebhookconfiguration kyverno-policy-webhook
  • Architectural Hardening:
    1. Always add namespaceSelector to exclude kube-system, kyverno, and other infrastructure namespaces from webhook evaluation.
    2. Run at least 3 replicas of admission webhooks with anti-affinity across multiple AZs and high-priority priorityClassName: system-cluster-critical.

Scenario 6: Azure AKS Workload Identity Failure (AADSTS70021: No matching federated identity record found)

The Symptom

Pods deployed on Azure Kubernetes Service (AKS) fail to mount secrets from Azure Key Vault via the Secret Store CSI Driver or fail to authenticate to Azure Service Bus, crashing with:

AADSTS70021: No matching federated identity record found for presented assertion.

Under-the-Hood Root Cause

Entra Workload Identity exchanges a projected Kubernetes ServiceAccount JWT for an Entra ID access token via OIDC federation. The exchange fails when:

  1. The issuer URL configured on the Azure User-Assigned Managed Identity (az identity federated-credential) is missing or has a mismatched trailing slash (/) compared to az aks show --query oidcIssuerProfile.issuerUrl.
  2. The subject string does not exactly match system:serviceaccount:<namespace>:<serviceaccount-name> (case-sensitive).
  3. The Pod specification is missing the mandatory webhook label azure.workload.identity/use: "true" under spec.template.metadata.labels, causing the mutating webhook to skip injecting AZURE_FEDERATED_TOKEN_FILE.

Diagnostic & Remediation Commands

# 1. Compare AKS OIDC Issuer URL against Federated Credential Issuer
az aks show -g rg-prod -n aks-prod --query "oidcIssuerProfile.issuerUrl" -o tsv
az identity federated-credential list -g rg-prod --identity-name id-payments-prod -o table

# 2. Verify Pod has the mutating webhook label and projected token volume
kubectl get pod <pod-name> -n payments-prod -o jsonpath="{.metadata.labels}"

Scenario 7: AKS Node NotReady (PLEG is not healthy) from Managed OS Disk IOPS Throttling

The Symptom

During a large microservice rollout or node scale-up, multiple AKS worker nodes simultaneously flip to NotReady with Kubelet stopped posting node status and PLEG is not healthy: pleg was last seen active 3m12s ago.

Under-the-Hood Root Cause

The node pool was provisioned with network-attached Managed OS Disks (StandardSSD_LRS or small Premium_LRS) instead of Ephemeral OS Disks. When 30 pods simultaneously pull multi-gigabyte container images and unpack layers in /var/lib/containerd while flushing container stdout logs, the OS disk exhausts its IOPS burst bucket and stalls kernel disk I/O—blocking containerd and the Kubelet runtime loop!

Diagnostic & Remediation Commands

# 1. Verify if Node Pool is using Managed vs Ephemeral OS Disks
az aks nodepool show -g rg-prod --cluster-name aks-prod -n userpool --query "osDiskType"

# 2. Permanent Fix: Recreate/migrate node pools with Ephemeral NVMe OS Disks (0 network IOPS latency, $0 cost)
az aks nodepool add -g rg-prod --cluster-name aks-prod -n userpoolv2 --node-osdisk-type Ephemeral --node-vm-size Standard_D8ds_v5