🚨 Enterprise Platform Troubleshooting & Incident Defense Hub

Welcome to the Platform Architect Incident Defense & Troubleshooting Hub. This guide provides staff-level diagnostic mental models, deterministic decision trees, and production-tested runbooks to isolate, triage, and remediate critical failures across multi-cloud infrastructure, Kubernetes clusters, service meshes, GitOps pipelines, and AI platforms.

🧭 The Architect’s Incident Response Mental Model

In high-stakes enterprise outages (P0/P1 incidents), junior engineers guess and restart services; Principal Platform Architects apply deterministic triage frameworks:

flowchart TD
    Alert["🚨 1. P0/P1 Alert Fires<br/>(PagerDuty / Opsgenie)"] --> Assess["⏱️ 2. 60-Second Blast Radius Triage<br/>• Who is impacted? (Single tenant vs entire region)<br/>• What changed? (Recent GitOps commit, Cloud provider event)<br/>• Which golden signal failed? (Latency, Errors, Saturation)"]
    
    Assess --> Contain["🛡️ 3. Immediate Containment (Stop the Bleed)<br/>• Shed non-critical traffic (WAF rate limit, drop background queues)<br/>• Flip DNS / Route 53 ARC routing control to secondary region<br/>• Abort canary rollout or pin previous stable release"]
    
    Contain --> Diagnose["🔬 4. Layer-by-Layer Root Cause Isolation<br/>• Layer 1: Hardware & Cloud Fabric (AWS Health, Quotas, Hypervisor)<br/>• Layer 2: Network & Routing (BGP, TGW, VPC Peering, MTU, DNS)<br/>• Layer 3: Kernel & OS (cgroups, OOM killer, conntrack, file descriptors)<br/>• Layer 4: Platform & Mesh (K8s API, etcd, Cilium eBPF, Ingress, IAM)<br/>• Layer 5: Application & State (DB connection pools, locks, schemas)"]
    
    Diagnose --> Stabilize["⚖️ 5. Stabilization & Controlled Warm-Up<br/>• Gradual traffic reintroduction (avoid thundering herd)<br/>• Verify error budgets & synthetic transactions return to green"]
    
    Stabilize --> Prevent["🔒 6. Automated Guardrails & RCA<br/>• Enforce preventative OPA/Kyverno policies & SCPs<br/>• Implement multi-window burn rate alerts in Prometheus<br/>• Publish blameless RCA within 48 hours"]

    classDef alert fill:#b91c1c,stroke:#f87171,stroke-width:2px,color:#ffffff;
    classDef step fill:#1e293b,stroke:#38bdf8,stroke-width:2px,color:#f8fafc;
    class Alert alert;
    class Assess,Contain,Diagnose,Stabilize,Prevent step;

📊 USE vs. RED Diagnostic Frameworks

When investigating platform anomalies, choose the right diagnostic methodology depending on whether you are inspecting resources (infrastructure) or workloads (services):

Dimension	The USE Method (Brendan Gregg)	The RED Method (Tom Wilkie)
Primary Target	Hardware, VMs, Nodes, Disks, Network Interfaces, Kernels	Microservices, APIs, Databases, HTTP Endpoints
Key Metric 1	Utilization: Time resource is busy (e.g. Disk % busy, CPU %)	Rate: Requests received per second (QPS/RPS)
Key Metric 2	Saturation: Extra work queued that cannot be serviced (Load avg, run queue, socket backlog)	Errors: Number of failed requests per second (HTTP 5xx, gRPC codes)
Key Metric 3	Errors: Hard failure events (disk I/O errors, dropped packets, memory ECC)	Duration: Latency distribution (p50, p95, p99 request time)
When to Use	When investigating node degradation, Karpenter Spot thrashing, or EBS I/O stalls	When investigating customer-facing slowness, API 504 timeouts, or Ingress anomalies

🔍 The Linux Kernel & Network Packet Forensics Decision Tree

When network traffic fails or stalls silently without explicit HTTP errors, trace the packet through the Linux kernel stack:

flowchart TD
    Start["Packet Fails to Reach Destination"] --> DNS{"1. Can the node resolve the hostname?"}
    DNS -- "No" --> DNSFix["• Test: dig +short my-service.internal @169.254.20.10<br/>• Check ndots:5 glibc query amplification<br/>• Check CoreDNS pod status & NodeLocal DNSCache"]
    DNS -- "Yes" --> Route{"2. Does a valid kernel route exist?"}
    Route -- "No" --> RouteFix["• Test: ip route get <target-ip><br/>• Check AWS VPC route table (0.0.0.0/0 to NAT GW / TGW)<br/>• Verify Transit Gateway route propagation"]
    Route -- "Yes" --> Netfilter{"3. Is packet dropped by iptables or eBPF?"}
    Netfilter -- "Dropped" --> NetfilterFix["• Test: iptables -nvL -t filter --line-numbers<br/>• Test: cilium monitor --type drop<br/>• Check default-deny CiliumNetworkPolicy missing port 53/Egress"]
    Netfilter -- "Allowed" --> Conntrack{"4. Is kernel conntrack table full?"}
    Conntrack -- "Full" --> ConntrackFix["• Test: dmesg -T | grep nf_conntrack: table full<br/>• Test: sysctl net.netfilter.nf_conntrack_count<br/>• Increase: sysctl -w net.netfilter.nf_conntrack_max=1048576"]
    Conntrack -- "Healthy" --> MTU{"5. Does packet stall only on large payloads?"}
    MTU -- "Yes" --> MTUFix["• Test: ping -M do -s 1472 <target-ip><br/>• Check MTU fragmentation (WireGuard/Geneve adds 50-60 bytes)<br/>• Clamp MTU to 1420 on overlay interfaces"]
    MTU -- "No" --> Socket["6. Check Application Socket Backlog<br/>• Test: ss -tlpn | grep <port><br/>• Check Recv-Q vs Send-Q (Application thread pool starved)"]

☸️ Kubernetes Pod Lifecycle Fast-Triage Matrix

Pod State	Most Common Root Cause	Fast Verification CLI Command	Immediate Break-Glass Remediation
`Pending`	• Node resource exhaustion (CPU/Memory) • Affinity/Toleration mismatch • StorageClass single-AZ deadlock	`kubectl describe pod <pod> \| grep -A 5 Events`	Add Karpenter NodePool capacity or switch StorageClass to `WaitForFirstConsumer`.
`CrashLoopBackOff`	• Missing config / secret key • Liveness probe timeout • Process exited with code 1 or 137 (OOM)	`kubectl logs <pod> --previous --tail=100`	Review container logs, increase liveness `initialDelaySeconds`, or increase memory limits.
`OOMKilled` (Exit 137)	• Memory usage exceeded `resources.limits.memory` • JVM heap sizing larger than cgroup limit	`kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[*].lastState.terminated.reason}'`	Increase memory limit or configure `-XX:MaxRAMPercentage=75.0` for JVM applications.
`Terminating` (Stuck)	• Finalizer blocking deletion • Node drained with unresponsive CSI driver	`kubectl get pod <pod> -o jsonpath='{.metadata.finalizers}'`	Clear finalizer if safe: `kubectl patch pod <pod> -p '{"metadata":{"finalizers":null}}' --type=merge`.
`CreateContainerConfigError`	• Secret or ConfigMap referenced in `envFrom` does not exist	`kubectl describe pod <pod> \| grep -i configmap`	Deploy missing Secret via External Secrets Operator or restore ConfigMap.
`ImagePullBackOff`	• Missing image tag • ECR token expired • Private registry rate limit (HTTP 429)	`kubectl describe pod <pod> \| grep -A 3 Failed`	Verify image digest exists in registry; verify ECR node IAM permissions (`ecr:GetAuthorizationToken`).

🛠️ Production Break-Glass CLI Cheat Sheet

1. Kernel, Sockets & Conntrack Diagnostics

# Check top socket consumers and queue saturation (Recv-Q > Send-Q indicates unhandled requests)
ss -tulpn

# Inspect kernel dmesg for OOM killer invocations and conntrack drops in the last 15 minutes
sudo dmesg -T | grep -E -i "oom|conntrack|drop|segfault" | tail -n 30

# Live network packet capture filtering only TCP RST flags (hunting unexpected resets)
sudo tcpdump -nn -i any "tcp[tcpflags] & (tcp-rst) != 0" -c 50

# Inspect kernel system call latency for a stuck process (D-state)
cat /proc/<PID>/stack
sudo strace -p <PID> -T -e trace=network,file -s 256

2. Kubernetes Node & Networking Forensics

# Identify top pods consuming memory/CPU across the entire cluster
kubectl top pods --all-namespaces --sort-by=memory | head -n 25

# Launch an ephemeral debug container with full host network privileges to inspect node interfaces
kubectl debug node/<node-name> -it --image=nicolaka/netshoot

# Query Cilium eBPF packet drop monitor
kubectl exec -n kube-system ds/cilium -c cilium-agent -- cilium monitor --type drop

# Inspect CoreDNS latency and query volume in real time
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=100 | grep -E "DENIED|SERVFAIL|NXDOMAIN"

3. GitOps & Cloud Drift Triage

# Check ArgoCD application sync errors and diff without touching the cluster
argocd app get <app-name> --refresh
argocd app diff <app-name>

# Check Crossplane Managed Resource reconciliation errors
kubectl get managed -o custom-columns=KIND:.kind,NAME:.metadata.name,READY:.status.conditions[?(@.type=="Ready")].status,SYNCED:.status.conditions[?(@.type=="Synced")].status,MESSAGE:.status.conditions[?(@.type=="Synced")].message

# Force emergency unlock on stuck Terraform state (use only after verifying no pipeline is executing)
terraform force-unlock <lock-id>

🎯 Common Tricky Issues Fast-Lookup Matrix

Looking for specific error messages, silent packet drops, or production traps? Consult the centralized index:

👉 The Staff Architect’s Compendium of 43 Common Tricky Issues & Respective Guides — Categorized by Silent Blackholes, Cascading Storms, Deadlocks & Quorum Loss, State Drift Traps, Identity Windows, and Hidden Sizing Bottlenecks.

📖 Deep-Dive Incident Playbooks

Explore the specialized troubleshooting guides below:

☸️ Kubernetes & Stateful Storage Triage Guide — Karpenter Spot disruptions, EBS/EFS CSI zone deadlocks, and PDB drain hangs.
🌐 Multi-Cloud & Enterprise Networking Triage Guide — Transit Gateway blackholes, Megaport BGP asymmetric routing, and AWS IAM OIDC failures.
🚀 GitOps, Control Planes & LLMOps Triage Guide — Argo Rollouts schema traps, Crossplane KCL composition validation, and Milvus/vLLM GPU memory exhaustion.
📜 IaC, Terraform, Vault & Policy-as-Code Triage Guide — DynamoDB state lock deadlocks, cyclic security group dependencies, and Vault auto-unseal failures.
🛡️ DevSecOps, Supply Chain & Fleet Patching Triage Guide — Air-gapped Cosign KMS signing, External Secrets Operator rate limits, and machine-id duplications.
🕸️ Service Mesh, eBPF & Chaos Engineering Triage Guide — Cilium WireGuard MTU drops, Chaos Mesh zombie tc netem rules, and Backstage catalog processor crashes.
🏛️ Enterprise SRE, Databases & FinOps Triage Guide — Aurora connection pool starvation, Route 53 ARC flapping, and OpenCost unallocated idle spend attribution.