🎯 The Staff Architect’s Compendium of Common Tricky Issues & Respective Guides
This compendium is a high-density, fast-lookup diagnostic directory of the 43 most tricky, subtle, and catastrophic production platform issues encountered in enterprise environments. Each issue outlines its diagnostic signature, under-the-hood root cause, immediate triage command, permanent architectural resolution, and a direct link to its specialized deep-dive guide.
🧠 Beginner Mental Model: How to Use This Diagnostic Directory
Think of this directory as an Emergency Room Symptom-to-Diagnosis Chart:
- Step 1 (Match the Symptom): Find the exact error message or latency pattern you see on your screen in the first column (
Symptom / Error Signature). - Step 2 (Understand the Hidden Trap): Read the
Under-the-Hood Root Causecolumn—in platform engineering, the real cause is usually 1 layer deeper than the error message suggests (e.g., a “DNS timeout” is often actually a Linux kernelconntracktable overflow!). - Step 3 (Run the 1-Line Triage Command): Copy and run the
Immediate Triage Commandto prove or disprove the hypothesis in under 30 seconds. - Step 4 (Execute the Deep-Dive Playbook): Click the linked guide in the last column for the full step-by-step remediation runbook.
🧭 Navigation by Operational Category
mindmap
root((Tricky Issues Directory))
Silent Blackholes
WireGuard MTU Blackhole
TGW Appliance Mode Asymmetry
CoreDNS conntrack Race
Megaport BGP MED Flapping
PrivateLink Cross-Zone Disabled
Chaos Mesh Zombie netem
Cascading Storms
Aurora Thundering Herd
ESO Secrets Manager Throttling
Prometheus TSDB Cardinality OOM
NAT Gateway SNAT Exhaustion
LiteLLM Upstream Retry Storm
TIME_WAIT Socket Leaks
Deadlocks & Quorum Loss
EBS CSI Multi-AZ Deadlock
Terraform DynamoDB State Lock
Vault Raft Quorum Loss
PDB Node Drain Hang
Admission Webhook Freeze
Security Group Cyclic Graph
State Drift & Schema Traps
Canary Dropped Column Abort
ArgoCD Mutating Webhook Loop
Crossplane Finalizer Deadlock
Golden AMI machine-id Duplicate
Git Force-Push Main Rewind
Git LFS 130-byte Pointer Build Fail
Identity & Permissions
Air-Gapped Cosign Rekor Timeout
Offboarded Employee Active STS
AWS IAM OIDC Thumbprint Expire
SCP Root Override Illusion
Lambda VPC ENI IP Starvation
Resource & Capacity Traps
CFS CPU Quota Latency Spike
Milvus HNSW 1.5x RAM OOM
vLLM PagedAttention CUDA OOM
ALB Deregistration Race 502s
OpenCost Unallocated Idle Spend🕳️ Category 1: Silent Blackholes & Kernel Packet Drops
| Issue & Error Signature | Root Cause Under the Hood | Fast Triage Command | Architectural Fix | Respective Deep-Dive Guide |
|---|---|---|---|---|
Cilium WireGuard MTU Silent BlackholeClient Hello sent... (hangs indefinitely) |
Tunnel headers (+60B) cause packets to exceed 1500B. DF bit set causes cloud fabric to drop packets silently without ICMP. | ping -M do -s 1420 <target-ip> |
Clamp Cilium MTU to 1420 in Helm values; enable TCP MSS clamping. |
🕸️ Service Mesh & Chaos Triage |
Transit Gateway Appliance Mode AsymmetryTCP connections stall during TLS or reset with TCP RST |
TGW hashes forward and return packets to different AZ firewalls. Firewall 2 has no connection state and drops SYN-ACK. | aws ec2 describe-transit-gateway-vpc-attachments --query "...ApplianceModeSupport" |
Enable ApplianceModeSupport=enable on the Inspection VPC attachment. |
🌐 Multi-Cloud Networking Triage |
CoreDNS conntrack UDP RaceIntermittent 5004ms HTTP timeouts in microservices |
ndots:5 triggers simultaneous A and AAAA queries. Netfilter races on UDP tuple insertion, silently dropping 1 packet. |
dmesg -T | grep nf_conntrack |
Deploy NodeLocal DNSCache DaemonSet (169.254.20.10); configure ndots: 2. |
☸️ Kubernetes & Stateful Triage |
Megaport MCR BGP Asymmetric Routing150ms+ latency spikes & TCP disconnects every 90s |
AWS and Azure advertise same subnets; unequal BGP Local Preference causes return traffic to route over public VPN. | traceroute -T -p 443 <dest-ip> |
Enforce deterministic BGP Local Preference (200 primary, 100 backup); prepend AS-path 3×. | 🌐 Multi-Cloud Networking Triage |
PrivateLink Cross-Zone Balancing Disabled50% of requests succeed, 50% time out after 30s |
NLB has targets in AZ-a only, but consumer sends packets to AZ-b endpoint ENI with cross-zone balancing off. | aws elbv2 describe-load-balancer-attributes |
Set load_balancing.cross_zone.enabled=true on the NLB. |
🌐 Multi-Cloud Networking Triage |
Chaos Mesh Zombie tc netem RulesDeleted chaos test leaves persistent 20% packet loss |
chaos-daemon pod crashed before flushing kernel traffic control (tc) queuing disciplines from the container cgroup. |
sudo nsenter -t $PID -n tc qdisc show dev eth0 |
Flush lingering rule: sudo nsenter -t $PID -n tc qdisc del dev eth0 root. |
🕸️ Service Mesh & Chaos Triage |
Route 53 Resolver Recursive Outbound LoopSluggish DNS & $10,000/week Route 53 bills |
Outbound resolver forwards .corp to on-prem; on-prem forwards .internal back to AWS Inbound endpoint. |
dig +trace non-existent.corp @10.100.0.2 |
Remove circular forwarders; enforce Route 53 Resolver DNS Firewall rules. | 🌐 Multi-Cloud Networking Triage |
🌪️ Category 2: Cascading Storms, Thundering Herds & Resource Exhaustion
| Issue & Error Signature | Root Cause Under the Hood | Fast Triage Command | Architectural Fix | Respective Deep-Dive Guide |
|---|---|---|---|---|
Aurora Thundering Herd on Pod RestartFATAL: remaining connection slots reserved |
Blanket pod reboot causes 120 pods to open 3,000 parallel database connections simultaneously, freezing CPU. | SELECT count(*), state FROM pg_stat_activity GROUP BY state; |
Place AWS RDS Proxy between EKS and Aurora; multiplex to a fixed pool of 50 backend connections. | 🏛️ Enterprise SRE & FinOps Triage |
ESO Secrets Manager API ThrottlingThrottlingException: Rate exceeded (TPS limit reached) |
500 ExternalSecret manifests configured with refreshInterval: 10s polling AWS Secrets Manager constantly. |
kubectl get externalsecrets -A |
Increase refreshInterval to 12h; switch to EventBridge webhook-driven secret updates. |
🛡️ DevSecOps & Patching Triage |
Prometheus TSDB Cardinality ExplosionPrometheus OOMKilled (Exit 137) during startup |
Developers exported unbounded labels (user_id, order_id), multiplying time series to 50M+ in memory. |
promtool tsdb analyze /prometheus/data |
Use metricRelabelings with action: labeldrop to strip high-cardinality labels before storage. |
☸️ Phase 4 Prometheus Lesson |
AWS NAT Gateway SNAT Port ExhaustionCloudWatch ErrorPortAllocation > 0 |
100 pods query single SaaS API without HTTP connection reuse, exhausting 55,000 source ports per IP. | Check CloudWatch metric ErrorPortAllocation |
Associate secondary Elastic IPs to the NAT Gateway (up to 8); mandate HTTP Keep-Alive connection pools. | 🧱 Phase 0 Networking 101 |
LiteLLM Upstream HTTP 429 Retry StormOpenAI 429 cascades into 504 Gateway Timeouts |
Gateway retries aggressively on 429 without exponential backoff or provider fallback routing. | Inspect LiteLLM logs for RateLimitError |
Configure exponential backoff with jitter; route fallbacks to Azure OpenAI & Claude on Bedrock. | 🚀 GitOps & LLMOps Triage |
TCP TIME_WAIT Socket Accumulationcannot assign requested address (EADDRNOTAVAIL) |
Short-lived connections close rapidly, leaving all 28k ephemeral ports in TIME_WAIT for 60 seconds. |
ss -tan state time-wait | wc -l |
Set sysctl -w net.ipv4.tcp_tw_reuse=1; expand ip_local_port_range="1024 65535". |
🧱 Phase 0 Networking 101 |
Dynamic Vault Secret Lease Expiration StormFATAL: password authentication failed for user v-token-* |
Vault short-lived lease (1h) expires; application connection pool held old credentials without refresh. | vault list sys/leases/lookup/database/creds |
Set HikariCP maxLifetime < Vault TTL; prefer 2-user static rotation for core databases. |
📜 IaC & Vault Triage |
🔒 Category 3: Deadlocks, Freezes & Quorum Split-Brains
| Issue & Error Signature | Root Cause Under the Hood | Fast Triage Command | Architectural Fix | Respective Deep-Dive Guide |
|---|---|---|---|---|
EBS CSI Multi-AZ Scheduling DeadlockVolumeZoneConflict: volume node affinity conflict |
StorageClass used Immediate binding; EBS volume provisioned in AZ-a, but Pod scheduled in AZ-b. |
kubectl get sc <name> -o yaml | grep volumeBindingMode |
Always configure volumeBindingMode: WaitForFirstConsumer on all block storage StorageClasses. |
☸️ Kubernetes & Stateful Triage |
Terraform DynamoDB State Lock StuckConditionalCheckFailedException: Lock Info: ID... |
CI/CD runner crashed or was canceled before Terraform could execute its exit cleanup handler. | aws dynamodb get-item --table-name terraform-locks ... |
Execute terraform force-unlock <Lock-ID>; wrap CI scripts in shell signal traps. |
📜 IaC & Vault Triage |
Vault Raft Storage Quorum Split-Brain[WARN] raft: no known peers, cannot start election |
AZ network partition separates 2 nodes from 1 in a 3-node cluster; nodes fail to establish majority quorum. | vault operator raft list-peers |
Break-glass single-node quorum recovery via /vault/data/peers.json. |
📜 IaC & Vault Triage |
PDB Node Drain Infinite HangCannot evict pod: would violate disruption budget |
minAvailable: 2 configured on a 2-replica deployment; evicting 1 pod drops healthy replicas to 1. |
kubectl get pdb -A |
Never set minAvailable == replicas; configure maxUnavailable: 1 or break-glass --disable-eviction. |
☸️ Kubernetes & Stateful Triage |
Admission Webhook Cluster Freezefailed calling webhook: context deadline exceeded |
Admission controller crashed, but webhook configured with failurePolicy: Fail across all namespaces. |
kubectl get validatingwebhookconfigurations |
Exclude kube-system via namespaceSelector; patch webhook to failurePolicy: Ignore in emergency. |
☸️ Kubernetes & Stateful Triage |
Terraform Security Group Cyclic GraphError: Cycle: aws_security_group.app, aws_security_group.db |
Inline ingress and egress blocks reference each other directly, preventing DAG creation. |
terraform plan |
Decouple rules into standalone aws_security_group_rule resources attaching to empty security groups. |
📜 IaC & Vault Triage |
Terragrunt Cold-Boot Output DeadlockModule /infra/eks depends on /infra/vpc, no outputs yet |
Fresh account bootstrap; VPC state doesn’t exist yet, crashing Terragrunt dependency parser. | terragrunt run-all plan |
Configure mock_outputs and mock_outputs_allowed_terraform_commands = ["validate", "plan"]. |
📜 IaC & Vault Triage |
🎭 Category 4: State Drift, Desync & Schema Traps
| Issue & Error Signature | Root Cause Under the Hood | Fast Triage Command | Architectural Fix | Respective Deep-Dive Guide |
|---|---|---|---|---|
Canary Abort with Breaking Database Schemacolumn "legacy_code" does not exist (HTTP 500) |
Canary release ran ALTER TABLE DROP COLUMN; when canary aborted back to v1.0, stable code crashed. |
Inspect database migration logs in CI | Enforce the Two-Phase Expand-and-Contract Pattern: add new column first, deprecate old, drop later. | 🚀 GitOps & LLMOps Triage |
ArgoCD Mutating Webhook Infinite Sync Stormlive state has limits, desired state has none |
Mutating webhook or VPA mutates live pod spec; ArgoCD detects drift and re-applies Git spec continuously. | argocd app diff <app-name> |
Declare ignoreDifferences in ArgoCD Application manifest for operator-managed fields. |
🚀 GitOps & LLMOps Triage |
Crossplane XR Finalizer DeadlockNamespace stuck in Terminating for hours |
AWS IAM role deleted or deletion protection enabled on RDS; Crossplane provider cannot delete cloud asset. | kubectl get managed -A | grep -v True |
Clear finalizer if safe: kubectl patch <resource> -p '{"metadata":{"finalizers":null}}' --type=merge. |
🚀 GitOps & LLMOps Triage |
Golden AMI /etc/machine-id DuplicationDHCPACK: duplicate IP assignment detected |
Builder VM created /etc/machine-id during bake; Packer forgot to truncate before taking snapshot. |
cat /etc/machine-id on multiple instances |
Run truncate -s 0 /etc/machine-id and cloud-init clean --all in Packer sysprep script. |
🛡️ DevSecOps & Patching Triage |
Accidental Git Force-Push Rewinds MainRemote branch rewound by 40 commits |
Developer ran git push --force instead of --force-with-lease after an out-of-date local rebase. |
git reflog show origin/main |
Recover commit SHA from reflog; enforce GitHub Branch Protection blocking force pushes. | 🧱 Phase 0 Git Fundamentals |
Git LFS 130-byte Text Pointer Build Failurecannot execute binary file: Exec format error |
Developer committed an executable without git lfs install; Git committed the plain-text pointer file. |
head -n 3 ./bin/<file> |
Run git lfs pull; enforce pre-commit hook requiring LFS installation. |
🧱 Phase 0 Git Fundamentals |
Backstage Catalog Processor OpenAPI CrashYAMLSyntaxError: Map keys must be unique |
Duplicate key in openapi.yaml parsed synchronously inside catalog worker transaction, stalling fleet. |
kubectl logs deploy/backstage |
Validate specs client-side in PR using npx @backstage/cli repo lint; isolate processor error boundaries. |
🕸️ Service Mesh & Chaos Triage |
🛡️ Category 5: Identity, Permissions & Token Expiration Windows
| Issue & Error Signature | Root Cause Under the Hood | Fast Triage Command | Architectural Fix | Respective Deep-Dive Guide |
|---|---|---|---|---|
Air-Gapped EKS Cosign Keyless Timeoutfailed to fetch rekor public keys: dial tcp 34... timeout |
Private VPC has no Internet Gateway; Kyverno cannot query public Sigstore Rekor ledger. | kubectl describe pod <pod> |
Switch to private AWS KMS Asymmetric Keys (awskms://arn:...) with AWS PrivateLink endpoints. |
🛡️ DevSecOps & Patching Triage |
Offboarded Employee Active 12h STS WindowTerminated employee continues running CLI commands |
IdP de-provisioning only blocks new logins; active AWS STS session tokens (ASIA...) remain valid until expiry. |
Audit CloudTrail for user’s STS role | Instantly revoke active tokens via IAM policy condition: "DateLessThan": {"aws:TokenIssueTime": "<timestamp>"}. |
☁️ Phase 1 Identity Center |
AWS IAM OIDC Thumbprint ExpirationOpenIDConnect provider's HTTPS cert cannot be verified |
GitHub/IdP rotated intermediate CA certificates; AWS IAM OIDC provider holds obsolete SHA-1 thumbprint. | openssl s_client -connect token.actions.githubusercontent.com:443 |
Update thumbprint list via aws iam update-open-id-connect-provider-thumbprint. |
🌐 Multi-Cloud Networking Triage |
SCP Root Administrator Override IllusionRoot user in member account fails with Access Denied |
Administrator assumes root bypasses SCPs; in AWS Organizations, SCPs define the account boundary first. |
aws organizations describe-policy |
Remove restrictive SCP from OU; never attach experimental SCPs to the Root Organization. | ☁️ Phase 1 Landing Zones |
Lambda VPC Subnet IP StarvationEC2 ENI allocation failed: Subnet has no available IPs |
Burst scaling of 500 concurrent Lambdas in shared /26 subnet exhausts IP pool for adjacent EKS nodes. |
aws ec2 describe-subnets |
Place Lambda in dedicated /20 subnets; share identical Security Group to maximize Hyperplane ENI reuse. |
📜 Phase 2 Python Lambda |
⚙️ Category 6: Runtime Throttling & Hidden Capacity Traps
| Issue & Error Signature | Root Cause Under the Hood | Fast Triage Command | Architectural Fix | Respective Deep-Dive Guide |
|---|---|---|---|---|
Linux CFS CPU Quota Latency Spike15% Host CPU but 200ms API p99 latency spikes |
Container resources.limits.cpu: "500m" exhausts 50ms quota in first 20ms; kernel freezes process for 80ms. |
cat /sys/fs/cgroup/cpu.stat | grep nr_throttled |
Eliminate CPU limits on latency-sensitive workloads; configure CPU requests for scheduling and burst freely. | 🧱 Phase 0 Kernel & cgroups |
Milvus QueryNode HNSW RAM OOM KillMilvus QueryNode Exit Code 137 (OOMKilled) |
Memory sized only for raw float vectors, omitting the 1.5× RAM overhead of HNSW graph pointers. | Check QueryNode Prometheus memory metrics | Size memory via M_total = N × D × 4 × 1.5; switch to DiskANN for massive collections. | 🚀 GitOps & LLMOps Triage |
vLLM PagedAttention CUDA Out of Memorytorch.cuda.OutOfMemoryError: CUDA out of memory |
gpu_memory_utilization: 0.98 pre-allocated too much KV cache, leaving zero buffer for dynamic activations. |
kubectl exec -it <pod> -- nvidia-smi |
Tune gpu_memory_utilization=0.90; add --swap-space 16 to route temporary spikes to host RAM. |
🚀 GitOps & LLMOps Triage |
ALB Pod Deregistration Race (502 Bad Gateway)Rolling deployment triggers 10s burst of HTTP 502s |
Pod receives SIGTERM and shuts down while AWS ALB deregistration delay (300s) is still draining. |
Check ALB Target Group 5xx metrics | Set deregistration_delay.timeout_seconds=30; add container preStop hook: sleep 15. |
☸️ Phase 4 Ingress ALB |
OpenCost Unallocated Idle Spend DisputeFinOps dashboard flags 48% ($45k) as unallocated waste |
OpenCost attributes costs strictly to pod requests, leaving Karpenter node spare buffer unallocated. | Check OpenCost deployment environment variables | Enable ALLOCATE_IDLE=true to distribute cluster buffer costs proportionally across tenant namespaces. |
🏛️ Enterprise SRE & FinOps Triage |
LLM-as-a-Judge Flapping in CI/CDPrompt PR scores 96% in Run 1, but 92% (Fail) in Run 2 |
Judge LLM invoked with non-zero temperature or exhibits positional bias favoring the first answer. | Inspect MLflow evaluation run parameters | Set judge temperature=0.0; implement swap-and-aggregate evaluation (averaging both orderings). |
🚀 Phase 8 MLflow Evaluation |
Azure AKS SNAT Port ExhaustionOutbound HTTPS timeouts to payment APIs at <20% CPU |
Standard Load Balancer pre-allocates only 1,024 SNAT ports per node; exhausted under high concurrency. | az monitor metrics list --metric UsedSNATPorts |
Attach Azure NAT Gateway (64,512 dynamic SNAT ports per IP). | 🌐 Multi-Cloud Networking Triage |
ExpressRoute BGP Overriding Firewall UDRAsymmetric TCP RSTs between On-Prem and Azure Spoke |
ExpressRoute advertises /24 BGP routes into Spoke, beating the 0.0.0.0/0 Azure Firewall UDR via LPM. |
az network nic show-effective-route-table |
Set disable-bgp-route-propagation=true on Spoke route tables + UDR on GatewaySubnet. |
🌐 Multi-Cloud Networking Triage |
AKS Entra Workload Identity AADSTS70021Key Vault CSI mount fails with no matching federated record |
Trailing / mismatch in OIDC Issuer URL, wrong system:serviceaccount:ns:sa subject, or missing pod label. |
az identity federated-credential list |
Match exact AKS issuerUrl and add label azure.workload.identity/use: "true" to Pod. |
☸️ K8s & Stateful Triage |
AKS Managed OS Disk IOPS StarvationNodes flip NotReady (PLEG unhealthy) during image pulls |
Network-attached Managed OS Disk exhausts IOPS burst during parallel container image unpacking. | az aks nodepool show --query osDiskType |
Use Ephemeral OS Disks (--node-osdisk-type Ephemeral) on local NVMe cache. |
☸️ K8s & Stateful Triage |
Cosmos DB Hot Partition HTTP 429 at 15% RURequestRateTooLarge despite 85% idle total RU/s |
Total RU/s is split evenly across N physical partitions (max 10k RU/s each); skewed key overloads 1 partition. | KQL query on partitionKeyRangeId_s where statusCode_s == 429 |
Use Hierarchical Partition Keys ([/tenantId, /userId]) or write-sharding suffixes. |
🏛️ Enterprise SRE & DB Triage |