⚡
Zero to Platform Architect2026
War-Room Triage & 43 Traps
War-Room Triage & 43 Traps⏱️ 10 min read⚡ Staff SRE Standard
🎯 Foundations Test Bank →
Quick Jump:🧭 CLI Playbook📝 Q&A Drill

🌐 Multi-Cloud & Enterprise Networking Triage Guide

This guide details the most intricate, tricky, and common multi-cloud networking failure modes encountered across AWS, Azure, on-premises datacenters, and cloud interconnects (Megaport/Equinix), accompanied by diagnostic commands and remediation architectures.

🧭 Beginner-Friendly First-5-Minutes Network War-Room Playbook (Step-by-Step SOP)

Mental Model (The Postal Delivery Chain): Every network packet needs 4 things to succeed: 1. DNS Name Resolution → 2. Local Route Table Path → 3. Firewall/Security Group Permission (Both Ways!) → 4. Matching Packet Size (MTU). Test them in that exact order!

Step 1: Isolate DNS vs. IP Connectivity (Minute 0–1)

dig +short <target-hostname>
# Verify whether DNS returns the expected Private IP (10.x.x.x) or accidentally returns a Public IP!

Step 2: Test Layer-4 TCP Handshake & Trace Path Asymmetry (Minute 1–2)

nc -zv -w 3 <target-ip> 443
traceroute -n -T -p 443 <target-ip>

Step 3: Test for Silent MTU Jumbo-Frame Blackholes (Minute 2–3)

# Send a 1472-byte payload (1500 MTU) with Do-Not-Fragment (-M do) set:
ping -c 3 -M do -s 1472 <target-ip>
  • What to look for: If small ping works but -s 1472 or curl https://... hangs during TLS ClientHello, you have an IPsec/VXLAN/WireGuard MTU blackhole!

Step 4: Check Transit Gateway Appliance Mode & BGP Route Tables (Minute 3–4)

aws ec2 search-transit-gateway-routes --transit-gateway-route-table-id <rtb-id>   --filters "Name=type,Values=propagated,static"

Step 5: Query VPC Flow Logs for REJECT or Asymmetric SYN-Only Drops (Minute 4–5)

  • Check whether ${tcp-flags} shows 2 (SYN only with no 18 SYN-ACK), proving asymmetric return routing dropped by a stateful firewall.

🎯 High-Impact Real-World Scenarios

Scenario 1: Transit Gateway Appliance Mode Asymmetry Dropping Stateful TCP

The Symptom

Inter-VPC or VPC-to-Internet traffic traversing an inspection VPC (containing Palo Alto / Fortinet / AWS Network Firewalls) experiences silent drops. Ping (ICMP) works intermittently, but HTTP/HTTPS connections stall during TLS handshakes or reset with TCP RST.

Under-the-Hood Root Cause

By default, AWS Transit Gateway selects an Availability Zone route table independently for the forward request and reverse response packets. In a dual-AZ inspection architecture:

  • Forward packet (VPC A → VPC B): TGW routes to Firewall 1 in us-east-1a. Firewall 1 establishes a TCP connection tracking session.
  • Return packet (VPC B → VPC A): TGW hashes and routes the return packet to Firewall 2 in us-east-1b. Firewall 2 has no record of the initial TCP SYN packet. Because it is a stateful firewall, it rejects the out-of-order TCP SYN-ACK or data packet and issues a TCP RST.
sequenceDiagram
    autonumber
    participant AppA as Workload VPC A
    participant TGW as AWS Transit Gateway
    participant FW1 as Firewall (AZ-a)
    participant FW2 as Firewall (AZ-b)
    participant AppB as Workload VPC B

    AppA->>TGW: TCP SYN (Port 443)
    TGW->>FW1: Routes to AZ-a Firewall
    Note over FW1: Records state in connection table ✅
    FW1->>AppB: Forwards TCP SYN to target
    AppB-->>TGW: Sends TCP SYN-ACK
    TGW->>FW2: ❌ Hashed to AZ-b Firewall (Asymmetric routing!)
    Note over FW2: No state table entry! Packet dropped with TCP RST! 🚨
    FW2-->>AppA: TCP RST (Connection Reset)

CLI Triage Commands

# 1. Inspect Transit Gateway Attachment configuration for the Inspection VPC
aws ec2 describe-transit-gateway-vpc-attachments \
  --filters "Name=vpc-id,Values=vpc-inspection-01" \
  --query "TransitGatewayVpcAttachments[*].Options.ApplianceModeSupport"

# 2. Check VPC Flow Logs for REJECT flags on port 443
aws ec2 get-flow-logs-integration --query "FlowLogs[?LogStatus=='ACTIVE']"

Remediation & Architectural Fix

  • The Permanent Fix: Enable Appliance Mode on the Transit Gateway attachment connecting the Inspection VPC. Appliance Mode guarantees that both the forward and reverse traffic flows pin to the exact same Availability Zone and ENI for the lifetime of the TCP connection:
aws ec2 modify-transit-gateway-vpc-attachment \
  --transit-gateway-attachment-id tgw-attach-0123456789abcdef0 \
  --options ApplianceModeSupport=enable

Scenario 2: Megaport MCR BGP Asymmetric Routing & MED Flapping

The Symptom

Traffic between AWS workloads (via Direct Connect) and Azure workloads (via ExpressRoute) over Megaport Cloud Router (MCR) suffers from severe packet loss and intermittent latency spikes of 150ms+. TCP connections randomly terminate every 90 seconds.

Under-the-Hood Root Cause

Both AWS and Azure advertise identical subnets (10.0.0.0/8) to the Megaport Cloud Router via BGP. However, BGP path selection metrics were asymmetric:

  • Azure preferred the direct Megaport link due to higher BGP Local Preference (120).
  • AWS Direct Connect received advertisements with identical AS-path lengths but differing Multi-Exit Discriminator (MED) values, causing route flapping when redundant virtual interfaces experienced micro-bursts. Packets from AWS traversed the direct MCR link, while return packets from Azure routed across an IPSec VPN backup link over the public Internet, breaking stateful firewall inspection.

CLI Triage Commands

# 1. Inspect BGP route table received by AWS Direct Connect Gateway
aws directconnect describe-direct-connect-gateway-associations \
  --direct-connect-gateway-id <dc-gw-id>

# 2. Trace route hops between AWS and Azure instances
traceroute -T -p 443 10.200.1.50

Remediation & Architectural Fix

  1. Enforce Deterministic BGP Local Preference: Configure Megaport MCR to set local-preference 200 on the primary interconnect and 100 on the backup.
  2. Apply BGP AS-Path Prepending on Secondary Links: Prepend the autonomous system number 3 times (as-path prepend 64512 64512 64512) on the backup tunnel to ensure all cloud routers deterministically prefer the direct high-speed interconnect.

The Symptom

A consumer VPC accesses an internal API service hosted in a producer VPC via AWS PrivateLink (Interface VPC Endpoint). Approximately 50% of requests succeed with 2ms latency, while the other 50% time out after 30 seconds.

Under-the-Hood Root Cause

The Network Load Balancer (NLB) backing the VPC Endpoint Service was created across two Availability Zones (us-east-1a and us-east-1b). However, backend target group pods were running only on nodes in us-east-1a, and Cross-Zone Load Balancing was disabled on the NLB.

When the consumer’s VPC Endpoint resolved DNS, it round-robined between the endpoint ENIs in AZ-a and AZ-b. When traffic hit the endpoint ENI in us-east-1b, the NLB refused to forward packets cross-zone to the healthy targets in AZ-a, silently dropping the connection.

CLI Triage Commands

# 1. Check cross-zone load balancing attribute on the NLB
aws elbv2 describe-load-balancer-attributes \
  --load-balancer-arn <nlb-arn> \
  --query "Attributes[?Key=='load_balancing.cross_zone.enabled'].Value"

# 2. Check health status of target group targets per AZ
aws elbv2 describe-target-health \
  --target-group-arn <target-group-arn> \
  --query "TargetHealthDescriptions[*].[Target.Id,Target.AvailabilityZone,TargetHealth.State]"

Remediation & Architectural Fix

  • Remediation: Enable Cross-Zone Load Balancing on the NLB:
aws elbv2 modify-load-balancer-attributes \
  --load-balancer-arn <nlb-arn> \
  --attributes Key=load_balancing.cross_zone.enabled,Value=true
  • Architectural Standard: Always distribute backend workloads evenly across all AZs using Kubernetes topologySpreadConstraints.

Scenario 4: Route 53 Resolver Outbound Loop Generating $10,000 AWS Bills

The Symptom

Internal DNS resolution becomes sluggish. At the end of the billing cycle, the platform team discovers a massive cost anomaly: Route 53 Resolver queries spiked to 800 million requests, costing thousands of dollars in a single week.

Under-the-Hood Root Cause

An engineer created a Route 53 Outbound Resolver Rule to forward all queries for .corp.internal to an on-premises Active Directory DNS server (10.50.0.10). However, on the Active Directory server, a forwarder was configured pointing .internal queries back to the AWS Route 53 Inbound Resolver endpoint (10.100.0.2).

When an application requested a non-existent domain (fake.corp.internal), AWS forwarded to on-premises; on-premises forwarded back to AWS; AWS forwarded to on-premises… creating an infinite recursive DNS forwarding loop until the TTL expired.

CLI Triage Commands

# 1. Inspect Route 53 Resolver query logging metrics
aws route53resolver list-resolver-query-log-configs

# 2. Query the resolver endpoint for recursion loops
dig +trace non-existent.corp.internal @10.100.0.2

Remediation & Architectural Fix

  1. Break the Forwarding Loop: Delete recursive forwarders on the on-premises DNS servers for subdomains owned by AWS.
  2. Apply Route 53 Resolver Firewall: Block queries matching wildcards for invalid internal patterns and configure strict query rate limits.
  3. Configure AWS CloudWatch Metric Alarms: Alert if Route 53 Resolver QueryVolume exceeds 50,000 queries per minute.

Scenario 5: AWS IAM OIDC Thumbprint Expiration Halting CI/CD

The Symptom

All GitHub Actions deployment workflows across 200 repositories fail simultaneously with:

An error occurred (AccessDenied) when calling the AssumeRoleWithWebIdentity operation: 
OpenIDConnect provider's HTTPS certificate cannot be verified.

Under-the-Hood Root Cause

AWS IAM requires an OpenID Connect (OIDC) identity provider configuration that includes the SHA-1 thumbprint of the IdP’s intermediate CA certificate. When GitHub or an enterprise IdP rotated their TLS certificate root/intermediate authorities, the thumbprint configured in AWS IAM became obsolete. AWS IAM rejected all incoming JWT token exchanges, completely shutting down automated deployments.

CLI Triage Commands

# 1. Inspect configured thumbprints on the AWS IAM OIDC Provider
aws iam get-open-id-connect-provider \
  --open-id-connect-provider-arn arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com

# 2. Fetch the current live thumbprint from GitHub's OIDC server
echo | openssl s_client -servername token.actions.githubusercontent.com -showcerts -connect token.actions.githubusercontent.com:443 2>/dev/null | openssl x509 -fingerprint -noout

Remediation & Architectural Fix

  • Immediate Fix: Update the OIDC Provider thumbprints in AWS IAM:
aws iam update-open-id-connect-provider-thumbprint \
  --open-id-connect-provider-arn arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com \
  --thumbprint-list "6938fd4d98bab03faadb97b34396831e3780aea1" "1c58a3a8518e8759bf075b76b750d4f2df264fcd"
  • Architectural Standard: AWS now automatically validates certificates signed by known public root CAs for GitHub Actions; verify that legacy manual thumbprint constraints are refreshed to avoid certificate rotation outages.

Scenario 5: Azure AKS / VM Outbound API Timeouts (SNAT Port Exhaustion)

The Symptom

During high-concurrency payment traffic, pods in AKS intermittently fail to establish outbound HTTPS connections to external payment gateways (Connection timed out), even though node CPU is < 25% and network bandwidth is < 10%.

Under-the-Hood Root Cause

The AKS cluster egresses through a standard Azure Load Balancer (outboundType: loadBalancer) without an Azure NAT Gateway. By default, an Azure Load Balancer pre-allocates a static ceiling of only 1,024 SNAT ports per backend node per frontend public IP. When a node opens more than 1,024 concurrent connections (including sockets lingering in TCP TIME_WAIT) to the same destination IP and port, the Azure SDN drops all subsequent SYN packets!

Diagnostic & Remediation Commands

# 1. Check Allocated vs Used SNAT Ports and Failed SNAT Connections on the AKS Load Balancer
az monitor metrics list --resource <aks-outbound-lb-id> --metric "UsedSNATPorts" "SNATConnectionCount" --aggregation Total

# 2. Permanent Fix: Attach an Azure NAT Gateway (dynamically pools 64,512 SNAT ports per IP across the subnet)
az aks update -g rg-prod -n aks-prod --outbound-type managedNATGateway --nat-gateway-managed-outbound-ip-count 2

Scenario 6: Azure ExpressRoute & Hub Firewall Asymmetric Routing (DisableBgpRoutePropagation Trap)

The Symptom

On-premises servers connecting to Azure Spoke VMs via ExpressRoute can send an initial SYN packet, but TCP handshakes hang and reset after 3 seconds.

Under-the-Hood Root Cause

  1. In the Spoke VNet route table, a User-Defined Route (UDR) sets 0.0.0.0/0 -> VirtualAppliance (Azure Firewall 10.0.1.4), but bgp_route_propagation_enabled was left true.
  2. ExpressRoute advertises specific /24 on-premises subnets via BGP into the Spoke route table. Because /24 is more specific than 0.0.0.0/0 (Longest Prefix Match), traffic from the Spoke to On-Premises bypasses the Azure Firewall, while On-Premises-to-Spoke traffic is routed through the Azure Firewall (or vice versa when GatewaySubnet lacks a Spoke UDR). The stateful Azure Firewall drops the single-sided TCP flow!

Diagnostic & Remediation Commands

# 1. Inspect Effective Routes on the Spoke VM/AKS NIC to spot BGP routes overriding the Firewall UDR
az network nic show-effective-route-table -g rg-spoke -n nic-vm-01 -o table

# 2. Permanent Fix: Disable BGP route propagation on Spoke route tables so all traffic traverses Azure Firewall
az network route-table update -g rg-spoke -n rt-spoke-prod --disable-bgp-route-propagation true