Lesson 6: AWS Cloud WAN, Dual-Stack IPv6 & Deep Network Forensics

🧠 The Concept (Explain Like I’m 5)

Imagine you are building a global airline logistics network spanning 50 countries:

• The Mesh Nightmare (TGW Inter-Region Peering): If you have 4 major airports (AWS Regions) and 200 airplane hangars (VPCs), you have to negotiate individual private flight paths between every single pair of airports. You must manually maintain hundreds of flight dispatch routing books. If you open Airport #5 in Tokyo, you spend a month re-writing static routes across the entire globe!
• The Global Super-Highway (AWS Cloud WAN): Instead of manual peerings, you lay down a single, unified global private fiber backbone called the Core Network. You write one single policy document that says: “All production hangars across the entire planet belong to the ‘Production Segment’ and can talk to each other automatically.” Cloud WAN handles global BGP propagation, edge routing, and encryption across all continents in seconds.
• IPv6 & Egress-Only Internet Gateway (EIGW): IPv4 is like a tiny telephone directory that only had 4 billion numbers—which humanity ran out of years ago. IPv6 gives us 340 undecillion addresses (3.4 × 10^38)—enough to assign a unique IP to every atom on Earth. An Egress-Only Internet Gateway (EIGW) is a magical one-way mirror: your internal servers can dial out to any website on the internet, but no hacker on the public internet can initiate a call into your servers—and unlike an IPv4 NAT Gateway, it costs $0.00!
• Network Forensics & TCP Flags: When a packet travels across the network, the TCP header contains small 1-bit flags (SYN, ACK, RST, FIN). By reading these flags in VPC Flow Logs, a network detective can instantly tell whether a connection was blocked by a firewall, timed out on the wire, or deliberately rejected by an application server.

🏢 The Enterprise Context (Why Network Architects Care)

1. Global Scale Without Operational Gridlock: Operating across US, Europe, and Asia-Pacific using Transit Gateways requires N(N-1)/2 regional peerings, complex routing tables, and manual route leak prevention. AWS Cloud WAN centralizes global network orchestration into a declarative JSON policy.
2. The IPv4 Depletion & NAT Gateway Tax Crisis: Public IPv4 addresses now cost $0.005/hour ($43.80/year per IP) on AWS, and NAT Gateways charge $0.045/GB. Migrating to dual-stack IPv6 with Egress-Only IGWs allows enterprises to eliminate dozens of NAT Gateways, slashing annual networking overhead by hundreds of thousands of dollars.
3. Regulatory Forensics & Zero-Trust Inspection: Under PCI-DSS 4.0, HIPAA, and FedRAMP, network architects must mathematically prove that untrusted traffic cannot reach database subnets, capture raw packet headers via Traffic Mirroring, and retain forensic audit trails.

🗺️ Visual Architecture: AWS Cloud WAN Global Core Network

flowchart TD
    subgraph GlobalPolicy ["AWS Cloud WAN Central Control Plane"]
        CNP["<b>Core Network Policy (policy.json)</b><br/>• Declarative Global Segments: 'prod', 'dev', 'egress'<br/>• Automated Tag-Based VPC Attachment Mapping<br/>• Global Dynamic BGP Route Propagation"]
    end

    subgraph RegionUS ["AWS Region: us-east-1"]
        EdgeUS["Cloud WAN Edge Location (us-east-1)"]
        VPC_Prod_US["Prod VPC (10.10.0.0/16)"]
        VPC_Dev_US["Dev VPC (10.20.0.0/16)"]
        EdgeUS --- VPC_Prod_US
        EdgeUS --- VPC_Dev_US
    end

    subgraph RegionEU ["AWS Region: eu-central-1"]
        EdgeEU["Cloud WAN Edge Location (eu-central-1)"]
        VPC_Prod_EU["Prod VPC (10.30.0.0/16)"]
        VPC_Dev_EU["Dev VPC (10.40.0.0/16)"]
        EdgeEU --- VPC_Prod_EU
        EdgeEU --- VPC_Dev_EU
    end

    subgraph RegionAPAC ["AWS Region: ap-southeast-1"]
        EdgeAPAC["Cloud WAN Edge Location (ap-southeast-1)"]
        VPC_Prod_APAC["Prod VPC (10.50.0.0/16)"]
        EdgeAPAC --- VPC_Prod_APAC
    end

    CNP ==> EdgeUS
    CNP ==> EdgeEU
    CNP ==> EdgeAPAC

    EdgeUS <-->|"AWS Global Fiber Dedicated Backbone"| EdgeEU
    EdgeEU <-->|"AWS Global Fiber Dedicated Backbone"| EdgeAPAC
    EdgeUS <-->|"AWS Global Fiber Dedicated Backbone"| EdgeAPAC

🔬 Global Backbone: AWS Cloud WAN vs. Transit Gateway Peering

🌐 Dual-Stack IPv6: Egress-Only Internet Gateway (EIGW), DNS64 & NAT64

1. IPv6 Address Architecture

• IPv4 Subnet: Typically a /24 (256 addresses) or /20 (4,096 addresses) leading to subnet exhaustion.
• AWS IPv6 Allocation: AWS assigns a /56 CIDR block to each VPC (2^72 addresses) and each subnet receives a standard /64 CIDR block (2^64 = 1.84 × 10^19 addresses!). Every container, pod, and virtual machine receives a globally unique, routable IPv6 address.

2. The Egress-Only Internet Gateway (EIGW): Stateful Zero-Cost Egress

In IPv4, private workloads require a NAT Gateway ($32/mo + $0.045/GB) to translate private RFC 1918 IPs into a public IP. In IPv6, every instance already has a globally unique public IPv6 address. To prevent inbound attacks:

• An Egress-Only Internet Gateway (EIGW) is a stateful firewall gateway.
• It permits outbound IPv6 connections initiated from internal instances to the public internet.
• It strictly blocks all unsolicited inbound connections from the public internet.
• FinOps Impact: EIGWs are 100% FREE! Zero hourly charges and zero data processing fees.

sequenceDiagram
    autonumber
    participant App as 📦 Private App Pod (IPv6: 2600:1f18:...:1)
    participant EIGW as 🛡️ Egress-Only Internet Gateway (EIGW)
    participant NATGW as 💸 Legacy IPv4 NAT Gateway ($0.045/GB)
    participant Web as 🌐 Internet API (github.com)

    Note over App,EIGW: Modern IPv6 Flow ($0.00 Cost, Direct & Fast)
    App->>EIGW: 1. Outbound TCP SYN to [2606:50c0:8000::]
    EIGW->>Web: 2. Forwards packet onto internet (Preserves real IPv6)
    Web-->>EIGW: 3. Return TCP SYN-ACK permitted (Existing state table entry)
    EIGW-->>App: 4. Delivered to App Pod ✅ (Zero NAT translation overhead!)

    Note over Web,EIGW: Malicious Public Internet Inbound Probe
    Web-x EIGW: 5. Unsolicited Inbound TCP SYN probe dropped at edge! ⛔

3. DNS64 and NAT64: Bridging IPv6-Only Workloads to IPv4 Services

When an IPv6-only pod needs to communicate with an external service that only has an IPv4 address (e.g., a legacy partner API without an AAAA record):

1. DNS64 (Route 53 Resolver): When the pod queries api.legacy-partner.com, Route 53 Resolver detects that the destination only has an IPv4 A record (203.0.113.50). DNS64 synthesizes an IPv6 address by prepending the Well-Known Prefix (64:ff9b::/96), returning 64:ff9b::cb00:7132.
2. NAT64 (AWS NAT Gateway): The pod routes packets destined for 64:ff9b::/96 to the AWS NAT Gateway. The NAT Gateway extracts the embedded IPv4 address, performs stateful translation, and forwards standard IPv4 packets to the destination!

🗄️ Enterprise IPAM & Overlapping CIDR Private NAT Solutions

1. AWS VPC IPAM (IP Address Manager) Hierarchy

In multi-account organizations, different squads inadvertently pick overlapping subnets. AWS IPAM solves this through hierarchical pool enforcement:

flowchart TD
    TopPool["<b>Corporate Top-Level IPAM Pool</b><br/>CIDR: 10.0.0.0/8 (Owned by Enterprise Network Team)"]
    
    RegUS["<b>Regional Pool: us-east-1</b><br/>CIDR: 10.10.0.0/12"]
    RegEU["<b>Regional Pool: eu-central-1</b><br/>CIDR: 10.30.0.0/12"]

    EnvProd["<b>Prod Allocation Pool</b><br/>Netmask Rule: /16 to /20<br/>Auto-tag: Environment=Prod"]
    EnvDev["<b>Dev Allocation Pool</b><br/>Netmask Rule: /20 to /24<br/>Auto-tag: Environment=Dev"]

    TopPool --> RegUS
    TopPool --> RegEU
    RegUS --> EnvProd
    RegUS --> EnvDev

2. Overlapping CIDRs in Mergers & Acquisitions: Private NAT Gateway

When Company A (10.100.0.0/16) acquires Company B (10.100.0.0/16), re-IPing either data center or cloud environment would take 18 months.

• The Solution: Deploy AWS Private NAT Gateways in the Transit Gateway transit route tables.
• Company A translates its outbound requests into an unallocated CGNAT pool (100.64.10.0/24).
• Company B receives the packets as if they originated from 100.64.10.x, enabling immediate, bidirectional Layer 3 communication without changing a single server IP!

🔍 Deep Network Telemetry & Forensics: Custom VPC Flow Logs, Athena & Traffic Mirroring

1. Advanced Custom VPC Flow Log Fields

Default VPC Flow Logs (version account-id interface-id srcaddr dstaddr srcport dstport protocol packets bytes start end action log-status) miss the critical Layer 4 telemetry needed for security diagnostics.

Production Custom VPC Flow Log Format:

${version} ${account-id} ${interface-id} ${srcaddr} ${dstaddr} ${srcport} ${dstport} ${protocol} ${packets} ${bytes} ${start} ${end} ${action} ${tcp-flags} ${pkt-srcaddr} ${pkt-dstaddr} ${flow-direction} ${traffic-path}

The Power of ${tcp-flags} Bitmask Analysis:

The TCP flag field is a decimal representation of the 8-bit TCP control flags:

• 1 = FIN (Connection teardown)
• 2 = SYN (Connection handshake initiation)
• 4 = RST (Connection reset / abruptly rejected)
• 16 = ACK (Acknowledgment)
• 18 = SYN-ACK (2 + 16, Remote host accepted the handshake)
• 20 = RST-ACK (4 + 16, Target rejected request or killed session)

Athena Diagnostic SQL Query: Isolating Firewall vs. Application Failures

-- Query Athena to determine whether connections were blocked by Security Groups or OS Firewalls
SELECT 
  srcaddr, 
  dstaddr, 
  dstport, 
  action, 
  tcp_flags,
  CASE 
    WHEN action = 'REJECT' THEN 'Dropped by AWS Security Group or Network ACL'
    WHEN action = 'ACCEPT' AND tcp_flags = 2 THEN 'SYN sent but no response (Routing issue / Target host down)'
    WHEN action = 'ACCEPT' AND tcp_flags = 18 THEN 'Handshake Successful (SYN-ACK returned)'
    WHEN action = 'ACCEPT' AND (tcp_flags = 4 OR tcp_flags = 20) THEN 'Connection Reset by Target OS or App (Port Closed)'
    ELSE 'Active Established Data Flow'
  END AS root_cause_diagnosis,
  COUNT(*) as flow_count
FROM enterprise_vpc_flow_logs
WHERE dstport = 443 
  AND start >= to_unixtime(now() - interval '1' hour)
GROUP BY srcaddr, dstaddr, dstport, action, tcp_flags
ORDER BY flow_count DESC;

2. VPC Traffic Mirroring (Raw Packet Capture via VXLAN UDP 4789)

When compliance mandates raw Deep Packet Inspection (DPI) for intrusion detection (Zeek / Suricata):

• AWS VPC Traffic Mirroring copies raw inbound and outbound packets from a workload’s Elastic Network Interface (ENI).
• It encapsulates the raw frames into VXLAN (Virtual Extensible LAN - RFC 7348) headers over UDP port 4789.
• Encapsulated packets stream across the VPC fabric directly to a Network Load Balancer fronting a fleet of open-source Suricata/Zeek sensors with zero performance impact on the production workload!

💻 Production Manifests: Cloud WAN Policy & Dual-Stack IPv6 Terraform

1. Cloud WAN Core Network Policy (core-network-policy.json)

{
  "version": "2021.12",
  "core-network-configuration": {
    "asn-ranges": ["64512-65534"],
    "vpn-ecmp-support": true,
    "edge-locations": [
      { "location": "us-east-1", "asn": 64512 },
      { "location": "eu-central-1", "asn": 64513 },
      { "location": "ap-southeast-1", "asn": 64514 }
    ]
  },
  "segments": [
    {
      "name": "prod",
      "description": "Production global enterprise workloads",
      "require-attachment-acceptance": false
    },
    {
      "name": "dev",
      "description": "Isolated development workloads",
      "require-attachment-acceptance": false
    },
    {
      "name": "egress",
      "description": "Central inspection and internet exit",
      "require-attachment-acceptance": false
    }
  ],
  "segment-actions": [
    {
      "action": "share",
      "mode": "attachment-route",
      "segment": "egress",
      "share-with": ["prod", "dev"]
    }
  ],
  "attachment-policies": [
    {
      "rule-number": 100,
      "condition-logic": "and",
      "conditions": [
        { "type": "tag-value", "operator": "equals", "key": "Environment", "value": "Production" }
      ],
      "action": {
        "association": { "segment": "prod" }
      }
    },
    {
      "rule-number": 200,
      "condition-logic": "and",
      "conditions": [
        { "type": "tag-value", "operator": "equals", "key": "Environment", "value": "Development" }
      ],
      "action": {
        "association": { "segment": "dev" }
      }
    }
  ]
}

2. Dual-Stack IPv6 VPC with Egress-Only IGW (ipv6_vpc.tf)

# 1. Dual-Stack VPC with Amazon-Provided IPv6 CIDR Block (/56)
resource "aws_vpc" "dual_stack_vpc" {
  cidr_block                       = "10.0.0.0/16"
  assign_generated_ipv6_cidr_block = true
  enable_dns_hostnames             = true
  enable_dns_support               = true

  tags = { Name = "enterprise-dualstack-vpc" }
}

# 2. Private Subnet with /64 IPv6 CIDR Block
resource "aws_subnet" "private_workload" {
  vpc_id                          = aws_vpc.dual_stack_vpc.id
  cidr_block                      = "10.0.1.0/24"
  ipv6_cidr_block                 = cidrsubnet(aws_vpc.dual_stack_vpc.ipv6_cidr_block, 8, 1) # /64
  assign_ipv6_address_on_creation = true

  tags = { Name = "private-ipv6-workload-subnet" }
}

# 3. Egress-Only Internet Gateway ($0.00 cost, stateful outbound IPv6)
resource "aws_egress_only_internet_gateway" "eigw" {
  vpc_id = aws_vpc.dual_stack_vpc.id

  tags = { Name = "corporate-eigw" }
}

# 4. Route Table: Route all outbound IPv6 traffic (::/0) to EIGW
resource "aws_route_table" "private_rt" {
  vpc_id = aws_vpc.dual_stack_vpc.id

  route {
    ipv6_cidr_block        = "::/0"
    egress_only_gateway_id = aws_egress_only_internet_gateway.eigw.id
  }

  tags = { Name = "private-ipv6-route-table" }
}

resource "aws_route_table_association" "private_assoc" {
  subnet_id      = aws_subnet.private_workload.id
  route_table_id = aws_route_table.private_rt.id
}

🧭 Step-by-Step Operational Playbook: Dual-Stack IPv6 Migration & Live Packet Forensics

Follow this step-by-step operational runbook to add IPv6 to an existing IPv4 VPC (eliminating NAT Gateway data fees) and capture live forensic packets during a security incident:

Step 1: Associate an Amazon-Provided /56 IPv6 CIDR Block to the VPC

1. Plain-English Goal: Give your VPC 2^72 globally unique IPv6 addresses alongside its existing IPv4 CIDR without disrupting running workloads.
2. Action:

   aws ec2 associate-vpc-cidr-block \
     --vpc-id vpc-0abc12345 \
     --amazon-provided-ipv6-cidr-block

Step 2: Carve /64 IPv6 Subnets & Attach an Egress-Only Internet Gateway (EIGW)

1. Plain-English Goal: Route outbound IPv6 traffic (::/0) from private subnets through a stateful Egress-Only Internet Gateway—which costs $0.00/GB compared to $0.045/GB on an IPv4 NAT Gateway!
2. Action:

   EIGW_ID=$(aws ec2 create-egress-only-internet-gateway --vpc-id vpc-0abc12345 --query "EgressOnlyInternetGateway.EgressOnlyInternetGatewayId" --output text)
   aws ec2 create-route --route-table-id rtb-private123 --destination-ipv6-cidr-block "::/0" --egress-only-internet-gateway-id "$EIGW_ID"

Step 3: Enable Enriched VPC Flow Logs with TCP Flag Telemetry

1. Plain-English Goal: Standard Flow Logs only say ACCEPT or REJECT. Custom Flow Logs capture ${tcp-flags} and ${pkt-src-aws-service} so you can distinguish a firewall block (SYN only) from an application crash (RST).
2. Action:

   aws ec2 create-flow-logs \
     --resource-type VPC \
     --resource-ids vpc-0abc12345 \
     --traffic-type ALL \
     --log-destination-type s3 \
     --log-destination "arn:aws:s3:::corp-network-forensics-logs" \
     --log-format '${version} ${vpc-id} ${subnet-id} ${srcaddr} ${dstaddr} ${srcport} ${dstport} ${protocol} ${tcp-flags} ${action} ${pkt-src-aws-service}'

Step 4: Activate Out-of-Band VPC Traffic Mirroring (VXLAN UDP 4789)

1. Plain-English Goal: When SecOps suspects a compromised pod or EC2 instance, clone its raw network packets at the Nitro hypervisor layer to a Suricata IDS sensor without logging into the suspect machine.
2. Action:

   aws ec2 create-traffic-mirror-session \
     --network-interface-id eni-suspect123 \
     --traffic-mirror-target-id tmt-suricata456 \
     --traffic-mirror-filter-id tmf-all-traffic \
     --session-number 1

🛠️ Real-World Troubleshooting & Failure Modes

1. The DNS64 / NAT64 MTU Fragmentation Hang

• Symptom: IPv6-only pods can query DNS64 successfully and resolve 64:ff9b::..., but HTTP connections to legacy IPv4 endpoints stall during large data downloads.
• Root Cause: NAT64 gateways translate IPv6 headers (40 fixed bytes) into IPv4 headers (20 bytes). When large packets hit maximum transmission limits, PMTUD or packet fragmentation bugs at the NAT64 boundary cause dropped packets.
• Fix: Enable Path MTU Discovery and ensure on-premise/host TCP MSS is set to 1220 bytes (IPv6 minimum MTU of 1280 bytes minus 60 bytes TCP/IPv6 header overhead).

2. Cloud WAN Policy Overwrite Race Condition

• Symptom: An engineer pushes an update to Cloud WAN policy; all cross-region traffic suddenly drops for 10 minutes.
• Root Cause: In Cloud WAN, updating a Core Network Policy creates a Change Set. If an automated pipeline executes execute-core-network-change-set without running a semantic diff, segment sharing rules (share-with) can be accidentally omitted, immediately tearing down global cross-region BGP routes!
• Fix: Enforce GitOps approvals with aws networkmanager get-core-network-change-set verification before applying policy executions.

📝 Staff Network Architect Interview Scenarios