Lesson 6: Azure Enterprise Landing Zones, Entra ID PIM, Virtual WAN & Private Link

🧠 The Concept (Explain Like I’m 5)

If you already know AWS, think of Microsoft Azure as a parallel cloud universe that solves the exact same enterprise problems, but with three major architectural advantages built directly into the platform:

1. Mandatory Resource Groups: In AWS, resources float loosely inside an account and you hope developers remember to tag them. In Azure, every single resource must live inside a Resource Group—like a mandatory shipping crate with its own lifecycle, RBAC permissions, and delete button. Delete the Resource Group, and every VM, disk, and NIC inside it is cleanly destroyed together.
2. Self-Healing Policies (DeployIfNotExists): In AWS, a Service Control Policy (SCP) can only say “No” (Deny). In Azure, Azure Policy doesn’t just block bad things—it has a DeployIfNotExists (DINE) superpower. When a developer creates a database or a Private Endpoint, Azure Policy can automatically wake up and deploy the missing diagnostic log forwarding or Private DNS record for them!
3. Just-In-Time Admin Access (Entra ID PIM): Instead of giving engineers permanent Owner access, Privileged Identity Management (PIM) keeps everyone at zero standing privilege. When an on-call SRE needs production access during an incident, they click “Activate” in PIM, pass a FIDO2 MFA check, enter a ticket number, and receive Contributor access for 60 minutes, after which Azure revokes it automatically.

🏢 The Enterprise Context: Azure Cloud Adoption Framework (CAF) Enterprise-Scale Landing Zones

In Tier-1 banks, GCCs (Walmart, Target, Goldman Sachs, Morgan Stanley), and multi-cloud enterprises, Azure is structured using the Microsoft Cloud Adoption Framework (CAF) Enterprise-Scale Landing Zone (ALZ) architecture:

flowchart TD
    TenantRoot["<b>Entra ID Tenant Root</b><br/>Global Identity Boundary"] --> RootMG["<b>Tenant Root Management Group</b>"]

    RootMG --> PlatformMG["<b>Platform Management Group</b><br/>Owned by Central Platform Team"]
    RootMG --> LandingZonesMG["<b>Landing Zones Management Group</b><br/>Application Workloads"]
    RootMG --> SandboxMG["<b>Sandbox Management Group</b><br/>R&D Experimentation (Strict Spend Caps)"]
    RootMG --> DecommMG["<b>Decommissioned MG</b><br/>Deny All Policy (Pending Deletion)"]

    subgraph PlatformSubs ["Core Platform Subscriptions"]
        MgmtSub["📊 <b>Management Subscription</b><br/>Log Analytics Workspace, Azure Monitor, Automation"]
        ConnSub["🌐 <b>Connectivity Subscription</b><br/>Virtual WAN / Hub VNet, Azure Firewall, ExpressRoute, Private DNS"]
        IdSub["🔐 <b>Identity Subscription</b><br/>Entra Domain Services, Key Vaults, AD Domain Controllers"]
    end

    subgraph WorkloadSubs ["Application Subscriptions (Vended via IaC)"]
        CorpMG["<b>Corp MG (Internal / Hybrid)</b><br/>VNet Peered to Hub, No Public IPs Allowed"]
        OnlineMG["<b>Online MG (Internet-Facing)</b><br/>Front Door + WAF Allowed, DDoS Protection"]
    end

    PlatformMG --> MgmtSub
    PlatformMG --> ConnSub
    PlatformMG --> IdSub

    LandingZonesMG --> CorpMG
    LandingZonesMG --> OnlineMG

Why Separate Corp vs. Online Management Groups?

• Corp Management Group: Hosts internal microservices, databases, and hybrid banking backends. Attached Azure Policies strictly Deny Public IP creation (Microsoft.Network/publicIPAddresses) and enforce mandatory VNet peering back to the central Connectivity Hub.
• Online Management Group: Hosts public-facing web/API tiers. Attached Azure Policies allow Azure Front Door and Application Gateway with WAF, and automatically enforce Azure DDoS Network Protection via DeployIfNotExists.

🛡️ Deep-Dive 1: Azure Policy Mechanics (The 7 Policy Effects)

In interviews, a classic Staff Architect question is: “How does Azure Policy differ architecturally from AWS Service Control Policies (SCPs)?”

While AWS SCPs are pure IAM guardrails (Allow/Deny on API actions), Azure Policy operates inside the Azure Resource Manager (ARM) control plane and inspects resource JSON payloads (properties.*) during PUT/PATCH requests as well as scanning existing resources continuously:

🔐 Deep-Dive 2: Entra ID, Managed Identities & Privileged Identity Management (PIM)

1. Eliminating Static Secrets: Managed Identities (System vs. User-Assigned)

In Azure, you should never create static Service Principal Client Secrets (client_secret) that expire and cause production outages. Instead, use Azure Managed Identities, which are backed by the Azure Instance Metadata Service (IMDS) at 169.254.169.254:

2. Entra ID Privileged Identity Management (PIM)

To pass RBI InfoSec, ISO 27001, and SOC 2 Type II audits with zero standing privileges:

• Eligible vs. Active Assignments: Platform engineers are assigned as Eligible for the Contributor or Owner role on production subscriptions. Under normal operations, they have read-only (Reader) access.
• Just-In-Time (JIT) Elevation: During an approved change window or P1 incident, the engineer requests activation in PIM:
  1. Enforces FIDO2 / Authenticator MFA.
  2. Requires an incident ticket ID and justification reason.
  3. Optional peer/manager approval workflow for Owner role.
  4. Automatically revokes access when the time window (1–4 hours) expires, logging every ARM API call to Azure Activity Logs.

🌐 Deep-Dive 3: Hub-and-Spoke Networking, Virtual WAN (vWAN), ExpressRoute & Private Link

flowchart LR
    OnPrem["🏢 On-Premises Data Center<br/>(172.16.0.0/12)"] -->|"ExpressRoute Private Peering<br/>(BGP over MPLS / Equinix)"| ERGw["<b>ExpressRoute Gateway</b><br/>(Or FastPath Direct to Spoke)"]

    subgraph ConnHub ["Connectivity Subscription (Hub VNet / vWAN Hub)"]
        ERGw --> ARS["<b>Azure Route Server</b><br/>(Dynamic BGP Engine)"]
        ARS <--> AzFW["🛡️ <b>Azure Firewall Premium</b><br/>IDPS + TLS Inspection + FQDN Filtering"]
        PrivDNS["📖 <b>Azure Private DNS Zones</b><br/>privatelink.postgres.database.azure.com<br/>+ Inbound/Outbound DNS Resolver"]
    end

    subgraph Spoke1 ["Spoke 1: Corp AKS VNet (10.10.0.0/16)"]
        UDR1["<b>UDR Route Table</b><br/>0.0.0.0/0 -> AzFW IP<br/>DisableBgpRoutePropagation: true"]
        AKS["☸️ AKS Private Cluster"]
        NATGw["<b>Azure NAT Gateway</b><br/>64,512 SNAT Ports / IP"]
    end

    subgraph Spoke2 ["Spoke 2: Data Tier VNet (10.20.0.0/16)"]
        PE["🔒 <b>Private Endpoint (NIC)</b><br/>10.20.1.5"] --> PgSQL[("Azure Database for<br/>PostgreSQL Flexible Server")]
    end

    AzFW <-->|"VNet Peering"| AKS
    AzFW <-->|"VNet Peering"| PE
    PrivDNS -.->|"VNet Link"| ConnHub
    PrivDNS -.->|"VNet Link"| Spoke1

Critical Azure Networking Traps Every Staff Architect Must Know:

1. VNet Peering Non-Transitivity & Gateway Transit Flags

Just like AWS VPC Peering, Azure VNet Peering is non-transitive (if Spoke A peers with Hub, and Hub peers with Spoke B, Spoke A cannot talk to Spoke B directly). To route Spoke-to-Spoke and Spoke-to-OnPrem traffic through the Hub:

• On the Hub-to-Spoke peering: Enable allow_gateway_transit = true (allows the Hub to share its ExpressRoute/VPN Gateway with the Spoke).
• On the Spoke-to-Hub peering: Enable use_remote_gateways = true and allow_forwarded_traffic = true (accepts packets forwarded by the Hub’s Azure Firewall).
• Attach a User-Defined Route (UDR) to all Spoke subnets setting 0.0.0.0/0 -> NextHopType: VirtualAppliance (pointing to the Azure Firewall private IP).

2. The BGP Route Propagation Override Trap (disable_bgp_route_propagation)

• The Trap: You configure a UDR on a Spoke subnet to force on-premises traffic (172.16.0.0/12) through Azure Firewall for deep packet inspection. However, because ExpressRoute is attached to the Hub, BGP advertises more specific /24 on-premises routes (172.16.1.0/24) directly to the Spoke! Because Longest Prefix Match (/24 beats /12) wins, Spoke traffic bypasses the Azure Firewall completely on the outbound path, while return traffic goes through the firewall—causing an immediate asymmetric routing TCP drop!
• The Fix: Set bgp_route_propagation_enabled = false (formerly disable_bgp_route_propagation = true) on Spoke route tables so only the Azure Firewall’s UDR routes are used, OR use Azure Virtual WAN (vWAN) Routing Intent, which automates symmetric firewall hair-pinning across ExpressRoute and VNets.

3. Azure Private Link & The DINE Private DNS Auto-Registration Pattern

When a developer creates an Azure Private Endpoint for Azure Key Vault (privatelink.vaultcore.azure.net) or PostgreSQL (privatelink.postgres.database.azure.com) in a Spoke VNet:

• In a multi-subscription enterprise, Spoke subscriptions must NOT create their own local Private DNS Zones (otherwise other spokes and on-premises servers cannot resolve the private IP!).
• The Enterprise Architecture:
  1. All privatelink.* Private DNS Zones live centrally in the Connectivity Subscription and are linked to the Hub VNet (which hosts the Azure DNS Private Resolver).
  2. When a developer creates a Private Endpoint in any Spoke subscription, an Azure Policy (DeployIfNotExists) triggers automatically and registers the A record (10.20.1.5) into the central Connectivity Private DNS Zone!

💻 Production Manifest: Terraform Azure Hub-Spoke, Firewall UDR & DINE Policy

# 1. Spoke Route Table forcing all traffic through Central Azure Firewall
#    CRITICAL: bgp_route_propagation_enabled = false prevents ExpressRoute BGP from bypassing the Firewall!
resource "azurerm_route_table" "spoke_udr" {
  name                          = "rt-spoke-payments-prod"
  location                      = azurerm_resource_group.spoke.location
  resource_group_name           = azurerm_resource_group.spoke.name
  bgp_route_propagation_enabled = false

  route {
    name                   = "DefaultRouteToHubFirewall"
    address_prefix         = "0.0.0.0/0"
    next_hop_type          = "VirtualAppliance"
    next_hop_in_ip_address = "10.0.1.4" # Private IP of Central Azure Firewall
  }
}

# 2. Spoke-to-Hub VNet Peering with Forwarded Traffic & Remote Gateways Enabled
resource "azurerm_virtual_network_peering" "spoke_to_hub" {
  name                         = "peer-spoke-payments-to-hub"
  resource_group_name          = azurerm_resource_group.spoke.name
  virtual_network_name         = azurerm_virtual_network.spoke.name
  remote_virtual_network_id    = data.azurerm_virtual_network.hub.id
  allow_virtual_network_access = true
  allow_forwarded_traffic      = true # Required to accept packets routed via Hub Azure Firewall
  use_remote_gateways          = true # Uses Hub ExpressRoute / VPN Gateway
}

# 3. Azure NAT Gateway to prevent SNAT Port Exhaustion (64,512 ports per Public IP)
resource "azurerm_nat_gateway" "spoke_egress" {
  name                    = "natgw-payments-prod"
  location                = azurerm_resource_group.spoke.location
  resource_group_name     = azurerm_resource_group.spoke.name
  sku_name                = "Standard"
  idle_timeout_in_minutes = 4
}

🧭 Step-by-Step Operational Playbook: Provisioning & Verifying an Azure Landing Zone Spoke

Follow this step-by-step operational sequence to vend a new production Azure Subscription and wire it into the central Connectivity Hub:

Step 1: Authenticate via Azure CLI & Set Management Group Context

1. Plain-English Goal: Log in with federated Entra ID credentials and verify the Management Group hierarchy (Platform vs LandingZones/Corp).
2. Action:

   az login --tenant <tenant-id>
   az account management-group list --output table

Step 2: Verify Azure Policy Compliance & DeployIfNotExists Remediation Tasks

1. Plain-English Goal: Confirm that organization guardrails (Region Restriction, Deny Public IP, Private DNS Auto-Registration) are active on the Corp Management Group.
2. Action:

   az policy state summarize --management-group "mg-corp-workloads"
   az policy remediation list --management-group "mg-corp-workloads" --output table

Step 3: Verify Effective Routes on a Spoke NIC (Catching Asymmetric BGP Leaks)

1. Plain-English Goal: Never guess how Azure routes packets—inspect the Effective Routes computed on the actual Network Interface (NIC) to verify 0.0.0.0/0 points to the Azure Firewall (10.0.1.4) and ExpressRoute BGP routes are not bypassing it.
2. Action:

   az network nic show-effective-route-table \
     --resource-group rg-payments-prod \
     --name nic-aks-nodepool-01 \
     --output table

Step 4: Verify Private Endpoint DNS Resolution Through Central Resolver

1. Plain-English Goal: Confirm that querying an Azure PaaS database FQDN returns the private 10.x.x.x Private Link IP via the privatelink.* CNAME chain.
2. Action:

   nslookup pg-payments-prod.postgres.database.azure.com
   # Expected: CNAME -> pg-payments-prod.privatelink.postgres.database.azure.com -> 10.20.1.5

🛠️ Real-World Troubleshooting & Failure Modes

Scenario 1: Outbound API Calls Intermittently Timing Out (SNAT Port Exhaustion)

• Symptom: During peak traffic, microservices in Azure fail to connect to external payment gateways (api.razorpay.com or api.stripe.com) with Connection timed out, even though CPU and bandwidth are below 20%.
• Under-the-Hood Root Cause: When multiple VMs or AKS nodes egress through an Azure Standard Load Balancer without a NAT Gateway, Azure pre-allocates a fixed pool of only 1,024 SNAT ports per node per public frontend IP. Once a node opens 1,024 concurrent TCP connections (including connections waiting in TIME_WAIT) to the same destination IP:Port, Azure drops all new outbound SYN packets!
• Immediate Triage & Permanent Fix:
  1. Check SNATConnectionCount (Filtered by ConnectionState = Failed) and UsedSNATPorts in Azure Monitor metrics on the Load Balancer.
  2. Attach an Azure NAT Gateway to the subnet (azurerm_subnet_nat_gateway_association). Unlike a Load Balancer, Azure NAT Gateway dynamically pools 64,512 SNAT ports per Public IP (up to 16 IPs = 1,032,192 concurrent SNAT ports) across all instances on demand without pre-allocation waste!

Scenario 2: ExpressRoute Asymmetric Routing Dropping Stateful TCP Through Azure Firewall

• Symptom: ping between an on-premises server and an Azure Spoke VM works briefly or fails after the first packet, and HTTPS connections reset with TCP RST after 3 seconds.
• Under-the-Hood Root Cause:
  1. Forward Path (On-Prem → Spoke): ExpressRoute Gateway receives the packet from on-premises. Because the GatewaySubnet in the Hub VNet is missing a UDR pointing the Spoke CIDR (10.10.0.0/16) to the Azure Firewall (10.0.1.4), the ExpressRoute Gateway routes the SYN packet directly to the Spoke VM, bypassing the Azure Firewall!
  2. Return Path (Spoke → On-Prem): The Spoke subnet has a UDR (0.0.0.0/0 -> 10.0.1.4), so the Spoke VM sends the SYN-ACK packet to the Azure Firewall. The stateful Azure Firewall sees a SYN-ACK for a session that never sent a SYN and drops it immediately!
• Permanent Fix: Attach a Route Table to the Hub GatewaySubnet (with bgp_route_propagation_enabled = true—never disable BGP on GatewaySubnet!) containing explicit /16 UDRs for each Spoke VNet pointing to the Azure Firewall IP (10.0.1.4), ensuring 100% symmetric inspection.

📝 Knowledge Check & Real-World Interview Scenarios