☁️ Round 3: IaC, Multi-Cloud Architecture, Vault & Policy-as-Code

In Staff Platform Architect and Principal Cloud Engineer interviews (PhonePe, Razorpay, Goldman Sachs IDC, Walmart Global Tech, CRED), Round 3 evaluates infrastructure engineering at scale.

Interviewers want to see how you orchestrate hundreds of AWS/GCP accounts, handle distributed state locking deadlocks, architect zero-trust security boundaries, and enforce enterprise governance without crippling developer velocity.

📌 Question 1: Terraform Distributed State Locking & Stale Lock Rescue

🎯 The Scenario & Interviewer’s Trap

“Your team manages cloud infrastructure across 40 engineers using Terraform with an S3 backend and DynamoDB state locking. A GitHub Actions runner applying a critical production VPC modification is suddenly killed (SIGKILL) due to a runner spot termination. Subsequent CI/CD runs fail with Error acquiring the state lock: ConditionalCheckFailedException. What data does DynamoDB store during a lock, why did this happen, and what is the exact protocol to safely unlock without causing state corruption?”

Interviewer’s Intent: To verify whether you know how Terraform guarantees ACID state modifications and whether you understand the danger of running force-unlock blindly in a multi-engineer team.

💡 Staff-Level Model Answer

sequenceDiagram
    autonumber
    participant CI as CI Runner (Terraform Apply)
    participant DDB as DynamoDB Lock Table
    participant S3 as S3 State Bucket (terraform.tfstate)

    CI->>DDB: PutItem: LockID="bucket/key-md5", Info={ID, Created, Path, Who}
    Note over DDB: ConditionalCheck: attribute_not_exists(LockID) -> SUCCESS!
    CI->>S3: Read current state (serial: 42)
    Note over CI: Spot Runner receives SIGKILL! Process dies instantly!
    Note over DDB: DynamoDB Lock RECORD PERSISTS INDEFINITELY!
    
    participant NewCI as New CI Runner / Engineer
    NewCI->>DDB: PutItem: LockID="bucket/key-md5"
    DDB-->>NewCI: ERROR! ConditionalCheckFailedException (Lock already held!)

1. How DynamoDB State Locking Works Under the Hood:

When Terraform executes terraform init/plan/apply with an S3 backend:

• It computes the primary key LockID: 📐 LockID = BucketName / StateFilePath-md5
• It attempts a DynamoDB PutItem operation using a Conditional Expression: attribute_not_exists(LockID)
• The payload stored in DynamoDB contains:

  {
    "LockID": {"S": "my-corp-tf-state/prod/vpc/terraform.tfstate-md5"},
    "Info": {
      "S": "{\"ID\":\"a1b2c3d4-e5f6-...\",\"Operation\":\"OperationTypeApply\",\"Info\":\"\",\"Who\":\"runner@gh-runner-04\",\"Version\":\"1.9.0\",\"Created\":\"2026-10-12T14:00:00Z\",\"Path\":\"prod/vpc/terraform.tfstate\"}"
    }
  }

• Because DynamoDB does not have an automatic lease TTL tied to the OS process, when the runner dies from SIGKILL, the lock record remains in DynamoDB indefinitely.

2. The Danger of Blind Force-Unlocking:

If an engineer runs terraform force-unlock <ID> while another engineer or background runner is actually executing a state write, both processes write to S3 concurrently, causing state corruption, lost resource IDs, and orphaned cloud resources!

3. The Staff-Level Safe Resolution Protocol:

1. Verify Process Death: Inspect the "Who" field in the error message and the GitHub Actions run history to confirm that the runner that acquired the lock is 100% terminated and no apply process is running.
2. Execute Force-Unlock with Exact Lock ID:

   terraform force-unlock a1b2c3d4-e5f6-7890-abcd-ef0123456789

3. Verify S3 State Serial Number:

   # Confirm state integrity and check current serial number:
   aws s3 cp s3://my-corp-tf-state/prod/vpc/terraform.tfstate - | jq .serial

4. Re-run Plan before Apply: Always run terraform plan first to ensure the in-memory state matches the real-world infrastructure before attempting another apply.

📌 Question 2: Transit Gateway Appliance Mode & Asymmetric Routing

🎯 The Scenario & Interviewer’s Trap

“You design a centralized Egress & East-West security inspection architecture using AWS Transit Gateway (TGW). Traffic from workload VPCs must pass through a dedicated Inspection VPC running a cluster of stateful Palo Alto / Fortinet firewalls across two Availability Zones (us-east-1a and us-east-1b). Workload pods in AZ-a can access external APIs, but workload pods in AZ-b experience random 50% packet drops and TCP resets. Why is this occurring, and what exact TGW configuration is missing?”

Interviewer’s Intent: To test if you understand stateful firewall connection tables and how AWS Transit Gateway balances traffic across ENIs without Appliance Mode.

💡 Staff-Level Model Answer

flowchart TD
    subgraph WorkloadVPC["Workload VPC (AZ-b)"]
        Pod["App Pod in AZ-b"]
    end

    subgraph TGW["AWS Transit Gateway (No Appliance Mode)"]
        TGW_Route["TGW Route Table"]
    end

    subgraph InspectVPC["Security Inspection VPC"]
        FW_A["Stateful Firewall A (AZ-a)<br/>Connection Table: [SYN NOT SEEN!]"]
        FW_B["Stateful Firewall B (AZ-b)<br/>Connection Table: [SYN: Active]"]
    end

    Pod -->|"1. Outbound SYN (AZ-b)"| TGW
    TGW -->|"2. Routed to local AZ ENI"| FW_B
    FW_B -->|"3. Forwarded to Internet"| Web["External SaaS API"]
    Web -->|"4. Return SYN-ACK packet"| TGW
    TGW -- "5. ASYMMETRIC HASH! Routes return to AZ-a ENI!" --> FW_A
    FW_A -- "6. DROP! Out-of-state packet (No prior SYN in table)!" --> Drop["TCP RESET / SILENT DROP!"]

1. The Root Cause: Asymmetric Flow Across Stateful Firewalls:

• The Outbound Path: The pod in AZ-b sends a TCP SYN. TGW routes the packet to the Inspection VPC attachment ENI located in AZ-b. Firewall B records the TCP connection in its state table and forwards the packet out to the internet.
• The Return Path: The external API sends the SYN-ACK back to TGW. By default, Transit Gateway uses an internal hash (Source IP, Destination IP, Source Port, Destination Port) to pick an attachment ENI.
• The Asymmetric Hash: Because TGW does not know that Firewall A and Firewall B are separate stateful devices, it may hash the return flow to the attachment ENI in AZ-a!
• The Silent Drop: Firewall A receives the SYN-ACK. Firewall A checks its connection state table, finds no record of a prior SYN for this 4-tuple, marks the packet as invalid/spoofed, and drops it immediately.

2. The Architectural Remediation:

Enable Appliance Mode on the Transit Gateway VPC Attachment for the Inspection VPC:

aws ec2 modify-transit-gateway-vpc-attachment \
    --transit-gateway-attachment-id tgw-attach-0123456789abcdef0 \
    --options ApplianceModeSupport=enable

In Terraform:

resource "aws_ec2_transit_gateway_vpc_attachment" "inspection" {
  transit_gateway_id = aws_ec2_transit_gateway.main.id
  vpc_id             = aws_vpc.inspection.id
  subnet_ids         = aws_subnet.inspection_firewall[*].id

  appliance_mode_support = "enable" # Guarantees bidirectional flow symmetry!
}

• How Appliance Mode Fixes It: When Appliance Mode is enabled, Transit Gateway guarantees that both outbound and return flows for a given TCP/UDP session are routed to the same attachment ENI in the same Availability Zone, ensuring the same firewall processes both legs of the conversation.

📌 Question 3: HashiCorp Vault Dynamic PostgreSQL Credentials & Lease Lifecycle

🎯 The Scenario & Interviewer’s Trap

“Your platform uses HashiCorp Vault to eliminate static database passwords. Spring Boot microservices authenticate to Vault using Kubernetes ServiceAccount tokens and receive dynamic PostgreSQL credentials with a 1-hour TTL. During a major AWS network incident, Vault loses network connectivity to the Aurora PostgreSQL database cluster for 2 hours. What happens to the microservices when their 1-hour lease expires? Does Vault cut active connections, and how do you recover?”

Interviewer’s Intent: To verify whether you know how Vault tracks dynamic secret leases, the role of lease renewal daemons, and failure modes when Vault cannot execute SQL revocation statements.

💡 Staff-Level Model Answer

sequenceDiagram
    autonumber
    participant App as Spring Boot Pod
    participant Vault as HashiCorp Vault Cluster
    participant DB as Aurora PostgreSQL

    App->>Vault: Read database/creds/payment-role
    Vault->>DB: CREATE ROLE "v-token-xyz" WITH PASSWORD '...' VALID UNTIL '...'
    Vault-->>App: Credentials + LeaseID (TTL: 3600s)
    Note over App: App establishes DB connection pool with dynamic user

    Note over Vault,DB: Network Partition: Vault cannot reach PostgreSQL!
    Note over App: 3600 seconds elapse! Lease expires!
    Vault->>DB: Attempt REVOKE & DROP ROLE "v-token-xyz"
    DB--xVault: Connection Timeout!
    Note over Vault: Vault places revocation into "Pending Revocation Queue"!
    Note over App,DB: ACTIVE CONNECTIONS PERSIST! (PostgreSQL does not terminate open sockets)

1. What Happens to Active Connections When the Lease Expires?

• PostgreSQL Behavior: When a user’s VALID UNTIL timestamp is reached in PostgreSQL, PostgreSQL prevents new connections from authenticating with that password. However, existing open TCP socket connections are NOT terminated!
• Microservices with established connection pools (HikariCP) continue executing queries on existing open sockets until the pool re-validates or cycles connections.
• Vault Revocation Failure: Vault attempts to connect to the database to execute the revocation SQL (REVOKE ALL PRIVILEGES... DROP ROLE...). Because Vault cannot reach PostgreSQL, the revocation fails.
• Vault marks the lease in an exponential backoff retry queue in its storage backend.

2. The Danger on Recovery (Thundering Herd Revocation):

When network connectivity is restored 2 hours later, Vault immediately executes thousands of accumulated pending revocation statements simultaneously, flooding PostgreSQL with DDL locks on the pg_authid catalog table, which can crash the database!

3. The Staff-Level Mitigation:

1. Graceful Lease Renewal (Vault Agent Sidecar): Applications should never hardcode credential fetching. Use a Vault Agent Sidecar configured to automatically renew leases before expiry (renew_percent: 75).
2. Circuit Breaking on Vault Failure: Configure connection pool timeouts so microservices don’t drop traffic immediately if credential rotation is delayed.
3. Emergency Vault Revocation Rate Limiting:

   # Check pending revocations:
   vault list sys/leases/lookup/database/creds/payment-role
   
   # Tune Vault revocation worker concurrency in vault config:
   # max_lease_ttl = "24h"

📌 Question 4: EKS IRSA (IAM Roles for Service Accounts) Security & OIDC Thumbprints

🎯 The Scenario & Interviewer’s Trap

“Explain the end-to-end cryptographic exchange that occurs when a Kubernetes pod with an IAM Role for Service Accounts (IRSA) calls s3.ListBuckets(). How does AWS STS verify the pod’s identity without any static AWS credentials stored on the node? In 2023, AWS rotated its OIDC root certificate and thousands of enterprise CI/CD pipelines broke globally. Why did that happen?”

Interviewer’s Intent: To verify whether you understand the OpenID Connect (OIDC) federation flow between Kubernetes Projected Service Account Tokens, STS AssumeRoleWithWebIdentity, and TLS CA thumbprints.

💡 Staff-Level Model Answer

sequenceDiagram
    autonumber
    participant Pod as Container / AWS SDK
    participant API as kube-apiserver (OIDC Issuer)
    participant STS as AWS Security Token Service (STS)
    participant S3 as Amazon S3 Service

    Note over Pod: kubelet mounts Projected SA Token at /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    Pod->>STS: AssumeRoleWithWebIdentity(RoleArn, WebIdentityToken)
    STS->>API: Fetch OIDC Discovery Document (/.well-known/openid-configuration)
    STS->>API: Fetch Public Keys (JWKS: /openid/v1/jwks)
    Note over STS: STS validates JWT Signature using Public Key & checks Issuer URL
    STS-->>Pod: Temporary AWS Credentials (AccessKey, SecretKey, SessionToken - 1h TTL)
    Pod->>S3: ListBuckets (Signed with Temporary Credentials)
    S3-->>Pod: 200 OK Response

1. The Step-by-Step IRSA Cryptographic Exchange:

1. Projected Volume Mount: kubelet projects a cryptographically signed, short-lived JSON Web Token (JWT) into the pod at /var/run/secrets/eks.amazonaws.com/serviceaccount/token.
2. SDK Interception: When the AWS SDK initializes, it reads the environment variables injected by the EKS Pod Identity Webhook:
   • AWS_ROLE_ARN=arn:aws:iam::123456789012:role/s3-reader
   • AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/...
3. STS Call: The SDK executes an AssumeRoleWithWebIdentity API call to AWS STS, passing the raw Kubernetes JWT.
4. Signature Verification via JWKS:
   • STS looks up the IAM Role’s Trust Policy, which points to the cluster’s OIDC Issuer URL.
   • STS contacts the cluster’s OIDC discovery endpoint (https://oidc.eks.us-east-1.amazonaws.com/id/.../openid/v1/jwks) and fetches the cluster’s public JSON Web Key Set (JWKS).
   • STS cryptographically verifies that the JWT was signed by the cluster’s private key and validates the aud (audience = sts.amazonaws.com) and sub (subject = system:serviceaccount:default:my-app-sa).
5. Temporary Credential Issuance: STS returns a 1-hour session token (ASIA...), allowing the pod to access AWS APIs securely.

2. The OIDC Thumbprint Incident:

• The Root Cause: When creating an IAM OIDC Identity Provider in AWS, AWS historically required a SHA-1 Thumbprint of the root Certificate Authority (CA) that signed the OIDC issuer’s TLS certificate.
• When the external Certificate Authority (or GitHub Actions / AWS EKS) rotated its intermediate or root TLS certificate to a newer CA, the TLS thumbprint changed!
• Because the IAM OIDC Provider had the old thumbprint hardcoded, STS rejected all AssumeRoleWithWebIdentity calls with: InvalidIdentityToken: OpenIDConnect provider's HTTPS certificate does not match the configured thumbprint.
• The Fix: AWS updated STS to automatically trust root CAs from major providers without strict single-thumbprint pinning. In Terraform, ensure multiple thumbprints or current official thumbprints are maintained.

📌 Question 5: Policy-as-Code: OPA Gatekeeper vs. Kyverno at Scale

🎯 The Scenario & Interviewer’s Trap

“You are designing policy enforcement for a 100-cluster multi-tenant Kubernetes platform. You must enforce CIS Benchmarks: prevent privileged containers, restrict hostPath mounts, block root users, and mandate non-empty resource requests. Compare OPA Gatekeeper and Kyverno across performance, syntax complexity, mutation capabilities, and operational maintenance. Which one do you choose and why?”

Interviewer’s Intent: To test if you have evaluated enterprise policy frameworks objectively, understanding the operational trade-off between Rego’s expressive power and Kyverno’s Kubernetes-native YAML simplicity.

💡 Staff-Level Model Answer

Staff Architectural Decision:

• Choose Kyverno if your requirements are Kubernetes-focused: CIS benchmark enforcement, image signature verification with Cosign, standard namespace resource generation, and team familiarity with YAML.
• Choose OPA Gatekeeper if you have a unified policy framework spanning Terraform (Conftest), Envoy microservice auth, and Kubernetes, where a single unified Rego language governs the entire enterprise tech stack.

📌 Summary: Cloud Architecture & Security Quick-Fire Reference

🧠 Complete Lesson-by-Lesson One-Line Memory Bank (Round 3 Scope)

☁️ Phase 1: Enterprise Cloud Governance & Identity

Lesson 1 & 3: AWS Orgs, Landing Zones & IAM Identity Center

🏗️ Phase 2: IaC & Policy-as-Code (All 7 Lessons)

Lesson 1: Terraform State & Modules

Lesson 2: Policy-as-Code (Checkov & OPA)

Lesson 3: Terragrunt Fleet Scaling

Lesson 4: HashiCorp Vault Secrets

Lesson 5, 6 & 7: Moved Blocks, HCL Mastery & Python Lambda

🛡️ Phase 3 & 3B: DevSecOps, Supply Chain & Fleet Patching (All 8 Lessons)

Phase 3: Cosign, OIDC, ESO & 2-User Rotation

Phase 3B: Golden AMIs, ASG Refresh & PDBs

Phase 1 — Lesson 6: Azure Enterprise Landing Zones, Entra ID PIM, Virtual WAN & Private Link