⚡
Zero to Platform Architect2026
War-Room Triage & 43 Traps
War-Room Triage & 43 Traps⏱️ 6 min read⚡ Staff SRE Standard
🎯 Operations Test Bank →
Quick Jump:📝 Q&A Drill

📜 IaC, Terraform, Vault & Policy-as-Code Triage Guide

This guide details the most common, tricky, and critical failure modes encountered across Infrastructure-as-Code (Terraform & Terragrunt), Secret Management (HashiCorp Vault), and Policy-as-Code engines (OPA & Checkov).

🎯 High-Impact Real-World Scenarios

Scenario 1: Terraform State Lock Stuck in DynamoDB (ConditionalCheckFailedException)

The Symptom

A developer pushes a change or triggers an automated CI/CD pipeline. The pipeline immediately halts with:

Error: Error acquiring the state lock: ConditionalCheckFailedException: 
The conditional request failed
Lock Info:
  ID:        e74b391a-8c10-4f92-9a5e-219803125d48
  Path:      terraform-states-prod/vpc/terraform.tfstate
  Operation: OperationTypeApply
  Who:       runner@github-runner-104.internal
  Created:   2026-10-12 14:15:22.182 UTC

Under-the-Hood Root Cause

Terraform writes a lock record to AWS DynamoDB (table terraform-locks) before executing plan or apply to prevent concurrent modifications. If a GitHub Actions runner or developer terminal process is killed abruptly (e.g. CI runner OOM, container cancellation, runner timeout, or spot termination) before Terraform can execute its exit cleanup handler, the lock record remains in DynamoDB indefinitely.

CLI Triage Commands

# 1. Verify that no other pipeline or engineer is actively running an apply on this state
# Check GitHub Actions / GitLab CI workflow runs for the repository

# 2. Inspect the lock item directly in AWS DynamoDB
aws dynamodb get-item \
  --table-name terraform-locks \
  --key '{"LockID": {"S": "terraform-states-prod/vpc/terraform.tfstate-md5"}}'

# 3. Force-unlock using the Lock ID provided in the error message
terraform force-unlock e74b391a-8c10-4f92-9a5e-219803125d48

Remediation & Architectural Guardrail

  • Never delete the DynamoDB item manually with the AWS Console or CLI unless terraform force-unlock fails.
  • Add Pipeline Traps: In CI/CD scripts, wrap Terraform runs in shell signal handlers (trap 'terraform force-unlock -force <ID>' SIGINT SIGTERM) to clean up locks if the job is canceled.

Scenario 2: Cyclic Graph Dependency Deadlock in Terraform Security Groups

The Symptom

Running terraform plan or terraform apply fails before evaluating any cloud resources:

Error: Cycle: aws_security_group.app, aws_security_group.db, aws_security_group.app

Under-the-Hood Root Cause

Security Group app defines an inline egress rule pointing to Security Group db. Security Group db defines an inline ingress rule pointing to Security Group app. Because both security group resources reference each other directly in their ingress / egress argument blocks, Terraform cannot construct a Directed Acyclic Graph (DAG) to determine which security group must be created first.

flowchart LR
    subgraph CyclicTrap ["❌ The Cyclic Trap (Inline Rules)"]
        SGApp["aws_security_group.app<br/>(references db in inline rule)"] <--> SGDB["aws_security_group.db<br/>(references app in inline rule)"]
    end

    subgraph DAGSolution ["✅ The Decoupled DAG (Standalone Rules)"]
        SGAppNew["aws_security_group.app<br/>(Empty shell)"]
        SGDBNew["aws_security_group.db<br/>(Empty shell)"]
        Rule1["aws_security_group_rule.app_to_db"]
        Rule2["aws_security_group_rule.db_from_app"]

        SGAppNew --> Rule1
        SGDBNew --> Rule1
        SGAppNew --> Rule2
        SGDBNew --> Rule2
    end

Remediation & Architectural Standard

  • Decouple Rules from Security Groups: Never embed inline ingress and egress blocks inside aws_security_group. Always provision empty security groups first, then define relationships using standalone aws_security_group_rule resources:
# 1. Standalone Security Groups (Zero mutual dependencies)
resource "aws_security_group" "app" {
  name   = "app-sg"
  vpc_id = aws_vpc.main.id
}

resource "aws_security_group" "db" {
  name   = "db-sg"
  vpc_id = aws_vpc.main.id
}

# 2. Standalone Rule attaching to app
resource "aws_security_group_rule" "app_egress_to_db" {
  type                     = "egress"
  from_port                = 5432
  to_port                  = 5432
  protocol                 = "tcp"
  security_group_id        = aws_security_group.app.id
  source_security_group_id = aws_security_group.db.id
}

# 3. Standalone Rule attaching to db
resource "aws_security_group_rule" "db_ingress_from_app" {
  type                     = "ingress"
  from_port                = 5432
  to_port                  = 5432
  protocol                 = "tcp"
  security_group_id        = aws_security_group.db.id
  source_security_group_id = aws_security_group.app.id
}

Scenario 3: HashiCorp Vault Auto-Unseal Failure & Raft Quorum Loss

The Symptom

Following a node reboot or cluster maintenance, all HashiCorp Vault instances report Sealed: true. Client applications fail with HTTP 503 Service Unavailable: Vault is sealed. Even automated KMS unseal fails with:

[ERROR] core: failed to retry unseal: error="kms:Decrypt: AccessDeniedException: 
User is not authorized to perform: kms:Decrypt on resource: arn:aws:kms:..."

Under-the-Hood Root Cause

Two cascading failure modes:

  1. KMS IAM Policy Revocation: The EC2 IAM role / IRSA role assumed by Vault lost kms:Decrypt permission on the AWS KMS customer-managed key used for transit auto-unseal (often caused by an overly aggressive SCP or IAM policy update).
  2. Raft Quorum Loss: While nodes were rebooting, two nodes out of a 3-node Integrated Storage (Raft) cluster could not establish peer heartbeat, losing the consensus quorum.

CLI Triage Commands

# 1. Check Vault status on all cluster nodes
vault status -address=http://127.0.0.1:8200

# 2. Inspect active Raft cluster peers
vault operator raft list-peers -address=http://127.0.0.1:8200

# 3. Verify KMS Decrypt permissions using AWS CLI from the Vault node
aws kms decrypt \
  --ciphertext-blob fileb://test_blob.bin \
  --key-id arn:aws:kms:us-east-1:123456789012:key/vault-unseal-key

Remediation Runbook

  • Restore IAM Permissions: Update the AWS IAM role policy or SCP to re-grant kms:Decrypt and kms:Encrypt on the unseal key.
  • Break-Glass Raft Quorum Recovery: If Raft permanently lost quorum, create a peers.json file in Vault’s data directory on the surviving leader node and restart:
[
  {
    "id": "vault-0",
    "address": "vault-0.vault-internal:8201",
    "leader": true
  }
]

Scenario 4: Dynamic Database Secret Lease Expiration Storm

The Symptom

At exactly 08:00 AM, hundreds of microservices crash simultaneously with database connection pool errors:

org.postgresql.util.PSQLException: FATAL: password authentication failed for user "v-token-app-abc123"

The AWS RDS PostgreSQL instance CPU utilization spikes to 100% as thousands of pods attempt to reconnect at once.

Under-the-Hood Root Cause

The platform team configured HashiCorp Vault’s Database Secrets Engine to issue dynamic, short-lived database credentials with a default TTL of 1 hour (default_lease_ttl = "1h"). When the lease expired, Vault automatically executed REVOKE on PostgreSQL. However, the microservices were using a legacy connection pool (HikariCP / Tomcat) that did not implement automated token renewal or reconnection listeners. When Vault revoked the user, all active pooled database connections were abruptly killed by PostgreSQL!

Remediation & Architectural Standard

  1. Extend Dynamic Secret Leases: Set dynamic credential TTLs to match maintenance cycles (e.g., 24 hours), or pair them with External Secrets Operator (ESO) with a defined refreshInterval: 12h.
  2. Enforce Staggered Rotation: Never configure identical lease expiration times across services. Add jitter to lease renewal cycles to prevent thundering herd reconnection storms on the database.
  3. Use 2-User Static Secret Rotation for mission-critical core databases rather than short-lived dynamic users.

Scenario 5: Terragrunt Cold-Boot Output Dependency Deadlock

The Symptom

During a Disaster Recovery (DR) exercise to rebuild a region from scratch, running terragrunt run-all apply fails with:

ERRO[0012] Module /infra/eks depends on /infra/vpc, which has no outputs yet. 
Cannot read output 'vpc_id' from dependency.

Under-the-Hood Root Cause

In Terragrunt, the eks module defines a dependency "vpc" block with config_path = "../vpc". In an existing environment, Terragrunt reads the output from the live S3 state file. However, in a completely fresh account (cold boot), the VPC has not yet been applied, so its state file does not exist. Without mock outputs, Terragrunt cannot parse the DAG and crashes before even creating the VPC!

Architectural Fix

Configure mock_outputs and mock_outputs_allowed_terraform_commands in all Terragrunt dependency blocks:

dependency "vpc" {
  config_path = "../vpc"

  # Allows 'terragrunt run-all plan' and cold-boot DAG parsing to succeed
  mock_outputs = {
    vpc_id             = "vpc-fake-id-for-plan"
    private_subnet_ids = ["subnet-fake-1", "subnet-fake-2"]
  }
  mock_outputs_allowed_terraform_commands = ["validate", "plan"]
}