Lesson 5: Declarative Refactoring with moved Blocks & Automated Drift Detection

🧠 The Concept (Explain Like I’m 5)

Imagine you live in a modern apartment building:

• The Refactoring Terror of the Past: In early versions of Terraform, if you reorganized your code (for instance, moving a database resource into a reusable module or renaming aws_instance.server to aws_instance.web), Terraform treated it as two separate actions: Destroy the old database, and create a brand-new empty one! If you ran terraform apply, your production customer data was deleted instantly. To prevent disaster, an engineer had to manually run fragile, error-prone terraform state mv commands from their laptop.
• The moved Block (The Change-of-Address Form): Instead of manual CLI hacking, Terraform 1.1+ introduced the declarative moved block. You leave a formal change-of-address note in your code: “Dear Terraform, aws_db_instance.db has moved to module.database.aws_db_instance.db.” When Terraform executes, it reads the note and quietly updates the address pointer in the state ledger with zero downtime, zero disruption, and zero risk of data loss.
• Infrastructure Drift (The Ghost in the Machine): At 2:00 AM on a Saturday, an on-call engineer fixes an urgent production outage by manually clicking around the AWS Console to open a Security Group port or expand an EBS volume. The real cloud has now drifted away from the blueprint stored in Git.

🏢 The Enterprise Context (Why Platform Architects Care)

1. Zero-Downtime Safe Refactoring at Scale: In an enterprise with 50 environments (Dev, Staging, Production across 30 AWS accounts), manual CLI terraform state mv commands cannot be safely coordinated. Declarative moved blocks are committed to Git, reviewed in Pull Requests, and applied automatically by CI/CD pipelines across every account without downtime.
2. Mitigating Plaintext Secrets in State: Marking a variable as sensitive = true in Terraform only hides it from your terminal console and CI logs; it does NOT encrypt the secret inside terraform.tfstate. If an RDS module outputs a master password, it sits in plaintext JSON in the S3 bucket. Platform architects must implement defense-in-depth: state encryption via KMS, strict IAM bucket policies, and dynamic credentials via HashiCorp Vault.
3. Automated Continuous Drift Detection: Manual changes made outside of Git violate SOC2 and ISO 27001 compliance. Scheduled drift detection pipelines run nightly to detect unmanaged cloud changes, alerting SRE teams before unmanaged drift causes production deployment failures.

🗺️ Visual Architecture: Declarative Refactoring & Drift Detection Pipelines

flowchart TD
    subgraph RefactorWorkflow ["Declarative Refactoring Pipeline"]
        CodeChange["Engineer moves resource into module in Git"] --> AddMoved["Add 'moved' block in main.tf"]
        AddMoved --> SubmitPR["Submit Pull Request & Run CI Plan"]
        SubmitPR --> PlanCheck{"Plan Output?"}
        PlanCheck -->|"Shows 0 to add, 0 to destroy, 1 to move ✅"| MergePR["Merge PR to Main Branch"]
        MergePR --> CIApply["CI Pipeline: terraform apply"]
        CIApply --> StateUpdated["State ledger pointer updated in S3!<br/>(Zero cloud downtime)"]
    end

    subgraph DriftWorkflow ["Automated Scheduled Drift Detection Pipeline"]
        CronTrigger["⏱️ Nightly Cron Trigger (02:00 UTC)"] --> Checkout["Checkout Main Branch in GitHub Actions"]
        Checkout --> OIDC["Assume AWS OIDC Role"]
        OIDC --> RunDriftPlan["terraform plan -detailed-exitcode"]
        RunDriftPlan --> EvaluateExit{"Process Exit Code"}
        
        EvaluateExit -->|"Exit 0: No Drift"| CleanLog["Log Clean: State matches reality ✅"]
        EvaluateExit -->|"Exit 2: Drift Detected"| SlackAlert["🚨 Fire PagerDuty / Slack Alert:<br/>'Manual Console Drift Detected in Prod!'"]
        EvaluateExit -->|"Exit 1: Error"| FailPipeline["❌ CI Execution Failure"]
        
        SlackAlert --> AutoPR["Auto-generate GitHub Issue / Remediation PR"]
    end

🔬 Under the Hood: moved Block Mechanics & Chaining

1. Anatomy of the moved Block

A moved block requires two parameters: from and to. Both represent resource addresses in Terraform’s execution graph:

moved {
  from = aws_security_group.web_tier
  to   = module.networking.aws_security_group.web_tier
}

2. Refactoring a Single Resource into a for_each Map

A common architectural evolution is transitioning a standalone resource into a multi-AZ collection:

# Before refactor:
# resource "aws_subnet" "public" { ... }

# After refactor:
# resource "aws_subnet" "public" {
#   for_each = var.subnets
#   ...
# }

# Declarative moved block maps single resource to a map element:
moved {
  from = aws_subnet.public
  to   = aws_subnet.public["us-east-1a"]
}

3. Chaining moved Blocks Across Module Versions

If infrastructure undergoes multiple refactoring cycles across months, you can chain moved blocks sequentially:

# Year 2024: Standalone resource moved into module
moved {
  from = aws_s3_bucket.logs
  to   = module.storage.aws_s3_bucket.logs
}

# Year 2026: Storage module refactored into a collection
moved {
  from = module.storage.aws_s3_bucket.logs
  to   = module.storage.aws_s3_bucket.buckets["audit_logs"]
}

How Terraform handles chains: If a resource is renamed from address A to address B in one sprint, and later from B to address C in a subsequent sprint, Terraform automatically traces the entire chain from A through B to C, updating the state directly to C in a single execution step.

4. Decommissioning moved Blocks

When can you safely delete a moved block from your codebase?

• Rule: A moved block must remain in Git until every environment, branch, and member account running that configuration has executed terraform apply at least once.
• Once all environments have converged to the new state address, the moved block can be safely removed.

🔒 Plaintext Secrets in State: Architecture & Mitigation

The Hard Truth: sensitive = true Does Not Protect State

When you declare:

output "db_password" {
  value     = aws_db_instance.main.password
  sensitive = true
}

Terraform merely redacts the value in the terminal CLI: db_password = (sensitive value). If you inspect the actual terraform.tfstate JSON file stored in AWS S3, the password exists in clear, unencrypted text:

"attributes": {
  "password": "MySuperSecretProductionPassword123!",
  "username": "admin"
}

Enterprise Defense-in-Depth Mitigation Matrix

1. S3 KMS Key Policies with Explicit Denies: Encrypt the remote state S3 bucket with a customer-managed KMS key. Deny kms:Decrypt to all users, even AWS account administrators, granting decryption permissions strictly to the dedicated CI/CD execution role.
2. Dynamic Ephemeral Secrets: Use HashiCorp Vault or AWS Secrets Manager to generate short-lived, dynamic database credentials on-the-fly rather than defining static credentials inside Terraform.
3. OpenTofu Native State Encryption (OpenTofu 1.7+): OpenTofu introduced native client-side state encryption. The state file is encrypted with AES-GCM before it leaves the runner memory, so even an attacker with full access to the S3 bucket sees only ciphertext.

🤖 Automated Nightly Drift Detection Pipeline (GitHub Actions)

The secret to automated drift detection is the -detailed-exitcode flag. Under standard execution, terraform plan returns exit code 0 even if pending changes exist. With -detailed-exitcode:

• 0: Succeeded, diff is empty (Infrastructure matches Git perfectly ✅).
• 1: Errored (Authentication failure, syntax bug ❌).
• 2: Succeeded, diff is non-empty (DRIFT DETECTED 🚨).

# .github/workflows/drift-detection.yml
name: Continuous Infrastructure Drift Detection

on:
  schedule:
    # Run every night at 02:00 AM UTC
    - cron: '0 2 * * *'
  workflow_dispatch: # Allow manual trigger from GitHub UI

jobs:
  detect-drift:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read

    steps:
      - name: Checkout Codebase
        uses: actions/checkout@v4

      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: 1.8.2

      - name: Authenticate to AWS via OIDC
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/TerraformDriftDetectorRole
          aws-region: us-east-1

      - name: Initialize Terraform
        run: terraform init

      - name: Execute Drift Detection Plan
        id: drift_check
        # continue-on-error is required because exit code 2 would otherwise fail the step immediately
        continue-on-error: true
        run: |
          terraform plan -detailed-exitcode -no-color > drift_report.log
          EXIT_CODE=$?
          echo "exit_code=${EXIT_CODE}" >> $GITHUB_OUTPUT

      - name: Evaluate Drift Status
        run: |
          if [ "${{ steps.drift_check.outputs.exit_code }}" -eq "0" ]; then
            echo "✅ No infrastructure drift detected. Reality matches Git."
          elif [ "${{ steps.drift_check.outputs.exit_code }}" -eq "2" ]; then
            echo "🚨 DRIFT DETECTED: Cloud resources have been modified outside of Git!"
            exit 2
          else
            echo "❌ Pipeline failed during plan execution."
            exit 1
          fi

      - name: Alert Security & SRE via Slack
        if: steps.drift_check.outputs.exit_code == '2'
        uses: slackapi/slack-github-action@v1.26.0
        with:
          payload: |
            {
              "text": "🚨 CRITICAL: Terraform Infrastructure Drift Detected in Production!",
              "blocks": [
                {
                  "type": "section",
                  "text": {
                    "type": "mrkdwn",
                    "text": "*Alert:* Production AWS resources have diverged from the Terraform configuration in `main`.\n*Action Required:* Review drift log to reconcile manual changes."
                  }
                },
                {
                  "type": "actions",
                  "elements": [
                    {
                      "type": "button",
                      "text": { "type": "plain_text", "text": "View Actions Run" },
                      "url": "${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
                    }
                  ]
                }
              ]
            }
        env:
          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_PLATFORM_ALERTS_WEBHOOK }}

🛠️ Step-by-Step Command Playbook

# 1. Run local plan with detailed exit code
terraform plan -detailed-exitcode
echo $? # Output: 0 = clean, 2 = drift detected

# 2. Reconcile drift: Refresh state against reality without applying changes
terraform apply -refresh-only

# 3. Target refactor verification: Verify moved blocks apply without destruction
terraform plan -target=module.networking

# 4. View raw JSON diff of planned modifications
terraform plan -out=tfplan.binary
terraform show -json tfplan.binary | jq '.resource_changes[] | select(.change.actions != ["no-op"])'

⚠️ Real-World Failure Modes & Troubleshooting Playbook

Scenario 1: False Positive Drift from Auto-Scaling & AWS Tags

• Symptom: The nightly drift detection job triggers false positive alerts every single night on an EKS Node Group or Auto Scaling Group (desired_capacity changed from 3 to 7).
• Root Cause: Karpenter or Kubernetes Cluster Autoscaler dynamically adjusts instance capacity based on pod traffic. Because this is managed by an in-cluster Kubernetes controller, Terraform detects it as external out-of-band drift.
• Resolution: Declare an explicit ignore_changes lifecycle rule:

  resource "aws_autoscaling_group" "nodes" {
    # ...
    lifecycle {
      ignore_changes = [
        desired_capacity,
        tag # If enterprise tagging bots inject billing tags dynamically
      ]
    }
  }

Scenario 2: Out-of-Band Cloud Deletion & Dangling State Lockup

• Symptom: An engineer manually deletes a misbehaving AWS VPC or RDS instance in the AWS Console during an emergency. The next scheduled Terraform CI run crashes with: Error: Provider produced inconsistent result after apply: When applying changes to aws_db_instance.main, provider returned a null value when not expected.
• Under-the-Hood Root Cause: The state file records that the RDS instance exists and expects sub-resources (like aws_db_parameter_group or aws_route53_record) to depend on it. Because the primary resource was purged out-of-band, Terraform attempts an in-place update or fails to build the destroy graph for dependent child resources.
• Triage & Remediation Runbook:

  # 1. Run refresh-only to sync state with reality without executing destructive changes
  terraform apply -refresh-only -auto-approve
  
  # 2. If the resource is still ghosted in state, cleanly surgically remove it from state
  terraform state rm aws_db_instance.main
  
  # 3. Run plan to re-create the resource cleanly or reconcile code
  terraform plan

📝 Staff Architect & SRE Interview Scenarios