Lesson 4: Zero-Downtime Secret Rotation (The 2-User Pattern), SLSA Level 3 & Exception Handling

🧠 The Concept (Explain Like I’m 5)

Imagine you are managing the combination lock on a bank vault while transactions are occurring:

• The Naive Password Change (The Crash): You walk up to the vault and immediately change the lock code from Code-1 to Code-2. For the next 15 minutes, cashiers trying to open the vault with Code-1 get locked out, alarms ring, and customers cannot withdraw money. (This is changing a production database password directly—all existing application connection pools instantly break!).
• The Two-User Alternating Pattern (The Relay Race):
  • The bank vault has two authorized keys: Key Blue and Key Green.
  • Today, all cashiers are using Key Blue. Key Green sits idle in the drawer.
  • On rotation day, the bank manager resets the lock code for Key Green and tests it.
  • The manager announces over the intercom: “Attention cashiers, beginning now, please use Key Green.”
  • Cashiers begin opening drawers with Key Green. Old cashiers finish their transactions with Key Blue.
  • Only when every cashier is verified using Key Green does the manager revoke Key Blue! Zero dropped transactions, zero customer downtime.
• SLSA (Supply-chain Levels for Software Artifacts): The government tamper-proof wax seal. It mathematically proves that a software container running in your cluster was compiled by your official GitHub Actions pipeline from an exact, verified Git commit, and was not tampered with by a rogue developer, compromised laptop, or malicious dependency.

🏢 The Enterprise Context (Why Platform Architects Care)

1. Mandatory 90-Day Rotation Compliance (PCI-DSS 4.0, SOC 2, ISO 27001): High-trust regulatory frameworks strictly mandate automated cryptographic credential rotation every 90 days. Manually coordinating database password rotations across 300 microservices would consume dozens of engineering hours every sprint. Automating it via AWS Secrets Manager and rotation Lambdas removes human toil completely.
2. Dependency Confusion & Build-Time Injection: In modern software development, attackers upload malicious packages to public repositories (npm, PyPI) with the exact same names as internal corporate libraries. If your build pipeline is not secured with SLSA Level 3 Provenance, malicious code can execute inside your CI runner and inject trojans into production binaries.
3. Stateful Workload Disruption During AMI Refresh: Rotating worker nodes that run stateful workloads (Kafka, PostgreSQL, Cassandra, Elasticsearch) is fundamentally different from stateless web apps. AWS EBS volumes cannot cross Availability Zone boundaries. Managing volume detachment latency and scheduling affinity is critical to avoiding permanent cluster partition.

🗺️ Visual Architecture: The Two-User Alternating Secret Rotation Cycle

sequenceDiagram
    autonumber
    participant SM as 🔐 AWS Secrets Manager
    participant Lambda as ⚡ Rotation Lambda Function
    participant RDS as 🗄️ AWS RDS PostgreSQL
    participant ESO as 🤖 External Secrets Operator
    participant App as 📦 Workload Application Pods

    Note over SM,RDS: Current Active State: User_Blue is AWSCURRENT. User_Green is idle.
    
    rect rgb(20, 30, 45)
    Note over SM,Lambda: 1. Step: createSecret
    SM->>Lambda: createSecret(SecretId)
    Lambda->>SM: PutSecretValue(AWSPENDING, new_password_for_green)
    end

    rect rgb(25, 40, 35)
    Note over SM,Lambda: 2. Step: setSecret
    SM->>Lambda: setSecret(SecretId)
    Lambda->>RDS: ALTER USER user_green WITH PASSWORD 'new_password_for_green';
    RDS-->>Lambda: Password updated in database engine ✅
    end

    rect rgb(35, 25, 45)
    Note over SM,Lambda: 3. Step: testSecret
    SM->>Lambda: testSecret(SecretId)
    Lambda->>RDS: Attempt login using user_green + new_password_for_green
    RDS-->>Lambda: Connection test succeeds (SELECT 1) ✅
    end

    rect rgb(45, 35, 25)
    Note over SM,Lambda: 4. Step: finishSecret
    SM->>Lambda: finishSecret(SecretId)
    SM->>SM: Shift Staging Labels: Promote user_green to AWSCURRENT.<br/>Demote user_blue to AWSPREVIOUS.
    end

    ESO->>SM: Syncs AWSCURRENT secret into Kubernetes etcd
    App->>App: Reads user_green credentials & opens new connection pool!
    Note over App,RDS: Zero dropped queries, zero downtime!

🔬 Under the Hood: AWS Secrets Manager 4-Step Rotation State Machine

AWS Secrets Manager uses Version Staging Labels to manage rotation states atomically:

• AWSCURRENT: The credentials currently active and consumed by production workloads.
• AWSPENDING: The new credentials currently being generated, configured, and tested.
• AWSPREVIOUS: The last-known working credentials retained for immediate rollback if needed.

The 4 Mandatory Lambda Steps

1. createSecret: Generates a cryptographically secure random string and creates a new version of the secret in Secrets Manager tagged with the AWSPENDING staging label.
2. setSecret: Connects to the target database engine as an administrative superuser and alters the password of the idle user to match the AWSPENDING secret.
3. testSecret: Verifies that the new credentials can successfully authenticate and execute queries against the database engine.
4. finishSecret: Moves the AWSCURRENT label from the old version to the new version. The old credentials receive the AWSPREVIOUS label.

💻 Production Implementation: Hardened Secrets Rotation & SLSA Provenance

1. Terraform Automated Rotation Configuration (rotation.tf)

resource "aws_secretsmanager_secret" "database_credentials" {
  name                    = "prod/workloads/postgres-cluster"
  description             = "Two-User alternating automated database credentials"
  recovery_window_in_days = 0 # Force immediate deletion if destroyed via IaC
  kms_key_id              = aws_kms_key.secrets_encryption_key.arn
}

resource "aws_secretsmanager_secret_rotation" "database_rotation" {
  secret_id           = aws_secretsmanager_secret.database_credentials.id
  rotation_lambda_arn = aws_lambda_function.postgres_rotator.arn

  rotation_rules {
    automatically_after_days = 30 # Rotates automatically every 30 days
  }
}

2. SLSA Level 3 Provenance Generation Workflow (.github/workflows/slsa-build.yml)

SLSA Level 3 requires non-falsifiable provenance generated on an isolated, ephemeral builder:

name: Production SLSA Level 3 Secure Build

on:
  push:
    branches: [main]

permissions:
  id-token: write
  contents: read
  actions: read

jobs:
  build:
    runs-on: ubuntu-latest
    outputs:
      hashes: ${{ steps.hash.outputs.hashes }}
    steps:
      - name: Checkout Code
        uses: actions/checkout@v4

      - name: Compile Statically Linked Go Binary
        run: |
          CGO_ENABLED=0 GOOS=linux go build -o payment-engine ./cmd/server
          sha256sum payment-engine > payment-engine.sha256

      - name: Generate Cryptographic Hash Digest
        id: hash
        run: |
          echo "hashes=$(sha256sum payment-engine | base64 -w0)" >> $GITHUB_OUTPUT

  # Generate Cryptographic SLSA Level 3 Attestation via Official Generator
  provenance:
    needs: [build]
    permissions:
      id-token: write
      contents: write
      actions: read
    uses: slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@v2.0.0
    with:
      base64-subjects: "${{ needs.build.outputs.hashes }}"
      upload-assets: true

⚠️ Real-World Failure Modes, Exceptions & Break-Glass Playbooks

Exception 1: Rotation Lambda Crash During setSecret (The Split-Brain Trap)

• The Catastrophe: During step 2 (setSecret), the Lambda successfully connects to PostgreSQL and alters user_green’s password. But before the Lambda can return HTTP 200 to Secrets Manager, the Lambda execution times out or crashes.
• The Failure:
  • The database engine has the new password.
  • AWS Secrets Manager does not know this step succeeded!
  • On retry, the Lambda attempts to test or re-set the password, but fails because its cached administrative connection credentials no longer match. The rotation halts in an error state with the secret permanently stuck in AWSPENDING.
• The Architectural Rule (Idempotency in Rotation Lambdas): Every rotation step must be strictly idempotent:
  1. The Lambda must first attempt to connect using the AWSPENDING credentials. If connection succeeds, the password was already updated in a previous interrupted attempt, and the Lambda can skip re-setting it and proceed cleanly to step 3 (testSecret).
  2. If connection fails with AWSPENDING, connect using administrative master credentials to update the password.

Exception 2: StatefulSet EBS Volume Multi-AZ Deadlock During Node Upgrades

• The Outage: An SRE initiates a rolling worker node upgrade. Node 1 in Availability Zone us-east-1a is drained and cordoned. A stateful PostgreSQL Pod terminates.
• The Hang: The PostgreSQL Pod remains stuck in Pending state indefinitely:

  0/12 nodes are available: 1 node has volume node affinity conflict, 11 nodes in different AZ.

• The Root Cause: AWS EBS volumes are physical block storage devices locked to a single Availability Zone (us-east-1a). They cannot cross AZ boundaries! If the EKS Autoscaler or Karpenter provisions a new replacement node in us-east-1b, Kubernetes cannot mount the us-east-1a EBS volume to that node!
• The Architectural Safeguards:
  1. Strict StorageClass Binding Mode: Always configure volumeBindingMode: WaitForFirstConsumer on the EBS StorageClass. This instructs Kubernetes to delay provisioning the EBS volume until the pod is scheduled to an instance, guaranteeing identical AZ placement.
  2. Volume Detachment Grace Windows: AWS EBS detachment takes 30 to 60 seconds. Set terminationGracePeriodSeconds: 60 on StatefulSets so the application flushes writes before the volume detach call is fired by the AWS CSI driver.

Exception 3: Emergency Break-Glass Credential Overwrite

• The Scenario: A rogue employee or security incident necessitates immediately locking out an active database user without waiting for the automated 30-day rotation schedule.
• The Break-Glass CLI Command:

  # Force immediate out-of-band secret rotation with no recovery delay
  aws secretsmanager rotate-secret \
    --secret-id prod/workloads/postgres-cluster \
    --rotate-immediately

📝 Staff Architect & SRE Interview Scenarios