Phase 3 & 3B: Hands-On Job-Essential Lab Solutions

This module contains end-to-end code solutions, configuration manifests, and verification testing steps for the Phase 3 & Phase 3B Job-Essential Exercises.

---

🛠️ Lab 1: The Pre-Commit Secret Blocker (TruffleHog)

Objective

Configure a Git pre-commit hook that scans for secrets, create an intentionally committed AWS API key, and verify that the commit is blocked by TruffleHog.

Execution Steps

1. Install the pre-commit framework:

   pip install pre-commit
   

2. Create .pre-commit-config.yaml:

   repos:
     - repo: https://github.com/trufflesecurity/trufflehog
       rev: v3.63.0
       hooks:
         - id: trufflehog
           name: TruffleHog Secret Scanner
           entry: trufflehog git file://. --since-commit HEAD --only-verified --fail
           stages: [commit]
   

3. Activate the Git hook:

   pre-commit install
   

4. Simulate an accidental secret leak:

   echo "AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" > credentials.env
   git add credentials.env
   git commit -m "chore: add environment secrets"
   

   Expected Output:

   TruffleHog Secret Scanner................................................Failed
   - hook id: trufflehog
   - exit code: 183
   🐷 Found unencrypted high-entropy credential in credentials.env:
      Detector: AWS
      Commit blocked by security pre-commit hook!
   

---

🛠️ Lab 2: The Cryptographic Gatekeeper (Cosign & Kyverno)

Objective

Deploy an unsigned container image to Kubernetes and prove that Kyverno blocks it. Then, sign the image using Cosign and prove that Kyverno admits the signed pod.

1. Insecure Deployment Test (Blocked)

# unsigned-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: unsigned-test-pod
  namespace: workloads
spec:
  containers:
    - name: nginx
      image: 123456789012.dkr.ecr.us-east-1.amazonaws.com/nginx:unsigned

kubectl apply -f unsigned-pod.yaml

Expected Output (Admission Blocked):

Error from server: admission webhook "check-image-signature.kyverno.svc" denied the request: 
policy check-image-signature/verify-signature failed: image 123456789012.dkr.ecr.us-east-1.amazonaws.com/nginx:unsigned 
failed cryptographic signature verification. No valid Cosign signature found from trusted issuer.

2. Sign the Image with Cosign & Re-Deploy

# Sign the container image using keyless OIDC authentication
cosign sign --yes 123456789012.dkr.ecr.us-east-1.amazonaws.com/nginx:unsigned

# Re-apply the pod
kubectl apply -f unsigned-pod.yaml

Expected Output: pod/unsigned-test-pod created. Pod successfully admitted!

---

🛠️ Lab 3: The Packer Golden AMI Factory

Objective

Build a hardened Golden AMI using HashiCorp Packer and Ansible, verifying that unnecessary services (Apache, Telnet) are purged and CIS Level 1 kernel settings are enforced.

# golden-image.pkr.hcl
packer {
  required_plugins {
    amazon = {
      version = ">= 1.3.0"
      source  = "github.com/hashicorp/amazon"
    }
  }
}

source "amazon-ebs" "cis_hardened" {
  ami_name      = "corp-hardened-linux-${formatdate("YYYYMMDDhhmm", timestamp())}"
  instance_type = "t3.small"
  region        = "us-east-1"
  source_ami    = "ami-0c55b159cbfafe1f0" # Base Ubuntu 22.04
  ssh_username  = "ubuntu"
  encrypt_boot  = true
}

build {
  sources = ["source.amazon-ebs.cis_hardened"]

  provisioner "shell" {
    inline = [
      "echo 'Applying CIS Benchmark Hardening...'",
      # 1. Disable legacy unencrypted protocols
      "sudo apt-get purge -y rsh-client talk telnet || true",
      # 2. Restrict su command access to wheel group
      "echo 'auth required pam_wheel.so use_uid' | sudo tee -a /etc/pam.d/su",
      # 3. Disable kernel IP forwarding unless configured as router
      "echo 'net.ipv4.ip_forward = 0' | sudo tee -a /etc/sysctl.d/99-cis.conf",
      "sudo sysctl -p /etc/sysctl.d/99-cis.conf"
    ]
  }
}

packer init golden-image.pkr.hcl
packer build golden-image.pkr.hcl

---

🛠️ Lab 4: The 0-Downtime Node Rotation Challenge

Objective

Deploy a 3-replica microservice protected by a PodDisruptionBudget (minAvailable: 2). Trigger a node drain (simulating patching) while running an automated load test, and verify zero HTTP 5xx errors.

1. In Terminal A: Run continuous HTTP traffic load test

while true; do 
  curl -s -o /dev/null -w "%{http_code}\n" http://api.corp.internal/healthz; 
  sleep 0.1; 
done

2. In Terminal B: Trigger node cordon and drain

kubectl cordon ip-10-0-1-45.ec2.internal
kubectl drain ip-10-0-1-45.ec2.internal --ignore-daemonsets --delete-emptydir-data

Verification: Terminal A never outputs a single 502 Bad Gateway or connection timeout because the PDB prevents the node drain from evicting more than 1 replica at a time, and the preStop hook allows in-flight connections to drain cleanly!