⚡
Zero to Platform Architect2026
Enterprise Platform & SRE Playbooks
Enterprise Platform & SRE Playbooks⏱️ 9 min read⚡ Staff SRE Standard
🎯 Operations Test Bank →
Quick Jump:🧠 Concept🗺️ Architecture🔬 Internals🧭 CLI Playbook📝 Q&A Drill

Enterprise Lesson 3: Production 3-Tier Kubernetes Reference Stack Manifest

🧠 The Concept (Explain Like I’m 5)

Imagine building a high-security skyscraper for a Swiss bank:

  • The Blueprint (The YAML Manifest): You don’t just build walls and hope for the best. You need structural earthquake dampening (Topology Spread across 3 Zones), emergency exits that never close (PodDisruptionBudget), bulletproof glass at the entrance (WAF & Ingress), security guards that strip all visitor weapons (Rootless Security Context), automatic elevators that scale with crowds (HPA v2 Autoscaling), and a private pneumatic tube directly to the vault (ExternalSecrets Operator).
  • If a junior engineer deploys a standard tutorial Deployment (kubectl create deployment nginx), it is equivalent to putting cardboard walls and leaving the front door unlocked.
  • This lesson provides the definitive, copy-paste production-grade reference manifest used by Staff Platform Architects at Fortune 500 enterprises.

🗺️ Visual Architecture: End-to-End Enterprise 3-Tier Microservice Stack

flowchart TD
    Client["🌍 Internet Client (HTTPS)"] --> WAF["🛡️ AWS WAF (Web ACL)"]
    WAF --> ALB["⚖️ AWS Application Load Balancer<br/>(TLS Terminated via cert-manager Let's Encrypt)"]

    subgraph EKSCluster ["EKS Production Cluster (Multi-AZ: az-1a, az-1b, az-1c)"]
        Ingress["AWS Load Balancer Controller (Ingress)"]
        ALB -->|"Direct Pod IP Target Group"| PodA

        subgraph PodReplicaSet ["PodTopologySpread: 3 Replicas Across 3 AZs"]
            PodA["Pod 1 (AZ-1a)<br/>• Non-Root (UID 10001)<br/>• ReadOnly RootFS<br/>• preStop: sleep 15"]
            PodB["Pod 2 (AZ-1b)<br/>• Non-Root (UID 10001)<br/>• ReadOnly RootFS<br/>• preStop: sleep 15"]
            PodC["Pod 3 (AZ-1c)<br/>• Non-Root (UID 10001)<br/>• ReadOnly RootFS<br/>• preStop: sleep 15"]
        end

        subgraph ResilienceControl ["Resilience & Autoscaling Controls"]
            PDB["<b>PodDisruptionBudget (PDB)</b><br/>minAvailable: 2"]
            HPA["<b>HPA v2 Autoscaler</b><br/>CPU 70% | Mem 80% | Custom RPS > 500"]
        end

        subgraph SecuritySecrets ["Zero-Trust Security & Secrets"]
            ESO["ExternalSecrets Operator"]
            NetPol["Cilium NetworkPolicy (Default Deny)"]
        end

        subgraph Observability ["Metrics & Scraping"]
            ServiceMonitor["Prometheus ServiceMonitor"]
        end
    end

    PodA -.-> NetPol
    PodReplicaSet -.-> PDB
    HPA -.-> PodReplicaSet
    ServiceMonitor -.-> PodReplicaSet

    ESO -->|"Syncs credentials via IRSA"| ASM[("AWS Secrets Manager / Vault")]
    ESO -->|"Injects Secret"| PodReplicaSet
    PodReplicaSet -->|"Allowed Egress: Port 5432"| AuroraDB[("Aurora PostgreSQL Database")]

📄 Complete Enterprise Production Manifest: production-service-stack.yaml

Below is the unified, fully-annotated production manifest ready for deployment on Amazon EKS or upstream Kubernetes:

# ==============================================================================
# 1. NAMESPACE & METADATA
# ==============================================================================
apiVersion: v1
kind: Namespace
metadata:
  name: payment-system
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
---
# ==============================================================================
# 2. EXTERNAL SECRET INTEGRATION (AWS SECRETS MANAGER VIA IRSA)
# ==============================================================================
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets-manager
  namespace: payment-system
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: payment-service-sa
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: payment-database-credentials
  namespace: payment-system
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: payment-db-secret
    creationPolicy: Owner
  data:
    - secretKey: DB_PASSWORD
      remoteRef:
        key: production/payment/rds
        property: password
---
# ==============================================================================
# 3. SERVICE ACCOUNT & WORKLOAD IAM (IRSA)
# ==============================================================================
apiVersion: v1
kind: ServiceAccount
metadata:
  name: payment-service-sa
  namespace: payment-system
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/payment-service-workload-role
---
# ==============================================================================
# 4. HARDENED ENTERPRISE PRODUCTION DEPLOYMENT
# ==============================================================================
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-service
  namespace: payment-system
  labels:
    app.kubernetes.io/name: payment-service
    app.kubernetes.io/part-of: payment-platform
spec:
  replicas: 3
  revisionHistoryLimit: 5
  selector:
    matchLabels:
      app.kubernetes.io/name: payment-service
  template:
    metadata:
      labels:
        app.kubernetes.io/name: payment-service
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8080"
    spec:
      serviceAccountName: payment-service-sa
      terminationGracePeriodSeconds: 60
      
      # Anti-Affinity & Multi-AZ Topology Spread
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: payment-service
              
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app.kubernetes.io/name: payment-service
                topologyKey: kubernetes.io/hostname

      # Pod-Level Security Context
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
        runAsGroup: 10001
        fsGroup: 10001
        seccompProfile:
          type: RuntimeDefault

      containers:
        - name: payment-api
          image: 123456789012.dkr.ecr.us-east-1.amazonaws.com/payment-service:v2.4.1
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080
              name: http
              protocol: TCP
              
          # Container-Level Hardening
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop:
                - ALL
                
          # Strict Resource Guarantees
          resources:
            requests:
              cpu: "500m"
              memory: "512Mi"
            limits:
              cpu: "1000m"
              memory: "1024Mi"
              
          # Graceful Connection Draining
          lifecycle:
            preStop:
              exec:
                command: ["/bin/sh", "-c", "sleep 15"]
                
          # Health & Startup Probes
          startupProbe:
            httpGet:
              path: /actuator/health/liveness
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 5
            failureThreshold: 12 # Allows up to 60s for slow JVM bootstrap
            
          livenessProbe:
            httpGet:
              path: /actuator/health/liveness
              port: 8080
            periodSeconds: 10
            timeoutSeconds: 2
            failureThreshold: 3
            
          readinessProbe:
            httpGet:
              path: /actuator/health/readiness
              port: 8080
            periodSeconds: 5
            timeoutSeconds: 2
            failureThreshold: 2

          env:
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: payment-db-secret
                  key: DB_PASSWORD
            - name: TMPDIR
              value: /tmp

          volumeMounts:
            - name: ephemeral-tmp
              mountPath: /tmp
              
      volumes:
        - name: ephemeral-tmp
          emptyDir:
            medium: Memory
            sizeLimit: 64Mi
---
# ==============================================================================
# 5. POD DISRUPTION BUDGET (PDB)
# ==============================================================================
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: payment-service-pdb
  namespace: payment-system
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: payment-service
---
# ==============================================================================
# 6. HORIZONTAL POD AUTOSCALER (HPA v2)
# ==============================================================================
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: payment-service-hpa
  namespace: payment-system
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: payment-service
  minReplicas: 3
  maxReplicas: 24
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 80
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Percent
          value: 100
          periodSeconds: 15
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Percent
          value: 10
          periodSeconds: 60
---
# ==============================================================================
# 7. SERVICE & INGRESS (AWS LOAD BALANCER CONTROLLER)
# ==============================================================================
apiVersion: v1
kind: Service
metadata:
  name: payment-service
  namespace: payment-system
  labels:
    app.kubernetes.io/name: payment-service
spec:
  type: ClusterIP
  ports:
    - port: 80
      targetPort: 8080
      protocol: TCP
      name: http
  selector:
    app.kubernetes.io/name: payment-service
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: payment-service-ingress
  namespace: payment-system
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
    alb.ingress.kubernetes.io/ssl-redirect: '443'
    alb.ingress.kubernetes.io/group.name: "shared-enterprise-alb"
    alb.ingress.kubernetes.io/wafv2-acl-arn: "arn:aws:wafv2:us-east-1:123456789012:regional/webacl/prod-waf/a1b2c3d4"
    alb.ingress.kubernetes.io/healthcheck-path: "/actuator/health/readiness"
spec:
  rules:
    - host: payments.company.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: payment-service
                port:
                  number: 80
---
# ==============================================================================
# 8. CILIUM ZERO-TRUST NETWORK POLICY
# ==============================================================================
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: payment-service-l7-policy
  namespace: payment-system
spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/name: payment-service
  ingress:
    # Allow incoming traffic from ALB Target Group
    - fromEntities:
        - world
      toPorts:
        - ports:
            - port: "8080"
              protocol: TCP
  egress:
    # Allow CoreDNS lookups
    - toEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: kube-system
            k8s-app: kube-dns
      toPorts:
        - ports:
            - port: "53"
              protocol: ANY
          rules:
            dns:
              - matchPattern: "*"
    # Allow DB traffic to Aurora PostgreSQL
    - toCIDR:
        - 10.0.100.0/24 # Aurora Subnet CIDR
      toPorts:
        - ports:
            - port: "5432"
              protocol: TCP
---
# ==============================================================================
# 9. PROMETHEUS SERVICEMONITOR
# ==============================================================================
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: payment-service-monitor
  namespace: payment-system
  labels:
    release: prometheus
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: payment-service
  endpoints:
    - port: http
      path: /actuator/prometheus
      interval: 15s
      scrapeTimeout: 10s

🔬 Under the Hood: Deep Explanations of Critical Directives

1. Graceful Connection Draining (preStop: sleep 15)

When a pod is terminating (during autoscaling scale-down or node draining), Kubernetes executes two tasks in parallel:

  1. Removes the Pod IP from the Service endpoints and ALB Target Group.
  2. Sends the SIGTERM signal to the container process.

Because propagating endpoint removal to the AWS ALB takes 5 to 10 seconds, without a preStop hook, the application receives SIGTERM and shuts down while the ALB is still forwarding new HTTP requests to it! This triggers immediate 502 Bad Gateway errors for real users.
The preStop: sleep 15 ensures the container stays alive and processes existing in-flight connections while the ALB cleanly completes deregistration.

2. ReadOnly Root Filesystem & In-Memory Scratch Space

Setting readOnlyRootFilesystem: true blocks attackers from writing rootkits, executing cryptocurrency miners, or modifying binaries even if they achieve remote code execution (RCE).
However, runtimes like Java, Node.js, and Python require a temporary directory for locks and caching (/tmp). Mounting an emptyDir backed by RAM (medium: Memory) provides a safe, ephemeral scratch space that vanishes on pod termination.

⚠️ Real-World Failure Modes & Exception Playbooks

Failure Mode 1: The OOMKilled CrashLoopBackOff on Java Heap

  • The Disaster: A Spring Boot pod with limits.memory: 1024Mi crashes with ExitCode: 137 (OOMKilled) every 20 minutes under high traffic.
  • The Root Cause: The JVM allocated -Xmx768m, but JVM metaspace, thread stacks (1MB per thread), and garbage collection overhead exceeded the Linux cgroup limit of 1024Mi.
  • The SRE Playbook:
    1. Add JVM container support flags: -XX:+UseContainerSupport -XX:MaxRAMPercentage=70.0.
    2. Increase the margin between memory requests and limits, or set limits.memory to 1.5× the heap allocation.

Failure Mode 2: Ingress 504 Gateway Timeout During High Node Churn

  • The Disaster: During a Karpenter spot consolidation event, API clients experience random 504 Gateway Timeouts.
  • The Root Cause: AWS ALB Target Group deregistration delay was set to the default 300 seconds, while Kubernetes terminationGracePeriodSeconds was only 30 seconds. The pod died 270 seconds before the load balancer stopped routing traffic to it!
  • The SRE Playbook:
    1. Synchronize ALB deregistration delay with pod grace periods via Ingress annotation: alb.ingress.kubernetes.io/target-group-attributes: deregistration_delay.timeout_seconds=30
    2. Ensure terminationGracePeriodSeconds: 60 and preStop: sleep 15.

🎤 Staff/Principal Architect Interview Questions

Q1: Why do we use topologySpreadConstraints with whenUnsatisfiable: DoNotSchedule instead of simple podAntiAffinity?

Model Answer:
podAntiAffinity is binary: it prevents scheduling multiple pods on the same host or zone, but it does not balance replicas evenly across 3 Availability Zones. If you have 6 replicas and 3 AZs, anti-affinity might place 4 replicas in AZ-A and 1 each in AZ-B and AZ-C without violating basic affinity.
topologySpreadConstraints with maxSkew: 1 and whenUnsatisfiable: DoNotSchedule mathematically enforces that the difference in replica count between any two zones never exceeds 1. If AZ-A has 2 pods, the scheduler strictly forbids scheduling a 3rd pod in AZ-A until AZ-B and AZ-C also have 2 pods, ensuring perfect high-availability equilibrium.

Q2: What happens if an external secret rotation fails while a deployment is scaling up?

Model Answer:
ExternalSecrets Operator synchronizes secrets into native Kubernetes Secret objects asynchronously. If AWS Secrets Manager is unavailable or IRSA credentials fail, ESO retains the last known valid Secret in the cluster cache.
Newly scaled pods will continue to boot successfully using the existing cached secret. Furthermore, ESO exposes Prometheus metrics (externalsecret_sync_error{name="payment-db-secret"}). An alert fires to on-call SREs before applications suffer any outage.