🧠 Operations & DevEx Architect Assessment: Exception & Failure Mode Matrix
Welcome to the Operations & DevEx Platform Architect Diagnostic Matrix. This assessment is specifically calibrated to evaluate your mastery of exceptions, failure modes, edge cases, and break-glass procedures across production Kubernetes, CI/CD supply chains, GitOps, eBPF, and Internal Developer Portals.
In Staff & Principal SRE interviews, you are tested on how systems break and how you engineer resilience against catastrophic edge cases.
🎯 How It Works
- Review the Left Column (Real-World Outage Symptoms, Exceptions, or Directives).
- Attempt to mentally map each block to its corresponding partner in the Right Column (Root Causes, Architectural Failure Modes, or Break-Glass Remediations).
- Expand the Answer & Architectural Rationale accordion to verify your understanding.
🟢 Tier 1: Low Difficulty (Operational Baseline)
Validates basic understanding of container digests, GitHub permissions, Kubernetes secret encoding, and health check types.
📌 Left Blocks: Primitives & Directives
A
Missing `id-token: write` in GitHub Actions
B
Deploying containers with `:latest` tag
C
Standard Kubernetes `kind: Secret`
D
ASG `health_check_type = "EC2"`
E
Missing `catalog-info.yaml` at repo root
🎯 Right Blocks: Behaviors & Mechanisms
1
Merely Base64-encoded strings (`base64 -d`), requiring etcd KMS envelope encryption.
2
Evaluates only VM hypervisor health; continues routing traffic to crashed web applications.
3
Runner agent refuses to mint OIDC JWT, failing cloud authentication with HTTP 403.
4
Missing metadata descriptor causing repository to vanish from Backstage catalog.
5
Mutable pointer allowing unauthorized image overwrites; violates immutable digest standard.
💡 Reveal Tier 1 Solution Key & Architectural Rationale Click to Expand
Verified Architectural Pairings:
A → 3 B → 5 C → 1 D → 2 E → 4
B → 5 (Tag Immutability): Tags are mutable pointers in registries. Node A and Node B can pull different code under the same tag. Immutable production deployments strictly require `@sha256:`.
C → 1 (Kubernetes Secrets): Kubernetes secrets are stored as plaintext Base64 in `etcd` by default. True encryption requires enabling AWS KMS envelope encryption at the control plane and using External Secrets Operator.
D → 2 (`EC2` vs `ELB` Health Checks): `EC2` checks only evaluate hypervisor ping. If NGINX or Python crashes, the instance stays marked healthy. Production web workloads must use `ELB` Target Group health checks.
E → 4 (`catalog-info.yaml`): The core entity descriptor parsed by the Backstage software catalog. Without it, the component exists as an unmanaged Git silo.
🟡 Tier 2: Medium Difficulty (Core Systems & Networking)
Evaluates Ingress routing modes, database secret rotation staging, compliance types, and sync sequencing.
📌 Left Blocks: Primitives & Directives
A
Ingress `target-type: ip`
B
AWS Secrets Manager `AWSPREVIOUS`
C
Detective Controls (AWS Config)
D
ArgoCD Sync Waves (`sync-wave: "1"`)
E
Kubernetes Gateway API Personas
🎯 Right Blocks: Behaviors & Mechanisms
1
Enforces deterministic phased rollouts (e.g. CRDs must report healthy before pods deploy).
2
Separates infrastructure management (`Gateway`) from developer routing (`HTTPRoute`).
3
Bypasses `iptables` and NodePort hops, routing packets directly to Pod IPs via VPC CNI.
4
Preserves immediate predecessor credentials for zero-downtime rollback if new passwords fail.
5
Continuously evaluates live cloud state for out-of-band console drift and triggers SSM self-healing.
💡 Reveal Tier 2 Solution Key & Architectural Rationale Click to Expand
Verified Architectural Pairings:
A → 3 B → 4 C → 5 D → 1 E → 2
B → 4 (`AWSPREVIOUS`): Ensures that if newly rotated credentials fail under concurrent load, SREs can execute an instant emergency rollback without restoring backups.
C → 5 (AWS Config): Unlike proactive CI/CD gates, detective controls continuously audit running infrastructure and execute automated remediation runbooks.
D → 1 (Sync Waves): Without sync waves, ArgoCD applies manifests in parallel, causing microservices to crash before operators or Custom Resource Definitions are established.
E → 2 (Gateway API): Solves the monolithic Ingress problem by separating Gateway infrastructure (managed by Ops) from HTTPRoute configurations (managed by developers).
🔴 Tier 3: Hard Difficulty (SRE & Infrastructure Exceptions)
Tests ASG capacity deadlocks, CoreDNS query amplification, GitOps fleet deletion traps, and eBPF DNS blackouts.
📌 Left Blocks: Primitives & Directives
A
ASG `min_healthy: 100%` + `max_size == desired`
B
CoreDNS `ndots: 5` Amplification
C
ApplicationSet without `preserveResourcesOnDeletion`
D
Cilium Default-Deny Egress Policy
E
Kyverno `failurePolicy: Fail` Webhook Timeout
🎯 Right Blocks: Behaviors & Mechanisms
1
Evaluates missing generator matches and permanently deletes 50 production applications.
2
Blocks UDP port 53 to CoreDNS, preventing all internal and external name resolution.
3
Surge instance cannot launch, deadlocking instance refresh because terminating any node violates 100%.
4
Webhook pod outage causes API server to reject all incoming Pod admissions across the cluster.
5
glibc evaluates 4 internal search paths for external domains, multiplying DNS queries by 500%.
💡 Reveal Tier 3 Solution Key & Architectural Rationale Click to Expand
Verified Architectural Pairings:
A → 3 B → 5 C → 1 D → 2 E → 4
A → 3 (ASG 100% Trap): If capacity is capped at 3, the ASG cannot scale to 4 before terminating an old instance. But terminating one drops capacity to 66%, violating `min_healthy = 100`. The refresh hangs indefinitely.
C → 1 (`preserveResourcesOnDeletion`): If an engineer renames a folder or label in a generator, ArgoCD considers the old applications unreferenced and deletes the live workloads unless resource preservation is true.
D → 2 (Cilium DNS Blackout): Blocking all egress without explicitly whitelisting UDP/TCP port 53 to `kube-system` cuts off DNS resolution, causing all application connection pools to crash.
E → 4 (Admission Webhook Deadlock): If an admission controller pod crashes and `failurePolicy: Fail` is set globally, the API Server rejects all new pods, preventing the admission controller from ever rescheduling.
🟣 Tier 4: Complex Difficulty (Staff Architect Edge Cases)
Explores irreversible database migrations during canary aborts, PDB node drain hangs, OIDC wildcards, and WireGuard MTU drops.
📌 Left Blocks: Primitives & Directives
A
Canary Abort with Breaking Database Schema
B
`kubectl drain` on 2-replica pod with `minAvailable: 2`
C
OIDC Trust Condition `repo:my-org/*:*`
D
WireGuard ClusterMesh MTU Fragmentation
E
Backstage Scaffolder Multi-Step Failure
🎯 Right Blocks: Behaviors & Mechanisms
1
Allows any repository in the GitHub organization to assume production cloud admin credentials.
2
60-byte tunnel overhead exceeds 1500-byte cloud MTU, silently dropping SSL handshakes.
3
Rollback restores old code that crashes because the canary's destructive SQL migration already executed.
4
Compensating rollback actions execute in reverse order (Saga pattern) to clean partial state.
5
Eviction API returns HTTP 429 Too Many Requests, causing node drain and OS patching to hang forever.
💡 Reveal Tier 4 Solution Key & Architectural Rationale Click to Expand
Verified Architectural Pairings:
A → 3 B → 5 C → 1 D → 2 E → 4
A → 3 (Canary Schema Trap): If a canary runs `DROP COLUMN`, and Argo Rollouts aborts back to stable v1.0, the stable pods crash because their code expects the dropped column. Requires **Expand-and-Contract (Two-Phase)** migrations.
B → 5 (PDB Eviction Deadlock): Evicting 1 pod drops healthy replicas to 1, violating `minAvailable: 2`. The Eviction API rejects the request, putting `kubectl drain` into an infinite hang. Requires `maxUnavailable: 1` or break-glass `--disable-eviction`.
C → 1 (OIDC Wildcard Escalation): Bounding trust to `*` allows any developer who creates a scratch repo to push a workflow that assumes production IAM credentials. Must scope to exact repos and branches.
D → 2 (MTU Fragmentation Blackhole): WireGuard adds 60 bytes. Packets with DF bit set exceeding 1500 bytes are dropped by cloud firewalls with no ICMP notification. Requires tuning Cilium MTU to 1420.
E → 4 (Scaffolder Saga Rollbacks): When multi-step provisioning fails halfway through (e.g. repo created but ArgoCD registration failed), custom compensating actions must delete the orphaned repository to maintain transactional hygiene.
🟤 Tier 5: Extreme Platform & State Exceptions (Principal / Fellow Edge Cases)
Tests EBS CSI single-AZ scheduling deadlocks, Chaos Mesh daemon zombie rules, OpenCost unallocated idle cost attribution, Crossplane KCL composition validation vs YAML bloat, and Megaport MCR asymmetric routing flap.
📌 Left Blocks: Primitives & Directives
A
EBS CSI `volumeBindingMode: Immediate` in Multi-AZ Cluster
B
Chaos Mesh `NetworkChaos` duration finishes but packet loss persists
C
OpenCost / Kubecost Reports 45% "Unallocated Idle" Spend
D
Megaport MCR Direct Peering between AWS DX and Azure ExpressRoute
E
Crossplane Patch-and-Transform YAML with Typo in Field Path
🎯 Right Blocks: Behaviors & Mechanisms
1
BGP Local Preference mismatch across clouds causes asymmetric routing and stateful TCP firewall resets.
2
StorageClass provisions volume in AZ-a; pod scheduled in AZ-b cannot attach disk (`VolumeZoneConflict`). Fix: `WaitForFirstConsumer`.
3
`chaos-daemon` crashed before tearing down `iptables` / `tc` netem rules; requires manual cgroup netem flush.
4
Cluster headroom over-provisioned or Karpenter `ALLOCATE_IDLE=true` not distributed across namespace tenants.
5
Fails silently at reconciliation time with partial resource orphans; resolved by strongly typed KCL Pipeline Functions.
💡 Reveal Tier 5 Solution Key & Architectural Rationale Click to Expand
Verified Architectural Pairings:
A → 2 B → 3 C → 4 D → 1 E → 5
A → 2 (EBS AZ Deadlock): With `Immediate`, the PV volume is dynamically created in an arbitrary AZ as soon as the PVC is created. If Kubernetes scheduler subsequently places the Pod in a different AZ, the EBS volume cannot attach across AZ boundaries. `WaitForFirstConsumer` delays volume creation until the Pod is placed in an AZ.
B → 3 (Chaos Mesh Zombie Rules): If a `chaos-daemon` Pod crashes or is evicted by the node OOM killer while a chaos experiment is running, the Linux kernel `tc` (traffic control) queuing discipline or `iptables` rules remain active in the target container cgroup. Triage requires inspecting the node network namespace (`tc qdisc show dev eth0`) or restarting the daemon.
C → 4 (Unallocated Idle Cost Attribution): In Karpenter/EKS clusters, compute nodes often have spare buffer capacity for burst scaling. If OpenCost is configured without `ALLOCATE_IDLE=true`, this cost remains unallocated in executive FinOps dashboards. Enabling weighted idle allocation distributes buffer costs proportionately across tenant workloads.
D → 1 (Multi-Cloud BGP Asymmetric Routing): When interconnecting AWS Direct Connect and Azure ExpressRoute via Megaport Cloud Router (MCR), unequal BGP AS-path prepending or BGP MED causes packets from AWS to Azure to traverse MCR, while return traffic routes across public Internet or alternate links, breaking stateful firewall tables.
E → 5 (Crossplane KCL vs Patch-and-Transform): Complex YAML patch-and-transform compositions lack build-time type checking. A typo in `transforms[].string.fmt` causes runtime reconciliation failure without clear errors. Crossplane Pipeline Composition Functions with KCL compile and validate types upfront, guaranteeing valid resource emission.
📊 Self-Evaluation Scoring Rubric
23 – 25 / 25 Correct
Distinguished / Fellow Architect
Total mastery of stateful storage deadlocks, chaos engineering traps, BGP interconnects, and FinOps models.
19 – 22 / 25 Correct
Staff SRE / Principal Architect
Comprehensive mastery of failure modes, edge cases, and break-glass production triage.
15 – 18 / 25 Correct
Senior Platform Engineer
Strong operational foundation; review Tier 4 & Tier 5 failure modes (canary schema traps, CSI zone locks).
11 – 14 / 25 Correct
Mid-Level Cloud Engineer
Solid conceptual basics; deepen understanding of kernel DNS amplification and OIDC trust policies.
< 11 / 25 Correct
Developing SRE
Re-read the masterclass lessons in Phase 3 (DevSecOps), Phase 4 (EKS & CSI), and Phase 5 (GitOps).
🧠 Complete Operations, K8s, DevEx & LLMOps One-Line Memory Bank (Lesson-by-Lesson)
Use these rapid-fire one-line questions and answers to lock every single lesson of Phases 3, 3B, 4, 5, 6, 7, and 8 into long-term memory.
🛡️ Phase 3: DevSecOps & Supply Chain 32 Rapid Q&As
Lesson 1: Image Security & Cosign
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | Why is deploying container tags like nginx:latest or app:1.2 dangerous in production? |
Image tags are mutable pointers that can be overwritten; production deployments must pin immutable @sha256:<digest> hashes. |
| 2 | What Trivy flag breaks a CI/CD pipeline only when a CRITICAL vulnerability actually has a published vendor patch? |
trivy image --exit-code 1 --severity CRITICAL --ignore-unfixed <image>. |
| 3 | What are the two standard formats for a Software Bill of Materials (SBOM)? | CycloneDX (cyclonedx-json) and SPDX (spdx-json). |
| 4 | How does Sigstore Cosign Keyless Signing sign images without storing long-lived private keys? | It exchanges the CI runner’s short-lived OIDC token with Fulcio (CA) for a 10-minute certificate and records the signature proof in Rekor (transparency log). |
| 5 | Why does Cosign verification fail in air-gapped Kubernetes clusters if Rekor transparency log verification is left at default? | Kyverno/Cosign tries to reach the public internet (rekor.sigstore.dev) and times out; air-gapped clusters require offline TUF roots or KMS-backed signing keys. |
| 6 | What Kyverno rule type verifies Cosign image signatures and mutates tags into immutable @sha256 digests at admission time? |
The verifyImages block inside a ClusterPolicy. |
| 7 | Where does Cosign store image signatures and SBOM attestations? | Directly inside the OCI container registry (ECR/GHCR) as attached .sig and .att artifact layers alongside the container image. |
| 8 | What command attaches a CycloneDX SBOM as a cryptographically signed in-toto attestation to an OCI image? | cosign attest --predicate sbom.json --type cyclonedx <image@sha256:digest>. |
Lesson 2: GitHub Actions & AWS OIDC
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | What mandatory permission must be set in a GitHub Actions workflow YAML to request an OIDC JWT token? | permissions: id-token: write. |
| 2 | What URL is configured as the OIDC Identity Provider URL in AWS IAM for GitHub Actions? | https://token.actions.githubusercontent.com with Audience sts.amazonaws.com. |
| 3 | What AWS STS API action does aws-actions/configure-aws-credentials call under the hood when using OIDC? |
sts:AssumeRoleWithWebIdentity. |
| 4 | Why is setting the IAM Trust Policy condition to "token.actions.githubusercontent.com:sub": "repo:my-org/*" a critical security vulnerability? |
The wildcard * allows any feature branch or pull request in any repository across the entire GitHub organization to assume your production IAM Role. |
| 5 | How do you lock an AWS IAM Role Trust Policy so only the main branch of payment-service can assume it? |
Use StringEquals on "token.actions.githubusercontent.com:sub": "repo:my-org/payment-service:ref:refs/heads/main". |
| 6 | Why must "token.actions.githubusercontent.com:aud": "sts.amazonaws.com" always be verified in the IAM Trust Policy? |
It prevents confused-deputy token replay attacks where a JWT minted for another audience is replayed against AWS STS. |
| 7 | What is the default validity duration of temporary AWS credentials returned by AssumeRoleWithWebIdentity in CI/CD? |
1 hour (3600 seconds), configurable down to 15 minutes. |
| 8 | Why must third-party GitHub Actions in enterprise workflows be pinned to a 40-character Git commit SHA (uses: actions/checkout@b4ffde6...) instead of @v4? |
Version tags (@v4) are mutable and can be hijacked if an upstream maintainer’s account is compromised (Supply Chain Tag Poisoning). |
Lesson 3: Secrets Scanning & ESO
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | Why does deleting a leaked secret in a new Git commit fail to secure the repository? | The secret remains permanently readable inside the .git/objects history of earlier commits and can be extracted with git log -p. |
| 2 | What tool scans Git commit history and pre-commit staged changes using regex and Shannon entropy analysis to block leaked keys? | Gitleaks (gitleaks protect --staged -v). |
| 3 | Are standard Kubernetes Secret objects encrypted by default in etcd? |
No, they are merely Base64-encoded strings unless KMS envelope encryption at rest is explicitly enabled on the EKS/K8s control plane. |
| 4 | What Kubernetes operator synchronizes external cloud secrets (AWS Secrets Manager / Vault) into native Kubernetes Secret objects? |
External Secrets Operator (ESO). |
| 5 | What is the difference between a SecretStore and a ClusterSecretStore in ESO? |
SecretStore is namespace-scoped, while ClusterSecretStore is cluster-wide and can project secrets into multiple namespaces with RBAC conditions. |
| 6 | Why did setting refreshInterval: 1s across 500 ExternalSecret objects cause an AWS outage and surprise $4,000 bill? |
It generated 500 GetSecretValue API calls per second, hitting AWS Secrets Manager account rate limits (ThrottlingException) and API call billing. |
| 7 | What is the recommended production refreshInterval for ESO ExternalSecret resources? |
1h (or event-driven refresh), combined with Reloader (reloader.stakater.com/auto: "true") to roll pods only when secret SHA hashes actually change. |
| 8 | How does ESO authenticate to AWS Secrets Manager without storing static AWS keys in the cluster? | Via EKS IRSA (IAM Roles for Service Accounts) or EKS Pod Identity attached to the ESO controller ServiceAccount. |
Lesson 4: 2-User Secret Rotation & SLSA
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | Why does single-user database password rotation cause application downtime? | Between the millisecond the password changes in RDS and the minute Kubernetes pods reload the secret, active pods fail authentication (FATAL: password authentication failed). |
| 2 | How does the Alternating 2-User (Dual-Account) Rotation Strategy achieve zero-downtime secret rotation? | User A remains active serving live traffic while User B’s password is rotated in the background; once updated, new pods switch to User B while User A stays valid during drain. |
| 3 | What are the 4 sequential steps executed by an AWS Secrets Manager rotation Lambda function? | createSecret → setSecret → testSecret → finishSecret. |
| 4 | What are the two staging labels used by AWS Secrets Manager during rotation? | AWSCURRENT (active credentials), AWSPENDING (new secret being tested), and AWSPREVIOUS (last known good secret for instant rollback). |
| 5 | What does SLSA (Supply-chain Levels for Software Artifacts) protect against? | Tampering of source code, build scripts, dependencies, or artifacts at any stage between a developer’s commit and production deployment. |
| 6 | What is the defining requirement of SLSA Level 3 provenance? | Builds must run on hardened, isolated, ephemeral build environments that generate unforgeable cryptographic provenance attestations signed by the control plane. |
| 7 | What file format does SLSA use to describe build inputs, builder identity, and output artifact digests? | in-toto Attestation Framework (https://slsa.dev/provenance/v1 JSON predicate). |
| 8 | How do you verify in Kubernetes that an image was built exclusively by your official GitHub Actions workflow and not on a developer’s laptop? | Configure Kyverno verifyImages to validate the SLSA provenance attestation issuer (token.actions.githubusercontent.com) and workflow repository URI. |
🛡️ Phase 3B: Compliance & Patching 29 Rapid Q&As
Lesson 1: Immutable Golden AMIs
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | Why is mutable infrastructure (running apt upgrade or Ansible via SSH on live production servers) an anti-pattern? |
It causes Configuration Drift (“snowflake servers”), slow auto-scaling boot times, and irreproducible outages when package mirrors fail mid-scale. |
| 2 | What is the core workflow of Immutable Infrastructure using HashiCorp Packer? | Packer boots a temporary builder VM, applies OS hardening and CIS benchmarks, strips SSH keys, snapshots a Golden AMI, and terminates the builder VM. |
| 3 | Why must /etc/machine-id be truncated (truncate -s 0 /etc/machine-id) before Packer snapshots a Golden AMI? |
If cloned nodes share the same /etc/machine-id, DHCP servers assign duplicate IPs, systemd journald merges logs, and Kubernetes node registration collides. |
| 4 | What is CIS (Center for Internet Security) Level 1 vs Level 2 OS hardening? | Level 1 applies safe production security defaults with zero performance impact; Level 2 applies extreme defense-in-depth (e.g., strict SELinux/FIPS) for regulated workloads. |
| 5 | Which kernel sysctl parameters must be baked into a Kubernetes worker Golden AMI for CNI networking to function? | net.bridge.bridge-nf-call-iptables = 1 and net.ipv4.ip_forward = 1. |
| 6 | What tool runs automated compliance tests (verifying file permissions, disabled kernel modules, and sysctls) inside Packer before creating the AMI? | OpenSCAP, Chef InSpec, or Goss. |
| 7 | Why do we disable swap (swapoff -a and remove swap from /etc/fstab) on Kubernetes worker Golden AMIs? |
Kubernetes QoS memory guarantees and deterministic OOM eviction require predictable RAM limits without kernel disk swapping latency. |
| 8 | How do you distribute a newly baked Golden AMI across 50 AWS accounts in an AWS Organization without copying snapshots? | Share the AMI via AWS RAM / Organization AMI sharing and grant the Organization principal kms:Decrypt permission on the AMI’s EBS Customer Managed KMS Key. |
Lesson 2: Automated Fleet Patching
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | How do you patch 500 immutable EC2 instances when a critical Linux kernel CVE is announced? | Bake a new Golden AMI, update the AWS Launch Template $Latest version via Terraform, and trigger an Auto Scaling Group (ASG) Instance Refresh. |
| 2 | What does MinHealthyPercentage = 90 and MaxHealthyPercentage = 110 mean in an ASG Instance Refresh? |
ASG surges capacity up to 110% by launching new patched instances before terminating up to 10% of old instances at a time, preserving 100% capacity. |
| 3 | Why is setting health_check_type = "EC2" on a production Auto Scaling Group a critical reliability trap? |
EC2 health checks only verify hypervisor power state; ELB health checks must be used so ASG waits until the application passes HTTP /healthz before terminating the next batch. |
| 4 | What mechanism pauses an ASG Instance Refresh automatically if a newly patched AMI starts throwing HTTP 5xx errors? | ASG CloudWatch Alarm Auto-Rollback—triggers an immediate rollback to the previous Launch Template version when the error alarm breaches. |
| 5 | What happens when an ASG terminates a Kubernetes worker node without an EC2 Lifecycle Hook? | The instance is killed abruptly, severing active user connections and leaving stateful volumes uncleanly attached. |
| 6 | How does an ASG Terminate Lifecycle Hook achieve zero-downtime Kubernetes node replacement? | It pauses instance termination in Terminating:Wait, sends an EventBridge notification to AWS Node Termination Handler / Karpenter, which runs kubectl cordon and kubectl drain before completing the hook. |
| 7 | How does Karpenter Drift & Expiration (ttlSecondsAfterEmpty / expireAfter) automate node patching without ASGs? |
When the underlying EKS AMI SSM Parameter updates, Karpenter marks existing nodes as Drifted and automatically cordons, drains, and replaces them respecting Disruption Budgets. |
| 8 | What is the difference between Canary Baking and Blue/Green Cluster Upgrades for fleet patching? | Canary Baking replaces 5% of nodes and monitors metrics for 1 hour before rolling out to 95%, whereas Blue/Green provisions an entirely separate node group and shifts pods across. |
Lesson 3: Continuous Compliance & PDBs
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | What is the difference between Preventive Controls and Detective Controls in cloud compliance? | Preventive blocks violations before creation (SCPs, OPA/Kyverno in CI); Detective audits live running state to catch console drift or newly disclosed CVEs (AWS Config, Falco). |
| 2 | How does AWS Config Auto-Remediation fix an S3 bucket that an admin accidentally made public in the console? | AWS Config detects s3-bucket-public-read-prohibited is NON_COMPLIANT and invokes an AWS Systems Manager (SSM) Automation Document (AWS-DisableS3BucketPublicReadWrite) within seconds. |
| 3 | What Kubernetes resource prevents automated node patching (kubectl drain) from terminating all replicas of a microservice simultaneously? |
PodDisruptionBudget (PDB). |
| 4 | What is the difference between minAvailable and maxUnavailable in a PodDisruptionBudget? |
minAvailable enforces the minimum number/percentage of pods that must remain running; maxUnavailable enforces the maximum number of pods that can be offline at once. |
| 5 | Why does a Deployment with replicas: 2 and a PDB with minAvailable: 2 (or maxUnavailable: 0) cause kubectl drain to hang forever? |
The Eviction API refuses to evict any pod because dropping from 2 to 1 violates minAvailable: 2, creating an Infinite Node Drain Deadlock. |
| 6 | Why is setting maxUnavailable: 1 (or 25%) generally safer than minAvailable: 100% for production deployments? |
maxUnavailable: 1 always permits at least one pod at a time to be gracefully evicted during node drains regardless of how low HPA scales the replica count (provided replicas >= 2). |
| 7 | Does a PodDisruptionBudget prevent pods from going down when a physical EC2 instance suffers an involuntary hardware crash? | No, PDBs only govern voluntary disruptions initiated via the Kubernetes Eviction API (kubectl drain, Karpenter consolidation, Cluster Autoscaler scale-down). |
| 8 | How do you break a stuck kubectl drain caused by a misconfigured PDB during an emergency security patch? |
Patch or delete the offending PDB (kubectl patch pdb <name> -p '{"spec":{"maxUnavailable":1}}'), or use kubectl drain --disable-eviction in a break-glass emergency. |
Hands-On Labs: Phase 3 & 3B Exercises
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | What command validates a Packer HCL template’s syntax and variable definitions before running a cloud build? | packer validate . (after packer init .). |
| 2 | How do you check which PodDisruptionBudgets in a cluster currently have ALLOWED DISRUPTIONS = 0 and will block node drains? |
Run kubectl get pdb -A and look for any PDB where ALLOWED DISRUPTIONS is 0. |
| 3 | How do you verify a Cosign keyless signature on a container image using the command line? | cosign verify --certificate-identity <workflow-uri> --certificate-oidc-issuer https://token.actions.githubusercontent.com <image@sha256:digest>. |
| 4 | How do you force External Secrets Operator to immediately re-sync an ExternalSecret from AWS Secrets Manager without waiting for refreshInterval? |
Annotate the resource with a current timestamp: kubectl annotate es <name> force-sync=$(date +%s) --overwrite. |
| 5 | How do you monitor the live percentage progress and status of an AWS Auto Scaling Group Instance Refresh? | Run aws autoscaling describe-instance-refreshes --auto-scaling-group-name <asg-name>. |
☸️ Phase 4: Kubernetes & Scaling 55 Rapid Q&As
Lesson 1: EKS Architecture & IRSA
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | Which control plane components are managed by AWS in an Amazon EKS cluster? | Multi-AZ kube-apiserver, etcd quorum, kube-scheduler, kube-controller-manager, and cloud-controller-manager. |
| 2 | How does the AWS VPC CNI (aws-node DaemonSet) differ from overlay CNIs like Flannel or Calico VXLAN? |
It allocates real first-class AWS VPC IPv4 addresses from the subnet directly to pods via secondary IPs on Elastic Network Interfaces (ENIs), eliminating VXLAN encapsulation overhead. |
| 3 | Why does an m5.large EC2 instance stop scheduling new pods at 29 pods even when CPU and RAM are 70% idle? |
Each m5.large supports a maximum of 3 ENIs × 10 IPs per ENI = 30 IPs (minus 1 for the node primary IP = 29 pods max). |
| 4 | How does AWS VPC CNI Prefix Delegation (ENABLE_PREFIX_DELEGATION = "true") solve pod density limits on EKS? |
Instead of assigning single /32 secondary IPs to ENIs slots, AWS assigns /28 IPv4 prefixes (16 IPs per slot), increasing an m5.large node’s capacity from 29 to 110 pods. |
| 5 | Why must production EKS worker nodes block access to EC2 Instance Metadata Service v1 (IMDSv1) and enforce IMDSv2 with http_put_response_hop_limit = 1? |
A hop limit of 1 ensures IMDS packets cannot be forwarded across the container’s veth router hop, preventing SSRF attacks in pods from stealing the Worker Node’s IAM Role credentials. |
| 6 | How does IRSA (IAM Roles for Service Accounts) inject AWS credentials into a specific pod? | The EKS Pod Identity Webhook mutates the pod to mount a projected OIDC JWT token and injects AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE environment variables. |
| 7 | What happens if an EBS PersistentVolumeClaim uses a StorageClass with volumeBindingMode: Immediate in a 3-AZ EKS cluster? |
The EBS volume is provisioned immediately in a random AZ (e.g., AZ-a) before the pod is scheduled; if the pod is later scheduled to AZ-b, it hangs in ContainerCreating forever. |
| 8 | Why is volumeBindingMode: WaitForFirstConsumer mandatory on all production AWS EBS StorageClasses? |
It delays creating the physical EBS volume until kube-scheduler picks a specific worker node, guaranteeing the volume is created in the exact same Availability Zone as the pod. |
| 9 | When must you use the Amazon EFS CSI Driver instead of the Amazon EBS CSI Driver? | When multiple pods across different worker nodes and Availability Zones must mount the exact same volume concurrently using ReadWriteMany (RWX). |
| 10 | How does EKS Pod Identity (introduced at re:Invent 2023) simplify IRSA at multi-cluster scale? | It uses an on-node agent (eks-pod-identity-agent on 169.254.170.23) and IAM Role Session Tags without requiring cluster-specific OIDC Provider thumbprints in IAM Trust Policies. |
Lesson 2: Karpenter & Spot Fleets
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | Why does Karpenter launch replacement worker nodes in 30–45 seconds while Cluster Autoscaler takes 3–5 minutes? |
Karpenter bypasses AWS Auto Scaling Groups (ASGs) entirely and calls the EC2 Fleet API directly based on real-time pending pod resource requests and topology constraints. |
| 2 | What are the two core Custom Resource Definitions (CRDs) used to configure Karpenter v1? | NodePool (defines pod scheduling constraints, taints, and disruption policies) and EC2NodeClass (defines AWS-specific AMI, subnets, security groups, and EBS block mappings). |
| 3 | Why must you specify a wide variety of instance families (m5, m6i, c5, c6i, r5) in a Karpenter Spot NodePool? |
Spot diversification across 15+ instance pools allows EC2 Fleet price-capacity-optimized allocation to pick the deepest capacity pool with the lowest interruption rate. |
| 4 | How does Karpenter allow a single NodePool to prioritize Spot instances while automatically falling back to On-Demand if Spot capacity is exhausted? |
Include both "spot" and "on-demand" in karpenter.sh/capacity-type; Karpenter always attempts cheaper Spot first and automatically falls back to On-Demand if InsufficientInstanceCapacity occurs. |
| 5 | What AWS infrastructure is required for Karpenter to gracefully drain Spot nodes before AWS reclaims them? | An Amazon EventBridge rule capturing EC2 Spot Instance Interruption Warnings (2-minute notice) and Rebalance Recommendations, routing them into an Amazon SQS queue polled by Karpenter. |
| 6 | What annotation on a critical stateful pod prevents Karpenter Consolidation from voluntarily evicting that pod to save money? | karpenter.sh/do-not-disrupt: "true". |
| 7 | How do you prevent Karpenter from consolidating nodes during peak daytime business traffic while allowing aggressive cost consolidation overnight? | Configure Disruption Budgets with Cron Schedules inside spec.disruption.budgets (e.g., nodes: "0" during peak business hours). |
| 8 | What Kubernetes construct ensures replicas of a deployment are distributed evenly across Availability Zones (maxSkew: 1) so a single Spot AZ reclamation doesn’t take down the service? |
topologySpreadConstraints keyed on topology.kubernetes.io/zone and kubernetes.io/hostname. |
Lesson 3: Helm & Kustomize
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | What is the fundamental architectural difference between Helm and Kustomize? | Helm is a Go-templating package manager with release lifecycle history ({{ .Values.x }}), while Kustomize is a template-free YAML overlay patching engine built natively into kubectl -k. |
| 2 | Where does Helm v3 store its release state history inside a Kubernetes cluster? | As compressed, double-Base64-encoded Kubernetes Secret objects labeled owner=helm inside the target release namespace. |
| 3 | What causes a Helm release to get stuck in pending-upgrade or pending-install, blocking all future deployments? |
A previous helm upgrade CI runner crashed or timed out before writing the final status back to the Helm release Secret. |
| 4 | How do you recover a Helm release stuck in pending-upgrade? |
Run helm history <release> and execute helm rollback <release> <last-deployed-revision> (or delete the stuck pending-upgrade Helm secret revision). |
| 5 | Why should helm upgrade --install --atomic be used safely with caution in automated pipelines? |
--atomic automatically rolls back the release if any pod fails readiness within --timeout, which is safe for stateless apps but can interrupt slow database migrations if timeout is too short. |
| 6 | What is the difference between a Strategic Merge Patch and a JSON 6902 Patch in Kustomize? | Strategic Merge looks like a partial Kubernetes YAML manifest that merges by key/name, while JSON 6902 uses explicit surgical operations (op: replace, path: /spec/replicas, value: 5). |
| 7 | How does Kustomize trigger an automatic rolling restart of a Deployment whenever a ConfigMap’s data changes? |
Kustomize configMapGenerator appends a 10-character content SHA hash suffix to the ConfigMap name (e.g., app-config-8g9f2k) and updates the Deployment’s volume reference automatically. |
| 8 | How do enterprise teams combine Helm and Kustomize for third-party charts like Prometheus or Ingress-Nginx? | Render the upstream Helm chart and apply environment-specific Kustomize overlays on top (using helmCharts in kustomization.yaml or Helm --post-renderer). |
Lesson 4: Ingress & AWS ALB Controller
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | What is the difference between alb.ingress.kubernetes.io/target-type: instance and target-type: ip? |
instance routes ALB traffic to a NodePort on EC2 nodes (adding an extra kube-proxy hop), whereas ip routes directly from the ALB to the Pod’s VPC IP address via AWS VPC CNI. |
| 2 | Why does deleting or creating a standard AWS Ingress per microservice become expensive at 100+ services? | By default, each Ingress object provisions a dedicated AWS Application Load Balancer (~$18–$25/month base fee each). |
| 3 | How do you share a single AWS Application Load Balancer across 50 Ingress objects in different namespaces? | Add the annotation alb.ingress.kubernetes.io/group.name: shared-prod-alb to all 50 Ingress manifests. |
| 4 | Why do users experience intermittent HTTP 502 / 504 Bad Gateway errors for 5–10 seconds during a zero-downtime Kubernetes rolling update behind an AWS ALB? | kubelet sends SIGTERM to the pod at the exact same millisecond the Endpoints controller notifies the ALB controller; the pod stops accepting connections before the ALB finishes removing its IP from the Target Group! |
| 5 | What two configurations completely eliminate the AWS ALB 502 Bad Gateway race condition during pod termination? | Add a container lifecycle.preStop hook (exec: command: ["/bin/sleep", "15"]) and enable Pod Readiness Gates (elbv2.k8s.aws/pod-readiness-gate-inject: enabled) on the namespace. |
| 6 | What must always satisfy the inequality terminationGracePeriodSeconds > preStop_sleep + app_drain_time? |
The Pod’s terminationGracePeriodSeconds (e.g., 60s > 15s preStop + 30s ALB deregistration delay) so kubelet does not SIGKILL the pod while in-flight requests are draining. |
| 7 | How does the AWS Load Balancer Controller automatically attach an SSL/TLS certificate and a WAFv2 WebACL to an ALB? | Via Ingress annotations alb.ingress.kubernetes.io/certificate-arn (ACM) and alb.ingress.kubernetes.io/wafv2-acl-arn. |
| 8 | How do you automatically sync Kubernetes Ingress and Service hostnames into AWS Route 53 Hosted Zones? |
Deploy ExternalDNS with an IRSA role granting route53:ChangeResourceRecordSets. |
Lesson 5: Prometheus & PromQL
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | What are the 4 core Prometheus metric types? | Counter (monotonic cumulative count), Gauge (goes up/down), Histogram (server-side bucketed latency), and Summary (client-side quantiles). |
| 2 | What is the exact difference between PromQL rate(metric[5m]) and irate(metric[5m])? |
rate() calculates the average per-second rate across all samples in the 5m window (smooth, used for alerts); irate() uses only the last 2 samples (spiky, used for high-resolution graphs). |
| 3 | How do you calculate the 99th percentile (p99) HTTP latency from a Prometheus Histogram in PromQL? | histogram_quantile(0.99, sum by (le, service) (rate(http_request_duration_seconds_bucket[5m]))). |
| 4 | Why must le (less-than-or-equal bucket label) always be included inside the sum by (le) clause of histogram_quantile()? |
Without le, PromQL collapses all histogram buckets into a single sum, destroying the bucket distribution required to interpolate quantiles. |
| 5 | What causes a Prometheus High-Cardinality Outage (OOMKill)? | Adding unbounded labels like user_id, email, request_id, or full URL paths creates millions of unique time series in the TSDB in-memory Head Block. |
| 6 | How do you immediately drop a rogue high-cardinality label at scrape time before it enters Prometheus TSDB? | Add a metric_relabel_configs rule with action: labeldrop and regex: "user_id" inside the ServiceMonitor or scrape_config. |
| 7 | Why can’t a single standalone Prometheus server scale horizontally or store 2 years of enterprise metrics natively? | Prometheus uses local node block storage without built-in distributed clustering; long-term HA requires Thanos or Grafana Mimir backed by Amazon S3. |
| 8 | How does Thanos Sidecar + Thanos Querier deduplicate metrics from two identical HA Prometheus replicas? | Both Prometheus servers attach an external_labels: {replica: "a/b"} label, and thanos-query runs with --query.replica-label=replica to merge and deduplicate series on the fly. |
Lesson 6: CoreDNS ndots & Cluster Upgrades
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | Why does a Kubernetes pod querying api.stripe.com generate 8 unnecessary DNS queries before resolving the real IP? |
Default /etc/resolv.conf sets ndots:5; since api.stripe.com has only 2 dots (2 < 5), glibc appends all 4 cluster search domains (in both IPv4 A and IPv6 AAAA) before querying the absolute domain. |
| 2 | What are the 3 fastest ways to eliminate CoreDNS ndots:5 amplification in high-throughput clusters? |
(1) Deploy NodeLocal DNSCache, (2) set dnsConfig.options: [{name: ndots, value: "2"}], or (3) append a trailing dot (api.stripe.com.) in application config to make it an explicit FQDN. |
| 3 | How does NodeLocal DNSCache prevent Linux kernel conntrack table exhaustion and 5-second UDP DNS timeouts? | It runs a local caching daemon on every node (169.254.20.10) that serves cached queries locally and upgrades upstream cache-misses to CoreDNS from lossy UDP to multiplexed TCP. |
| 4 | Can you upgrade an Amazon EKS cluster directly from version 1.28 to 1.31 in one step? |
No, Kubernetes strictly prohibits skipping minor versions; you must upgrade sequentially (1.28 → 1.29 → 1.30 → 1.31). |
| 5 | What is the official Kubernetes Version Skew Policy between kube-apiserver and worker node kubelet? |
kubelet may be up to 3 minor versions older than kube-apiserver (in EKS, keep nodes within 1–2 minor versions), and must never be newer than kube-apiserver. |
| 6 | What CLI tool scans a live cluster and Git manifests to detect removed or deprecated Kubernetes API versions (like autoscaling/v2beta2 or networking.k8s.io/v1beta1) before upgrading? |
Pluto (pluto detect-all-in-cluster) or Kubent (kube-no-trouble). |
| 7 | In what exact sequence must components be upgraded during a production EKS cluster upgrade? | (1) Audit deprecated APIs → (2) Upgrade EKS Control Plane → (3) Upgrade core add-ons (vpc-cni, coredns, kube-proxy, ebs-csi) → (4) Roll/replace Worker Nodes. |
| 8 | Why should you verify PodDisruptionBudgets and topologySpreadConstraints before triggering a worker node group upgrade? |
A single misconfigured PDB with ALLOWED DISRUPTIONS = 0 will cause the managed node group rolling upgrade to stall and fail with NodeEvictionFailure. |
Hands-On Labs: Phase 4 Exercises
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | What command launches a temporary debugging container attached to the process and network namespace of a running distroless pod? | kubectl debug -it <pod-name> --image=nicolaka/netshoot --target=<container-name>. |
| 2 | How do you inspect the exact AWS IAM Role ARN attached to a Kubernetes ServiceAccount for IRSA? | kubectl get sa <sa-name> -o jsonpath='{.metadata.annotations.eks\.amazonaws\.com/role-arn}'. |
| 3 | How do you identify the top 10 highest-cardinality metric names inside a Prometheus TSDB? | Query /api/v1/status/tsdb in the Prometheus UI or run promtool tsdb analyze /prometheus. |
| 4 | What PromQL query detects pods that are experiencing >25% Linux CFS CPU quota throttling? | rate(container_cpu_cfs_throttled_periods_total[5m]) / rate(container_cpu_cfs_periods_total[5m]) > 0.25. |
| 5 | What command shows real-time node provisioning decisions and bin-packing events from Karpenter? | kubectl logs -n kube-system -l app.kubernetes.io/name=karpenter -f. |
🔄 Phase 5: GitOps & Control Planes 45 Rapid Q&As
Lesson 1: ArgoCD App of Apps
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | What are the 3 core architectural microservices that make up an ArgoCD installation? | argocd-server (API/UI), argocd-repo-server (clones Git & renders Helm/Kustomize), and argocd-application-controller (StatefulSet comparing live K8s state against Git). |
| 2 | What is the difference between automated.prune: true and automated.selfHeal: true in an ArgoCD Application? |
prune: true deletes cluster resources when their YAML files are deleted from Git; selfHeal: true overwrites manual kubectl edit changes on the cluster back to match Git. |
| 3 | What is the ArgoCD App of Apps Pattern? | Bootstrapping a cluster with a single root Application CRD whose Git directory contains child Application manifests for all other platform add-ons and workloads. |
| 4 | How do you force ArgoCD to install Custom Resource Definitions (CRDs) before deploying operators, and operators before deploying custom resources? | Annotate manifests with ArgoCD Sync Waves (argocd.argoproj.io/sync-wave: "-1", "0", "1"), which applies lower numbers first and waits for health checks between waves. |
| 5 | What ArgoCD Hook runs a database schema migration Job automatically before deploying new application pods? |
argocd.argoproj.io/hook: PreSync combined with argocd.argoproj.io/hook-delete-policy: BeforeHookCreation. |
| 6 | What causes an ArgoCD Infinite OutOfSync Loop where a Deployment continuously syncs every 3 seconds? | A cluster controller (like HPA mutating spec.replicas or a MutatingWebhook injecting defaults) modifies fields that conflict with the static values in Git. |
| 7 | How do you fix an ArgoCD Infinite Sync Loop caused by HPA modifying spec.replicas? |
Remove spec.replicas from the Git YAML manifest or add ignoreDifferences for /spec/replicas with RespectIgnoreDifferences=true in syncOptions. |
| 8 | Why does deleting an ArgoCD Application with metadata.finalizers: [resources-finalizer.argocd.argoproj.io] perform a cascading delete of all child Kubernetes resources? |
The finalizer instructs the controller to delete every managed Deployment, Service, and PVC before removing the Application object itself; omitting the finalizer performs an Orphan Delete. |
Lesson 2: Argo Rollouts & Canaries
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | Why is the native Kubernetes Deployment RollingUpdate strategy insufficient for mission-critical FinTech releases? |
Native RollingUpdate only checks basic pod readinessProbe status, cannot split traffic by exact percentages (e.g., 2%), and cannot auto-rollback based on Prometheus error-rate SLIs. |
| 2 | What CRD replaces kind: Deployment when adopting progressive delivery with Argo Rollouts? |
kind: Rollout (apiVersion: argoproj.io/v1alpha1). |
| 3 | What is the difference between Blue/Green and Canary deployment strategies in Argo Rollouts? | Blue/Green provisions 100% duplicate capacity (activeService vs previewService) and flips traffic instantly; Canary shifts traffic incrementally (5% → 20% → 50% → 100%) with automated metric pauses. |
| 4 | Without a service mesh or ingress controller, why can’t a 4-replica Kubernetes Deployment achieve a 5% canary traffic split? |
Native Service routing distributes traffic per pod (1 / 4 = 25% minimum increment); achieving 5% requires L7 weighted routing via AWS ALB, NGINX, Cilium, or Istio Gateway API. |
| 5 | What Argo Rollouts CRD queries Prometheus or Datadog during a canary rollout and triggers an automatic rollback if error rates breach 1%? | AnalysisTemplate (instantiated at runtime as an AnalysisRun). |
| 6 | In an AnalysisTemplate, what is the difference between failureLimit: 2 and inconclusiveLimit? |
failureLimit: 2 aborts and rolls back the canary after 2 failed metric checks; inconclusiveLimit pauses the rollout when Prometheus returns NaN (e.g., zero traffic during off-hours). |
| 7 | What causes the Canary Database Schema Trap where an automated canary rollback brings down 100% of production? | The canary executed a destructive SQL migration (e.g., renaming or dropping a column) that broke backward compatibility with the stable v1 pods. |
| 8 | How do you guarantee safe canary rollbacks when evolving relational database schemas? | Strictly enforce the Expand-and-Contract (Parallel Change) Pattern—never drop or rename columns in the same release that deploys new code. |
Lesson 3: Crossplane Control Planes
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | What is the fundamental architectural difference between Terraform and Crossplane? | Terraform is a CLI/pipeline tool that runs once at CI time, whereas Crossplane runs inside Kubernetes as an active 24/7 reconciliation loop that continuously self-heals cloud drift. |
| 2 | Define the 4 core building blocks of Crossplane (Provider, MR, XRD, Composition). |
Provider (AWS/GCP API controller), Managed Resource (MR) (1:1 cloud resource like an RDS instance), XRD (custom API schema), and Composition (template combining multiple MRs). |
| 3 | Why did Crossplane introduce Composition Functions (KCL / Python / Go) to replace traditional Patch-and-Transform YAML? | Traditional YAML patching became thousands of lines of untestable spaghetti lacking loops or conditionals; KCL Functions allow concise, strongly typed programmatic logic with unit testing. |
| 4 | How does a developer provision an AWS RDS database and S3 bucket using Crossplane without knowing AWS IAM or VPC subnet IDs? | The developer commits a 10-line Composite Resource Claim (XRC) YAML specifying storageGB: 50, and the Platform Team’s Composition translates it into hardened AWS resources. |
| 5 | What happens if someone manually deletes a Crossplane-managed Security Group in the AWS Console? | Within ~60 seconds, the Crossplane Provider reconciles live AWS state against the Kubernetes desired state and automatically recreates the deleted Security Group. |
| 6 | What causes a Crossplane Finalizer Deadlock where a deleted Composite Resource hangs in Terminating forever? |
Either the ProviderConfig credentials were deleted before the cloud resource was destroyed, or AWS rejected deletion due to child dependencies (e.g., non-empty S3 bucket or active ENI). |
| 7 | What flag on a Crossplane Managed Resource prevents kubectl delete from accidentally destroying a production cloud database? |
spec.deletionPolicy: Orphan (alongside spec.managementPolicies: ["Observe", "Create", "Update"] omitting "Delete"). |
| 8 | How does Crossplane deliver newly generated RDS database credentials directly to the application pod? | Using writeConnectionSecretToRef, which writes the endpoint, username, and password into a native Kubernetes Secret inside the application’s namespace. |
Lesson 4: ApplicationSets & Matrix Generators
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | What problem does the ArgoCD ApplicationSet Controller solve at fleet scale? |
It eliminates manually writing hundreds of repetitive Application YAML manifests by dynamically generating them from a single template using Generators. |
| 2 | Name the 4 most common ArgoCD ApplicationSet Generators. | List Generator, Cluster Generator (matches cluster labels), Git Generator (directory or file discovery), and Pull Request Generator (ephemeral preview environments). |
| 3 | What does the Matrix Generator do in an ArgoCD ApplicationSet? |
It computes the Cartesian Product (M × N) of two generators—for example, combining 10 Clusters (Cluster Generator) × 15 Add-ons (Git Generator) = 150 Applications from one file. |
| 4 | What does the Merge Generator do when combined with a Matrix Generator? | It allows per-cluster or per-environment configuration overrides (e.g., overriding replica counts or versions for prod-india while keeping defaults for dev). |
| 5 | What catastrophic outage can happen if a developer deletes a directory in Git watched by an ApplicationSet with default settings? |
The ApplicationSet deletes the generated Application CRDs, which triggers a cascading delete of all production workloads across every cluster! |
| 6 | How do you protect production clusters from accidental mass-deletion by an ApplicationSet? |
Configure spec.syncPolicy.preserveResourcesOnDeletion: true (or set applicationsSync: create-update) on the ApplicationSet. |
| 7 | How do you prevent an ApplicationSet from upgrading Cilium or CoreDNS across all 50 production clusters at the exact same second? |
Enable ApplicationSet Progressive Syncs (RollingSync), which rolls updates out in staged batches (dev → staging → 10% prod → 100% prod) waiting for Healthy status between steps. |
| 8 | In the Crossplane + ArgoCD Golden Pattern, what is the exact division of responsibilities? | ArgoCD syncs YAML from Git into the cluster, while Crossplane reconciles those custom YAML CRDs into actual AWS/GCP cloud infrastructure. |
Lesson 5: Hub-and-Spoke ArgoCD & DR
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | Why do enterprises prefer a Centralized Hub-and-Spoke ArgoCD Architecture over running a separate ArgoCD instance inside every single workload cluster? | A single Hub control plane provides unified RBAC/SSO, a single pane of glass for 50+ spoke clusters, and removes duplicate ArgoCD compute overhead from workload clusters. |
| 2 | Where does the Hub ArgoCD store credentials and endpoints for remote Spoke clusters? | Inside the argocd namespace as Kubernetes Secret objects carrying the label argocd.argoproj.io/secret-type: cluster. |
| 3 | How does Hub ArgoCD authenticate to remote Amazon EKS Spoke clusters without storing static long-lived bearer tokens? | Using aws-iam-authenticator / EKS IRSA Cross-Account Role Assumption configured inside the cluster Secret’s execProviderConfig. |
| 4 | Why does argocd-application-controller get OOMKilled or lag when managing 50+ spoke clusters and 3,000 Applications? |
A single controller replica cannot hold the Informer cache watches for 50 clusters in memory; you must enable Controller Dynamic Sharding (replicas: 5) across spoke clusters. |
| 5 | Why must the Hub ArgoCD never access Spoke cluster API servers over the public internet? | Exposing kube-apiserver publicly creates an attack surface; Hub-to-Spoke traffic must route over private AWS Transit Gateway, VPC Peering, or AWS PrivateLink. |
| 6 | What happens to running pods on the 50 Spoke clusters if the Central Hub ArgoCD cluster goes completely offline for 2 hours? | Zero impact to live traffic—Spoke workloads continue running normally; only new Git deployments and drift reconciliation are paused until the Hub recovers. |
| 7 | How do you recover the entire Hub ArgoCD control plane into a warm standby region (ap-south-2) in under 10 minutes without restoring etcd backups? |
Because ArgoCD is 100% declarative in Git, bootstrap the standby Hub cluster with the root App of Apps manifest and Cluster Secrets; ArgoCD reconciles against live spokes without restarting pods. |
| 8 | What Redis component in ArgoCD must run in HA mode (argocd-redis-ha) for production Hub clusters? |
The Redis cache storing live Kubernetes resource trees and UI session state across argocd-server and argocd-repo-server replicas. |
Hands-On Labs: Phase 5 Exercises
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | What CLI command manually promotes a paused Argo Rollout canary to the next step? | kubectl argo rollouts promote <rollout-name>. |
| 2 | What CLI command immediately aborts a degraded canary rollout and shifts 100% of traffic back to the stable ReplicaSet? | kubectl argo rollouts abort <rollout-name>. |
| 3 | How do you trace all child Managed Resources created by a Crossplane Composite Resource from the command line? | Run crossplane beta trace <xrd-kind> <name>. |
| 4 | What condition fields must both equal True on a Crossplane Managed Resource to confirm it is healthy and synced with AWS? |
SYNCED = True and READY = True. |
| 5 | How do you preview the exact Application manifests an ApplicationSet will generate before applying it to the cluster? |
Run argocd appset generate <appset.yaml>. |
🐝 Phase 6: Service Mesh & eBPF 32 Rapid Q&As
Lesson 1: Cilium & eBPF
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | Why does legacy kube-proxy running in iptables mode degrade severely when a cluster scales past 5,000 Kubernetes Services? |
iptables evaluates rules sequentially in an O(N) linear list and must rewrite the entire multi-megabyte rule table atomically on every endpoint update. |
| 2 | How does Cilium (kubeProxyReplacement: true) achieve O(1) service load-balancing at any cluster scale? |
It attaches eBPF programs to kernel socket and TC hooks that look up Service-to-Pod backends directly inside O(1) eBPF Hash Maps in kernel memory. |
| 3 | What ensures an eBPF program loaded into the Linux kernel cannot crash the kernel or enter an infinite loop? | The In-Kernel eBPF Verifier, which performs static DAG code analysis, bounds-checking, and memory safety verification before JIT compilation. |
| 4 | How does Cilium Socket-Level Load Balancing (bpf_sockops) accelerate pod-to-Service communication? |
It intercepts the connect() syscall at the socket layer and rewrites the ClusterIP directly to the backend Pod IP before packets even enter the TCP/IP packet stack. |
| 5 | Why does traditional sidecar service mesh architecture (Istio Envoy per pod) add latency and double memory consumption? | Every packet traverses the Linux TCP/IP stack 4 times per hop (App → local Envoy → Wire → remote Envoy → remote App), and every pod runs a dedicated Envoy proxy container. |
| 6 | How does Cilium provides transparent node-to-node encryption across the cluster without sidecars? | By enabling Kernel-Mode WireGuard Encryption (encryption.enabled=true, encryption.type=wireguard), which encrypts all pod-to-pod traffic between worker nodes at the network interface layer. |
| 7 | What observability tool built into Cilium provides real-time eBPF network flow logs, DNS query visibility, and service dependency maps? | Hubble (hubble observe). |
| 8 | What happens to existing active pod network connections if the cilium-agent DaemonSet pod restarts or crashes? |
Zero packet loss—compiled eBPF programs and maps persist inside the Linux kernel datapath independently of the user-space cilium-agent control plane. |
Lesson 2: Zero-Trust L7 Policies
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | What is the default network security posture of a fresh Kubernetes cluster without NetworkPolicies? | Flat allow-all network—every pod in every namespace can freely send packets to every other pod across the entire cluster. |
| 2 | Why are Pod IP addresses unreliable for enforcing security firewall rules in Kubernetes? | Pod IPs are ephemeral and churn constantly during autoscaling; Cilium uses cryptographic Security Identities (numeric ID) derived from pod labels instead of IP addresses. |
| 3 | What layers of the OSI model can a standard Kubernetes NetworkPolicy filter vs a CiliumNetworkPolicy? |
Standard NetworkPolicy is strictly L3/L4 (IPs, Pod Labels, TCP/UDP ports), whereas CiliumNetworkPolicy filters L3, L4, and L7 (HTTP methods/paths, gRPC, Kafka topics, DNS FQDNs). |
| 4 | How do you configure a Zero-Trust Default Deny-All baseline for a namespace in Kubernetes? | Apply a NetworkPolicy with an empty podSelector: {} and policyTypes: ["Ingress", "Egress"] with no allow rules. |
| 5 | What critical traffic is accidentally broken if you apply a Default Deny-All Egress policy without an explicit DNS exception? | Pods immediately lose the ability to resolve DNS names because UDP/TCP Port 53 traffic to kube-dns / coredns in kube-system is blocked! |
| 6 | How does a CiliumNetworkPolicy allow a pod to egress only to https://api.stripe.com even when Stripe’s CDN rotates IP addresses every minute? |
Using an L7 DNS proxy rule (toFQDNs: [{matchName: "api.stripe.com"}]); Cilium intercepts the pod’s DNS lookup, learns the returned IPs dynamically, and updates the pod’s eBPF allow map in real time. |
| 7 | How does a CiliumNetworkPolicy allow frontend to call GET /api/v1/catalog on backend, while blocking DELETE /api/v1/catalog? |
By specifying an L7 HTTP rule (rules: http: [{method: "GET", path: "/api/v1/catalog"}]), which transparently routes matching flows through an embedded node-level Envoy listener. |
| 8 | What command shows live packets being dropped by network policies across the cluster with the exact drop reason? | hubble observe --verdict DROPPED -f. |
Lesson 3: Mesh Resilience & Circuit Breaking
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | Why do naive unlimited retries during a downstream database slowdown trigger a catastrophic Retry Storm? | If 3 upstream tiers each configure 3 retries, a single failing request multiplies into 3 × 3 × 3 = 27× traffic amplification, DDoSing the recovering backend. |
| 2 | What two mathematical techniques must always be paired with retries to prevent synchronized retry waves? | Exponential Backoff (2^n × base_delay) combined with Random Jitter + a Retry Budget (capping retries at max 10–20% of total traffic). |
| 3 | Which HTTP status codes or methods should never be automatically retried without an explicit Idempotency Key? | Non-idempotent mutations like POST /charge-payment that timed out after already reaching the upstream server. |
| 4 | What is Circuit Breaking / Outlier Detection in Envoy and Cilium Service Mesh? | Automatically ejecting a degraded backend pod IP from the load-balancing pool for baseEjectionTime (e.g., 30s) when it returns consecutive 5xx errors or breaches max pending connections. |
| 5 | What are the 3 states of a Circuit Breaker? | Closed (normal traffic flows), Open (fast-fails immediately with 503 Overflow without calling backend), and Half-Open (allows a probe request to test recovery). |
| 6 | Why must upstream microservices always enforce shorter timeouts as requests travel deeper down a call chain (Timeout Budget Propagation)? | If API Gateway waits 2s, Service A waits 5s, and Service B waits 10s, Service B continues wasting CPU/DB connections for 8 seconds after the user already disconnected! |
| 7 | What Envoy response flag in access logs proves that a request was fast-failed by a circuit breaker (max_pending_requests overflow)? |
UO (Upstream Overflow). |
| 8 | What is Mutual TLS (mTLS) in Cilium Service Mesh, and how does Cilium separate authentication from encryption? | Cilium uses SPIFFE/SPIRE x509 identities to mutually authenticate workload identities in control plane handshakes while encrypting data-plane packets at line rate via kernel WireGuard/IPsec. |
Lesson 4: Cilium ClusterMesh & Gateway API
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | Why is the new Kubernetes Gateway API (gateway.networking.k8s.io) replacing the legacy Ingress resource? |
Legacy Ingress relied on vendor-specific annotation sprawl and lacked role separation, native traffic weighting, header routing, or gRPC/TCP support. |
| 2 | Name the 3 role-oriented CRDs of the Kubernetes Gateway API and who owns each. | GatewayClass (Infrastructure Provider), Gateway (Platform/SRE Team — listeners & TLS), and HTTPRoute / GRPCRoute (Application Developers — paths & weights). |
| 3 | How does HTTPRoute prevent developers in the marketing namespace from hijacking traffic meant for the payments namespace? |
The platform team’s Gateway controls allowed namespace bindings (allowedRoutes.namespaces), and cross-namespace backend references require an explicit ReferenceGrant in the target namespace. |
| 4 | What is Cilium ClusterMesh? | An eBPF multi-cluster networking technology that links multiple Kubernetes clusters across regions/clouds, enabling pod-to-pod routing and cross-cluster Service discovery without ingress gateways. |
| 5 | What prerequisite must be met by Pod CIDRs across all clusters participating in a Cilium ClusterMesh? | Each cluster must have a non-overlapping Pod CIDR and a unique cluster.id (1–255) so eBPF can route directly to remote pod IPs. |
| 6 | How do you turn a standard Kubernetes Service into a global multi-cluster service across Cilium ClusterMesh? |
Add the annotation service.cilium.io/global: "true". |
| 7 | By default, does a global ClusterMesh service balance traffic across all regions equally, or can it prefer local pods first? | Annotating service.cilium.io/affinity: "local" instructs Cilium to route 100% of traffic to healthy pods inside the local cluster, failing over to the remote region cluster only if all local pods die. |
| 8 | Why must Cilium MTU be reduced to 1420 when running ClusterMesh with WireGuard encryption over cloud VPC networks? |
Standard VPC MTU is 1500 (or 9001 jumbo); WireGuard adds 60–80 bytes of encapsulation headers, which will drop large TLS handshake packets if inner MTU is not clamped. |
🎭 Phase 7: Internal Developer Portals 40 Rapid Q&As
Lesson 1: Backstage Software Catalog
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | What core cognitive problem does an Internal Developer Portal (IDP) like Spotify Backstage solve? | It eliminates developer cognitive load and tribal knowledge silos by unifying service ownership, APIs, docs, CI/CD, and golden-path self-service into a single portal. |
| 2 | What YAML file placed at the root of a Git repository registers a microservice into the Backstage Software Catalog? | catalog-info.yaml. |
| 3 | Name the 3 Core Entity Kinds and 2 Grouping Kinds in the Backstage System Model. | Core: Component (service/website), API (OpenAPI/gRPC), Resource (DB/S3). Grouping: System (collection of components/resources) and Domain (business unit). |
| 4 | How does Backstage keep its PostgreSQL catalog database updated without requiring manual UI registrations? | Via Entity Providers (GitHub Org Discovery, GitHub Webhook processor, and Kubernetes/AWS Discovery processors) that continuously ingest and stitch entities into a graph. |
| 5 | What causes an entity in Backstage to show an Orphan Warning? | The catalog-info.yaml file was deleted or moved in the source Git repository, leaving a stale cached record in Backstage’s database. |
| 6 | What is the difference between spec.lifecycle: experimental and spec.lifecycle: production in catalog-info.yaml? |
It classifies the operational maturity of the service, driving automated production readiness scorecards and on-call paging rules. |
| 7 | How do two Backstage entities express a dependency relationship in the catalog graph? | Using spec.dependsOn: ["resource:default/orders-postgres-db"] or spec.providesApis / spec.consumesApis. |
| 8 | What causes the Backstage Catalog Processor to crash or reject an API entity during ingestion? | A malformed syntax error or unescaped $ref circular loop inside the referenced OpenAPI (swagger.yaml) or AsyncAPI specification file. |
Lesson 2: Golden Path Templates
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | What is a “Golden Path” (Paved Road) in Platform Engineering? | An opinionated, automated self-service workflow that makes the secure, compliant, observable way of building and deploying software the easiest default path for developers. |
| 2 | What Backstage component executes multi-step infrastructure and code provisioning workflows (kind: Template)? |
The Backstage Scaffolder (@backstage/plugin-scaffolder-backend). |
| 3 | What are the 3 main sections of a Backstage Template YAML definition? |
spec.parameters (React JSON Schema UI form inputs), spec.steps (sequential actions: fetch, template, publish PR, register), and spec.output (links to created repo/ArgoCD). |
| 4 | Why should a Backstage Golden Path template create a Git Pull Request containing Crossplane/Terraform manifests instead of calling AWS APIs directly from Node.js? | Committing declarative manifests to Git preserves GitOps auditability, policy-as-code PR checks, and automatic drift reconciliation. |
| 5 | What templating engine does Backstage fetch:template use to inject user form parameters into skeleton source files? |
Nunjucks (${{ parameters.service_name }}). |
| 6 | What happens if Step 3 (publish:github) succeeds in creating a GitHub repo, but Step 4 (catalog:register) fails in a Scaffolder run? |
Without compensating cleanup or idempotency, retrying the template fails with Repository already exists; production custom actions must implement idempotent checks or Saga rollbacks. |
| 7 | How do you prevent a developer from typing an invalid Kubernetes service name (e.g., uppercase letters or spaces) into a Backstage Scaffolder form? | Add a regex pattern: '^[a-z0-9-]+$' validation rule inside the spec.parameters JSON Schema property. |
| 8 | What custom Scaffolder action allows platform teams to trigger an external GitHub Actions workflow or Jenkins job during scaffolding? | github:actions:dispatch (or custom HTTP webhook actions). |
Lesson 3: TechDocs & Docs-as-Code
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | Why do traditional enterprise Confluence wikis fail for operational runbooks? | Documentation decoupled from source code rots immediately because developers rarely update a separate wiki when refactoring code or APIs. |
| 2 | How does Backstage TechDocs (Docs-as-Code) solve documentation drift? | Markdown docs (docs/index.md + mkdocs.yml) live inside the exact same Git repository as the microservice code and are reviewed in the same Pull Request. |
| 3 | What annotation in catalog-info.yaml tells Backstage where to find an entity’s TechDocs source files? |
backstage.io/techdocs-ref: dir:.. |
| 4 | Why must you never configure techdocs.builder: 'local' in a production Backstage deployment? |
'local' forces the Backstage Node.js backend pod to spawn heavy Python mkdocs subprocesses on the fly when users view pages, causing CPU starvation and OOMKills. |
| 5 | What is the recommended production architecture for Backstage TechDocs (techdocs.builder: 'external')? |
CI/CD Pre-Building—GitHub Actions runs techdocs-cli generate and techdocs-cli publish to push compiled static HTML/CSS to an Amazon S3 / GCS bucket, which Backstage serves instantly. |
| 6 | What MkDocs extension renders architecture-as-code diagrams (flowchart, sequenceDiagram) natively inside TechDocs pages? |
Mermaid.js (pymdownx.superfences with custom mermaid fence). |
| 7 | How do you enforce that a developer cannot change an API endpoint without updating the service’s TechDocs or Runbook? | Configure a GitHub Actions PR gate or CODEOWNERS check that flags changes to /api/* or /terraform/* lacking corresponding updates in /docs/*. |
| 8 | Where should every microservice’s operational runbook link so on-call SREs can find it at 3:00 AM? | Directly inside the Prometheus Alert annotations.runbook_url pointing to the service’s Backstage TechDocs URL. |
Lesson 4: Backstage RBAC & Scorecards
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | How does Backstage authenticate users and automatically map them to their engineering teams (kind: Group)? |
Via OIDC / SAML SSO (Okta, Azure Entra ID, Google Workspace) paired with an automated Org Discovery Processor that syncs users and group memberships. |
| 2 | In the Backstage Permission Framework, what is the difference between a Basic Permission and a Resource Permission? | A Basic Permission controls global actions (catalog.entity.create), while a Resource Permission evaluates fine-grained conditions against a specific entity (e.g., isEntityOwner). |
| 3 | How do you restrict a destructive Scaffolder template (like "Decommission Production Database") so only Staff SREs can see or execute it? |
Tag the template with tags: ["restricted-sre"] and apply a Backstage RBAC Conditional Policy denying scaffolder.template.parameter.read and task.create unless the user is in group:default/sre-leads. |
| 4 | What is a Production Readiness Scorecard in an Internal Developer Portal? | An automated grading engine (Bronze / Silver / Gold) that continuously audits every microservice against security, reliability, observability, and documentation standards. |
| 5 | Name 5 automated checks typically evaluated by a Gold-Tier Production Readiness Scorecard. | (1) Valid PodDisruptionBudget, (2) SLO + Runbook URL defined, (3) Zero CRITICAL Trivy CVEs, (4) Cosign image signature verified, and (5) PagerDuty rotation attached. |
| 6 | How do you turn passive Backstage Scorecards into active Deployment Gates during major sales events (like IPL or Diwali)? | Query the Scorecard API from GitHub Actions or ArgoCD PreSync hooks and block production promotion for any service scoring below Silver/Gold. |
| 7 | How does Backstage prevent a developer in Team A from viewing sensitive runtime secrets or triggering deployments on Team B’s Kubernetes pods? | By applying the isEntityOwner RBAC condition on the Kubernetes and ArgoCD plugins, scoping actions strictly to spec.owner == user.claims.groups. |
| 8 | What audit trail must be enabled on the Backstage backend to satisfy SOC2 and ISO27001 compliance? | Structured Audit Logging capturing every Scaffolder task execution, RBAC permission decision (ALLOW/DENY), user identity, and timestamp sent to SIEM. |
Lesson 5: Chaos Engineering (Chaos Mesh)
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | What is the core discipline of Chaos Engineering? | Designing controlled, hypothesis-driven fault injection experiments to uncover systemic weaknesses before they cause unpredicted production outages. |
| 2 | What must always be defined before injecting any fault in a Chaos Engineering experiment? | The Steady-State Hypothesis (measurable business/technical SLIs such as "Checkout p99 < 250ms and HTTP 5xx < 0.1%"). |
| 3 | Name the 4 most common fault CRDs in Chaos Mesh. | PodChaos (pod-kill/container-kill), NetworkChaos (latency/loss/partition), StressChaos (CPU/memory burn via stress-ng), and IOChaos / HTTPChaos. |
| 4 | How does the Chaos Mesh chaos-daemon inject a 150ms network delay into a target pod without modifying the pod’s container image? |
The privileged chaos-daemon on the host uses nsenter to enter the target pod’s network namespace (netns) and configures a Linux kernel Traffic Control (tc) netem qdisc on its eth0 interface. |
| 5 | What happens if the chaos-daemon pod crashes or is OOMKilled while a NetworkChaos experiment is actively injecting 50% packet loss? |
The Linux kernel tc qdisc netem rule remains orphaned inside the target pod’s network namespace (Zombie Chaos Rule), causing permanent packet loss even after the NetworkChaos CRD is deleted! |
| 6 | How do you manually clear a Zombie NetworkChaos rule from an affected pod without restarting the pod? |
Enter the pod’s network namespace from the worker node and delete the root queuing discipline: nsenter -t <pod-pid> -n tc qdisc del dev eth0 root. |
| 7 | How do you prevent a Chaos Mesh experiment from accidentally targeting kube-system or critical control plane namespaces? |
Configure Chaos Mesh controller flags FILTER_NAMESPACE / annotation enforcement (chaos-mesh.org/inject: "true" required on target namespaces) and block system namespaces via Kyverno. |
| 8 | How do you automate an immediate Emergency Stop (Abort) of a Chaos Mesh Workflow if real customer error budgets start burning? |
Pair the Chaos Workflow with a Prometheus Alertmanager webhook (or Argo Rollouts Analysis) that automatically deletes the active Chaos CRD as soon as any production SLO alert fires. |
🧠 Phase 8: LLMOps Infrastructure 29 Rapid Q&As
Lesson 1: Vector DBs & AI Gateways
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | What is RAG (Retrieval-Augmented Generation), and why is it preferred over fine-tuning for dynamic enterprise data? | RAG retrieves relevant document chunks from a Vector DB at query time and injects them into the LLM prompt context, preventing hallucinations and enforcing real-time document ACLs without expensive weight retraining. |
| 2 | Why does a Milvus Vector Database QueryNode get OOMKilled when loading 50 million 1536-dimension float32 vectors (50M × 1536 × 4B ≈ 307GB)? |
In-memory HNSW (Hierarchical Navigable Small World) graph indices require 1.5× to 2× the raw vector size in RAM to store multi-layer graph adjacency pointers alongside raw vectors. |
| 3 | How do you reduce Vector Database memory consumption by 75% to 90% when scaling to billions of vectors? | Use Vector Quantization (SQ8 Scalar Quantization to 8-bit integers or PQ Product Quantization) or DiskANN (NVMe SSD-backed indexing). |
| 4 | Why should enterprise microservices never call OpenAI, Anthropic, or AWS Bedrock APIs directly from application code? | Direct calls leak API keys across dozens of repos, lack unified token cost tracking per team, and cannot automatically fail over when OpenAI returns HTTP 429 or 503. |
| 5 | What are the 4 core functions of an Enterprise AI Gateway (LiteLLM / Envoy AI Gateway)? | (1) Multi-provider automatic failover (OpenAI → Azure OpenAI → Bedrock), (2) Token-based rate limiting (TPM/RPM), (3) Per-tenant FinOps cost chargeback, and (4) Semantic Caching. |
| 6 | How does Semantic Caching (using Redis Vector Search) differ from traditional exact-match key-value caching? | Instead of requiring identical prompt strings, it embeds the user’s query into a vector and returns a cached response if Cosine Similarity exceeds a threshold (e.g., > 0.95), saving 30–50% of LLM spend. |
| 7 | Why do traditional HTTP load balancers and rate limiters (requests/sec) fail to protect LLM gateways? | One LLM request with a 100,000-token context window consumes 1,000× more GPU compute and quota than a 100-token request; LLM gateways must rate-limit by Tokens Per Minute (TPM). |
| 8 | What causes a LiteLLM Retry Storm during an upstream OpenAI outage, and how do you prevent it? | Setting high retry counts across multiple fallback regions; fix by setting num_retries: 2, configuring a cooldown_time: 60s circuit breaker on degraded deployments, and enforcing request timeouts. |
Lesson 2: MLflow & Prompt Evaluation
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | What are the 4 core components of an enterprise MLflow platform? | MLflow Tracking (experiments/params/metrics), MLflow Model Registry (versioning & aliases like @champion), MLflow Projects, and Artifact Store (S3) + Backend Store (PostgreSQL). |
| 2 | Why do traditional unit tests (assert output == "expected") fail when testing LLM applications in CI/CD? |
LLMs are probabilistic and non-deterministic—two semantically identical correct answers rarely have identical word-for-word string representations. |
| 3 | What are the 4 core metrics of the RAGAS (Retrieval-Augmented Generation Assessment) evaluation framework? | Context Precision (signal-to-noise of retrieved chunks), Context Recall (did retrieval find all facts?), Faithfulness (did LLM hallucinate outside context?), and Answer Relevance. |
| 4 | Which RAGAS metric specifically detects LLM Hallucinations (claims generated by the model that cannot be inferred from the retrieved context)? | Faithfulness (scored from 0.0 to 1.0; production CI gates typically require Faithfulness >= 0.90). |
| 5 | Why did an LLM-as-a-Judge CI evaluation pipeline pass a PR at 10:00 AM and fail the exact same code at 10:15 AM? | The judge LLM was invoked with temperature > 0 or an unpinned floating model name; LLM-as-a-Judge must always set temperature: 0.0, seed: 42, and pin exact model snapshots. |
| 6 | How do you version prompts in a GitOps workflow instead of hardcoding prompt strings inside Python application code? | Store prompts as versioned YAML templates in Git (or MLflow Prompt Registry), evaluate them against a Golden Dataset in CI using DeepEval/RAGAS, and promote via @production aliases. |
| 7 | What is Data Drift / Embedding Drift in a production RAG system? | When live user queries diverge semantically from the historical document corpus or when changing embedding models (text-embedding-ada-002 → text-embedding-3-large) invalidates existing vector distances. |
| 8 | Why can’t you query vectors generated by text-embedding-ada-002 using a query embedded with text-embedding-3-large? |
Different embedding models map semantics into completely incompatible vector spaces; switching embedding models requires re-indexing the entire document corpus into a new collection (Blue/Green Vector Re-indexing). |
Lesson 3: GPU Slicing, vLLM & Guardrails
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | What is the structural difference between GPU Time-Slicing and NVIDIA MIG (Multi-Instance GPU) on an A100/H100 GPU? | Time-Slicing shares one GPU via context-switching with zero memory or fault isolation (one OOM crashes all pods); MIG physically partitions the silicon into up to 7 hardware-isolated GPU instances with dedicated VRAM and compute cores. |
| 2 | How much GPU VRAM is required just to load the model weights of a 70B parameter LLM in FP16 (16-bit = 2 bytes per parameter)? |
≈ 140 GB of VRAM (70B × 2 bytes), plus an additional 20–40 GB for the KV (Key-Value) Cache and activations—requiring at least 2× 80GB H100 GPUs or AWQ/GPTQ 4-bit quantization. |
| 3 | What breakthrough memory technology in vLLM eliminates 60–80% of GPU VRAM waste during concurrent LLM serving? | PagedAttention—it splits the attention KV Cache into non-contiguous virtual memory blocks (like OS virtual memory pages), eliminating contiguous memory fragmentation and enabling Continuous Batching. |
| 4 | What causes a vLLM CUDA Out-Of-Memory (OOM) crash during traffic bursts even though the model weights loaded cleanly at startup? | Setting --max-model-len too high (e.g., 128k context) or --gpu-memory-utilization to 0.98, leaving zero VRAM headroom for PyTorch activation spikes during large prefill batches. |
| 5 | How do you shard a 70B model across 4 GPUs on the same physical node in vLLM? |
Set Tensor Parallelism --tensor-parallel-size 4 (TP=4), which splits matrix multiplications across all 4 GPUs simultaneously over high-speed NVLink. |
| 6 | Why does autoscaling vLLM pods based on Kubernetes CPU or GPU Memory % fail completely? | vLLM pre-allocates 90% of GPU VRAM at startup (so memory is permanently constant at 90%), and CPU stays low; you must autoscale vLLM with KEDA on vllm:num_requests_waiting (queue depth) and vllm:gpu_cache_usage_perc. |
| 7 | Why do LLM GPU pods take 10+ minutes to start during scale-out, and how do you cut cold-start time to under 30 seconds? | Pulling 140GB of Safetensors weights over the internet or slow EBS throttles network I/O; store model weights on Amazon FSx for Lustre or pre-cached NVMe instance stores with --load-format safetensors. |
| 8 | What runtime security layer sits in front of an LLM to block Prompt Injection (Jailbreaks) and redact PII (Aadhaar, PAN, Credit Cards) before prompts leave the cluster? | LLM Guardrails (NVIDIA NeMo Guardrails, Llama Guard 3, and Microsoft Presidio PII redaction filters). |
Hands-On Labs: Phase 8 Exercises
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | What Prometheus exporter deployed by the NVIDIA GPU Operator exposes hardware metrics like SM utilization (DCGM_FI_DEV_DEC_UTIL), VRAM temperature, and XID hardware errors? |
NVIDIA DCGM Exporter (dcgm-exporter). |
| 2 | What does an NVIDIA XID Error 79 (“GPU has fallen off the bus”) or 48 (“Double Bit ECC Error”) on a worker node indicate? |
A fatal physical hardware/PCIe/VRAM failure on the GPU; an automated Node Problem Detector remediation must cordon, drain, and terminate the unhealthy GPU node. |
| 3 | What two core latency SLIs define user experience in streaming LLM inference? | TTFT (Time To First Token) (prefill latency, target < 500ms) and ITL / TPOT (Inter-Token Latency / Time Per Output Token) (decode throughput, target < 30ms/token). |
| 4 | How do you enable 4-bit AWQ (Activation-aware Weight Quantization) in vLLM to fit a 70B model onto a single 80GB GPU? | Pass --quantization awq when launching the vLLM server with an AWQ-quantized model checkpoint (~38GB VRAM for weights). |
| 5 | What Kubernetes scheduling extension ensures an 8-GPU distributed training job schedules all 8 pods together or none at all to prevent resource deadlocks? | Gang Scheduling via Kueue or Volcano Scheduler. |