โšก
Zero to Platform Architect2026
Staff Battleground & Test Banks
Staff Battleground & Test Banksโฑ๏ธ 120 min readโšก Staff SRE Standard
๐ŸŽฏ Memory Bank →

๐Ÿง  Master All-Lessons One-Line Memory Bank (All Phases 0โ€“10)

Welcome to the Complete Curriculum One-Line Memory Bank. Every single lesson across Foundations (Phases 0โ€“2), Operations & Kubernetes (Phases 3โ€“5), Service Mesh, DevEx & LLMOps (Phases 6โ€“8), Enterprise Architecture & SRE (Phase 10), and the Capstone Engine is distilled here into crisp, rapid-fire questions and answers for active recall drills.

Active Recall Engine 540+ Production Q&As Phases 0 through 10

โšก Active Recall & Pre-Interview Memory Drill Instructions

Staff and Principal interview panels in Tier-1 Tech, Indian High-Concurrency Unicorns, and GCCs probe deep architectural reflexes. Use the active controls below to switch between Active Recall Mode (which masks answers so you formulate them in your mind before clicking to reveal) and Review Mode (which shows answers for speed-reading).

๐Ÿ™ˆ 1. Active Recall Mode

Answers are masked. Read the question, say the answer out loud, then click to verify.

<div class="p-3 rounded-xl bg-white dark:bg-slate-800/80 border border-slate-200 dark:border-white/10">
  <div class="font-bold text-slate-900 dark:text-white mb-1 flex items-center gap-1.5">
    <span>๐Ÿ”</span> <span>2. Instant Live Search</span>
  </div>
  <p class="text-slate-500 dark:text-slate-400">Search 540+ questions across all 10 phases by keyword (e.g. <code>cgroups</code>, <code>CIDR</code>, <code>etcd</code>, <code>OIDC</code>).</p>
</div>

<div class="p-3 rounded-xl bg-white dark:bg-slate-800/80 border border-slate-200 dark:border-white/10">
  <div class="font-bold text-slate-900 dark:text-white mb-1 flex items-center gap-1.5">
    <span>๐ŸŽฒ</span> <span>3. Random Drill</span>
  </div>
  <p class="text-slate-500 dark:text-slate-400">Test unexpected interview curveballs by jumping to a randomly selected production question.</p>
</div>
๐Ÿง Phase 0: The Primitives 60 Rapid Q&As

Lesson 1: Linux Primitives & systemd

# One-Line Question One-Line Answer
1 What numeric value represents Read, Write, and Execute permissions in Linux octal notation? Read (r) = 4, Write (w) = 2, Execute (x) = 1.
2 Why is the Execute (x) bit mandatory on a Linux directory for a user to cd into it? The x bit on a directory controls inode traversal (cd); without it, users can list file names (r) but cannot enter or stat files.
3 What is the difference between SIGTERM (15) and SIGKILL (9)? SIGTERM is a catchable signal allowing graceful cleanup, whereas SIGKILL is handled directly by the kernel to terminate the process immediately.
4 Why does a process in Uninterruptible Sleep (D state) ignore kill -9? Signals are only delivered when a process returns from kernel space to user space, and a D-state process is blocked inside a kernel hardware I/O syscall.
5 Why does df -h show 100% disk usage when du -sh shows only 40% used? An open process is holding a file descriptor to a deleted file, preventing the kernel from reclaiming disk blocks until the process closes the fd or restarts.
6 How do you find deleted files still held open by active processes? Run lsof +L1 to list all open files with a link count less than 1.
7 Where should custom systemd unit files be placed to avoid being overwritten by OS package upgrades? Place custom unit files in /etc/systemd/system/, which takes precedence over /lib/systemd/system/.
8 What systemd directive ensures a crashed daemon waits 5 seconds before restarting? Set Restart=always (or on-failure) combined with RestartSec=5s under the [Service] block.
9 How do you stream live systemd logs for a specific service starting from the most recent entries? Run journalctl -u <service-name> -f -n 50.
10 What file defines the maximum number of open file descriptors permitted per process at the kernel level? /proc/sys/fs/file-max (system-wide) and LimitNOFILE= inside the systemd service unit (per-service).
# One-Line Question One-Line Answer
1 What is the mathematical formula to calculate total IPv4 addresses in a /N CIDR block? Total IPs = 2^(32 - N).
2 How many total IPs and AWS-usable IPs exist in a /24 subnet? 256 total IPs, and 251 usable IPs because AWS reserves 5 IPs per subnet.
3 Which 5 IP addresses does AWS reserve in every VPC subnet (e.g., 10.0.1.0/24)? .0 (Network), .1 (VPC Router), .2 (AWS DNS), .3 (Future use), and .255 (Broadcast).
4 What are the three packets exchanged during a TCP 3-way handshake? SYN (Client โ†’ Server), SYN-ACK (Server โ†’ Client), and ACK (Client โ†’ Server).
5 What is the fundamental difference between AWS Security Groups and Network ACLs (NACLs)? Security Groups are stateful (ENI-level, allow rules only), whereas NACLs are stateless (subnet-level, explicit allow/deny rules required for both directions).
6 Why must a stateless NACL allow inbound traffic on ports 1024โ€“65535? Outbound requests originate from random ephemeral source ports (1024โ€“65535), so return responses arrive targeting those ephemeral ports.
7 What command displays all listening TCP sockets along with their owning process PIDs on modern Linux? ss -tulpn (replaces legacy netstat -tulpn).
8 What does a TCP socket stuck in SYN_SENT state indicate? The client sent a TCP SYN packet, but no SYN-ACK was receivedโ€”typically caused by a silent firewall drop or blackhole route.
9 What does a buildup of CLOSE_WAIT sockets on a server indicate? The remote peer closed the connection (FIN), but the local application has a code bug and failed to execute close() on the socket descriptor.
10 What port and protocol does standard DNS use, and when does it switch to TCP? UDP Port 53 by default; it switches to TCP Port 53 for zone transfers (AXFR) or when responses exceed 512 bytes (or EDNS0 buffer size).
# One-Line Question One-Line Answer
1 What are the 4 core principles of OpenGitOps? Declarative state, Versioned & Immutable storage, Pulled automatically, and Continuously Reconciled.
2 What is the difference between git merge and git rebase? git merge preserves chronological history with a merge commit, whereas git rebase rewrites commit hashes onto a linear tip.
3 Why should you never run git rebase on a shared public branch like main? Rebasing generates brand-new SHA-1 hashes for existing commits, diverging history and breaking local clones for every other developer.
4 What is the exact difference between git reset --soft HEAD~1 and git reset --hard HEAD~1? --soft moves HEAD back one commit while keeping changes staged; --hard permanently destroys uncommitted changes in both the index and working tree.
5 Why is git revert <commit> preferred over git reset for production rollbacks in GitOps? git revert creates a new forward commit that cleanly inverses the bad change, preserving an immutable audit trail for ArgoCD/Flux.
6 How do you recover a commit that was accidentally deleted by a git reset --hard? Run git reflog to locate the orphaned commitโ€™s SHA hash and execute git checkout or git reset --hard <sha>.
7 Why is --force-with-lease safer than git push --force? --force-with-lease aborts the push if another teammate has pushed new commits to the remote branch since your last fetch.
8 What command uses binary search across commit history to pinpoint the exact commit that introduced a bug? git bisect start, followed by git bisect bad and git bisect good <sha>.
9 Why is Pull-based GitOps (ArgoCD inside cluster) more secure than Push-based CI (kubectl apply in GitHub Actions)? Pull-based GitOps keeps cluster admin credentials inside the cluster, eliminating the need to expose the Kubernetes API or store god-mode tokens in external CI runners.
10 How do you sign Git commits cryptographically to verify author identity? Configure GPG or SSH signing keys and commit using git commit -S -m "message".
# One-Line Question One-Line Answer
1 Are containers lightweight virtual machines with their own kernel? No, containers are isolated host Linux processes sharing the underlying host OS kernel via namespaces and cgroups.
2 What filesystem driver does Docker use to stack immutable image layers with a writable container layer? OverlayFS (lowerdir for read-only image layers, upperdir for writable changes, and merged for the unified view).
3 How does OverlayFS handle deleting a file that exists in a read-only lower layer? It creates a character 0/0 whiteout file in upperdir that masks the underlying file in the merged directory view.
4 What is the primary security and size benefit of a Multi-Stage Dockerfile? It compiles code in a heavy builder stage and copies only the compiled binary into a minimal runtime stage, stripping compilers, source code, and build secrets.
5 Why is a distroless base image more secure than alpine or ubuntu? Distroless images contain zero shell (/bin/sh) and zero package managers (apt/apk), preventing attackers from spawning interactive reverse shells.
6 Why should COPY package.json . and RUN npm ci be placed before COPY . . in a Dockerfile? Docker caches layers sequentially; copying dependency manifests first prevents source code edits from invalidating the expensive dependency installation cache.
7 What is the difference between ENTRYPOINT and CMD in a Dockerfile? ENTRYPOINT defines the fixed executable binary run by the container, while CMD supplies default arguments that can be easily overridden at runtime.
8 Why should you always use exec form ["app", "--flag"] instead of shell form "app --flag" in ENTRYPOINT? Shell form wraps the process in /bin/sh -c as PID 1, which swallows SIGTERM signals and prevents graceful application shutdown.
9 How do you prevent a compromised container process from having root privileges on the host kernel? Add a non-root user (USER 10001) in the Dockerfile and set allowPrivilegeEscalation: false at runtime.
10 How does docker run -p 8080:80 route host traffic into the containerโ€™s isolated network namespace? It injects an iptables Destination NAT (DNAT) rule in the hostโ€™s PREROUTING chain pointing to the containerโ€™s virtual Ethernet (veth) IP.
# One-Line Question One-Line Answer
1 What is the exact division of responsibility between Linux Namespaces and Control Groups (cgroups)? Namespaces isolate visibility (what a process can see), while cgroups restrict resource consumption (how much CPU/RAM/I/O a process can use).
2 Name the 7 Linux Kernel Namespaces used by container runtimes. pid (Process IDs), net (Networking), mnt (Mount points), uts (Hostname), ipc (Inter-process comms), user (UIDs), and cgroup (cgroup root).
3 What Linux system call creates a new process inside brand-new isolated namespaces? clone() (passed with flags like `CLONE_NEWPID
4 What Linux command allows an SRE to enter the network namespace of a running container using its host PID? nsenter -t <host-pid> -n ip addr (allows running host network debugging tools inside a distroless containerโ€™s netns).
5 How does a container network namespace communicate with the host root network namespace? Through a virtual Ethernet (veth) pipe pair, where one end sits inside the container netns and the other attaches to a host bridge (docker0 or cni0).
6 What is the difference between cpu.shares (CPU requests) and cpu.cfs_quota_us (CPU limits)? cpu.shares is a soft proportional weight enforced only during CPU contention, whereas cpu.cfs_quota_us is a hard ceiling enforced every 100ms regardless of idle host CPU.
7 What architectural upgrade did cgroups v2 introduce over cgroups v1? Unified single hierarchy design and the ability to safely restrict buffered Page Cache and writeback disk I/O alongside anonymous memory.
8 What is Linux Pressure Stall Information (PSI) in /proc/pressure/{cpu,memory,io}? A kernel metric measuring the exact percentage of wall-clock time tasks are stalled waiting for hardware resources (some vs full).
9 What happens when a containerโ€™s memory usage (anon + non-reclaimable cache) breaches memory.max? The kernel invokes the cgroup OOM killer (mem_cgroup_out_of_memory), sending SIGKILL (Exit Code 137) to the highest-scoring process.
10 What system call changes the root directory of a container more securely than chroot? pivot_root(), which unmounts the old host root filesystem entirely so privileged processes cannot break out via file descriptors.
# One-Line Question One-Line Answer
1 How do you manually create an isolated network namespace named red using iproute2? Run sudo ip netns add red.
2 How do you create a virtual Ethernet (veth) cable connecting veth-red and veth-host? Run sudo ip link add veth-red type veth peer name veth-host.
3 How do you move one end of a veth interface into the red network namespace? Run sudo ip link set veth-red netns red.
4 Why does ping 127.0.0.1 fail inside a newly created network namespace? The loopback interface (lo) is down by default in a new netns and requires ip netns exec red ip link set lo up.
5 What host kernel sysctl must be set to 1 for Linux to route packets between interfaces like a router? sudo sysctl -w net.ipv4.ip_forward=1.
6 What iptables rule masks private container IPs behind the hostโ€™s primary interface IP for internet access? sudo iptables -t nat -A POSTROUTING -s 10.200.1.0/24 ! -o veth-host -j MASQUERADE.
7 How do you manually restrict a cgroup v2 group to 50% of one CPU core (50ms per 100ms)? Run echo "50000 100000" > /sys/fs/cgroup/my-group/cpu.max.
8 How do you move an active shell process into a specific cgroup v2 directory? Write its PID into cgroup.procs: echo $$ > /sys/fs/cgroup/my-group/cgroup.procs.
9 How do you inspect the exact namespaces a process belongs to on the Linux filesystem? Run ls -la /proc/<pid>/ns/.
10 How do you verify image layer sizes and build history for a local Docker image? Run docker history <image-name>.
โ˜๏ธ Phase 1: Enterprise Multi-Cloud 55 Rapid Q&As

Lesson 1: AWS Orgs & Landing Zones

# One-Line Question One-Line Answer
1 Why do enterprises use a Multi-Account Landing Zone instead of a single AWS account with many VPCs? Accounts provide the hardest blast-radius boundary for IAM permissions, AWS API rate limits, security audits, and FinOps billing isolation.
2 What are the 4 standard core accounts in an AWS Security Reference Architecture (SRA) Landing Zone? Management Account, Log Archive Account, Security Tooling (Audit) Account, and Shared Network Hub Account.
3 Do AWS Service Control Policies (SCPs) grant IAM permissions to users? No, SCPs never grant permissions; they act as organizational guardrails defining the maximum permission ceiling for member accounts (including Account Root).
4 Does an AWS Service Control Policy (SCP) affect users or roles inside the Organization Management Account? No, SCPs never apply to the Management Accountโ€”which is why workloads must never run in the Management Account.
5 What AWS evaluation logic applies when an SCP, Permission Boundary, and IAM Role Policy are all present? An explicit Deny in any policy overrides everything; otherwise, an action is allowed only if all applicable policies (SCP โˆฉ Boundary โˆฉ IAM Policy) explicitly Allow it.
6 How do you prevent a compromised admin from deleting centralized CloudTrail logs in the Log Archive S3 bucket? Apply an SCP denying s3:DeleteBucket and enable S3 Object Lock in Compliance Mode on the Log Archive bucket.
7 What service automates multi-account vending and baseline guardrail provisioning in AWS? AWS Control Tower (using Account Factory for Terraform โ€” AFT).
8 How do you prevent developers from launching resources outside approved Indian regions (ap-south-1, ap-south-2)? Attach a Region Restriction SCP with Effect: Deny and a StringNotEquals condition on aws:RequestedRegion (while excluding global services like IAM/Route53).
9 Why must global services like iam:*, sts:*, and cloudfront:* be exempted in a Region Restriction SCP? Global AWS services always execute API calls against us-east-1 control planes regardless of your workload region.
10 What IAM mechanism allows a central CI/CD runner to prevent a developer from creating an IAM role with higher privileges than the runner itself? IAM Permissions Boundaries attached mandatory via condition keys on iam:CreateRole.
# One-Line Question One-Line Answer
1 Why does VPC Peering fail at enterprise scale (N > 10 VPCs)? VPC Peering is non-transitive and requires N(N-1)/2 full-mesh connections (e.g., 100 VPCs require 4,950 peerings).
2 How does AWS Transit Gateway (TGW) solve the O(N^2) peering mesh problem? TGW acts as a centralized L3 regional hub router where each spoke VPC connects once (O(N) topology).
3 What is the difference between a TGW Route Table Association and Propagation? An Association binds an attachment to a single route table for its outbound routing decisions; a Propagation dynamically injects an attachmentโ€™s CIDR into target route tables.
4 How do you isolate Dev VPCs from Prod VPCs while allowing both to reach a Shared Services VPC via one TGW? Create separate Dev and Prod TGW Route Tables that propagate only the Shared Services CIDR, omitting cross-environment propagations.
5 What is a Blackhole route in an AWS Transit Gateway route table? A static route configured to drop matching packets immediately, used to block unauthorized inter-VPC CIDR traffic explicitly.
6 Why do enterprises centralize outbound internet traffic through a dedicated Egress Inspection VPC? It eliminates paying hourly fees for redundant NAT Gateways in 100+ spoke VPCs and enforces centralized L7 DPI/FQDN firewall inspection.
7 What causes 50% packet drops when routing multi-AZ traffic through stateful firewalls in an Inspection VPC? Asymmetric routingโ€”outbound packets pass through Firewall AZ-a, while return packets hash to Firewall AZ-b which lacks the TCP state entry.
8 What single Transit Gateway setting prevents asymmetric routing across multi-AZ stateful inspection firewalls? Enable Appliance Mode (ApplianceModeSupport = enable) on the Inspection VPCโ€™s Transit Gateway attachment.
9 How many concurrent connections can a single AWS NAT Gateway support to a single destination IP:Port? 55,000 connections per destination tuple before throwing port allocation errors.
10 How do you eliminate NAT Gateway data processing charges ($0.045/GB) for high-volume S3 and DynamoDB traffic? Provision free Gateway VPC Endpoints for S3 and DynamoDB, which route traffic directly over the private AWS backbone via route table entries.
# One-Line Question One-Line Answer
1 What is the difference between SAML 2.0/OIDC and SCIM in enterprise identity federation? SAML 2.0 / OIDC handles Authentication (verifying identity at login), while SCIM handles Lifecycle Provisioning (auto-creating/disabling users and groups).
2 Why are static IAM User access keys (AKIA...) banned in enterprise cloud environments? They never expire automatically and are the #1 vector for credential leaks in Git repositories; temporary STS tokens (ASIA...) expire in 1โ€“12 hours.
3 What is an AWS IAM Identity Center Permission Set? A centralized role template managed in the Organization that provisions corresponding IAM roles into target AWS accounts upon assignment.
4 What is the Azure architectural equivalent of AWS Organizations and Organizational Units (OUs)? Azure Management Groups (organized in a hierarchy up to 6 levels deep above Azure Subscriptions).
5 What is the Azure equivalent of an AWS Service Control Policy (SCP)? Azure Policy (which can audit, deny, or auto-remediate non-compliant resources across Management Groups).
6 What is the key difference between AWS SCPs and Azure Policy? AWS SCPs filter IAM API actions before execution, whereas Azure Policy evaluates ARM resource properties and can auto-deploy missing configurations (DeployIfNotExists).
7 How do you authenticate an AWS workload to Azure Entra ID (or vice versa) without storing client secrets? Configure Workload Identity Federation using OIDC trust between the AWS IAM/EKS OIDC issuer and an Azure Managed Identity / App Registration.
8 What happens to an offboarded engineerโ€™s active AWS CLI session when you disable them in Okta/Entra ID? Existing issued AWS STS session tokens remain active until their TTL expires unless an inline Deny policy with aws:TokenIssueTime is injected.
9 What is Azure Virtual WAN (vWAN)? Microsoftโ€™s managed hub-and-spoke networking service equivalent to AWS Transit Gateway + Cloud WAN.
10 How do you enforce Just-In-Time (JIT) temporary privileged access for production break-glass scenarios? Use Azure Entra ID Privileged Identity Management (PIM) or AWS IAM Identity Center temporary elevated access requiring ticket approval and MFA.
# One-Line Question One-Line Answer
1 How does AWS PrivateLink differ from VPC Peering or Transit Gateway? PrivateLink is unidirectional L4 TCP exposure of a single service via an ENI without exposing the entire VPC CIDR or requiring non-overlapping IPs.
2 Can two VPCs with identical, overlapping 10.0.0.0/16 CIDRs communicate securely? Yes, by exposing the target application behind a Network Load Balancer (NLB) and connecting via an AWS PrivateLink Interface Endpoint.
3 What protocol dynamically exchanges routing tables between on-premises datacenters and AWS Direct Connect? BGP (Border Gateway Protocol) over TCP Port 179.
4 In what order does a router evaluate BGP attributes to select the best outbound path? Weight (highest, Cisco/AWS local) โ†’ Local Preference (highest, outbound) โ†’ AS_PATH (shortest) โ†’ MED (lowest, inbound).
5 How do you force AWS to prefer Direct Connect Link A over Link B for traffic returning to your datacenter? Advertise a shorter AS_PATH (avoiding AS-Path prepending) or a lower MED (Multi-Exit Discriminator) on Link A.
6 What protocol is paired with BGP to detect physical link failures in under 300ms instead of waiting 90s for BGP Hold timers? BFD (Bidirectional Forwarding Detection).
7 What is the difference between a Route 53 Inbound Resolver Endpoint and an Outbound Resolver Endpoint? Inbound allows on-prem DNS servers to resolve AWS Private Hosted Zones; Outbound forwards queries from AWS VPCs to on-prem DNS servers.
8 What causes a fatal DNS query loop between Route 53 Outbound Endpoints and on-premises BIND/Windows DNS servers? Forwarding corp.internal from AWS to on-prem while on-prem simultaneously forwards corp.internal back to the AWS Inbound Endpoint.
9 How do you connect AWS Direct Connect and Google Cloud Partner Interconnect privately without physical datacenter routers? Provision a software-defined virtual router (MCR) on Megaport or Equinix Fabric to bridge BGP sessions across both cloud fabrics at Layer 3.
10 Why does enabling Private DNS on a VPC Interface Endpoint work seamlessly for zero-code SDK migration? It creates a hidden Route 53 Private Hosted Zone that resolves the public service FQDN (e.g., sqs.us-east-1.amazonaws.com) directly to the local ENIโ€™s private IP.
# One-Line Question One-Line Answer
1 How does AWS Cloud WAN improve upon managing multi-region Transit Gateways manually? It replaces manual inter-region TGW peering and static route tables with a single declarative Global Core Network Policy written in JSON.
2 Why are enterprises adopting Dual-Stack IPv6 (56 VPC prefixes) for Kubernetes clusters? Large enterprises exhaust RFC 1918 private IPv4 space (10.0.0.0/8) when running thousands of high-density Kubernetes pods per node.
3 What device enables IPv6-only Kubernetes pods to communicate with legacy IPv4-only external APIs? NAT64 paired with DNS64 (which synthesizes IPv6 64:ff9b::/96 records from IPv4 A records).
4 What is an Egress-Only Internet Gateway (EOIGW) in AWS? The IPv6 equivalent of a NAT Gatewayโ€”it allows outbound IPv6 traffic from private subnets while blocking unsolicited inbound IPv6 connections.
5 Why doesnโ€™t IPv6 require NAT (Network Address Translation) for internet routing? Every IPv6 address assigned in an AWS VPC is a globally unique public unicast address (GUA), relying on stateful firewalls (EOIGW/Security Groups) rather than NAT.
6 What extra metadata fields must be enabled in custom VPC Flow Logs (v5) to debug Transit Gateway drops? pkt-srcaddr, pkt-dstaddr, flow-direction, traffic-path, and sublocation-id.
7 Why are standard VPC Flow Logs (srcaddr, dstaddr) insufficient when packets traverse an NLB or Transit Gateway? Standard fields only show the intermediate NLB/TGW ENI IP addresses, whereas pkt-srcaddr and pkt-dstaddr reveal the original pod and final destination IPs.
8 What tcpdump flag prevents DNS reverse-lookup delays and displays numeric ports immediately during an outage? -nn (e.g., tcpdump -i any -nn port 443).
9 What do the [S], [S.], [P.], and [R] flags mean in tcpdump output? [S] = SYN, [S.] = SYN-ACK, [P.] = PUSH-ACK (data payload), and [R] = RST (Connection Reset).
10 What AWS feature clones raw network packets from an EC2 instanceโ€™s ENI and streams them via VXLAN to an IDS sensor? AWS VPC Traffic Mirroring.
# One-Line Question One-Line Answer
1 How do you test if an IAM policy or SCP blocks an action without actually executing the destructive API call? Run aws iam simulate-principal-policy --policy-source-arn <arn> --action-names <action>.
2 What AWS CLI tool traces the network topology path between two ENIs and identifies the exact blocking Security Group or Route Table? AWS VPC Reachability Analyzer (aws ec2 start-network-insights-analysis).
3 How do you query VPC Flow Logs at scale across 50 accounts without paying for expensive CloudWatch Logs ingestion? Deliver VPC Flow Logs directly to S3 in Apache Parquet format with hourly partitions and query via Amazon Athena.
4 What dig flag traces DNS delegation step-by-step from the root . nameservers down to the authoritative server? dig +trace <domain>.
5 How do you verify the exact MTU size supported along a network path between two Linux hosts? Run ping -M do -s 1472 <dest-ip> (1472 payload + 20 IP + 8 ICMP = 1500 bytes).
๐Ÿ—๏ธ Phase 2: IaC & Policy-as-Code 70 Rapid Q&As

Lesson 1: Terraform State & Modules

# One-Line Question One-Line Answer
1 Why does Terraform require a state file (terraform.tfstate) unlike Kubernetes manifests? Cloud APIs lack a built-in declarative reconciliation loop, so Terraform uses .tfstate to map HCL code blocks to real-world cloud resource IDs.
2 Why must terraform.tfstate never be committed to Git? It stores sensitive resource attributes and secrets in unencrypted plaintext JSON and causes merge conflicts when teams run parallel applies.
3 What are the two AWS services used together for an enterprise Terraform Remote Backend? Amazon S3 (encrypted state storage with versioning) and Amazon DynamoDB (distributed mutual-exclusion locking via LockID).
4 How does Terraform build its execution order across hundreds of resources? It constructs a Directed Acyclic Graph (DAG) from implicit resource references and explicit depends_on edges, executing independent nodes in parallel (default parallelism = 10).
5 What is the difference between count and for_each in Terraform resource blocks? count indexes resources by numeric list position ([0], [1]), whereas for_each indexes resources by stable map keys (["az-a"]).
6 Why is for_each strongly preferred over count for production subnets or IAM users? Deleting the first item in a count list shifts all subsequent indices down by 1, causing Terraform to destroy and recreate every remaining resource.
7 How do you pin a Terraform module to an immutable release version in Git? Reference the module source with a Git tag or commit SHA: source = "git::https://github.com/org/vpc.git?ref=v1.4.2".
8 What command upgrades provider plugins while respecting version constraints in .terraform.lock.hcl? terraform init -upgrade.
9 Why must .terraform.lock.hcl always be committed to version control? It pins the exact cryptographic SHA-256 hashes of provider binaries so every developer and CI runner executes identical provider code.
10 What replaces the deprecated terraform taint command to force recreating a single broken resource? terraform apply -replace="aws_instance.web[0]".
# One-Line Question One-Line Answer
1 What is the architectural difference between Checkov and Open Policy Agent (OPA)? Checkov performs fast static graph analysis on raw HCL files, while OPA (Conftest) evaluates complex Rego policies against the compiled terraform plan JSON.
2 How do you export a binary terraform plan into a machine-readable JSON file for OPA evaluation? Run terraform plan -out=tfplan.binary && terraform show -json tfplan.binary > tfplan.json.
3 What happens in OPA Rego when a rule references a JSON field that does not exist in input? The expression evaluates to undefined (not false), halting execution of the current rule body immediately.
4 How do you prevent the โ€œRego Silent Bypass Trapโ€ when writing allow/deny rules? Always declare an explicit default (default allow = false) and write deny[msg] rules that accumulate violation strings.
5 In Rego, how is logical AND vs logical OR written? Logical AND is lines inside the same rule body; Logical OR is defining multiple rules with the exact same name.
6 What symbol acts as an existential quantifier (โ€œfor some index iโ€) when iterating over arrays in Rego? The underscore wildcard [_] (e.g., rc := input.resource_changes[_]).
7 How do you document an approved exception to a Checkov rule directly inside a Terraform resource block? Add an inline comment with a mandatory justification: #checkov:skip=CKV_AWS_20:Public static assets bucket approved by SecOps-104.
8 How do you unit-test Rego policies in a CI/CD pipeline before deploying them? Create *_test.rego files with mocked with input as {...} payloads and run opa test ./policy -v.
9 What CLI tool runs OPA Rego policies specifically against structured configuration files (Terraform JSON, K8s YAML, Dockerfiles)? Conftest (conftest test tfplan.json -p policy/).
10 What property in tfplan.json tells you whether Terraform intends to create, update, or delete a resource? input.resource_changes[_].change.actions.
# One-Line Question One-Line Answer
1 Why do enterprises split a single monolithic Terraform state file into smaller layered states (โ€œTerraservicesโ€)? Monolithic states take 30+ minutes to refresh, hit AWS API rate limits, and risk wiping out the entire companyโ€™s infrastructure from a single bad apply.
2 What core problem does Terragrunt solve when managing 50+ AWS accounts and environments? It keeps backend configurations, provider blocks, and module invocations DRY (Donโ€™t Repeat Yourself) via hierarchical terragrunt.hcl inheritance.
3 How does a child terragrunt.hcl inherit the S3 remote state configuration from the root directory? Using include "root" { path = find_in_parent_folders() }.
4 What dynamically generates provider.tf and backend.tf files on the fly before Terragrunt calls Terraform? The Terragrunt generate block (generate "provider" { path = "provider.tf" ... }).
5 How does Terragrunt pass the vpc_id output from the vpc module into the eks module without hardcoding terraform_remote_state data sources? Using a dependency "vpc" { config_path = "../vpc" } block and referencing dependency.vpc.outputs.vpc_id.
6 What causes a Terragrunt Cold-Boot Deadlock when running terragrunt run-all plan on a brand-new environment? The eks module tries to read dependency.vpc.outputs.vpc_id, which does not exist yet because the vpc module has never been applied.
7 How do you solve the Terragrunt Cold-Boot dependency failure during plan and validate? Define mock_outputs = { vpc_id = "vpc-mock123" } and mock_outputs_allowed_terraform_commands = ["validate", "plan"] inside the dependency block.
8 What command executes terraform apply across an entire directory tree of modules in topological dependency order? terragrunt run-all apply.
9 How do you limit Terragrunt concurrency so run-all does not trigger AWS API throttling? Pass --terragrunt-parallelism 4.
10 How do you deploy the exact same Terraform module version to dev, staging, and prod while promoting versions safely? Override the module source version tag (?ref=v1.2.0 in dev, ?ref=v1.1.0 in prod) in each environmentโ€™s env.hcl.
# One-Line Question One-Line Answer
1 What is the cryptographic mechanism Vault uses by default to split its Master Key into N unseal shards? Shamirโ€™s Secret Sharing Algorithm (requiring a threshold of K shards out of N to reconstruct the Master Key in RAM).
2 How do production Vault clusters unseal automatically upon pod restart without human operators typing unseal keys? Cloud KMS Auto-Unsealโ€”Vault encrypts its Master Key using an AWS KMS / GCP Cloud KMS key and decrypts it via IAM role identity on boot.
3 What storage backend replaced external Consul clusters as the modern standard for HA Vault deployments? Integrated Storage (Raft), which embeds Raft consensus directly inside Vault nodes across 3 or 5 AZs.
4 What is the difference between Vault Static KV-v2 secrets and Vault Dynamic Secrets? Static secrets exist until manually rotated, whereas Dynamic Secrets are generated on-demand with a short lease TTL and automatically revoked upon expiry.
5 How does a CI/CD machine or VM authenticate to Vault using the AppRole auth method? By presenting a RoleID (like a username, baked into config) and a single-use wrapped SecretID (like a password, delivered securely at runtime).
6 What is Response Wrapping (-wrap-ttl=60s) in HashiCorp Vault? Vault stores the secret inside a single-use cubbyhole token that can only be unwrapped once; if an attacker intercepts and unwraps it first, the real appโ€™s unwrap fails and triggers a security alert.
7 How does Vault verify a Kubernetes podโ€™s identity during Kubernetes Auth login? The pod sends its ServiceAccount JWT to Vault, and Vault verifies it against the Kubernetes API Serverโ€™s TokenReview endpoint.
8 What happens to active PostgreSQL connections if a Vault dynamic credential lease expires? Vault connects to PostgreSQL and executes DROP ROLE / REVOKE, terminating or invalidating the temporary database user.
9 What is Vault Transit Secrets Engine (โ€œEncryption as a Serviceโ€)? Microservices send plaintext to Vaultโ€™s /transit/encrypt API and receive ciphertext back without ever storing or handling encryption keys in application memory.
10 Where must Vault audit logs be sent, and what happens if all audit log devices fail to write? Audit logs go to immutable storage/SIEM; if all configured audit sinks fail to write, Vault immediately halts and refuses to serve any API requests to guarantee zero unaudited access.
# One-Line Question One-Line Answer
1 What happens if you rename resource "aws_db_instance" "db" to "primary_db" in HCL without state migration? Terraform plans to destroy the existing production database and create a brand-new empty one!
2 Why are declarative moved { from = ... to = ... } blocks superior to running terraform state mv manually? moved blocks are committed to Git, reviewed in PRs, and execute automatically across dev, staging, and prod during standard CI/CD pipelines.
3 What declarative block added in Terraform 1.5+ imports existing brownfield cloud resources without running terraform import CLI commands? The import { to = aws_s3_bucket.logs, id = "my-bucket-name" } block.
4 What CLI flag paired with import blocks automatically generates the matching HCL resource code from live cloud infrastructure? terraform plan -generate-config-out=generated_resources.tf.
5 What declarative block removes a resource from Terraform state tracking without destroying the real cloud resource? The removed { from = aws_s3_bucket.legacy, lifecycle { destroy = false } } block (replaces terraform state rm).
6 What exit code does terraform plan -detailed-exitcode return when infrastructure drift is detected? Exit code 0 = No changes; Exit code 1 = Error; Exit code 2 = Drift / Changes detected.
7 What is the difference between terraform plan -refresh-only and terraform plan? -refresh-only only compares live cloud APIs against .tfstate to detect out-of-band console changes, ignoring unapplied HCL code modifications.
8 When should you use lifecycle { ignore_changes = [tags] }? When an external controller (like Kubernetes Karpenter, AWS Auto Scaling, or a Lambda) dynamically mutates specific attributes outside of Terraform.
9 What lifecycle meta-argument causes terraform plan to abort with a hard error if any change would destroy a critical database? lifecycle { prevent_destroy = true }.
10 How does lifecycle { create_before_destroy = true } prevent downtime during resource replacement? It provisions the new replacement resource first and updates downstream references before destroying the old resource.
# One-Line Question One-Line Answer
1 What HCL construct dynamically generates multiple nested configuration blocks (such as ingress rules inside a aws_security_group)? The dynamic "ingress" { for_each = var.rules ... content { ... } } block.
2 How do you write a for expression in HCL that filters a map to include only production subnets? [for k, v in var.subnets : v.id if v.env == "prod"].
3 What is the HCL Splat operator ([*]), and what does aws_instance.web[*].private_ip return? It iterates over a list of resources and returns a flat list of all their private_ip attributes.
4 What HCL function safely accesses deeply nested map keys and returns a fallback default instead of throwing an error if a key is missing? try(var.config.database.port, 5432).
5 What HCL function calculates CIDR subnet ranges dynamically from a base VPC CIDR? cidrsubnet(prefix, newbits, netnum) (e.g., cidrsubnet("10.0.0.0/16", 8, 1) โ†’ "10.0.1.0/24").
6 How do you validate input variables in Terraform so invalid CIDRs or instance types fail immediately before API calls? Add a validation { condition = can(cidrnetmask(var.cidr)) error_message = "..." } block inside the variable definition.
7 What is the difference between precondition and postcondition inside a resource lifecycle block? precondition validates assumptions before creating a resource; postcondition verifies live attributes of the resource after creation (e.g., ensuring encrypted == true).
8 What built-in Terraform testing framework (introduced in v1.6) runs .tftest.hcl unit and integration tests natively? terraform test (using run and assert blocks with mock_provider).
9 What HCL function merges multiple maps of AWS tags together, where later maps override earlier keys? merge(local.common_tags, var.extra_tags).
10 What HCL function converts a list of strings into an unordered set required by for_each? toset(var.subnet_ids).
# One-Line Question One-Line Answer
1 Why must boto3.client('s3') be initialized outside the lambda_handler(event, context) function in AWS Lambda? Code outside the handler runs once during container Cold Start and is reused across subsequent Warm Invocations, avoiding expensive TLS handshake overhead on every call.
2 Why does a basic s3.list_objects_v2() or ec2.describe_instances() call silently miss resources in large production accounts? AWS APIs paginate responses (typically returning max 1,000 items); you must use Boto3 Paginators (client.get_paginator().paginate()) to fetch all pages.
3 How do you prevent Boto3 scripts from crashing when AWS APIs return ThrottlingException (HTTP 429)? Configure botocore.config.Config(retries={'max_attempts': 10, 'mode': 'adaptive'}), which implements client-side token-bucket rate limiting and exponential backoff with jitter.
4 What is the primary danger of executing AWS Lambda inside private VPC subnets without capacity planning? Each unique (SecurityGroup, Subnet) combination requires a Hyperplane ENI; if subnets use small /28 CIDRs, Lambda can exhaust available subnet IPs during scaling bursts.
5 Why must every production Lambda function interacting with SQS or EventBridge have an idempotency key check? Cloud event queues guarantee at-least-once delivery, meaning the exact same event payload can be delivered twice and must not execute duplicate state changes.
6 How does AWS Powertools for Python enforce idempotency in Lambda functions? Using the @idempotent decorator backed by a DynamoDB table that locks and caches results by payload hash.
7 What happens to messages that repeatedly fail processing in an SQS-triggered Lambda function? After exceeding maxReceiveCount, SQS moves the poison message to a Dead-Letter Queue (DLQ) so it stops blocking the main queue.
8 Why should you use ReportBatchItemFailures when processing SQS batches in Lambda? If 1 message out of a batch of 10 fails, returning only the failed itemIdentifier prevents SQS from reprocessing the 9 messages that already succeeded.
9 What Python standard library module should be used instead of print() in Lambda to enable CloudWatch Logs Insights JSON queries? Structured JSON logging via logging (or aws_lambda_powertools.Logger) including correlation_id and aws_request_id.
10 How does a Python AWS Config Custom Rule remediation Lambda fix non-compliant resources in real time? EventBridge captures the ComplianceChangeNotification, triggers the Lambda, and the Lambda uses boto3 to apply the missing encryption or security policy automatically.
๐Ÿ›ก๏ธ Phase 3: DevSecOps & Supply Chain 32 Rapid Q&As

Lesson 1: Image Security & Cosign

# One-Line Question One-Line Answer
1 Why is deploying container tags like nginx:latest or app:1.2 dangerous in production? Image tags are mutable pointers that can be overwritten; production deployments must pin immutable @sha256:<digest> hashes.
2 What Trivy flag breaks a CI/CD pipeline only when a CRITICAL vulnerability actually has a published vendor patch? trivy image --exit-code 1 --severity CRITICAL --ignore-unfixed <image>.
3 What are the two standard formats for a Software Bill of Materials (SBOM)? CycloneDX (cyclonedx-json) and SPDX (spdx-json).
4 How does Sigstore Cosign Keyless Signing sign images without storing long-lived private keys? It exchanges the CI runnerโ€™s short-lived OIDC token with Fulcio (CA) for a 10-minute certificate and records the signature proof in Rekor (transparency log).
5 Why does Cosign verification fail in air-gapped Kubernetes clusters if Rekor transparency log verification is left at default? Kyverno/Cosign tries to reach the public internet (rekor.sigstore.dev) and times out; air-gapped clusters require offline TUF roots or KMS-backed signing keys.
6 What Kyverno rule type verifies Cosign image signatures and mutates tags into immutable @sha256 digests at admission time? The verifyImages block inside a ClusterPolicy.
7 Where does Cosign store image signatures and SBOM attestations? Directly inside the OCI container registry (ECR/GHCR) as attached .sig and .att artifact layers alongside the container image.
8 What command attaches a CycloneDX SBOM as a cryptographically signed in-toto attestation to an OCI image? cosign attest --predicate sbom.json --type cyclonedx <image@sha256:digest>.
# One-Line Question One-Line Answer
1 What mandatory permission must be set in a GitHub Actions workflow YAML to request an OIDC JWT token? permissions: id-token: write.
2 What URL is configured as the OIDC Identity Provider URL in AWS IAM for GitHub Actions? https://token.actions.githubusercontent.com with Audience sts.amazonaws.com.
3 What AWS STS API action does aws-actions/configure-aws-credentials call under the hood when using OIDC? sts:AssumeRoleWithWebIdentity.
4 Why is setting the IAM Trust Policy condition to "token.actions.githubusercontent.com:sub": "repo:my-org/*" a critical security vulnerability? The wildcard * allows any feature branch or pull request in any repository across the entire GitHub organization to assume your production IAM Role.
5 How do you lock an AWS IAM Role Trust Policy so only the main branch of payment-service can assume it? Use StringEquals on "token.actions.githubusercontent.com:sub": "repo:my-org/payment-service:ref:refs/heads/main".
6 Why must "token.actions.githubusercontent.com:aud": "sts.amazonaws.com" always be verified in the IAM Trust Policy? It prevents confused-deputy token replay attacks where a JWT minted for another audience is replayed against AWS STS.
7 What is the default validity duration of temporary AWS credentials returned by AssumeRoleWithWebIdentity in CI/CD? 1 hour (3600 seconds), configurable down to 15 minutes.
8 Why must third-party GitHub Actions in enterprise workflows be pinned to a 40-character Git commit SHA (uses: actions/checkout@b4ffde6...) instead of @v4? Version tags (@v4) are mutable and can be hijacked if an upstream maintainerโ€™s account is compromised (Supply Chain Tag Poisoning).
# One-Line Question One-Line Answer
1 Why does deleting a leaked secret in a new Git commit fail to secure the repository? The secret remains permanently readable inside the .git/objects history of earlier commits and can be extracted with git log -p.
2 What tool scans Git commit history and pre-commit staged changes using regex and Shannon entropy analysis to block leaked keys? Gitleaks (gitleaks protect --staged -v).
3 Are standard Kubernetes Secret objects encrypted by default in etcd? No, they are merely Base64-encoded strings unless KMS envelope encryption at rest is explicitly enabled on the EKS/K8s control plane.
4 What Kubernetes operator synchronizes external cloud secrets (AWS Secrets Manager / Vault) into native Kubernetes Secret objects? External Secrets Operator (ESO).
5 What is the difference between a SecretStore and a ClusterSecretStore in ESO? SecretStore is namespace-scoped, while ClusterSecretStore is cluster-wide and can project secrets into multiple namespaces with RBAC conditions.
6 Why did setting refreshInterval: 1s across 500 ExternalSecret objects cause an AWS outage and surprise $4,000 bill? It generated 500 GetSecretValue API calls per second, hitting AWS Secrets Manager account rate limits (ThrottlingException) and API call billing.
7 What is the recommended production refreshInterval for ESO ExternalSecret resources? 1h (or event-driven refresh), combined with Reloader (reloader.stakater.com/auto: "true") to roll pods only when secret SHA hashes actually change.
8 How does ESO authenticate to AWS Secrets Manager without storing static AWS keys in the cluster? Via EKS IRSA (IAM Roles for Service Accounts) or EKS Pod Identity attached to the ESO controller ServiceAccount.
# One-Line Question One-Line Answer
1 Why does single-user database password rotation cause application downtime? Between the millisecond the password changes in RDS and the minute Kubernetes pods reload the secret, active pods fail authentication (FATAL: password authentication failed).
2 How does the Alternating 2-User (Dual-Account) Rotation Strategy achieve zero-downtime secret rotation? User A remains active serving live traffic while User Bโ€™s password is rotated in the background; once updated, new pods switch to User B while User A stays valid during drain.
3 What are the 4 sequential steps executed by an AWS Secrets Manager rotation Lambda function? createSecret โ†’ setSecret โ†’ testSecret โ†’ finishSecret.
4 What are the two staging labels used by AWS Secrets Manager during rotation? AWSCURRENT (active credentials), AWSPENDING (new secret being tested), and AWSPREVIOUS (last known good secret for instant rollback).
5 What does SLSA (Supply-chain Levels for Software Artifacts) protect against? Tampering of source code, build scripts, dependencies, or artifacts at any stage between a developerโ€™s commit and production deployment.
6 What is the defining requirement of SLSA Level 3 provenance? Builds must run on hardened, isolated, ephemeral build environments that generate unforgeable cryptographic provenance attestations signed by the control plane.
7 What file format does SLSA use to describe build inputs, builder identity, and output artifact digests? in-toto Attestation Framework (https://slsa.dev/provenance/v1 JSON predicate).
8 How do you verify in Kubernetes that an image was built exclusively by your official GitHub Actions workflow and not on a developerโ€™s laptop? Configure Kyverno verifyImages to validate the SLSA provenance attestation issuer (token.actions.githubusercontent.com) and workflow repository URI.
๐Ÿ›ก๏ธ Phase 3B: Compliance & Patching 29 Rapid Q&As

Lesson 1: Immutable Golden AMIs

# One-Line Question One-Line Answer
1 Why is mutable infrastructure (running apt upgrade or Ansible via SSH on live production servers) an anti-pattern? It causes Configuration Drift (โ€œsnowflake serversโ€), slow auto-scaling boot times, and irreproducible outages when package mirrors fail mid-scale.
2 What is the core workflow of Immutable Infrastructure using HashiCorp Packer? Packer boots a temporary builder VM, applies OS hardening and CIS benchmarks, strips SSH keys, snapshots a Golden AMI, and terminates the builder VM.
3 Why must /etc/machine-id be truncated (truncate -s 0 /etc/machine-id) before Packer snapshots a Golden AMI? If cloned nodes share the same /etc/machine-id, DHCP servers assign duplicate IPs, systemd journald merges logs, and Kubernetes node registration collides.
4 What is CIS (Center for Internet Security) Level 1 vs Level 2 OS hardening? Level 1 applies safe production security defaults with zero performance impact; Level 2 applies extreme defense-in-depth (e.g., strict SELinux/FIPS) for regulated workloads.
5 Which kernel sysctl parameters must be baked into a Kubernetes worker Golden AMI for CNI networking to function? net.bridge.bridge-nf-call-iptables = 1 and net.ipv4.ip_forward = 1.
6 What tool runs automated compliance tests (verifying file permissions, disabled kernel modules, and sysctls) inside Packer before creating the AMI? OpenSCAP, Chef InSpec, or Goss.
7 Why do we disable swap (swapoff -a and remove swap from /etc/fstab) on Kubernetes worker Golden AMIs? Kubernetes QoS memory guarantees and deterministic OOM eviction require predictable RAM limits without kernel disk swapping latency.
8 How do you distribute a newly baked Golden AMI across 50 AWS accounts in an AWS Organization without copying snapshots? Share the AMI via AWS RAM / Organization AMI sharing and grant the Organization principal kms:Decrypt permission on the AMIโ€™s EBS Customer Managed KMS Key.
# One-Line Question One-Line Answer
1 How do you patch 500 immutable EC2 instances when a critical Linux kernel CVE is announced? Bake a new Golden AMI, update the AWS Launch Template $Latest version via Terraform, and trigger an Auto Scaling Group (ASG) Instance Refresh.
2 What does MinHealthyPercentage = 90 and MaxHealthyPercentage = 110 mean in an ASG Instance Refresh? ASG surges capacity up to 110% by launching new patched instances before terminating up to 10% of old instances at a time, preserving 100% capacity.
3 Why is setting health_check_type = "EC2" on a production Auto Scaling Group a critical reliability trap? EC2 health checks only verify hypervisor power state; ELB health checks must be used so ASG waits until the application passes HTTP /healthz before terminating the next batch.
4 What mechanism pauses an ASG Instance Refresh automatically if a newly patched AMI starts throwing HTTP 5xx errors? ASG CloudWatch Alarm Auto-Rollbackโ€”triggers an immediate rollback to the previous Launch Template version when the error alarm breaches.
5 What happens when an ASG terminates a Kubernetes worker node without an EC2 Lifecycle Hook? The instance is killed abruptly, severing active user connections and leaving stateful volumes uncleanly attached.
6 How does an ASG Terminate Lifecycle Hook achieve zero-downtime Kubernetes node replacement? It pauses instance termination in Terminating:Wait, sends an EventBridge notification to AWS Node Termination Handler / Karpenter, which runs kubectl cordon and kubectl drain before completing the hook.
7 How does Karpenter Drift & Expiration (ttlSecondsAfterEmpty / expireAfter) automate node patching without ASGs? When the underlying EKS AMI SSM Parameter updates, Karpenter marks existing nodes as Drifted and automatically cordons, drains, and replaces them respecting Disruption Budgets.
8 What is the difference between Canary Baking and Blue/Green Cluster Upgrades for fleet patching? Canary Baking replaces 5% of nodes and monitors metrics for 1 hour before rolling out to 95%, whereas Blue/Green provisions an entirely separate node group and shifts pods across.
# One-Line Question One-Line Answer
1 What is the difference between Preventive Controls and Detective Controls in cloud compliance? Preventive blocks violations before creation (SCPs, OPA/Kyverno in CI); Detective audits live running state to catch console drift or newly disclosed CVEs (AWS Config, Falco).
2 How does AWS Config Auto-Remediation fix an S3 bucket that an admin accidentally made public in the console? AWS Config detects s3-bucket-public-read-prohibited is NON_COMPLIANT and invokes an AWS Systems Manager (SSM) Automation Document (AWS-DisableS3BucketPublicReadWrite) within seconds.
3 What Kubernetes resource prevents automated node patching (kubectl drain) from terminating all replicas of a microservice simultaneously? PodDisruptionBudget (PDB).
4 What is the difference between minAvailable and maxUnavailable in a PodDisruptionBudget? minAvailable enforces the minimum number/percentage of pods that must remain running; maxUnavailable enforces the maximum number of pods that can be offline at once.
5 Why does a Deployment with replicas: 2 and a PDB with minAvailable: 2 (or maxUnavailable: 0) cause kubectl drain to hang forever? The Eviction API refuses to evict any pod because dropping from 2 to 1 violates minAvailable: 2, creating an Infinite Node Drain Deadlock.
6 Why is setting maxUnavailable: 1 (or 25%) generally safer than minAvailable: 100% for production deployments? maxUnavailable: 1 always permits at least one pod at a time to be gracefully evicted during node drains regardless of how low HPA scales the replica count (provided replicas >= 2).
7 Does a PodDisruptionBudget prevent pods from going down when a physical EC2 instance suffers an involuntary hardware crash? No, PDBs only govern voluntary disruptions initiated via the Kubernetes Eviction API (kubectl drain, Karpenter consolidation, Cluster Autoscaler scale-down).
8 How do you break a stuck kubectl drain caused by a misconfigured PDB during an emergency security patch? Patch or delete the offending PDB (kubectl patch pdb <name> -p '{"spec":{"maxUnavailable":1}}'), or use kubectl drain --disable-eviction in a break-glass emergency.
# One-Line Question One-Line Answer
1 What command validates a Packer HCL templateโ€™s syntax and variable definitions before running a cloud build? packer validate . (after packer init .).
2 How do you check which PodDisruptionBudgets in a cluster currently have ALLOWED DISRUPTIONS = 0 and will block node drains? Run kubectl get pdb -A and look for any PDB where ALLOWED DISRUPTIONS is 0.
3 How do you verify a Cosign keyless signature on a container image using the command line? cosign verify --certificate-identity <workflow-uri> --certificate-oidc-issuer https://token.actions.githubusercontent.com <image@sha256:digest>.
4 How do you force External Secrets Operator to immediately re-sync an ExternalSecret from AWS Secrets Manager without waiting for refreshInterval? Annotate the resource with a current timestamp: kubectl annotate es <name> force-sync=$(date +%s) --overwrite.
5 How do you monitor the live percentage progress and status of an AWS Auto Scaling Group Instance Refresh? Run aws autoscaling describe-instance-refreshes --auto-scaling-group-name <asg-name>.
โ˜ธ๏ธ Phase 4: Kubernetes & Scaling 55 Rapid Q&As

Lesson 1: EKS Architecture & IRSA

# One-Line Question One-Line Answer
1 Which control plane components are managed by AWS in an Amazon EKS cluster? Multi-AZ kube-apiserver, etcd quorum, kube-scheduler, kube-controller-manager, and cloud-controller-manager.
2 How does the AWS VPC CNI (aws-node DaemonSet) differ from overlay CNIs like Flannel or Calico VXLAN? It allocates real first-class AWS VPC IPv4 addresses from the subnet directly to pods via secondary IPs on Elastic Network Interfaces (ENIs), eliminating VXLAN encapsulation overhead.
3 Why does an m5.large EC2 instance stop scheduling new pods at 29 pods even when CPU and RAM are 70% idle? Each m5.large supports a maximum of 3 ENIs ร— 10 IPs per ENI = 30 IPs (minus 1 for the node primary IP = 29 pods max).
4 How does AWS VPC CNI Prefix Delegation (ENABLE_PREFIX_DELEGATION = "true") solve pod density limits on EKS? Instead of assigning single /32 secondary IPs to ENIs slots, AWS assigns /28 IPv4 prefixes (16 IPs per slot), increasing an m5.large nodeโ€™s capacity from 29 to 110 pods.
5 Why must production EKS worker nodes block access to EC2 Instance Metadata Service v1 (IMDSv1) and enforce IMDSv2 with http_put_response_hop_limit = 1? A hop limit of 1 ensures IMDS packets cannot be forwarded across the containerโ€™s veth router hop, preventing SSRF attacks in pods from stealing the Worker Nodeโ€™s IAM Role credentials.
6 How does IRSA (IAM Roles for Service Accounts) inject AWS credentials into a specific pod? The EKS Pod Identity Webhook mutates the pod to mount a projected OIDC JWT token and injects AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE environment variables.
7 What happens if an EBS PersistentVolumeClaim uses a StorageClass with volumeBindingMode: Immediate in a 3-AZ EKS cluster? The EBS volume is provisioned immediately in a random AZ (e.g., AZ-a) before the pod is scheduled; if the pod is later scheduled to AZ-b, it hangs in ContainerCreating forever.
8 Why is volumeBindingMode: WaitForFirstConsumer mandatory on all production AWS EBS StorageClasses? It delays creating the physical EBS volume until kube-scheduler picks a specific worker node, guaranteeing the volume is created in the exact same Availability Zone as the pod.
9 When must you use the Amazon EFS CSI Driver instead of the Amazon EBS CSI Driver? When multiple pods across different worker nodes and Availability Zones must mount the exact same volume concurrently using ReadWriteMany (RWX).
10 How does EKS Pod Identity (introduced at re:Invent 2023) simplify IRSA at multi-cluster scale? It uses an on-node agent (eks-pod-identity-agent on 169.254.170.23) and IAM Role Session Tags without requiring cluster-specific OIDC Provider thumbprints in IAM Trust Policies.
# One-Line Question One-Line Answer
1 Why does Karpenter launch replacement worker nodes in 30โ€“45 seconds while Cluster Autoscaler takes 3โ€“5 minutes? Karpenter bypasses AWS Auto Scaling Groups (ASGs) entirely and calls the EC2 Fleet API directly based on real-time pending pod resource requests and topology constraints.
2 What are the two core Custom Resource Definitions (CRDs) used to configure Karpenter v1? NodePool (defines pod scheduling constraints, taints, and disruption policies) and EC2NodeClass (defines AWS-specific AMI, subnets, security groups, and EBS block mappings).
3 Why must you specify a wide variety of instance families (m5, m6i, c5, c6i, r5) in a Karpenter Spot NodePool? Spot diversification across 15+ instance pools allows EC2 Fleet price-capacity-optimized allocation to pick the deepest capacity pool with the lowest interruption rate.
4 How does Karpenter allow a single NodePool to prioritize Spot instances while automatically falling back to On-Demand if Spot capacity is exhausted? Include both "spot" and "on-demand" in karpenter.sh/capacity-type; Karpenter always attempts cheaper Spot first and automatically falls back to On-Demand if InsufficientInstanceCapacity occurs.
5 What AWS infrastructure is required for Karpenter to gracefully drain Spot nodes before AWS reclaims them? An Amazon EventBridge rule capturing EC2 Spot Instance Interruption Warnings (2-minute notice) and Rebalance Recommendations, routing them into an Amazon SQS queue polled by Karpenter.
6 What annotation on a critical stateful pod prevents Karpenter Consolidation from voluntarily evicting that pod to save money? karpenter.sh/do-not-disrupt: "true".
7 How do you prevent Karpenter from consolidating nodes during peak daytime business traffic while allowing aggressive cost consolidation overnight? Configure Disruption Budgets with Cron Schedules inside spec.disruption.budgets (e.g., nodes: "0" during peak business hours).
8 What Kubernetes construct ensures replicas of a deployment are distributed evenly across Availability Zones (maxSkew: 1) so a single Spot AZ reclamation doesnโ€™t take down the service? topologySpreadConstraints keyed on topology.kubernetes.io/zone and kubernetes.io/hostname.
# One-Line Question One-Line Answer
1 What is the fundamental architectural difference between Helm and Kustomize? Helm is a Go-templating package manager with release lifecycle history ({{ .Values.x }}), while Kustomize is a template-free YAML overlay patching engine built natively into kubectl -k.
2 Where does Helm v3 store its release state history inside a Kubernetes cluster? As compressed, double-Base64-encoded Kubernetes Secret objects labeled owner=helm inside the target release namespace.
3 What causes a Helm release to get stuck in pending-upgrade or pending-install, blocking all future deployments? A previous helm upgrade CI runner crashed or timed out before writing the final status back to the Helm release Secret.
4 How do you recover a Helm release stuck in pending-upgrade? Run helm history <release> and execute helm rollback <release> <last-deployed-revision> (or delete the stuck pending-upgrade Helm secret revision).
5 Why should helm upgrade --install --atomic be used safely with caution in automated pipelines? --atomic automatically rolls back the release if any pod fails readiness within --timeout, which is safe for stateless apps but can interrupt slow database migrations if timeout is too short.
6 What is the difference between a Strategic Merge Patch and a JSON 6902 Patch in Kustomize? Strategic Merge looks like a partial Kubernetes YAML manifest that merges by key/name, while JSON 6902 uses explicit surgical operations (op: replace, path: /spec/replicas, value: 5).
7 How does Kustomize trigger an automatic rolling restart of a Deployment whenever a ConfigMapโ€™s data changes? Kustomize configMapGenerator appends a 10-character content SHA hash suffix to the ConfigMap name (e.g., app-config-8g9f2k) and updates the Deploymentโ€™s volume reference automatically.
8 How do enterprise teams combine Helm and Kustomize for third-party charts like Prometheus or Ingress-Nginx? Render the upstream Helm chart and apply environment-specific Kustomize overlays on top (using helmCharts in kustomization.yaml or Helm --post-renderer).
# One-Line Question One-Line Answer
1 What is the difference between alb.ingress.kubernetes.io/target-type: instance and target-type: ip? instance routes ALB traffic to a NodePort on EC2 nodes (adding an extra kube-proxy hop), whereas ip routes directly from the ALB to the Podโ€™s VPC IP address via AWS VPC CNI.
2 Why does deleting or creating a standard AWS Ingress per microservice become expensive at 100+ services? By default, each Ingress object provisions a dedicated AWS Application Load Balancer (~$18โ€“$25/month base fee each).
3 How do you share a single AWS Application Load Balancer across 50 Ingress objects in different namespaces? Add the annotation alb.ingress.kubernetes.io/group.name: shared-prod-alb to all 50 Ingress manifests.
4 Why do users experience intermittent HTTP 502 / 504 Bad Gateway errors for 5โ€“10 seconds during a zero-downtime Kubernetes rolling update behind an AWS ALB? kubelet sends SIGTERM to the pod at the exact same millisecond the Endpoints controller notifies the ALB controller; the pod stops accepting connections before the ALB finishes removing its IP from the Target Group!
5 What two configurations completely eliminate the AWS ALB 502 Bad Gateway race condition during pod termination? Add a container lifecycle.preStop hook (exec: command: ["/bin/sleep", "15"]) and enable Pod Readiness Gates (elbv2.k8s.aws/pod-readiness-gate-inject: enabled) on the namespace.
6 What must always satisfy the inequality terminationGracePeriodSeconds > preStop_sleep + app_drain_time? The Podโ€™s terminationGracePeriodSeconds (e.g., 60s > 15s preStop + 30s ALB deregistration delay) so kubelet does not SIGKILL the pod while in-flight requests are draining.
7 How does the AWS Load Balancer Controller automatically attach an SSL/TLS certificate and a WAFv2 WebACL to an ALB? Via Ingress annotations alb.ingress.kubernetes.io/certificate-arn (ACM) and alb.ingress.kubernetes.io/wafv2-acl-arn.
8 How do you automatically sync Kubernetes Ingress and Service hostnames into AWS Route 53 Hosted Zones? Deploy ExternalDNS with an IRSA role granting route53:ChangeResourceRecordSets.
# One-Line Question One-Line Answer
1 What are the 4 core Prometheus metric types? Counter (monotonic cumulative count), Gauge (goes up/down), Histogram (server-side bucketed latency), and Summary (client-side quantiles).
2 What is the exact difference between PromQL rate(metric[5m]) and irate(metric[5m])? rate() calculates the average per-second rate across all samples in the 5m window (smooth, used for alerts); irate() uses only the last 2 samples (spiky, used for high-resolution graphs).
3 How do you calculate the 99th percentile (p99) HTTP latency from a Prometheus Histogram in PromQL? histogram_quantile(0.99, sum by (le, service) (rate(http_request_duration_seconds_bucket[5m]))).
4 Why must le (less-than-or-equal bucket label) always be included inside the sum by (le) clause of histogram_quantile()? Without le, PromQL collapses all histogram buckets into a single sum, destroying the bucket distribution required to interpolate quantiles.
5 What causes a Prometheus High-Cardinality Outage (OOMKill)? Adding unbounded labels like user_id, email, request_id, or full URL paths creates millions of unique time series in the TSDB in-memory Head Block.
6 How do you immediately drop a rogue high-cardinality label at scrape time before it enters Prometheus TSDB? Add a metric_relabel_configs rule with action: labeldrop and regex: "user_id" inside the ServiceMonitor or scrape_config.
7 Why canโ€™t a single standalone Prometheus server scale horizontally or store 2 years of enterprise metrics natively? Prometheus uses local node block storage without built-in distributed clustering; long-term HA requires Thanos or Grafana Mimir backed by Amazon S3.
8 How does Thanos Sidecar + Thanos Querier deduplicate metrics from two identical HA Prometheus replicas? Both Prometheus servers attach an external_labels: {replica: "a/b"} label, and thanos-query runs with --query.replica-label=replica to merge and deduplicate series on the fly.
# One-Line Question One-Line Answer
1 Why does a Kubernetes pod querying api.stripe.com generate 8 unnecessary DNS queries before resolving the real IP? Default /etc/resolv.conf sets ndots:5; since api.stripe.com has only 2 dots (2 < 5), glibc appends all 4 cluster search domains (in both IPv4 A and IPv6 AAAA) before querying the absolute domain.
2 What are the 3 fastest ways to eliminate CoreDNS ndots:5 amplification in high-throughput clusters? (1) Deploy NodeLocal DNSCache, (2) set dnsConfig.options: [{name: ndots, value: "2"}], or (3) append a trailing dot (api.stripe.com.) in application config to make it an explicit FQDN.
3 How does NodeLocal DNSCache prevent Linux kernel conntrack table exhaustion and 5-second UDP DNS timeouts? It runs a local caching daemon on every node (169.254.20.10) that serves cached queries locally and upgrades upstream cache-misses to CoreDNS from lossy UDP to multiplexed TCP.
4 Can you upgrade an Amazon EKS cluster directly from version 1.28 to 1.31 in one step? No, Kubernetes strictly prohibits skipping minor versions; you must upgrade sequentially (1.28 โ†’ 1.29 โ†’ 1.30 โ†’ 1.31).
5 What is the official Kubernetes Version Skew Policy between kube-apiserver and worker node kubelet? kubelet may be up to 3 minor versions older than kube-apiserver (in EKS, keep nodes within 1โ€“2 minor versions), and must never be newer than kube-apiserver.
6 What CLI tool scans a live cluster and Git manifests to detect removed or deprecated Kubernetes API versions (like autoscaling/v2beta2 or networking.k8s.io/v1beta1) before upgrading? Pluto (pluto detect-all-in-cluster) or Kubent (kube-no-trouble).
7 In what exact sequence must components be upgraded during a production EKS cluster upgrade? (1) Audit deprecated APIs โ†’ (2) Upgrade EKS Control Plane โ†’ (3) Upgrade core add-ons (vpc-cni, coredns, kube-proxy, ebs-csi) โ†’ (4) Roll/replace Worker Nodes.
8 Why should you verify PodDisruptionBudgets and topologySpreadConstraints before triggering a worker node group upgrade? A single misconfigured PDB with ALLOWED DISRUPTIONS = 0 will cause the managed node group rolling upgrade to stall and fail with NodeEvictionFailure.
# One-Line Question One-Line Answer
1 What command launches a temporary debugging container attached to the process and network namespace of a running distroless pod? kubectl debug -it <pod-name> --image=nicolaka/netshoot --target=<container-name>.
2 How do you inspect the exact AWS IAM Role ARN attached to a Kubernetes ServiceAccount for IRSA? kubectl get sa <sa-name> -o jsonpath='{.metadata.annotations.eks\.amazonaws\.com/role-arn}'.
3 How do you identify the top 10 highest-cardinality metric names inside a Prometheus TSDB? Query /api/v1/status/tsdb in the Prometheus UI or run promtool tsdb analyze /prometheus.
4 What PromQL query detects pods that are experiencing >25% Linux CFS CPU quota throttling? rate(container_cpu_cfs_throttled_periods_total[5m]) / rate(container_cpu_cfs_periods_total[5m]) > 0.25.
5 What command shows real-time node provisioning decisions and bin-packing events from Karpenter? kubectl logs -n kube-system -l app.kubernetes.io/name=karpenter -f.
๐Ÿ”„ Phase 5: GitOps & Control Planes 45 Rapid Q&As

Lesson 1: ArgoCD App of Apps

# One-Line Question One-Line Answer
1 What are the 3 core architectural microservices that make up an ArgoCD installation? argocd-server (API/UI), argocd-repo-server (clones Git & renders Helm/Kustomize), and argocd-application-controller (StatefulSet comparing live K8s state against Git).
2 What is the difference between automated.prune: true and automated.selfHeal: true in an ArgoCD Application? prune: true deletes cluster resources when their YAML files are deleted from Git; selfHeal: true overwrites manual kubectl edit changes on the cluster back to match Git.
3 What is the ArgoCD App of Apps Pattern? Bootstrapping a cluster with a single root Application CRD whose Git directory contains child Application manifests for all other platform add-ons and workloads.
4 How do you force ArgoCD to install Custom Resource Definitions (CRDs) before deploying operators, and operators before deploying custom resources? Annotate manifests with ArgoCD Sync Waves (argocd.argoproj.io/sync-wave: "-1", "0", "1"), which applies lower numbers first and waits for health checks between waves.
5 What ArgoCD Hook runs a database schema migration Job automatically before deploying new application pods? argocd.argoproj.io/hook: PreSync combined with argocd.argoproj.io/hook-delete-policy: BeforeHookCreation.
6 What causes an ArgoCD Infinite OutOfSync Loop where a Deployment continuously syncs every 3 seconds? A cluster controller (like HPA mutating spec.replicas or a MutatingWebhook injecting defaults) modifies fields that conflict with the static values in Git.
7 How do you fix an ArgoCD Infinite Sync Loop caused by HPA modifying spec.replicas? Remove spec.replicas from the Git YAML manifest or add ignoreDifferences for /spec/replicas with RespectIgnoreDifferences=true in syncOptions.
8 Why does deleting an ArgoCD Application with metadata.finalizers: [resources-finalizer.argocd.argoproj.io] perform a cascading delete of all child Kubernetes resources? The finalizer instructs the controller to delete every managed Deployment, Service, and PVC before removing the Application object itself; omitting the finalizer performs an Orphan Delete.
# One-Line Question One-Line Answer
1 Why is the native Kubernetes Deployment RollingUpdate strategy insufficient for mission-critical FinTech releases? Native RollingUpdate only checks basic pod readinessProbe status, cannot split traffic by exact percentages (e.g., 2%), and cannot auto-rollback based on Prometheus error-rate SLIs.
2 What CRD replaces kind: Deployment when adopting progressive delivery with Argo Rollouts? kind: Rollout (apiVersion: argoproj.io/v1alpha1).
3 What is the difference between Blue/Green and Canary deployment strategies in Argo Rollouts? Blue/Green provisions 100% duplicate capacity (activeService vs previewService) and flips traffic instantly; Canary shifts traffic incrementally (5% โ†’ 20% โ†’ 50% โ†’ 100%) with automated metric pauses.
4 Without a service mesh or ingress controller, why canโ€™t a 4-replica Kubernetes Deployment achieve a 5% canary traffic split? Native Service routing distributes traffic per pod (1 / 4 = 25% minimum increment); achieving 5% requires L7 weighted routing via AWS ALB, NGINX, Cilium, or Istio Gateway API.
5 What Argo Rollouts CRD queries Prometheus or Datadog during a canary rollout and triggers an automatic rollback if error rates breach 1%? AnalysisTemplate (instantiated at runtime as an AnalysisRun).
6 In an AnalysisTemplate, what is the difference between failureLimit: 2 and inconclusiveLimit? failureLimit: 2 aborts and rolls back the canary after 2 failed metric checks; inconclusiveLimit pauses the rollout when Prometheus returns NaN (e.g., zero traffic during off-hours).
7 What causes the Canary Database Schema Trap where an automated canary rollback brings down 100% of production? The canary executed a destructive SQL migration (e.g., renaming or dropping a column) that broke backward compatibility with the stable v1 pods.
8 How do you guarantee safe canary rollbacks when evolving relational database schemas? Strictly enforce the Expand-and-Contract (Parallel Change) Patternโ€”never drop or rename columns in the same release that deploys new code.
# One-Line Question One-Line Answer
1 What is the fundamental architectural difference between Terraform and Crossplane? Terraform is a CLI/pipeline tool that runs once at CI time, whereas Crossplane runs inside Kubernetes as an active 24/7 reconciliation loop that continuously self-heals cloud drift.
2 Define the 4 core building blocks of Crossplane (Provider, MR, XRD, Composition). Provider (AWS/GCP API controller), Managed Resource (MR) (1:1 cloud resource like an RDS instance), XRD (custom API schema), and Composition (template combining multiple MRs).
3 Why did Crossplane introduce Composition Functions (KCL / Python / Go) to replace traditional Patch-and-Transform YAML? Traditional YAML patching became thousands of lines of untestable spaghetti lacking loops or conditionals; KCL Functions allow concise, strongly typed programmatic logic with unit testing.
4 How does a developer provision an AWS RDS database and S3 bucket using Crossplane without knowing AWS IAM or VPC subnet IDs? The developer commits a 10-line Composite Resource Claim (XRC) YAML specifying storageGB: 50, and the Platform Teamโ€™s Composition translates it into hardened AWS resources.
5 What happens if someone manually deletes a Crossplane-managed Security Group in the AWS Console? Within ~60 seconds, the Crossplane Provider reconciles live AWS state against the Kubernetes desired state and automatically recreates the deleted Security Group.
6 What causes a Crossplane Finalizer Deadlock where a deleted Composite Resource hangs in Terminating forever? Either the ProviderConfig credentials were deleted before the cloud resource was destroyed, or AWS rejected deletion due to child dependencies (e.g., non-empty S3 bucket or active ENI).
7 What flag on a Crossplane Managed Resource prevents kubectl delete from accidentally destroying a production cloud database? spec.deletionPolicy: Orphan (alongside spec.managementPolicies: ["Observe", "Create", "Update"] omitting "Delete").
8 How does Crossplane deliver newly generated RDS database credentials directly to the application pod? Using writeConnectionSecretToRef, which writes the endpoint, username, and password into a native Kubernetes Secret inside the applicationโ€™s namespace.
# One-Line Question One-Line Answer
1 What problem does the ArgoCD ApplicationSet Controller solve at fleet scale? It eliminates manually writing hundreds of repetitive Application YAML manifests by dynamically generating them from a single template using Generators.
2 Name the 4 most common ArgoCD ApplicationSet Generators. List Generator, Cluster Generator (matches cluster labels), Git Generator (directory or file discovery), and Pull Request Generator (ephemeral preview environments).
3 What does the Matrix Generator do in an ArgoCD ApplicationSet? It computes the Cartesian Product (M ร— N) of two generatorsโ€”for example, combining 10 Clusters (Cluster Generator) ร— 15 Add-ons (Git Generator) = 150 Applications from one file.
4 What does the Merge Generator do when combined with a Matrix Generator? It allows per-cluster or per-environment configuration overrides (e.g., overriding replica counts or versions for prod-india while keeping defaults for dev).
5 What catastrophic outage can happen if a developer deletes a directory in Git watched by an ApplicationSet with default settings? The ApplicationSet deletes the generated Application CRDs, which triggers a cascading delete of all production workloads across every cluster!
6 How do you protect production clusters from accidental mass-deletion by an ApplicationSet? Configure spec.syncPolicy.preserveResourcesOnDeletion: true (or set applicationsSync: create-update) on the ApplicationSet.
7 How do you prevent an ApplicationSet from upgrading Cilium or CoreDNS across all 50 production clusters at the exact same second? Enable ApplicationSet Progressive Syncs (RollingSync), which rolls updates out in staged batches (dev โ†’ staging โ†’ 10% prod โ†’ 100% prod) waiting for Healthy status between steps.
8 In the Crossplane + ArgoCD Golden Pattern, what is the exact division of responsibilities? ArgoCD syncs YAML from Git into the cluster, while Crossplane reconciles those custom YAML CRDs into actual AWS/GCP cloud infrastructure.
# One-Line Question One-Line Answer
1 Why do enterprises prefer a Centralized Hub-and-Spoke ArgoCD Architecture over running a separate ArgoCD instance inside every single workload cluster? A single Hub control plane provides unified RBAC/SSO, a single pane of glass for 50+ spoke clusters, and removes duplicate ArgoCD compute overhead from workload clusters.
2 Where does the Hub ArgoCD store credentials and endpoints for remote Spoke clusters? Inside the argocd namespace as Kubernetes Secret objects carrying the label argocd.argoproj.io/secret-type: cluster.
3 How does Hub ArgoCD authenticate to remote Amazon EKS Spoke clusters without storing static long-lived bearer tokens? Using aws-iam-authenticator / EKS IRSA Cross-Account Role Assumption configured inside the cluster Secretโ€™s execProviderConfig.
4 Why does argocd-application-controller get OOMKilled or lag when managing 50+ spoke clusters and 3,000 Applications? A single controller replica cannot hold the Informer cache watches for 50 clusters in memory; you must enable Controller Dynamic Sharding (replicas: 5) across spoke clusters.
5 Why must the Hub ArgoCD never access Spoke cluster API servers over the public internet? Exposing kube-apiserver publicly creates an attack surface; Hub-to-Spoke traffic must route over private AWS Transit Gateway, VPC Peering, or AWS PrivateLink.
6 What happens to running pods on the 50 Spoke clusters if the Central Hub ArgoCD cluster goes completely offline for 2 hours? Zero impact to live trafficโ€”Spoke workloads continue running normally; only new Git deployments and drift reconciliation are paused until the Hub recovers.
7 How do you recover the entire Hub ArgoCD control plane into a warm standby region (ap-south-2) in under 10 minutes without restoring etcd backups? Because ArgoCD is 100% declarative in Git, bootstrap the standby Hub cluster with the root App of Apps manifest and Cluster Secrets; ArgoCD reconciles against live spokes without restarting pods.
8 What Redis component in ArgoCD must run in HA mode (argocd-redis-ha) for production Hub clusters? The Redis cache storing live Kubernetes resource trees and UI session state across argocd-server and argocd-repo-server replicas.
# One-Line Question One-Line Answer
1 What CLI command manually promotes a paused Argo Rollout canary to the next step? kubectl argo rollouts promote <rollout-name>.
2 What CLI command immediately aborts a degraded canary rollout and shifts 100% of traffic back to the stable ReplicaSet? kubectl argo rollouts abort <rollout-name>.
3 How do you trace all child Managed Resources created by a Crossplane Composite Resource from the command line? Run crossplane beta trace <xrd-kind> <name>.
4 What condition fields must both equal True on a Crossplane Managed Resource to confirm it is healthy and synced with AWS? SYNCED = True and READY = True.
5 How do you preview the exact Application manifests an ApplicationSet will generate before applying it to the cluster? Run argocd appset generate <appset.yaml>.
๐Ÿ Phase 6: Service Mesh & eBPF 32 Rapid Q&As

Lesson 1: Cilium & eBPF

# One-Line Question One-Line Answer
1 Why does legacy kube-proxy running in iptables mode degrade severely when a cluster scales past 5,000 Kubernetes Services? iptables evaluates rules sequentially in an O(N) linear list and must rewrite the entire multi-megabyte rule table atomically on every endpoint update.
2 How does Cilium (kubeProxyReplacement: true) achieve O(1) service load-balancing at any cluster scale? It attaches eBPF programs to kernel socket and TC hooks that look up Service-to-Pod backends directly inside O(1) eBPF Hash Maps in kernel memory.
3 What ensures an eBPF program loaded into the Linux kernel cannot crash the kernel or enter an infinite loop? The In-Kernel eBPF Verifier, which performs static DAG code analysis, bounds-checking, and memory safety verification before JIT compilation.
4 How does Cilium Socket-Level Load Balancing (bpf_sockops) accelerate pod-to-Service communication? It intercepts the connect() syscall at the socket layer and rewrites the ClusterIP directly to the backend Pod IP before packets even enter the TCP/IP packet stack.
5 Why does traditional sidecar service mesh architecture (Istio Envoy per pod) add latency and double memory consumption? Every packet traverses the Linux TCP/IP stack 4 times per hop (App โ†’ local Envoy โ†’ Wire โ†’ remote Envoy โ†’ remote App), and every pod runs a dedicated Envoy proxy container.
6 How does Cilium provides transparent node-to-node encryption across the cluster without sidecars? By enabling Kernel-Mode WireGuard Encryption (encryption.enabled=true, encryption.type=wireguard), which encrypts all pod-to-pod traffic between worker nodes at the network interface layer.
7 What observability tool built into Cilium provides real-time eBPF network flow logs, DNS query visibility, and service dependency maps? Hubble (hubble observe).
8 What happens to existing active pod network connections if the cilium-agent DaemonSet pod restarts or crashes? Zero packet lossโ€”compiled eBPF programs and maps persist inside the Linux kernel datapath independently of the user-space cilium-agent control plane.
# One-Line Question One-Line Answer
1 What is the default network security posture of a fresh Kubernetes cluster without NetworkPolicies? Flat allow-all networkโ€”every pod in every namespace can freely send packets to every other pod across the entire cluster.
2 Why are Pod IP addresses unreliable for enforcing security firewall rules in Kubernetes? Pod IPs are ephemeral and churn constantly during autoscaling; Cilium uses cryptographic Security Identities (numeric ID) derived from pod labels instead of IP addresses.
3 What layers of the OSI model can a standard Kubernetes NetworkPolicy filter vs a CiliumNetworkPolicy? Standard NetworkPolicy is strictly L3/L4 (IPs, Pod Labels, TCP/UDP ports), whereas CiliumNetworkPolicy filters L3, L4, and L7 (HTTP methods/paths, gRPC, Kafka topics, DNS FQDNs).
4 How do you configure a Zero-Trust Default Deny-All baseline for a namespace in Kubernetes? Apply a NetworkPolicy with an empty podSelector: {} and policyTypes: ["Ingress", "Egress"] with no allow rules.
5 What critical traffic is accidentally broken if you apply a Default Deny-All Egress policy without an explicit DNS exception? Pods immediately lose the ability to resolve DNS names because UDP/TCP Port 53 traffic to kube-dns / coredns in kube-system is blocked!
6 How does a CiliumNetworkPolicy allow a pod to egress only to https://api.stripe.com even when Stripeโ€™s CDN rotates IP addresses every minute? Using an L7 DNS proxy rule (toFQDNs: [{matchName: "api.stripe.com"}]); Cilium intercepts the podโ€™s DNS lookup, learns the returned IPs dynamically, and updates the podโ€™s eBPF allow map in real time.
7 How does a CiliumNetworkPolicy allow frontend to call GET /api/v1/catalog on backend, while blocking DELETE /api/v1/catalog? By specifying an L7 HTTP rule (rules: http: [{method: "GET", path: "/api/v1/catalog"}]), which transparently routes matching flows through an embedded node-level Envoy listener.
8 What command shows live packets being dropped by network policies across the cluster with the exact drop reason? hubble observe --verdict DROPPED -f.
# One-Line Question One-Line Answer
1 Why do naive unlimited retries during a downstream database slowdown trigger a catastrophic Retry Storm? If 3 upstream tiers each configure 3 retries, a single failing request multiplies into 3 ร— 3 ร— 3 = 27ร— traffic amplification, DDoSing the recovering backend.
2 What two mathematical techniques must always be paired with retries to prevent synchronized retry waves? Exponential Backoff (2^n ร— base_delay) combined with Random Jitter + a Retry Budget (capping retries at max 10โ€“20% of total traffic).
3 Which HTTP status codes or methods should never be automatically retried without an explicit Idempotency Key? Non-idempotent mutations like POST /charge-payment that timed out after already reaching the upstream server.
4 What is Circuit Breaking / Outlier Detection in Envoy and Cilium Service Mesh? Automatically ejecting a degraded backend pod IP from the load-balancing pool for baseEjectionTime (e.g., 30s) when it returns consecutive 5xx errors or breaches max pending connections.
5 What are the 3 states of a Circuit Breaker? Closed (normal traffic flows), Open (fast-fails immediately with 503 Overflow without calling backend), and Half-Open (allows a probe request to test recovery).
6 Why must upstream microservices always enforce shorter timeouts as requests travel deeper down a call chain (Timeout Budget Propagation)? If API Gateway waits 2s, Service A waits 5s, and Service B waits 10s, Service B continues wasting CPU/DB connections for 8 seconds after the user already disconnected!
7 What Envoy response flag in access logs proves that a request was fast-failed by a circuit breaker (max_pending_requests overflow)? UO (Upstream Overflow).
8 What is Mutual TLS (mTLS) in Cilium Service Mesh, and how does Cilium separate authentication from encryption? Cilium uses SPIFFE/SPIRE x509 identities to mutually authenticate workload identities in control plane handshakes while encrypting data-plane packets at line rate via kernel WireGuard/IPsec.
# One-Line Question One-Line Answer
1 Why is the new Kubernetes Gateway API (gateway.networking.k8s.io) replacing the legacy Ingress resource? Legacy Ingress relied on vendor-specific annotation sprawl and lacked role separation, native traffic weighting, header routing, or gRPC/TCP support.
2 Name the 3 role-oriented CRDs of the Kubernetes Gateway API and who owns each. GatewayClass (Infrastructure Provider), Gateway (Platform/SRE Team โ€” listeners & TLS), and HTTPRoute / GRPCRoute (Application Developers โ€” paths & weights).
3 How does HTTPRoute prevent developers in the marketing namespace from hijacking traffic meant for the payments namespace? The platform teamโ€™s Gateway controls allowed namespace bindings (allowedRoutes.namespaces), and cross-namespace backend references require an explicit ReferenceGrant in the target namespace.
4 What is Cilium ClusterMesh? An eBPF multi-cluster networking technology that links multiple Kubernetes clusters across regions/clouds, enabling pod-to-pod routing and cross-cluster Service discovery without ingress gateways.
5 What prerequisite must be met by Pod CIDRs across all clusters participating in a Cilium ClusterMesh? Each cluster must have a non-overlapping Pod CIDR and a unique cluster.id (1โ€“255) so eBPF can route directly to remote pod IPs.
6 How do you turn a standard Kubernetes Service into a global multi-cluster service across Cilium ClusterMesh? Add the annotation service.cilium.io/global: "true".
7 By default, does a global ClusterMesh service balance traffic across all regions equally, or can it prefer local pods first? Annotating service.cilium.io/affinity: "local" instructs Cilium to route 100% of traffic to healthy pods inside the local cluster, failing over to the remote region cluster only if all local pods die.
8 Why must Cilium MTU be reduced to 1420 when running ClusterMesh with WireGuard encryption over cloud VPC networks? Standard VPC MTU is 1500 (or 9001 jumbo); WireGuard adds 60โ€“80 bytes of encapsulation headers, which will drop large TLS handshake packets if inner MTU is not clamped.
๐ŸŽญ Phase 7: Internal Developer Portals 40 Rapid Q&As

Lesson 1: Backstage Software Catalog

# One-Line Question One-Line Answer
1 What core cognitive problem does an Internal Developer Portal (IDP) like Spotify Backstage solve? It eliminates developer cognitive load and tribal knowledge silos by unifying service ownership, APIs, docs, CI/CD, and golden-path self-service into a single portal.
2 What YAML file placed at the root of a Git repository registers a microservice into the Backstage Software Catalog? catalog-info.yaml.
3 Name the 3 Core Entity Kinds and 2 Grouping Kinds in the Backstage System Model. Core: Component (service/website), API (OpenAPI/gRPC), Resource (DB/S3). Grouping: System (collection of components/resources) and Domain (business unit).
4 How does Backstage keep its PostgreSQL catalog database updated without requiring manual UI registrations? Via Entity Providers (GitHub Org Discovery, GitHub Webhook processor, and Kubernetes/AWS Discovery processors) that continuously ingest and stitch entities into a graph.
5 What causes an entity in Backstage to show an Orphan Warning? The catalog-info.yaml file was deleted or moved in the source Git repository, leaving a stale cached record in Backstageโ€™s database.
6 What is the difference between spec.lifecycle: experimental and spec.lifecycle: production in catalog-info.yaml? It classifies the operational maturity of the service, driving automated production readiness scorecards and on-call paging rules.
7 How do two Backstage entities express a dependency relationship in the catalog graph? Using spec.dependsOn: ["resource:default/orders-postgres-db"] or spec.providesApis / spec.consumesApis.
8 What causes the Backstage Catalog Processor to crash or reject an API entity during ingestion? A malformed syntax error or unescaped $ref circular loop inside the referenced OpenAPI (swagger.yaml) or AsyncAPI specification file.
# One-Line Question One-Line Answer
1 What is a โ€œGolden Pathโ€ (Paved Road) in Platform Engineering? An opinionated, automated self-service workflow that makes the secure, compliant, observable way of building and deploying software the easiest default path for developers.
2 What Backstage component executes multi-step infrastructure and code provisioning workflows (kind: Template)? The Backstage Scaffolder (@backstage/plugin-scaffolder-backend).
3 What are the 3 main sections of a Backstage Template YAML definition? spec.parameters (React JSON Schema UI form inputs), spec.steps (sequential actions: fetch, template, publish PR, register), and spec.output (links to created repo/ArgoCD).
4 Why should a Backstage Golden Path template create a Git Pull Request containing Crossplane/Terraform manifests instead of calling AWS APIs directly from Node.js? Committing declarative manifests to Git preserves GitOps auditability, policy-as-code PR checks, and automatic drift reconciliation.
5 What templating engine does Backstage fetch:template use to inject user form parameters into skeleton source files? Nunjucks (${{ parameters.service_name }}).
6 What happens if Step 3 (publish:github) succeeds in creating a GitHub repo, but Step 4 (catalog:register) fails in a Scaffolder run? Without compensating cleanup or idempotency, retrying the template fails with Repository already exists; production custom actions must implement idempotent checks or Saga rollbacks.
7 How do you prevent a developer from typing an invalid Kubernetes service name (e.g., uppercase letters or spaces) into a Backstage Scaffolder form? Add a regex pattern: '^[a-z0-9-]+$' validation rule inside the spec.parameters JSON Schema property.
8 What custom Scaffolder action allows platform teams to trigger an external GitHub Actions workflow or Jenkins job during scaffolding? github:actions:dispatch (or custom HTTP webhook actions).
# One-Line Question One-Line Answer
1 Why do traditional enterprise Confluence wikis fail for operational runbooks? Documentation decoupled from source code rots immediately because developers rarely update a separate wiki when refactoring code or APIs.
2 How does Backstage TechDocs (Docs-as-Code) solve documentation drift? Markdown docs (docs/index.md + mkdocs.yml) live inside the exact same Git repository as the microservice code and are reviewed in the same Pull Request.
3 What annotation in catalog-info.yaml tells Backstage where to find an entityโ€™s TechDocs source files? backstage.io/techdocs-ref: dir:..
4 Why must you never configure techdocs.builder: 'local' in a production Backstage deployment? 'local' forces the Backstage Node.js backend pod to spawn heavy Python mkdocs subprocesses on the fly when users view pages, causing CPU starvation and OOMKills.
5 What is the recommended production architecture for Backstage TechDocs (techdocs.builder: 'external')? CI/CD Pre-Buildingโ€”GitHub Actions runs techdocs-cli generate and techdocs-cli publish to push compiled static HTML/CSS to an Amazon S3 / GCS bucket, which Backstage serves instantly.
6 What MkDocs extension renders architecture-as-code diagrams (flowchart, sequenceDiagram) natively inside TechDocs pages? Mermaid.js (pymdownx.superfences with custom mermaid fence).
7 How do you enforce that a developer cannot change an API endpoint without updating the serviceโ€™s TechDocs or Runbook? Configure a GitHub Actions PR gate or CODEOWNERS check that flags changes to /api/* or /terraform/* lacking corresponding updates in /docs/*.
8 Where should every microserviceโ€™s operational runbook link so on-call SREs can find it at 3:00 AM? Directly inside the Prometheus Alert annotations.runbook_url pointing to the serviceโ€™s Backstage TechDocs URL.
# One-Line Question One-Line Answer
1 How does Backstage authenticate users and automatically map them to their engineering teams (kind: Group)? Via OIDC / SAML SSO (Okta, Azure Entra ID, Google Workspace) paired with an automated Org Discovery Processor that syncs users and group memberships.
2 In the Backstage Permission Framework, what is the difference between a Basic Permission and a Resource Permission? A Basic Permission controls global actions (catalog.entity.create), while a Resource Permission evaluates fine-grained conditions against a specific entity (e.g., isEntityOwner).
3 How do you restrict a destructive Scaffolder template (like "Decommission Production Database") so only Staff SREs can see or execute it? Tag the template with tags: ["restricted-sre"] and apply a Backstage RBAC Conditional Policy denying scaffolder.template.parameter.read and task.create unless the user is in group:default/sre-leads.
4 What is a Production Readiness Scorecard in an Internal Developer Portal? An automated grading engine (Bronze / Silver / Gold) that continuously audits every microservice against security, reliability, observability, and documentation standards.
5 Name 5 automated checks typically evaluated by a Gold-Tier Production Readiness Scorecard. (1) Valid PodDisruptionBudget, (2) SLO + Runbook URL defined, (3) Zero CRITICAL Trivy CVEs, (4) Cosign image signature verified, and (5) PagerDuty rotation attached.
6 How do you turn passive Backstage Scorecards into active Deployment Gates during major sales events (like IPL or Diwali)? Query the Scorecard API from GitHub Actions or ArgoCD PreSync hooks and block production promotion for any service scoring below Silver/Gold.
7 How does Backstage prevent a developer in Team A from viewing sensitive runtime secrets or triggering deployments on Team Bโ€™s Kubernetes pods? By applying the isEntityOwner RBAC condition on the Kubernetes and ArgoCD plugins, scoping actions strictly to spec.owner == user.claims.groups.
8 What audit trail must be enabled on the Backstage backend to satisfy SOC2 and ISO27001 compliance? Structured Audit Logging capturing every Scaffolder task execution, RBAC permission decision (ALLOW/DENY), user identity, and timestamp sent to SIEM.
# One-Line Question One-Line Answer
1 What is the core discipline of Chaos Engineering? Designing controlled, hypothesis-driven fault injection experiments to uncover systemic weaknesses before they cause unpredicted production outages.
2 What must always be defined before injecting any fault in a Chaos Engineering experiment? The Steady-State Hypothesis (measurable business/technical SLIs such as "Checkout p99 < 250ms and HTTP 5xx < 0.1%").
3 Name the 4 most common fault CRDs in Chaos Mesh. PodChaos (pod-kill/container-kill), NetworkChaos (latency/loss/partition), StressChaos (CPU/memory burn via stress-ng), and IOChaos / HTTPChaos.
4 How does the Chaos Mesh chaos-daemon inject a 150ms network delay into a target pod without modifying the podโ€™s container image? The privileged chaos-daemon on the host uses nsenter to enter the target podโ€™s network namespace (netns) and configures a Linux kernel Traffic Control (tc) netem qdisc on its eth0 interface.
5 What happens if the chaos-daemon pod crashes or is OOMKilled while a NetworkChaos experiment is actively injecting 50% packet loss? The Linux kernel tc qdisc netem rule remains orphaned inside the target podโ€™s network namespace (Zombie Chaos Rule), causing permanent packet loss even after the NetworkChaos CRD is deleted!
6 How do you manually clear a Zombie NetworkChaos rule from an affected pod without restarting the pod? Enter the podโ€™s network namespace from the worker node and delete the root queuing discipline: nsenter -t <pod-pid> -n tc qdisc del dev eth0 root.
7 How do you prevent a Chaos Mesh experiment from accidentally targeting kube-system or critical control plane namespaces? Configure Chaos Mesh controller flags FILTER_NAMESPACE / annotation enforcement (chaos-mesh.org/inject: "true" required on target namespaces) and block system namespaces via Kyverno.
8 How do you automate an immediate Emergency Stop (Abort) of a Chaos Mesh Workflow if real customer error budgets start burning? Pair the Chaos Workflow with a Prometheus Alertmanager webhook (or Argo Rollouts Analysis) that automatically deletes the active Chaos CRD as soon as any production SLO alert fires.
๐Ÿง  Phase 8: LLMOps Infrastructure 29 Rapid Q&As

Lesson 1: Vector DBs & AI Gateways

# One-Line Question One-Line Answer
1 What is RAG (Retrieval-Augmented Generation), and why is it preferred over fine-tuning for dynamic enterprise data? RAG retrieves relevant document chunks from a Vector DB at query time and injects them into the LLM prompt context, preventing hallucinations and enforcing real-time document ACLs without expensive weight retraining.
2 Why does a Milvus Vector Database QueryNode get OOMKilled when loading 50 million 1536-dimension float32 vectors (50M ร— 1536 ร— 4B โ‰ˆ 307GB)? In-memory HNSW (Hierarchical Navigable Small World) graph indices require 1.5ร— to 2ร— the raw vector size in RAM to store multi-layer graph adjacency pointers alongside raw vectors.
3 How do you reduce Vector Database memory consumption by 75% to 90% when scaling to billions of vectors? Use Vector Quantization (SQ8 Scalar Quantization to 8-bit integers or PQ Product Quantization) or DiskANN (NVMe SSD-backed indexing).
4 Why should enterprise microservices never call OpenAI, Anthropic, or AWS Bedrock APIs directly from application code? Direct calls leak API keys across dozens of repos, lack unified token cost tracking per team, and cannot automatically fail over when OpenAI returns HTTP 429 or 503.
5 What are the 4 core functions of an Enterprise AI Gateway (LiteLLM / Envoy AI Gateway)? (1) Multi-provider automatic failover (OpenAI โ†’ Azure OpenAI โ†’ Bedrock), (2) Token-based rate limiting (TPM/RPM), (3) Per-tenant FinOps cost chargeback, and (4) Semantic Caching.
6 How does Semantic Caching (using Redis Vector Search) differ from traditional exact-match key-value caching? Instead of requiring identical prompt strings, it embeds the userโ€™s query into a vector and returns a cached response if Cosine Similarity exceeds a threshold (e.g., > 0.95), saving 30โ€“50% of LLM spend.
7 Why do traditional HTTP load balancers and rate limiters (requests/sec) fail to protect LLM gateways? One LLM request with a 100,000-token context window consumes 1,000ร— more GPU compute and quota than a 100-token request; LLM gateways must rate-limit by Tokens Per Minute (TPM).
8 What causes a LiteLLM Retry Storm during an upstream OpenAI outage, and how do you prevent it? Setting high retry counts across multiple fallback regions; fix by setting num_retries: 2, configuring a cooldown_time: 60s circuit breaker on degraded deployments, and enforcing request timeouts.
# One-Line Question One-Line Answer
1 What are the 4 core components of an enterprise MLflow platform? MLflow Tracking (experiments/params/metrics), MLflow Model Registry (versioning & aliases like @champion), MLflow Projects, and Artifact Store (S3) + Backend Store (PostgreSQL).
2 Why do traditional unit tests (assert output == "expected") fail when testing LLM applications in CI/CD? LLMs are probabilistic and non-deterministicโ€”two semantically identical correct answers rarely have identical word-for-word string representations.
3 What are the 4 core metrics of the RAGAS (Retrieval-Augmented Generation Assessment) evaluation framework? Context Precision (signal-to-noise of retrieved chunks), Context Recall (did retrieval find all facts?), Faithfulness (did LLM hallucinate outside context?), and Answer Relevance.
4 Which RAGAS metric specifically detects LLM Hallucinations (claims generated by the model that cannot be inferred from the retrieved context)? Faithfulness (scored from 0.0 to 1.0; production CI gates typically require Faithfulness >= 0.90).
5 Why did an LLM-as-a-Judge CI evaluation pipeline pass a PR at 10:00 AM and fail the exact same code at 10:15 AM? The judge LLM was invoked with temperature > 0 or an unpinned floating model name; LLM-as-a-Judge must always set temperature: 0.0, seed: 42, and pin exact model snapshots.
6 How do you version prompts in a GitOps workflow instead of hardcoding prompt strings inside Python application code? Store prompts as versioned YAML templates in Git (or MLflow Prompt Registry), evaluate them against a Golden Dataset in CI using DeepEval/RAGAS, and promote via @production aliases.
7 What is Data Drift / Embedding Drift in a production RAG system? When live user queries diverge semantically from the historical document corpus or when changing embedding models (text-embedding-ada-002 โ†’ text-embedding-3-large) invalidates existing vector distances.
8 Why canโ€™t you query vectors generated by text-embedding-ada-002 using a query embedded with text-embedding-3-large? Different embedding models map semantics into completely incompatible vector spaces; switching embedding models requires re-indexing the entire document corpus into a new collection (Blue/Green Vector Re-indexing).
# One-Line Question One-Line Answer
1 What is the structural difference between GPU Time-Slicing and NVIDIA MIG (Multi-Instance GPU) on an A100/H100 GPU? Time-Slicing shares one GPU via context-switching with zero memory or fault isolation (one OOM crashes all pods); MIG physically partitions the silicon into up to 7 hardware-isolated GPU instances with dedicated VRAM and compute cores.
2 How much GPU VRAM is required just to load the model weights of a 70B parameter LLM in FP16 (16-bit = 2 bytes per parameter)? โ‰ˆ 140 GB of VRAM (70B ร— 2 bytes), plus an additional 20โ€“40 GB for the KV (Key-Value) Cache and activationsโ€”requiring at least 2ร— 80GB H100 GPUs or AWQ/GPTQ 4-bit quantization.
3 What breakthrough memory technology in vLLM eliminates 60โ€“80% of GPU VRAM waste during concurrent LLM serving? PagedAttentionโ€”it splits the attention KV Cache into non-contiguous virtual memory blocks (like OS virtual memory pages), eliminating contiguous memory fragmentation and enabling Continuous Batching.
4 What causes a vLLM CUDA Out-Of-Memory (OOM) crash during traffic bursts even though the model weights loaded cleanly at startup? Setting --max-model-len too high (e.g., 128k context) or --gpu-memory-utilization to 0.98, leaving zero VRAM headroom for PyTorch activation spikes during large prefill batches.
5 How do you shard a 70B model across 4 GPUs on the same physical node in vLLM? Set Tensor Parallelism --tensor-parallel-size 4 (TP=4), which splits matrix multiplications across all 4 GPUs simultaneously over high-speed NVLink.
6 Why does autoscaling vLLM pods based on Kubernetes CPU or GPU Memory % fail completely? vLLM pre-allocates 90% of GPU VRAM at startup (so memory is permanently constant at 90%), and CPU stays low; you must autoscale vLLM with KEDA on vllm:num_requests_waiting (queue depth) and vllm:gpu_cache_usage_perc.
7 Why do LLM GPU pods take 10+ minutes to start during scale-out, and how do you cut cold-start time to under 30 seconds? Pulling 140GB of Safetensors weights over the internet or slow EBS throttles network I/O; store model weights on Amazon FSx for Lustre or pre-cached NVMe instance stores with --load-format safetensors.
8 What runtime security layer sits in front of an LLM to block Prompt Injection (Jailbreaks) and redact PII (Aadhaar, PAN, Credit Cards) before prompts leave the cluster? LLM Guardrails (NVIDIA NeMo Guardrails, Llama Guard 3, and Microsoft Presidio PII redaction filters).
# One-Line Question One-Line Answer
1 What Prometheus exporter deployed by the NVIDIA GPU Operator exposes hardware metrics like SM utilization (DCGM_FI_DEV_DEC_UTIL), VRAM temperature, and XID hardware errors? NVIDIA DCGM Exporter (dcgm-exporter).
2 What does an NVIDIA XID Error 79 (โ€œGPU has fallen off the busโ€) or 48 (โ€œDouble Bit ECC Errorโ€) on a worker node indicate? A fatal physical hardware/PCIe/VRAM failure on the GPU; an automated Node Problem Detector remediation must cordon, drain, and terminate the unhealthy GPU node.
3 What two core latency SLIs define user experience in streaming LLM inference? TTFT (Time To First Token) (prefill latency, target < 500ms) and ITL / TPOT (Inter-Token Latency / Time Per Output Token) (decode throughput, target < 30ms/token).
4 How do you enable 4-bit AWQ (Activation-aware Weight Quantization) in vLLM to fit a 70B model onto a single 80GB GPU? Pass --quantization awq when launching the vLLM server with an AWQ-quantized model checkpoint (~38GB VRAM for weights).
5 What Kubernetes scheduling extension ensures an 8-GPU distributed training job schedules all 8 pods together or none at all to prevent resource deadlocks? Gang Scheduling via Kueue or Volcano Scheduler.
๐Ÿ›๏ธ Enterprise Architecture & SRE Masterclass 96 Rapid Q&As

Lesson 1: Fleet Patching & Deployment Strategies

# One-Line Question One-Line Answer
1 What are the 5 progressive deployment rings used in hyperscale enterprise fleet rollouts? Ring 0 (Canary/Internal) โ†’ Ring 1 (Dev/QA) โ†’ Ring 2 (Single-AZ Early Prod) โ†’ Ring 3 (Low-Traffic Region) โ†’ Ring 4 (Global Production).
2 Why is Shadow / Dark Traffic Mirroring superior to synthetic staging tests for validating high-risk financial microservices? It duplicates 100% of real production user payloads asynchronously to the v2 shadow deployment without returning shadow responses to real users.
3 Why must shadow deployments stub out or block downstream write calls (like charging credit cards or sending SMS)? Without write-blocking mocks, replaying mirrored production requests would double-charge customers and send duplicate notifications!
4 What is the difference between Deployment and Release in modern platform engineering? Deployment pushes binary code to production servers (dark/inactive), while Release exposes features to users dynamically via Feature Flags (OpenFeature / LaunchDarkly).
5 What is Bake Time in a progressive delivery pipeline? A mandatory soak window (e.g., 30โ€“60 minutes) between deployment rings to catch slow memory leaks, connection pool exhaustion, or delayed cron failures.
6 How do you patch stateful database clusters (PostgreSQL / Redis Sentinel) with zero downtime? Patch all Read Replicas / Standby nodes first, verify replication sync (0 lag), execute a Controlled Primary Switchover (<5s), and finally patch the demoted old primary.
7 What emergency mechanism kills a buggy new feature in < 200 milliseconds globally without waiting 15 minutes for a CI/CD container rollback? A Feature Flag Kill-Switch pushed via streaming SSE/WebSockets from the feature management control plane.
8 Why should automated rollbacks always be triggered by Composite SLIs rather than single CPU/memory metrics? A broken deployment often throws fast HTTP 500 errors while using almost zero CPU or memory; rollbacks must watch error rate, p99 latency, and business conversion drops.
# One-Line Question One-Line Answer
1 Define RPO (Recovery Point Objective) vs RTO (Recovery Time Objective) in one sentence. RPO measures how much data (time) you can afford to lose, while RTO measures how quickly service must be restored after an outage starts.
2 List the 4 standard Disaster Recovery (DR) tiers in order of increasing cost and decreasing RTO/RPO. Backup & Restore (hours) โ†’ Pilot Light (10โ€“30m) โ†’ Warm Standby (1โ€“5m) โ†’ Multi-Region Active-Active (<5s).
3 What is the exact difference between Pilot Light and Warm Standby DR architectures? Pilot Light keeps only the data replication layer running in DR (compute is scaled to zero), whereas Warm Standby keeps a scaled-down but live functional copy of all microservices running 24/7.
4 Why should you never use Route 53 Control Plane APIs (ChangeResourceRecordSets) to execute an emergency Disaster Recovery DNS failover? Route 53 Control Plane lives in us-east-1 and is throttled/degraded during major AWS outages; you must use Route 53 Application Recovery Controller (ARC) Routing Control, which runs on a highly available data-plane cluster across 5 global regions.
5 What is the physical replication lag and RPO of Amazon Aurora Global Database across regions (ap-south-1 โ†’ ap-south-2)? Under 1 second (RPO < 1s) using dedicated storage-layer replication independent of database CPU, with managed cross-region failover RTO under 1 minute.
6 Why is Multi-Region Active-Active with Synchronous Replication (RPO = 0) mathematically and physically impractical between Mumbai and Hyderabad (700 km)? Speed-of-light fiber latency adds ~15โ€“20ms RTT to every SQL commit lock, slashing write throughput and risking split-brain deadlocks under the CAP / PACELC Theorem.
7 How does Cell-Based Architecture (Sharded Active-Active) achieve multi-region active-active scale without cross-region distributed write locks? It deterministically partitions users by user_id hash or geography into independent regional cells so a userโ€™s writes only ever hit one home region at a time.
8 What causes Split-Brain Data Corruption during an automated DR failover, and how do you prevent it? Both the old primary DB and promoted standby DB accept writes simultaneously; prevent it using STONITH (Shoot The Other Node In The Head) fencing tokens and revoking old primary VIP/route entries first.
# One-Line Question One-Line Answer
1 Why must production Kubernetes containers set securityContext.readOnlyRootFilesystem: true? It prevents attackers who exploit an RCE vulnerability from downloading malware binaries or modifying application files on the container filesystem.
2 If readOnlyRootFilesystem: true is enabled, how do applications like NGINX or Spring Boot write temporary runtime files or caches? By mounting an ephemeral in-memory or scratch emptyDir: {} volume specifically at /tmp and /var/run.
3 What is the exact difference between startupProbe, readinessProbe, and livenessProbe in Kubernetes? startupProbe waits for slow legacy boot; readinessProbe removes a temporarily overloaded pod from Service endpoints without killing it; livenessProbe restarts a deadlocked container.
4 Why must a livenessProbe never check downstream dependencies like PostgreSQL or Redis connectivity? If the database blips for 3 seconds, every microservice podโ€™s livenessProbe will fail simultaneously, causing Kubernetes to restart all pods at once in a catastrophic Cascading Restart Storm!
5 How do you guarantee a Kubernetes pod receives the Guaranteed Quality of Service (QoS) class so it is protected from node memory pressure evictions? Set resources.requests strictly equal to resources.limits for both CPU and Memory across every container in the Pod.
6 Why do Staff SREs recommend setting resources.requests.cpu without resources.limits.cpu (or using Guaranteed with pinned cores) for latency-critical Go/Java APIs? Hard CPU limits enforce 100ms Linux CFS bandwidth quotas that artificially throttle multi-threaded bursts even when 80% of the host nodeโ€™s CPU cores are sitting completely idle.
7 What Kubernetes Linux capability drop is considered the mandatory Zero-Trust baseline for every production container? securityContext.capabilities.drop: ["ALL"], combined with runAsNonRoot: true and seccompProfile.type: RuntimeDefault.
8 How do you prevent Kubernetes from automatically mounting the ServiceAccount API token into pods that donโ€™t talk to kube-apiserver? Set automountServiceAccountToken: false on the Pod spec or ServiceAccount.
# One-Line Question One-Line Answer
1 How do you prevent attackers from bypassing Amazon CloudFront + AWS WAF and sending requests directly to your origin AWS Application Load Balancer IP? Use CloudFront VPC Origins (keeping the ALB 100% private) or restrict the ALB Security Group to the AWS-managed com.amazonaws.global.cloudfront.origin-facing prefix list + verify a rotated custom secret header in WAF.
2 What is CloudFront Origin Shield, and how does it protect backend Kubernetes clusters during traffic spikes? An additional centralized caching layer placed in front of your origin that collapses duplicate cache misses from 600+ global Edge PoPs into a single origin fetch.
3 What replaces legacy S3 Origin Access Identity (OAI) for securing private S3 buckets behind CloudFront with KMS support? Origin Access Control (OAC)โ€”supports SSE-KMS encrypted S3 buckets, dynamic PUT/POST requests, and AWS Signature Version 4 (SigV4).
4 What is the difference between CloudFront Functions and Lambda@Edge? CloudFront Functions run in < 1ms at all 600+ Edge PoPs for lightweight header/URL rewrites at 1/6th the cost; Lambda@Edge runs full Node.js/Python at 13 Regional Edge Caches for heavy origin/network logic.
5 Why does ingesting 20 TB/day of raw debug logs into Datadog, Splunk, or CloudWatch Logs bankrupt enterprise observability budgets? Commercial log indexers charge $1.50โ€“$3.00 per GB ingested + retention fees, turning 20 TB/day into a $15M+/year bill.
6 How does the Hot / Warm / Cold Tiered Log Architecture cut observability spend by 80%? Vector / FluentBit routes ERROR/SECURITY logs to Hot OpenSearch (7 days), operational metrics to Prometheus, and streams 100% of raw verbose logs compressed in Zstandard Parquet directly to Cold S3 queried via Amazon Athena.
7 What agent written in Rust processes and transforms logs at 10ร— the throughput and a fraction of the memory of legacy Logstash/Fluentd? Vector (by Datadog) or Fluent Bit (written in C).
8 What HTTP Cache-Control directive allows CloudFront to immediately serve a slightly expired cached object to the user while asynchronously refreshing it from the origin in the background? Cache-Control: max-age=60, stale-while-revalidate=300, stale-if-error=86400.
# One-Line Question One-Line Answer
1 Why can backend APM dashboards be 100% green (200 OK) while 30% of real customers in India are staring at a broken blank screen? Backend metrics cannot see client-side ISP BGP blackholes (Jio/Airtel routing issues), CDN SSL handshake failures, or fatal React JavaScript crashesโ€”requiring Real User Monitoring (RUM).
2 What are the 3 Google Core Web Vitals tracked by RUM SDKs? LCP (Largest Contentful Paint < 2.5s), INP (Interaction to Next Paint < 200ms), and CLS (Cumulative Layout Shift < 0.1).
3 What W3C standard HTTP header connects a userโ€™s browser click in RUM all the way through API Gateways, Kafka queues, and database queries? traceparent (00-<32-hex-trace-id>-<16-hex-span-id>-<flags>).
4 What is the critical difference between Head-Based Sampling and Tail-Based Sampling in OpenTelemetry? Head-based makes a random coin-flip decision (1%) when the request enters (missing 99% of rare errors); Tail-based buffers spans in the OTel Collector and keeps 100% of traces containing errors or high latency while sampling normal 200 OK traces at 1%.
5 Why must all spans sharing the same trace_id be routed to the exact same OpenTelemetry Collector pod when using Tail-Based Sampling? Tail sampling requires inspecting the complete trace tree in memory; a Load-Balancing OTel Exporter (routing_key: traceID) hashes spans by trace_id to consistent collector pods.
6 Define SLI, SLO, and SLA in one crisp line. SLI is the actual measurement (99.92% good requests), SLO is the internal engineering target (99.95%), and SLA is the external legal contract with financial penalties (99.90%).
7 How many minutes of downtime per 30-day month are allowed by a 99.9% (โ€œThree Ninesโ€) vs 99.99% (โ€œFour Ninesโ€) SLO? 99.9% = 43.2 minutes/month; 99.99% = 4.32 minutes/month.
8 Why does the Google SRE Multi-Window Multi-Burn-Rate Alert check both a 1-hour window and a 5-minute window (14.4x burn rate) before paging an engineer? The 1h window confirms a real threat consuming 2% of monthly budget, while the 5m window ensures the alert stops firing immediately as soon as the outage ends.
# One-Line Question One-Line Answer
1 What is the fundamental difference between a KPI (Key Performance Indicator) and a KRI (Key Risk Indicator)? A KPI measures how efficiently you are achieving goals (e.g., Deployments/Day), whereas a KRI is a leading early-warning metric measuring how close you are to a catastrophic failure or breach (e.g., % of unpatched critical CVEs or certificates expiring in <7 days).
2 What are the 4 DORA (DevOps Research and Assessment) metrics used to measure engineering platform velocity and stability? Deployment Frequency, Lead Time for Changes, Change Failure Rate (CFR), and Failed Deployment Recovery Time (MTTR).
3 Why does traditional perimeter security (โ€œtrusted internal VPC subnetโ€) fail in modern cloud breaches? Once an attacker compromises a single public pod via SSRF or dependency hijacking, a flat trusted VPC allows unrestricted lateral movement across all internal databases.
4 What are the 3 pillars of Zero-Trust Architecture (NIST 800-207) inside a Kubernetes and Cloud platform? (1) Verify cryptographic workload identity on every call (SPIFFE/mTLS), (2) Enforce L7 least-privilege access, and (3) Assume breach with continuous runtime behavioral monitoring (eBPF/Falco).
5 How does Falco or Cilium Tetragon detect a container breakout or reverse shell in real time? By attaching eBPF probes to Linux kernel syscalls (execve, connect, openat), alerting or killing the process in kernel-space the millisecond a web container spawns /bin/sh or reads /etc/shadow.
6 What is Continuous Compliance as Code compared to traditional manual point-in-time SOC2/PCI-DSS spreadsheet audits? Automated agents (AWS Security Hub, Config, Kyverno, Kubescape) continuously stream cryptographic evidence of control enforcement into an immutable audit lake 24/7.
7 How do you guarantee that even a rogue AWS Organization Admin cannot delete or alter historical CloudTrail security logs during a breach? Stream Organization CloudTrail to a dedicated Log Archive Account S3 bucket with S3 Object Lock in Compliance Mode and an SCP blocking s3:PutBucketObjectLockConfiguration.
8 What KRI metric warns a Platform Architect that an automated Kubernetes node-autoscaling cliff is imminent? Subnet IPv4 Exhaustion KRI (Available IPv4 Addresses per Subnet < 15%) and AWS Service Quota Utilization KRI (vCPU Quota > 80%).
# One-Line Question One-Line Answer
1 How do you prevent 700 AWS accounts from creating overlapping 10.0.0.0/16 VPC CIDRs without maintaining a manual Excel spreadsheet? Deploy AWS VPC IPAM (IP Address Manager) at the Organization root to programmatically allocate non-overlapping CIDRs from regional pools via Terraform (aws_vpc_ipam_pool_cidr_allocation).
2 Why is Shared VPC (AWS RAM Subnet Sharing) preferred over creating 700 separate VPCs for 700 microservices? Central Networking owns the VPC, route tables, and TGW attachments in one Network Account and shares specific subnets via AWS Resource Access Manager (RAM) into participant accounts, slashing NAT/endpoint sprawl and centralizing firewall governance.
3 How is an AWS account vended automatically in under 15 minutes when an engineering team submits a request in Backstage? Account Factory for Terraform (AFT) triggers a GitOps pipeline that creates the account, moves it into the target OU, attaches SCPs, allocates an IPAM VPC, and registers it with Central Hub ArgoCD.
4 What happens when AWS GuardDuty detects cryptocurrency mining or IAM credential exfiltration inside one of your 700 AWS accounts? EventBridge routes the High-Severity finding to a central SOAR Step Function that automatically attaches a Quarantine Deny-All Security Group, snapshots the EBS volume for forensics, and revokes active IAM Role sessions.
5 How do you revoke an active compromised AWS IAM Role session immediately across all running instances? Attach the managed inline policy AWSRevokeOlderSessions, which denies all API actions (*:*) where aws:TokenIssueTime is earlier than the current timestamp.
6 Why do enterprises enforce Just-In-Time (JIT) Break-Glass Access with zero standing admin privileges in production accounts? Standing AdministratorAccess roles are permanent targets for phishing and session hijacking; JIT grants temporary elevated access for 1โ€“2 hours tied to an active Sev-1 PagerDuty incident ID.
7 How do you deploy a mandatory security baseline (like AWS Config rules or a forensic IAM role) across all 700 accounts automatically? Use CloudFormation StackSets with Service-Managed Permissions or AFT Global Account Customizations at the AWS Organization root.
8 What AWS service aggregates GuardDuty, Inspector, Macie, and CIS benchmark findings from 700 accounts into a single normalized dashboard? AWS Security Hub (configured with a Delegated Administrator account and cross-region aggregation).
# One-Line Question One-Line Answer
1 Why must you never execute database schema migrations (flyway migrate / alembic upgrade) inside the application containerโ€™s startup entrypoint script? When Kubernetes scales up 20 pods simultaneously, all 20 pods race to run DDL migrations at the exact same millisecond, causing schema lock contention, timeouts, and CrashLoopBackOff.
2 What is the industry-standard pattern for running database migrations in Kubernetes GitOps? Run a single-pod ArgoCD PreSync Job (or Helm pre-upgrade hook) backed by an explicit database advisory lock (pg_advisory_lock) before rolling out new application pods.
3 Why does running ALTER TABLE orders ADD COLUMN status VARCHAR(20) DEFAULT 'NEW' in older DB engines (or adding a NOT NULL constraint without NOT VALID in PostgreSQL) take down a high-traffic table? It acquires an ACCESS EXCLUSIVE lock that blocks both reads (SELECT) and writes (INSERT/UPDATE) while scanning the table, piling up application threads in seconds!
4 How do you add a FOREIGN KEY or CHECK constraint to a 500-million-row PostgreSQL table with zero table locking downtime? Run ALTER TABLE orders ADD CONSTRAINT fk_user FOREIGN KEY (user_id) REFERENCES users(id) NOT VALID; (instant metadata lock), followed by ALTER TABLE orders VALIDATE CONSTRAINT fk_user; (which takes only a non-blocking SHARE UPDATE EXCLUSIVE lock).
5 What mandatory SQL session timeout must be set at the top of every production migration script to prevent a blocked DDL statement from queuing all application SELECT queries? SET lock_timeout = '2s'; SET statement_timeout = '10s';โ€”if the migration cannot acquire the table lock within 2 seconds, it aborts harmlessly instead of freezing production!
6 Why must you always use CREATE INDEX CONCURRENTLY instead of CREATE INDEX on live PostgreSQL production tables? Standard CREATE INDEX locks the table against all writes (INSERT/UPDATE/DELETE) for the entire build duration; CONCURRENTLY builds the index in the background without blocking concurrent writes.
7 What are the 3 phases of the Expand-and-Contract (Parallel Change) Pattern for renaming or splitting a column with zero downtime? (1) Expand: Add new nullable column & deploy dual-write code โ†’ (2) Migrate: Backfill historical rows in small batches & switch reads to new column โ†’ (3) Contract: Remove old column references in code, then drop old column in a future release.
8 How do you break a Terraform / Microservice Cyclic Dependency Deadlock (Service A needs Service Bโ€™s URL, and Service B needs Service Aโ€™s URL, or Security Group A and B reference each other)? Decouple creation from wiring: provision deterministic Route 53 Internal DNS records / SSM Parameters or empty aws_security_group shells first, then attach aws_vpc_security_group_ingress_rule resources separately.
# One-Line Question One-Line Answer
1 Why is tracking โ€œTotal Monthly Cloud Billโ€ a flawed metric for high-growth companies, and what replaces it? Total spend naturally rises when customer traffic doubles; Staff FinOps Architects track Unit Economics (e.g., Cloud Cost per 1,000 Checkout Transactions or Cost per Active User).
2 What tool integrates into GitHub Pull Requests to calculate the exact monthly dollar impact (+$1,420/mo) of a Terraform code change before it is merged? Infracost (paired with OPA/Conftest policies to block PRs exceeding budget thresholds without FinOps approval).
3 What open-source CNCF project runs inside Kubernetes to calculate real-time cost per namespace, deployment, and label by combining node billing APIs with cgroup usage? OpenCost (the open-source engine powering Kubecost).
4 What causes the โ€œKubernetes Slack Cost / Over-Provisioning Gapโ€ where nodes are 85% allocated in kubectl describe node but actual hardware CPU usage is only 12%? Developers set inflated resources.requests.cpu values (โ€œjust to be safeโ€), forcing Karpenter/Autoscaler to rent EC2 nodes for reserved capacity that is never actually consumed.
5 How do you safely automate rightsizing inflated Kubernetes CPU and Memory requests across 500 microservices? Deploy Goldilocks / Vertical Pod Autoscaler (VPA) in Recommender mode alongside OpenCost, and enforce rightsizing patches aligned to p95 historical consumption + 20% buffer.
6 What is the difference between AWS Compute Savings Plans and EC2 Instance Savings Plans? Compute Savings Plans automatically apply across any EC2 instance family, size, OS, region, Fargate, or Lambda (up to 66% discount); EC2 Instance Savings Plans lock you to a specific instance family in one region for a slightly higher discount (up to 72%).
7 Why is targeting 100% Savings Plan / Reserved Instance coverage a dangerous FinOps anti-pattern? It leaves zero room for seasonal traffic dips, architecture optimizations, or Spot adoption; the enterprise standard is 75โ€“85% Commitment Coverage purchased in staggered quarterly tranches (Commitment Laddering).
8 What are the 4 highest-ROI technical cost optimizations a Platform Architect can execute in week 1 without touching application business logic? (1) Migrate x86 to AWS Graviton3/4 (m7g/c7g), (2) Convert all EBS gp2 volumes to gp3, (3) Add S3/DynamoDB Gateway VPC Endpoints to bypass NAT Gateways, and (4) Enable S3 Intelligent-Tiering.
# One-Line Question One-Line Answer
1 What are the 7 Rs of Cloud Migration, and which two yield the highest long-term cloud-native ROI vs fastest datacenter exit? Retain, Retire, Relocate, Rehost (Lift-and-Shift โ€” fastest exit), Replatform (Lift-and-Tinker), Repurchase, and Refactor / Re-architect (Cloud-Native โ€” highest ROI).
2 What is the Strangler Fig Pattern (coined by Martin Fowler) for migrating a 15-year-old monolithic ERP/Banking application to Kubernetes microservices? Place an API Gateway / Envoy Proxy in front of the legacy monolith and incrementally peel off one bounded domain route at a time (/api/payments, /api/users) to new cloud-native microservices until the monolith shrinks away.
3 What is an Anti-Corruption Layer (ACL) in a Strangler Fig migration? A translation adapter layer between the new clean microservice domain model and the legacy monolithโ€™s messy database schema so legacy technical debt does not pollute the new architecture.
4 How do you migrate a 20 TB on-premises Oracle or PostgreSQL database to AWS Aurora with less than 60 seconds of cutover downtime? Use AWS Database Migration Service (DMS) in Full Load + Continuous CDC (Change Data Capture) mode, which streams ongoing binary transaction logs until cutover.
5 Why must you drop or disable secondary indexes and foreign keys on the target database during the DMS Full Load phase, but rebuild them before enabling CDC? Indexes slow bulk INSERT speed by 10ร— during Full Load; however, if indexes are missing when CDC starts applying UPDATE/DELETE statements, the target DB performs full table scans on every row change and stalls!
6 What is the safest way to validate that a newly migrated cloud database and microservice produce identical financial results to the on-prem system before cutover? Run Parallel Run / Dual-Read Comparison (Diffy pattern) where live traffic is processed by both systems and an asynchronous comparator verifies 100% payload parity.
7 How do you guarantee an instant < 30-second rollback if an unexpected bug appears 2 hours after cutting over writes to the new cloud database? Set up Reverse CDC Replication (replicating changes from the new Cloud DB back to the old On-Prem DB) so the on-prem database remains warm and up-to-date for instant fallback.
8 How do you solve the IP Hardcoding Trap where legacy on-premises apps have hardcoded private IPs (192.168.10.55) in compiled binaries? Extend the Layer-2/Layer-3 subnet via AWS Transit Gateway + Direct Connect during transition or intercept connections using local iptables / Envoy Virtual IPs pointing to cloud PrivateLink endpoints.
# One-Line Question One-Line Answer
1 State Littleโ€™s Law (L = ฮป ร— W) and explain how it dictates microservice pod capacity during a traffic spike. The number of concurrent in-flight requests (L) equals arrival rate (ฮป req/s) ร— response latency (W seconds); if downstream latency W jumps 10ร— (from 50ms to 500ms), required thread/connection capacity L instantly explodes 10ร— at the exact same traffic rate!
2 What is the PostgreSQL / HikariCP mathematical formula for sizing optimal active database connections per database server? Optimal Connections = (CPU Core Count ร— 2) + Effective Spindle Count (e.g., a 16-vCPU RDS instance performs best at ~35โ€“50 active concurrent query threads, not 5,000!).
3 If 500 Kubernetes pods each open a pool of 20 database connections (500 ร— 20 = 10,000 connections), how do you prevent PostgreSQL from crashing from connection context-switching? Place PgBouncer (in transaction pooling mode) or AWS RDS Proxy in front of PostgreSQL to multiplex 10,000 client sockets over 50 warm backend database connections.
4 What causes AWS RDS Proxy Connection Pinning, which silently disables transaction multiplexing and ruins connection pooling? Using session-level state such as SET variables, prepared statements, temporary tables, or advisory locks pins the client to a dedicated DB connection for the entire session.
5 Why do Reactive Auto-Scalers (HPA + Karpenter) fail during an Indian IPL Cricket toss or Flash Sale where traffic jumps from 10k RPS to 500k RPS in 8 seconds? Even fast Karpenter + container cold starts take 45โ€“60 seconds; by the time new nodes register, the 500k RPS wave has already OOMKilled and cascaded across existing pods!
6 How do PhonePe, Hotstar, and Swiggy survive instant 50x IPL/Festival spikes without waiting for reactive autoscaling? Predictive Pre-Scaling (Scheduled KEDA Cron Scalers + Pre-Warmed AWS ALBs) 30 minutes before the event, combined with Edge Waiting Rooms and Graceful Feature Degradation.
7 What is the XFetch Algorithm (Probabilistic Early Expiration), and how does it prevent a Redis Cache Stampede (Dogpile Effect) when a celebrity itemโ€™s cache TTL expires? Instead of waiting for TTL = 0 (when 10,000 threads miss cache simultaneously), a single worker probabilistically recomputes the cache slightly before expiration using Time - ฮฒ ยท ฮด ยท ln(rand()) โ‰ฅ TTL.
8 What Go concurrency primitive collapses 5,000 simultaneous identical cache-miss requests inside a pod into a single shared database query? singleflight.Group (golang.org/x/sync/singleflight).
# One-Line Question One-Line Answer
1 What is the golden rule of a Sev-1 War Room during the first 15 minutes of a catastrophic outage? Mitigate Customer Impact First, Diagnose Root Cause Secondโ€”stop the bleeding via rollback, traffic failover, feature flag kill-switch, or load shedding before debugging code.
2 What are the 3 distinct roles in an incident command structure (ICS)? Incident Commander (IC) (single decision authority, runs no CLI commands), Operations Lead (Ops) (executes technical mitigations), and Communications Lead (Comms) (updates status page & executives every 15m).
3 Why must the Incident Commander strictly forbid engineers from running unannounced kubectl delete or config edits during a Sev-1 call? Uncoordinated mutations destroy forensic evidence and frequently compound a single-zone degradation into a total global outage.
4 What is Load Shedding vs Graceful Degradation during a saturation crisis? Load Shedding drops excess requests cheaply at the edge (HTTP 503 + Retry-After) to protect core server survival; Graceful Degradation disables non-essential features (recommendations/reviews) so core checkout succeeds.
5 Why must a Load Shedder use a LIFO (Last-In, First-Out) or CoDel queue instead of a long FIFO queue during an overload? In a deep FIFO queue, every request waits 10 seconds only to be processed after the client already timed out; LIFO serves fresh requests immediately in <50ms and drops stale expired requests.
6 What is Metastable Failure (Latching Overload), where a cluster remains 100% dead at 5xx even after external traffic drops back down to normal low levels? Retry storms, exhausted connection pools, and JVM garbage collection thrashing lock the system into a self-sustaining failure loop that cannot recover without shedding 100% of load or resetting retry queues.
7 In a Blameless Postmortem (COE โ€” Correction of Errors), why is โ€œHuman Error โ€” Engineer ran wrong commandโ€ never accepted as a Root Cause? Human error is always a symptom of missing system guardrails; the true root cause is the systemic gap that allowed a single command or unvalidated config to reach production without automated safety gates.
8 What are the two categories of Postmortem Action Items required before closing a Sev-1 COE? Prevention Items (reducing probability of recurrence, e.g., OPA policy gate) and Detection / Mitigation Items (reducing MTTR, e.g., automated canary rollback in <60s).
๐Ÿ† Capstone & Incident Triage Directory 5 Rapid Q&As

Capstone: The Self-Service Cloud Engine

# One-Line Question One-Line Answer
1 Trace the complete end-to-end flow of the Self-Service Cloud Engine Capstone when a developer clicks "Create Production Service" in Backstage. Backstage Scaffolder opens a Git PR โ†’ Checkov/OPA validates policy โ†’ ArgoCD ApplicationSet syncs manifests โ†’ Crossplane KCL provisions AWS RDS/S3 โ†’ ESO/Vault injects secrets โ†’ Cilium + Karpenter runs the workload.
2 Why does the Capstone architecture use Terraform for the base Landing Zone / EKS bootstrap and Crossplane for developer-requested databases and queues? Terraform excels at foundational static cluster/network bootstrapping, while Crossplane empowers self-service developers to request app-scoped cloud resources via Kubernetes YAML reconciled continuously by ArgoCD.
3 How does the Capstone architecture guarantee that no microservice can ever be deployed to production without an SBOM, a Cosign signature, and an active SLO? GitHub Actions signs the image with Cosign and attaches the CycloneDX SBOM, Kyverno verifies both at admission time, and Backstage Scorecards gate ArgoCD promotion on valid SLO definitions.
4 How does the Capstone platform achieve zero-trust multi-tenancy across engineering teams sharing the same EKS cluster? Each team gets an isolated namespace governed by ResourceQuotas, Cilium L7 Default-Deny NetworkPolicies, dedicated IRSA roles, and OpenCost chargeback tagging.
5 What makes this Capstone architecture the ultimate Staff/Principal interview portfolio piece for the Indian and global tech market? It unifies Day-0 Architecture (Landing Zones/TGW), Day-1 Golden Paths (Backstage/Crossplane/ArgoCD), and Day-2 SRE Operations (eBPF/Chaos Mesh/OpenCost/vLLM) into one cohesive production platform.