☸️ Round 2: Kubernetes Internals, Controllers & Production Scheduling

In Staff Platform Architect and Principal SRE interviews across Indian tech leaders (Flipkart, Swiggy, Razorpay, Uber IDC, Atlassian, Google Cloud), Round 2 moves past manifest syntax into control plane mechanics.

Interviewers expect you to explain how the API Server handles concurrency via MVCC, how etcd executes Raft consensus, how the scheduler scores nodes, how kubelet coordinates with CRI/CNI/CSI, and how to recover from fatal webhook freezes.

📌 Question 1: Trace kubectl apply -f deployment.yaml Under the Hood

🎯 The Scenario & Interviewer’s Trap

“You execute kubectl apply -f deployment.yaml against an Amazon EKS / Google GKE cluster. Trace the lifecycle of this request from the moment the CLI executes until the container’s process is running and accepting traffic on the worker node. List every subsystem, validation phase, and controller involved.”

Interviewer’s Intent: To test if you understand the asynchronous, decoupled, level-triggered architecture of Kubernetes or if you think kube-apiserver directly creates containers.

💡 Staff-Level Model Answer

flowchart TD
    subgraph Client["1. Client Phase"]
        CLI["kubectl apply"] --> ClientGo["client-go discovers OpenAPI schema & serializes JSON"]
    end

    subgraph APIServer["2. kube-apiserver Pipeline"]
        AuthN["Authentication (TLS Cert / OIDC Webhook / AWS IAM)"]
        AuthZ["Authorization (RBAC ClusterRole / Webhook)"]
        Mutating["Mutating Admission Webhooks (Linkerd, Kyverno, etc.)"]
        Schema["Object Schema Validation (OpenAPI v3)"]
        Validating["Validating Admission Webhooks (OPA Gatekeeper)"]
        AuthN --> AuthZ --> Mutating --> Schema --> Validating
    end

    subgraph Storage["3. Persistent State"]
        Validating --> etcd["etcd Raft Consensus (WAL fsync + MVCC Commit)"]
    end

    subgraph Controllers["4. Control Plane Reconciliation"]
        etcd -->|Watch Event| DepCtrl["Deployment Controller (creates ReplicaSet)"]
        DepCtrl -->|Watch Event| RSCtrl["ReplicaSet Controller (creates Pod specs with Pending status)"]
        RSCtrl -->|Watch Event| Sched["kube-scheduler (Filter -> Score -> Reserve -> Bind)"]
    end

    subgraph Worker["5. Node Execution (kubelet)"]
        Sched -->|Assigns nodeName| Kubelet["kubelet PLEG (Pod Lifecycle Event Generator)"]
        Kubelet --> CRI["CRI (containerd): Pull Image, Create Namespaces"]
        Kubelet --> CNI["CNI (AWS VPC CNI / Cilium): Allocate IP & veth pair"]
        Kubelet --> CSI["CSI (EBS CSI): Attach & Format/Mount Filesystem"]
        CRI --> Container["Application Container Process Starts (PID in netns)"]
    end

    ClientGo --> AuthN

Step-by-Step Breakdown:

1. Client-Side Processing: kubectl loads ~/.kube/config, contacts the API discovery endpoint (/openapi/v2), converts YAML to JSON, and executes a PATCH or POST request over TLS.
2. API Server Request Pipeline:
   • Authentication (AuthN): Validates caller identity via x509 client certificates, OIDC bearer tokens, or Webhook Token Authenticators (e.g., aws-iam-authenticator).
   • Authorization (AuthZ): Evaluates RBAC rules (RoleBinding / ClusterRoleBinding) to verify verbs (create, patch) against the resource (deployments.apps).
   • Mutating Webhooks: Invoked sequentially. Webhooks can inject sidecars (Istio, Vault Agent) or modify fields.
   • Schema Validation: Verifies required fields against registered OpenAPI schemas.
   • Validating Webhooks: Invoked in parallel (e.g., OPA Gatekeeper, Kyverno). If any webhook rejects with allow: false, the entire request is aborted with HTTP 400/403.
3. etcd Persistence:
   • The API Server writes the validated Deployment resource to etcd.
   • etcd writes the record to its Write-Ahead Log (WAL), executes Raft consensus across a majority quorum (N/2 + 1), and commits the change to its bbolt MVCC key-value store, returning a new resourceVersion.
4. Asynchronous Controller Reconciliation Loops:
   • Deployment Controller: Watches etcd for deployment changes via HTTP/2 streaming. It sees the new Deployment, creates a corresponding ReplicaSet, and writes it to the API Server.
   • ReplicaSet Controller: Sees the ReplicaSet, calculates desired vs current replicas, creates the requested number of Pod objects with spec.nodeName: "" (Pending state), and writes them to etcd.
5. Scheduling Phase (kube-scheduler):
   • The scheduler watches for unassigned pods (nodeName == "").
   • Filtering (Predicates): Filters out nodes lacking resources (CPU/RAM), failing Taints/Tolerations, or failing NodeAffinity rules.
   • Scoring (Priorities): Ranks candidate nodes using scoring algorithms (LeastAllocated, ImageLocality, TopologySpread).
   • Reserve & Permit: Temporarily reserves node resources and executes any Permit plugins.
   • Binding: Writes a Binding object back to the API Server, populating pod.spec.nodeName.
6. Worker Node Execution (kubelet):
   • The kubelet on the target worker node detects the pod assigned to its node via its Informer watch loop.
   • CSI Attachment & Mount: Kubelet calls the CSI driver to attach remote block storage (e.g., EBS) and mount the filesystem into /var/lib/kubelet/pods/<pod-uid>/volumes.
   • CRI Container Runtime: Kubelet instructs containerd via gRPC to pull container images, set up Linux cgroups, and initialize namespaces.
   • CNI Network Provisioning: containerd calls the CNI plugin (e.g., AWS VPC CNI, Cilium) to allocate an IP address, configure host-to-pod veth interfaces, and set up routing rules.
   • Container Start: containerd launches the container process and marks the Pod Running.

📌 Question 2: etcd Raft Consensus, Quorum Loss & Linearizable Reads

🎯 The Scenario & Interviewer’s Trap

“Your production Kubernetes control plane runs a 3-node etcd cluster spread across 3 AWS Availability Zones. AZ-a suffers a catastrophic network isolation event. What happens to etcd? How does Raft handle leader election, what is the mathematical quorum requirement, and what is the difference between a serializable read and a linearizable read?”

Interviewer’s Intent: To verify whether you understand distributed consensus theory, split-brain protection, and how etcd avoids serving stale reads during network partitions.

💡 Staff-Level Model Answer

flowchart TD
    subgraph Cluster["3-Node etcd Cluster across 3 AZs"]
        AZ1["Node 1 (Leader) in AZ-a<br/>(ISOLATED!)"]
        AZ2["Node 2 (Follower) in AZ-b"]
        AZ3["Node 3 (Follower) in AZ-c"]
    end

    AZ1 -.->|Network Partition| AZ2
    AZ1 -.->|Network Partition| AZ3
    AZ2 <-->|Heartbeats Active| AZ3

    subgraph Election["Raft Quorum: ceil((3+1)/2) = 2 nodes"]
        AZ2 -->|Election Timeout fires| AZ2_Cand["Node 2 becomes Candidate"]
        AZ2_Cand -->|Requests Vote| AZ3
        AZ3 -->|Grants Vote| AZ2_Cand
        AZ2_Cand --> NewLeader["Node 2 becomes New Raft Leader!"]
    end

1. Quorum Mathematics:

In etcd (and Raft in general), the quorum requirement is: 📐 Quorum = ⌊ N / 2 ⌋ + 1

• For a 3-node cluster (N=3): Quorum is ⌊ 3/2 ⌋ + 1 = 2 nodes.
• Fault Tolerance: Can survive f node failures, where N = 2f + 1:
  • 3 nodes survive f = 1 failure.
  • 5 nodes survive f = 2 failures.
• Why an odd number of nodes? A 4-node cluster also requires a quorum of ⌊ 4/2 ⌋ + 1 = 3 nodes. Adding a 4th node does not increase fault tolerance (it still tolerates only 1 failure) while increasing network communication overhead!

2. Network Partition Behavior:

• The Minority Partition (Node 1 in AZ-a):
  • Node 1 cannot contact Nodes 2 and 3. When the API Server attempts a write to Node 1, Node 1 cannot achieve quorum replication. The write fails or hangs.
• The Majority Partition (Nodes 2 & 3 in AZ-b and AZ-c):
  • Nodes 2 and 3 stop receiving leader heartbeats. Node 2’s randomized election timeout expires, increments its Raft term, transitions to Candidate, and requests votes from Node 3.
  • Node 3 votes for Node 2. Node 2 receives 2/3 votes, achieves quorum, and becomes the new leader.
  • The cluster continues processing writes and reads without interruption.

3. Serializable Reads vs. Linearizable Reads:

• Serializable Read (Stale Read Risk):
  • The local node satisfies the read directly from its in-memory bbolt database without communicating with other nodes.
  • Danger: If Node 1 is partitioned, an API server querying Node 1 with serializable reads would receive stale state that does not reflect recent writes committed by Node 2.
• Linearizable Read (Default in Kubernetes):
  • The leader node confirms with a quorum of peers (via a Raft ReadIndex heartbeat exchange) that it is still the legitimate leader before serving the read.
  • Guarantees strong consistency: clients are guaranteed to see the latest committed write.

How to Triage etcd Health:

# Check cluster membership and health:
etcdctl endpoint health --cluster -w table

# Check leader status and Raft term:
etcdctl endpoint status --cluster -w table

# Check WAL disk fsync latency (must be < 10ms):
etcdctl check perf

📌 Question 3: The Fatal Admission Webhook Freeze & Rescue

🎯 The Scenario & Interviewer’s Trap

“An engineer configures a ValidatingWebhookConfiguration enforcing custom compliance rules. The webhook server runs as a pod inside the cluster. Overnight, all worker nodes reboot due to an automated AMI patch. When the nodes restart, the webhook pods fail to start. Suddenly, the entire Kubernetes cluster freezes: no pods can start, kubectl apply hangs indefinitely, and even kubectl delete validatingwebhookconfiguration times out. How did this deadlock occur, and how do you recover the cluster?”

Interviewer’s Intent: To test if you understand the cyclic dependency between in-cluster admission webhooks and API Server request processing, and whether you know how to execute emergency surgical recovery.

💡 Staff-Level Model Answer

flowchart TD
    Reboot["Worker Nodes Reboot"] --> Evict["Webhook Pods Evicted / Destroyed"]
    Evict --> AttemptStart["Kubelet attempts to recreate Webhook Pods"]
    AttemptStart --> APICall["Kubelet sends Pod Create request to API Server"]
    APICall --> Intercept["API Server intercepts request: ValidatingWebhookConfiguration has failurePolicy: Fail"]
    Intercept --> CallWebhook["API Server attempts to contact Webhook Service at 10.100.20.5:443"]
    CallWebhook --> Timeout["Webhook Pod is NOT RUNNING! Connection Times Out after 30s!"]
    Timeout --> Reject["API Server REJECTS the pod creation!"]
    Reject --> Deadlock["DEADLOCK: Webhook cannot start because Webhook is required to start itself!"]

1. Why the Deadlock Occurred:

• The ValidatingWebhookConfiguration was configured with:

  failurePolicy: Fail
  rules:
    - operations: ["CREATE", "UPDATE"]
      apiGroups: ["*"]
      apiVersions: ["*"]
      resources: ["pods", "deployments"]

• When nodes rebooted, kubelet attempted to launch the webhook pod.
• The API Server intercepted the Pod creation request. Because failurePolicy: Fail was set, the API Server required approval from the webhook service.
• But the webhook service has 0 ready endpoints because its pods are currently trying to start!
• The API Server waited for the webhook timeout (default 10–30s) and aborted the pod creation. Classic chicken-and-egg cyclic deadlock.

2. The 3 Recovery Procedures:

Strategy A: Emergency Master Node Manifest Bypass (Fastest)

If you have SSH access to control plane nodes (or self-hosted control planes):

# Temporarily edit the kube-apiserver static pod manifest:
vim /etc/kubernetes/manifests/kube-apiserver.yaml

# Add this flag to temporarily disable admission plugins:
--disable-admission-plugins=ValidatingAdmissionWebhook,MutatingAdmissionWebhook

# kubelet automatically restarts kube-apiserver.
# Delete the broken webhook:
kubectl delete validatingwebhookconfiguration <broken-webhook-name>

# Re-enable webhooks in kube-apiserver.yaml

Strategy B: Direct etcd Surgical Purge

If the API Server is completely non-responsive to API requests:

# Directly query and delete the webhook object from etcd:
ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  del /registry/validatingwebhookconfigurations/<broken-webhook-name>

Strategy C: Architectural Prevention (The Staff Answer)

Never run webhooks that intercept core system namespaces without strict safeguards:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: security-validator
webhooks:
  - name: validator.corp.com
    failurePolicy: Ignore     # <-- Non-blocking during outages!
    timeoutSeconds: 3
    namespaceSelector:
      matchExpressions:
        - key: kubernetes.io/metadata.name
          operator: NotIn
          values: ["kube-system", "kube-node-lease", "webhook-system"] # Exclude system namespaces!

📌 Question 4: StatefulSets & EBS CSI Multi-AZ Scheduling Deadlocks

🎯 The Scenario & Interviewer’s Trap

“You run a Kafka StatefulSet on Amazon EKS backed by AWS EBS gp3 volumes. Worker Node-1 in us-east-1a suffers an AWS hardware failure and crashes. Karpenter immediately provisions a new replacement EC2 instance in us-east-1b because that AZ has cheaper spot capacity. The Kafka pod is scheduled on the new node, but stays permanently trapped in ContainerCreating with VolumeAttachment failed: VolumeInUse or ZoneMismatch. What happened, and how do you design storage classes to prevent this?”

Interviewer’s Intent: To test if you understand that EBS block volumes are physically tied to a single AWS Availability Zone, and how Kubernetes topology-aware volume scheduling works.

💡 Staff-Level Model Answer

flowchart TD
    subgraph AZa["Availability Zone: us-east-1a"]
        EBS["EBS Volume: vol-0123<br/>(Hard-locked to us-east-1a SAN fabric)"]
        Node1["Node-1 (FAILED / CRASHED)"]
        EBS -.->|Attached| Node1
    end

    subgraph AZb["Availability Zone: us-east-1b"]
        Node2["Node-2 (New Karpenter Node launched in AZ-b)"]
        Pod["Kafka-0 Pod scheduled here!"]
    end

    Pod -->|Requires vol-0123| EBS
    EBS -- "CANNOT CROSS PHYSICAL AZ BOUNDARY!" --> CrossErr["Error: VolumeAttachment Deadlock!"]

1. The Root Cause:

• EBS is Single-AZ: Unlike EFS (NFS) or S3 (Object Storage), Amazon EBS volumes exist on physical Storage Area Network (SAN) infrastructure inside a single Availability Zone (us-east-1a). They cannot cross physical datacenter boundaries.
• The Karpenter Race: When Node-1 died, Karpenter saw a pending pod. If the StorageClass used default immediate binding, or if the pod lacked AZ node affinity, Karpenter launched the replacement node in us-east-1b.
• The Detach Hang: Furthermore, AWS EBS has a volume lock timeout. If Node-1 crashed abruptly without cleanly unmounting the volume, the AWS EC2 API reports the volume in busy or attaching state for up to 6–10 minutes before force-detaching.

2. The Architectural Solution:

1. Topology-Aware StorageClass (WaitForFirstConsumer):

   apiVersion: storage.k8s.io/v1
   kind: StorageClass
   metadata:
     name: ebs-gp3-topology
   provisioner: ebs.csi.aws.com
   volumeBindingMode: WaitForFirstConsumer  # Crucial! Defers volume creation until pod is scheduled
   allowedTopologies:
     - matchLabelExpressions:
         - key: topology.ebs.csi.aws.com/zone
           values:
             - us-east-1a
             - us-east-1b
             - us-east-1c

2. StatefulSet Node Affinity / NodeSelector: Ensure StatefulSet pods maintain an affinity to the zone where their PersistentVolume was originally provisioned:

   spec:
     template:
       spec:
         affinity:
           nodeAffinity:
             requiredDuringSchedulingIgnoredDuringExecution:
               nodeSelectorTerms:
                 - matchExpressions:
                     - key: topology.kubernetes.io/zone
                       operator: In
                       values: ["us-east-1a"]

📌 Question 5: Karpenter vs. Cluster Autoscaler Architectural Comparison

🎯 The Scenario & Interviewer’s Trap

“Why did your engineering team migrate from Kubernetes Cluster Autoscaler to Karpenter? Detail the architectural differences in how they communicate with the cloud provider, how they handle bin-packing, and how Karpenter handles Spot instance interruptions without causing service disruptions.”

Interviewer’s Intent: To verify whether you have actually operated modern, cost-efficient Kubernetes clusters at scale, or if you still rely on legacy Auto Scaling Group (ASG) patterns.

💡 Staff-Level Model Answer

Karpenter Spot Interruption Graceful Workflow:

sequenceDiagram
    autonumber
    participant AWS as AWS EC2 Spot Fabric
    participant SQS as SQS Interruption Queue
    participant Karpenter as Karpenter Controller
    participant K8s as Kubelet & Pods

    AWS->>SQS: Spot Interruption Warning (2-minute countdown)
    SQS->>Karpenter: Event: EC2 Spot Interruption Notice
    Karpenter->>AWS: Immediately provisions replacement node (parallel spin-up)
    Karpenter->>K8s: Taints expiring node (NoSchedule) to block new pods
    Karpenter->>K8s: Issues graceful eviction (respects PDBs)
    Note over K8s: Replacement node is READY before the 2-minute deadline expires!
    K8s->>Karpenter: Pods rescheduled on new node with ZERO dropped requests!

📌 Summary: Kubernetes Internals Quick-Fire Reference

🧠 Complete Lesson-by-Lesson One-Line Memory Bank (Round 2 Scope)

☸️ Phase 4: Kubernetes & Scaling (All Lessons)

Lesson 1: EKS Architecture & IRSA

Lesson 2: Karpenter & Spot Fleets

Lesson 3: Helm & Kustomize

Lesson 4: Ingress & AWS ALB Controller

Lesson 5: Prometheus & PromQL

Lesson 6: CoreDNS ndots & Cluster Upgrades

🔄 Phase 5: GitOps & Control Planes (All Lessons)

Lesson 1: ArgoCD App of Apps

Lesson 2: Argo Rollouts & Canaries

Lesson 3: Crossplane Control Planes

Lesson 4 & 5: ApplicationSets & Hub-and-Spoke ArgoCD

🐝 Phase 6: Service Mesh & eBPF (All Lessons)

Lesson 1–4: Cilium, eBPF, L7 Policies & ClusterMesh

Phase 4 — Lesson 7: Azure AKS Internals, Cilium CNI Overlay & Azure Dependent PaaS Ecosystem