Lesson 6: CoreDNS ndots Amplification, EKS Version Upgrades & Exception Handling

🧠 The Concept (Explain Like I’m 5)

Imagine you work in a huge corporate office in Dallas, and you want to call an external vendor named stripe.com:

The ndots: 5 Problem (The Inefficient Receptionist): When you dial stripe.com, the company receptionist refuses to call the outside world first. Instead, the receptionist assumes stripe.com must be an internal employee!
1. She pages: “Is stripe.com.marketing.dallas.office here?” (No).
2. She pages: “Is stripe.com.sales.dallas.office here?” (No).
3. She pages: “Is stripe.com.it.dallas.office here?” (No).
4. She pages: “Is stripe.com.dallas.office here?” (No).
5. Only after 4 wasted internal intercom calls does she finally dial the external telephone network to reach stripe.com!
In Kubernetes, this happens on every single outbound HTTP request, multiplying your DNS traffic by 500% and causing mysterious 5-second network latency freezes across your entire cluster.
The EKS Upgrade (Changing the Airplane Engine Mid-Flight): Upgrading Kubernetes from version 1.28 to 1.30 while 500 microservices are processing customer credit card payments without dropping a single packet.

🏢 The Enterprise Context (Why Platform Architects Care)

“It’s Always DNS”: DNS is responsible for over 70% of intermittent latency spikes in production Kubernetes clusters. A misconfigured ndots setting combined with the Linux kernel’s UDP connection-tracking (conntrack) race condition causes random 5.0-second request timeouts that plague microservice communication.
Kubernetes API Deprecation Incompatibilities: Every minor Kubernetes release (e.g., 1.25, 1.28, 1.30) permanently removes deprecated APIs (such as PodSecurityPolicy, flowcontrol.apiserver.k8s.io, or older Ingress specifications). Upgrading an EKS cluster without auditing your Helm charts causes workloads to fail admission, paralyzing deployments.
The EKS Version Deprecation Deadline: AWS deprecates older Kubernetes versions every 14 months. If an enterprise fails to upgrade, AWS imposes an automatic Extended Support Surcharge ($430+ per cluster per month) and eventually forces automated in-place upgrades that can break non-compliant workloads without warning.

🗺️ Visual Architecture: The `ndots: 5` DNS Amplification Storm

sequenceDiagram
    autonumber
    actor Pod as 📦 Microservice Pod
    participant Resolver as 🐧 Linux glibc Resolver (/etc/resolv.conf)
    participant CoreDNS as 🌐 CoreDNS Service (172.20.0.10)
    participant Route53 as ☁️ External Amazon DNS (169.254.169.253)

    Pod->>Resolver: getaddrinfo("api.stripe.com")
    Note over Resolver: Domain has 2 dots ('api' . 'stripe' . 'com' < 5 dots).<br/>glibc iterates through all internal search domains first!

    Resolver->>CoreDNS: Query 1: api.stripe.com.default.svc.cluster.local
    CoreDNS-->>Resolver: NXDOMAIN (Non-Existent Domain) ❌
    Resolver->>CoreDNS: Query 2: api.stripe.com.svc.cluster.local
    CoreDNS-->>Resolver: NXDOMAIN ❌
    Resolver->>CoreDNS: Query 3: api.stripe.com.cluster.local
    CoreDNS-->>Resolver: NXDOMAIN ❌
    Resolver->>CoreDNS: Query 4: api.stripe.com.ec2.internal
    CoreDNS-->>Resolver: NXDOMAIN ❌

    Note over Resolver: 4 wasted UDP queries executed!<br/>Finally queries the absolute external domain:
    Resolver->>CoreDNS: Query 5: api.stripe.com
    CoreDNS->>Route53: Forward external query to upstream DNS
    Route53-->>CoreDNS: Returns 54.187.205.12 ✅
    CoreDNS-->>Resolver: Returns IP Address to Pod

🔬 Under the Hood: The `ndots` Math & Linux Resolver Rules

Inside every single Pod running in Kubernetes, the kubelet automatically synthesizes /etc/resolv.conf:

nameserver 172.20.0.10
search default.svc.cluster.local svc.cluster.local cluster.local ec2.internal
options ndots:5

The `ndots: 5` Specification

What ndots means: If a domain name contains strictly fewer than ndots dots, the system resolver appends each item in the search domain list sequentially before querying the raw domain name.
The Calculation:
- For internal service payment-service: Dots = 0 (< 5). Resolves on query 1: payment-service.default.svc.cluster.local ✅.
- For external service api.stripe.com: Dots = 2 (< 5). glibc fires queries 1, 2, 3, and 4 to CoreDNS before query 5 succeeds!
The Result: If your microservice executes 10,000 HTTP requests per second to external APIs (Stripe, Twilio, OpenAI, Datadog), your pods generate 50,000 DNS queries per second, flooding CoreDNS, exhausting Linux kernel conntrack tables, and triggering random 5-second timeouts!

🛠️ The Architectural Solutions to DNS Amplification

1. The Trailing Dot Pattern (Instant Code Fix)

In your application code or configuration, append a trailing dot to absolute external hostnames:

https://api.stripe.com./v1/charges

Why it works: The trailing dot declares an FQDN (Fully Qualified Domain Name). The Linux resolver recognizes this as an absolute root lookup and skips the internal search path entirely, executing 1 single DNS query instead of 5!

2. Custom Pod `dnsConfig` Optimization

For high-throughput microservices, override ndots in the Kubernetes Pod spec:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-service
  namespace: workloads
spec:
  replicas: 5
  template:
    metadata:
      labels:
        app: payment-service
    spec:
      dnsConfig:
        options:
          - name: ndots
            value: "2" # If domain has >= 2 dots, query external DNS immediately!
          - name: single-request-reopen # Prevents Linux UDP conntrack race condition
      containers:
        - name: app
          image: 123456789012.dkr.ecr.us-east-1.amazonaws.com/payment:v2.1.0

3. Deploying NodeLocal DNSCache

Instead of routing every DNS request over the network to centralized CoreDNS pods, deploy NodeLocal DNSCache as a DaemonSet:

It runs a local caching agent on 169.254.20.10 directly inside every worker node’s memory.
Resolves cached DNS queries in sub-millisecond latency with zero network overhead and bypasses iptables connection tracking completely!

Production NodeLocal DNSCache DaemonSet Manifest (`nodelocaldns.yaml`)

apiVersion: v1
kind: ConfigMap
metadata:
  name: node-local-dns
  namespace: kube-system
data:
  Corefile: |
    cluster.local:53 {
        errors
        cache {
            success 9984 30
            denial 9984 5
        }
        reload
        loop
        bind 169.254.20.10
        forward . 172.20.0.10 { # Upstream CoreDNS Service ClusterIP
            force_tcp           # Bypasses UDP conntrack race conditions completely!
        }
        prometheus :9253
        health 169.254.20.10:8080
    }
    in-addr.arpa:53 {
        errors
        cache 30
        reload
        loop
        bind 169.254.20.10
        forward . 172.20.0.10 {
            force_tcp
        }
        prometheus :9253
    }
    .:53 {
        errors
        cache 30
        reload
        loop
        bind 169.254.20.10
        forward . /etc/resolv.conf # Direct VPC Resolver (e.g., 10.0.0.2)
        prometheus :9253
    }
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-local-dns
  namespace: kube-system
  labels:
    k8s-app: node-local-dns
spec:
  selector:
    matchLabels:
      k8s-app: node-local-dns
  template:
    metadata:
      labels:
        k8s-app: node-local-dns
    spec:
      priorityClassName: system-node-critical
      serviceAccountName: node-local-dns
      hostNetwork: true
      dnsPolicy: Default # Don't use cluster DNS for the caching agent itself
      tolerations:
        - operator: Exists
      containers:
        - name: node-cache
          image: registry.k8s.io/dns/k8s-dns-node-cache:1.22.28
          resources:
            requests:
              cpu: 50m
              memory: 30Mi
            limits:
              cpu: 200m
              memory: 100Mi
          args:
            - -localip
            - 169.254.20.10
            - -conf
            - /etc/Corefile
            - -upstreamsvc
            - kube-dns
          securityContext:
            capabilities:
              add:
                - NET_ADMIN # Required to configure link-local dummy interface on node
          volumeMounts:
            - mountPath: /etc/Corefile
              name: config-volume
              subPath: Corefile
          ports:
            - containerPort: 53
              name: dns
              protocol: UDP
            - containerPort: 53
              name: dns-tcp
              protocol: TCP
            - containerPort: 9253
              name: metrics
              protocol: TCP
      volumes:
        - name: config-volume
          configMap:
            name: node-local-dns

🗺️ EKS Cluster Version Upgrades: End-to-End State Machine

Upgrading an enterprise EKS cluster without downtime requires following an unalterable operational order of operations:

flowchart TD
    Audit["<b>Phase 1: Pre-Upgrade Audit</b><br/>• Scan deprecated APIs (Pluto / Kubent)<br/>• Audit EKS Addon compatibility matrix"] --> UpgradeAddons["<b>Phase 2: Upgrade EKS Managed Addons</b><br/>• VPC CNI (aws-node)<br/>• CoreDNS<br/>• kube-proxy"]
    
    UpgradeAddons --> UpgradeCP["<b>Phase 3: EKS Control Plane Upgrade</b><br/>• AWS upgrades Kubernetes API Server & etcd<br/>• Takes ~15–20 minutes (Zero downtime)"]
    
    UpgradeCP --> UpgradeNodes["<b>Phase 4: Rolling Node Group Upgrade</b><br/>• Launch new nodes with updated EKS AMI<br/>• Drain & cordoning old nodes (PDB safe)"]
    
    UpgradeNodes --> PostVerification["<b>Phase 5: Post-Upgrade Verification</b><br/>• Verify Pod health & synthetic canary traffic"]

⚠️ Real-World Failure Modes, Exceptions & Break-Glass Playbooks

Exception 1: The 5-Second DNS Latency Freeze (Linux Kernel Conntrack Race)

The Outage: Microservice HTTP calls intermittently freeze for exactly 5.000 seconds, followed by a successful HTTP 200.
The Root Cause: GNU glibc sends A (IPv4) and AAAA (IPv6) DNS queries concurrently over the exact same UDP socket tuple (Source IP, Source Port, Dest IP, Dest Port 53). In the Linux kernel, the netfilter connection tracking subsystem (nf_conntrack) attempts to insert both UDP packets into the conntrack table at the exact same millisecond. Due to a kernel spinlock collision, one packet is silently dropped! The client waits for the default 5-second DNS retry timeout before re-transmitting.
The Fix: Add options single-request-reopen to /etc/resolv.conf (via dnsConfig in Kubernetes). This instructs glibc to close and re-open the UDP socket between queries, completely preventing the conntrack race.

Exception 2: EKS Upgrade Webhook Lockout (Control Plane Paralysis)

The Outage: An engineer triggers an EKS control plane upgrade from 1.28 to 1.29. Halfway through the upgrade, the Kubernetes API Server starts rejecting all deployments, pod creations, and scale operations with:
```
Internal error occurred: failed calling webhook "validate.kyverno.svc": Post "https://kyverno.kyverno.svc:443/validate": dial tcp 10.100.42.12:443: i/o timeout
```
The Root Cause: A legacy ValidatingWebhookConfiguration has failurePolicy: Fail. During the upgrade, the webhook pod was either restarted or relies on an API that was deprecated in 1.29. Because the API Server cannot reach the webhook, it refuses to admit any pod in the cluster!

The Break-Glass Emergency Fix: When a broken webhook is paralyzing your cluster, execute an immediate administrative bypass to delete the blocking admission configuration:

# List all registered validating webhooks
kubectl get validatingwebhookconfigurations

# Delete the blocking webhook configuration immediately to unfreeze the API server
kubectl delete validatingwebhookconfiguration kyverno-policy-validating-webhook-cfg

Exception 3: Deprecated API Removal Brick (e.g. `k8s.gcr.io` or `policy/v1beta1`)

The Outage: After upgrading to EKS 1.25+, existing Helm deployments crash during CI/CD with:

Error: unable to build kubernetes objects: unable to recognize "": no matches for kind "PodSecurityPolicy" in version "policy/v1beta1"

The Prevention: Run automated deprecation scanning in your CI/CD pipelines before any upgrade is initiated:

# Scan live cluster for deprecated APIs
kubent

# Scan Helm charts in Git for deprecated APIs before upgrading
pluto detect-helm -d ./charts/

📝 Staff Architect & SRE Interview Scenarios

❓ 🟢 Low Difficulty: What happens if you upgrade worker nodes BEFORE the control plane?

Question: In Kubernetes, what is the Version Skew Policy rule, and what breaks if an engineer upgrades worker nodes to Kubernetes 1.30 while the control plane is still running 1.28?

Answer: Kubernetes officially enforces the N-2 Version Skew Policy: kubelet worker node versions can never be newer than the kube-apiserver control plane version. If worker nodes running 1.30 attempt to join an API server running 1.28, the kubelet will fail to register with the cluster, reporting unsupported API feature gates and schema mismatches. The control plane must ALWAYS be upgraded first, followed by worker nodes.

❓ 🟡 Medium Difficulty: CoreDNS Memory Leak Under High Traffic

Question: During peak holiday traffic, CoreDNS pods in an EKS cluster suddenly crash with Exit Code 137 (OOMKilled). Why does this happen, and how do you configure CoreDNS to survive high query volumes?

Answer: By default, CoreDNS caches DNS lookups in memory. Under massive query spikes with high client entropy, the cache map grows rapidly, exhausting the modest default memory limits (typically 170Mi).

The Fix:

Tune the CoreDNS Corefile cache plugin with explicit entry limits:
```
cache 30 {
    success 9984
    denial  9984
}
```
Deploy Horizontal Pod Autoscaler (HPA) or cluster-proportional-autoscaler on CoreDNS to scale CoreDNS replicas dynamically based on the total number of worker nodes and cluster cores.

❓ 🔴 Hard Difficulty: The CoreDNS `autopath` Plugin Mechanism

Question: What does the CoreDNS autopath plugin do, and why do some enterprise platform teams disable it despite its ability to solve the ndots: 5 problem?

Answer: autopath solves ndots by performing server-side search path resolution inside CoreDNS: instead of returning NXDOMAIN and forcing the client to try all 4 search paths, CoreDNS walks the search path internally and returns the final IP in a single response.

Why Teams Disable It: autopath requires CoreDNS to query the Kubernetes API Server to discover each client Pod’s namespace in real time. Under massive cluster load, this puts intense read pressure on kube-apiserver and etcd. If CoreDNS experiences a race condition with newly spawned pods, it can return stale or incorrect DNS responses. Deploying NodeLocal DNSCache is considered the superior, risk-free alternative.

❓ 🟣 Complex Difficulty: Cross-AZ Latency in EKS CoreDNS Routing

Question: If CoreDNS is deployed with 2 replicas in us-east-1a, but a high-throughput microservice pod runs in us-east-1b, what network penalty is incurred on every DNS lookup, and how is it resolved?

Answer: In standard Kubernetes networking, kube-proxy load-balances Service IP traffic randomly across all available CoreDNS endpoints regardless of Availability Zone. This causes 50% of DNS queries to cross AZ boundaries, incurring cross-AZ data transfer costs ($0.01/GB) and adding 1 to 2 milliseconds of network round-trip latency to every query.

The Fix: Configure Kubernetes Topology Aware Routing (formerly Topology Aware Hints) on the CoreDNS service by adding the annotation:

service.kubernetes.io/topology-mode: Auto

This instructs the CNI and kube-proxy to route DNS traffic strictly to CoreDNS replicas running within the exact same Availability Zone as the requesting Pod, keeping DNS latency sub-millisecond and eliminating cross-AZ data fees.