Lesson 6: Linux Kernel Under the Hood: Namespaces, cgroups v2 & Container Primitives

🧠 The Concept (Explain Like I’m 5)

• What is a Container, really? A container is NOT a lightweight virtual machine. There is no hypervisor, no virtualized BIOS, and no guest operating system. A container is simply an ordinary Linux process wearing two pairs of high-tech glasses:
  • Namespaces (What the process can SEE): The illusion of solitude. The process looks in the mirror and believes it is the sole program running on the entire computer. It gets its own process tree (seeing itself as PID 1), its own network card (eth0), and its own isolated file system (/).
  • cgroups (What the process can USE): The strict financial allowance. A supervisor standing over the process saying: “You are allocated 2 CPU cores and 4GB of RAM. If you attempt to consume more RAM, the Linux Out-Of-Memory (OOM) Killer will execute you immediately.”
• OverlayFS (The Tracing Paper Stack): Imagine 5 sheets of translucent tracing paper stacked together (read-only base image layers). On top sits one clear plastic sheet (the container writable layer). You can sketch or erase on the top plastic sheet; the original drawings underneath remain untouched.

🏢 The Enterprise Context (Why Platform Architects Care)

1. The Shared Kernel Security Boundary: Because every container on a host shares the exact same underlying Linux kernel, kernel vulnerabilities (such as dirty COW, eBPF verifier bypasses, or unshare privilege escalations) allow malicious processes to break container isolation and achieve total host takeover.
2. Kubernetes Quality of Service (QoS) Direct Mapping: Kubernetes Pod QoS classes (Guaranteed, Burstable, BestEffort) directly configure the Linux kernel’s cgroups v2 controllers. Setting requests and limits incorrectly causes severe CPU throttling and random OOM pod terminations.
3. Advanced Triage Without Docker: In modern production Kubernetes environments (running containerd or CRI-O), container images are stripped of debugging utilities (curl, netstat, tcpdump are missing). Platform architects use Linux kernel primitives like nsenter and ip netns to inject diagnostic tools directly into container namespaces from the host OS.

🗺️ Visual Architecture: Namespaces, cgroups v2 & OverlayFS

flowchart TD
    subgraph HostNode ["Bare-Metal / EC2 Host (Single Shared Linux Kernel)"]
        KernelCore["<b>Linux Kernel Core</b><br/>(Scheduler, Memory Management, VFS, Netfilter)"]

        subgraph CgroupsTree ["cgroups v2 Unified Tree (/sys/fs/cgroup)"]
            CPUCont["<b>cpu.max</b>: 200000 100000 (2 CPU Quota / 100ms CFS Window)<br/><b>cpu.stat</b>: tracks nr_throttled"]
            MemCont["<b>memory.max</b>: 4GB (Hard OOM boundary)<br/><b>memory.high</b>: 3.5GB (Kernel Throttling Zone)"]
        end

        subgraph KernelNamespaces ["Linux Kernel Isolation Namespaces"]
            PID_NS["<b>PID Namespace</b><br/>Process sees itself as PID 1<br/>Host sees process as PID 48201"]
            NET_NS["<b>NET Namespace</b><br/>Virtual eth0, private routing table, isolated loopback"]
            MNT_NS["<b>MNT Namespace</b><br/>Isolated VFS Mount Points (chroot / pivot_root)"]
            USER_NS["<b>USER Namespace</b><br/>Container root (UID 0) maps to Host UID 10001"]
        end

        subgraph OverlayFSStack ["OverlayFS Union Filesystem (/var/lib/containerd/...)"]
            Merged["<b>merged/</b>: Unified rootfs visible to container (/bin, /etc, /app)"]
            Upper["<b>upperdir/</b>: Ephemeral Container Read/Write Layer"]
            Work["<b>workdir/</b>: Scratch space for atomic operations"]
            Lower["<b>lowerdir/</b>: Immutable Read-Only Image Layers (Base OS + Dependencies)"]
            
            Upper --> Merged
            Lower --> Merged
        end

        KernelCore --> CgroupsTree
        KernelCore --> KernelNamespaces
        MNT_NS --> Merged
    end

🔬 Under the Hood: The 7 Linux Kernel Namespaces

The Linux kernel isolates processes across 7 distinct resource dimensions:

💻 Hands-On: Building a “Container” with Native Linux Commands

You do not need Docker or containerd to create a container. You can assemble one natively using Linux kernel system calls in under 60 seconds:

# 1. Create a dedicated container root filesystem directory
mkdir -p /tmp/mini-container/rootfs

# 2. Extract an Alpine Linux root filesystem into the folder
curl -sSL https://dl-cdn.alpinelinux.org/alpine/v3.19/releases/x86_64/alpine-minirootfs-3.19.1-x86_64.tar.gz | tar -xz -C /tmp/mini-container/rootfs

# 3. Create a dedicated Network Namespace
sudo ip netns add container-net

# 4. Create a virtual ethernet cable (veth pair) connecting host to container
sudo ip link add veth-host type veth peer name veth-guest

# 5. Move one end of the cable into the container network namespace
sudo ip link set veth-guest netns container-net

# 6. Configure IP addresses on both ends of the virtual link
sudo ip addr add 192.168.100.1/24 dev veth-host
sudo ip link set veth-host up

sudo ip netns exec container-net ip addr add 192.168.100.2/24 dev veth-guest
sudo ip netns exec container-net ip link set veth-guest up
sudo ip netns exec container-net ip link set lo up

# 7. Spawn the isolated process using 'unshare' (PID, Mount, and UTS namespaces)
sudo unshare --mount --uts --ipc --pid --fork chroot /tmp/mini-container/rootfs /bin/sh

Inside this shell:

• Run ps aux: You see only /bin/sh as PID 1!
• Run hostname my-isolated-box: Modifies only the container hostname; the host hostname remains unchanged.

⚙️ cgroups v1 vs. cgroups v2 Architecture

In early Linux kernels (and legacy Kubernetes nodes), resource control was managed by cgroups v1. Modern operating systems (RHEL 9, Ubuntu 22.04+, Amazon Linux 2023) use cgroups v2.

flowchart TD
    subgraph CgroupsV1 ["Legacy cgroups v1 (Fragmented Hierarchies)"]
        direction TB
        V1_Root["/sys/fs/cgroup"]
        V1_CPU["/cpu (CPU Controller)"]
        V1_Mem["/memory (Memory Controller)"]
        V1_Blk["/blkio (I/O Controller)"]
        
        V1_Root --> V1_CPU
        V1_Root --> V1_Mem
        V1_Root --> V1_Blk
        V1_CPU -.->|"No unified tracking"| V1_Mem
    end

    subgraph CgroupsV2 ["Modern cgroups v2 (Unified Hierarchy)"]
        direction TB
        V2_Root["/sys/fs/cgroup"]
        V2_Slice["kubepods.slice"]
        V2_Pod["pod-uuid.scope<br/>• cpu.max<br/>• memory.max<br/>• memory.high<br/>• io.weight"]
        
        V2_Root --> V2_Slice
        V2_Slice --> V2_Pod
    end

Why cgroups v2 is Essential for Production

1. Unified Resource Accounting: In cgroups v1, memory and block I/O were tracked in independent trees. If a process performed heavy disk writes, the kernel buffered them in page cache memory under /memory, but couldn’t associate them with /blkio. This caused severe writeback priority inversions.
2. Pressure Stall Information (PSI): cgroups v2 introduces /proc/pressure/{cpu,memory,io}. This gives platforms real-time visibility into the exact percentage of time threads are stalled waiting on memory pages or CPU cycles before failures happen.
3. Graceful Throttling (memory.high): Instead of immediately killing a process when it hits a memory ceiling, cgroups v2 introduces memory.high. When a container crosses this threshold, the kernel aggressively reclaims page cache and throttles memory allocation speeds, giving application garbage collectors time to run before triggering the catastrophic OOM Killer.

🧮 CPU CFS Quota Mathematics & The Throttling Trap

In Linux, container CPU limits are enforced by the Completely Fair Scheduler (CFS) using two values:

• cpu.cfs_period_us: The scheduling enforcement window (standard default: 100,000 microseconds = 100 milliseconds).
• cpu.cfs_quota_us: The total CPU time the container’s threads may consume within that period.

📐 Allocated CPU Cores = cpu.cfs_quota_us / cpu.cfs_period_us

The Throttling Trap

Suppose you configure a container limit of 0.5 CPUs (cpu.cfs_quota_us = 50,000, period = 100,000):

1. The container receives an incoming HTTP request and utilizes 1 full host CPU core at 100% capacity.
2. In the first 50ms of the scheduling window, it exhausts its 50ms quota.
3. The kernel halts the container process for the remaining 50ms of the window!
4. Even though the host EC2 instance has 32 idle CPU cores, your application experiences an artificial 50ms latency spike on every request!

Real-Time Kubernetes Resource Spec Mapping to CFS Quota

# Kubernetes Resource Spec translating directly to Linux CFS Cgroup values:
# cpu.cfs_period_us = 100000 (100ms)
# cpu.cfs_quota_us  = 200000 (2 cores * 100ms)
resources:
  requests:
    cpu: "1000m" # CFS CPU Shares: 1024 (guaranteed baseline priority)
    memory: "2Gi"
  limits:
    cpu: "2000m" # CFS Quota: 200,000 us (hard ceiling causing throttling)
    memory: "4Gi"

# Check if a container is experiencing CPU throttling in cgroups v2
cat /sys/fs/cgroup/system.slice/docker-<CONTAINER_ID>.scope/cpu.stat
# Key metrics to alert on:
# nr_periods 14820
# nr_throttled 3410       <-- If this is > 5% of nr_periods, latency is degrading!
# throttled_usec 152891000

💀 Deep Dive: The Linux Out-Of-Memory (OOM) Killer

When available physical RAM and swap space are completely exhausted, the Linux kernel calls out_of_memory(). The kernel scores every running process to decide which one to terminate:

📐 OOM Score = Normalized RAM Usage (0 to 1000) + oom_score_adj

• The process with the highest score is abruptly terminated with SIGKILL (Exit Code 137 = 128 + 9).
• Kubernetes sets oom_score_adj based on Pod QoS:
  • Guaranteed Pods: oom_score_adj = -997 (Protected from termination).
  • Burstable Pods: Scaled between 2 and 999 based on memory requested vs limits.
  • BestEffort Pods: oom_score_adj = 1000 (First to be terminated by the kernel!).

Real-Time Guaranteed QoS Pod Manifest (Immunizing against OOM Termination)

To lock the container’s oom_score_adj at -997 and protect mission-critical payment pods from kernel kills, set requests identical to limits:

apiVersion: v1
kind: Pod
metadata:
  name: payment-engine-guaranteed-qos
spec:
  containers:
    - name: payment-api
      image: payment-service:v2.4
      resources:
        # Exact parity qualifies pod for "Guaranteed QoS" (oom_score_adj = -997)
        requests:
          cpu: "2"
          memory: "4Gi"
        limits:
          cpu: "2"
          memory: "4Gi"

# 1. Inspect kernel ring buffer for OOM events
dmesg -T | grep -E -i "oom[-_]killer|killed process"

# Sample Output:
# [Mon Oct 12 11:20:04 2026] Memory cgroup out of memory: Killed process 41820 (java) 
# total-vm:4210432kB, anon-rss:2097152kB, file-rss:1048kB, shmem-rss:0kB
# oom_score_adj:998

# 2. View a running process's OOM score adjustment
cat /proc/<PID>/oom_score_adj

🛠️ Debugging Live Containers with nsenter

When a container in production lacks basic networking tools (curl, tcpdump, ip), do not modify the container image. Enter the container’s network namespace directly from the host node:

# 1. Identify the container process ID (PID) on the host node
CONTAINER_PID=$(docker inspect --format '{{.State.Pid}}' payment-api)

# 2. Enter the container's network namespace using nsenter
# -t <PID>: target process
# -n: enter network namespace only
sudo nsenter -t ${CONTAINER_PID} -n ip addr show

# 3. Capture live network traffic inside the container using the host's tcpdump!
sudo nsenter -t ${CONTAINER_PID} -n tcpdump -i eth0 -nn port 8080 -w /tmp/capture.pcap

# 4. Enter BOTH network and mount namespaces to inspect internal files
sudo nsenter -t ${CONTAINER_PID} -m -n /bin/sh

🚨 Git Disaster Recovery & Emergency Operations

When production pipelines break due to corrupted repositories or bad merges, platform architects utilize deep Git recovery commands:

# 1. Inspect the absolute log of all HEAD movements (even deleted commits and branches!)
git reflog

# Sample Output:
# 4a19b21 HEAD@{0}: reset: moving to HEAD~1
# d39c80f HEAD@{1}: commit: Critical deployment hotfix (Accidentally deleted!)

# 2. Resurrect an accidentally deleted branch or commit
git branch hotfix-recovery d39c80f

# 3. Automated Binary Regression Hunt with git bisect
# Automatically identifies the exact commit that introduced a defect
git bisect start
git bisect bad                 # Current commit is broken
git bisect good v2.4.0         # Known working release

# Run automated test script against each checked-out commit:
git bisect run ./test_suite.sh
# Git automatically outputs: "d39c80f is the first bad commit"
git bisect reset

🚨 Real-World Troubleshooting & Tricky Production Scenarios

Scenario 1: The “15% Host CPU But 200ms Latency” Throttling Anomaly

• Symptom: A Node.js API experiences severe p99 latency spikes of 150–200ms. Datadog shows the host EC2 instance CPU utilization is only at 18%, and the pod is averaging only 350m of CPU usage.
• Under-the-Hood Root Cause: The pod manifest has resources.limits.cpu: "500m". The Linux Completely Fair Scheduler (CFS) enforces this via cpu.cfs_quota_us = 50000 with a period_us = 100000 (100ms). When incoming web traffic arrives in bursts, Node.js worker threads consume the 50ms allowance in the first 20ms of the window. The kernel abruptly freezes (throttles) the process for the remaining 80ms, adding an artificial 80ms wait to every pending HTTP response!
• Triage & Remediation Runbook:

  # 1. Identify if the container cgroup is being throttled
  kubectl exec -it <pod-name> -- cat /sys/fs/cgroup/cpu.stat
  # Look for nr_throttled and throttled_usec:
  # nr_periods 12050
  # nr_throttled 4810  <-- 40% of periods throttled!
  
  # 2. Query Prometheus rate of throttled periods
  # rate(container_cpu_cfs_throttled_periods_total[5m]) / rate(container_cpu_cfs_periods_total[5m]) * 100
  
  # 3. Remediation: Remove CPU limits or increase quota significantly
  # Best Practice: Keep requests for scheduling, eliminate limits to allow bursting

Scenario 2: Memory Page Cache Accumulation Triggering Eviction

• Symptom: A batch processing pod writing log chunks to local disk is terminated by the kubelet with Evicted: The node was low on resource: memory.
• Under-the-Hood Root Cause: Linux aggressively caches file I/O in the Page Cache (file-rss). In cgroups v1, memory limits count both anonymous memory (anon-rss = application heap/stack) and page cache. While page cache is technically reclaimable, sudden memory allocations by other processes can trigger node memory pressure before the kernel can sync dirty pages to disk, causing the kubelet to evict the pod.
• Triage & Fix:

  # 1. Inspect memory breakdown in cgroups v2
  cat /sys/fs/cgroup/memory.stat | grep -E "anon|file|inactive_file"
  
  # 2. Force drop reclaimable page cache during testing (requires host root)
  sync; echo 3 > /proc/sys/vm/drop_caches
  
  # 3. Permanent Fix: In cgroups v2 (modern K8s 1.28+ on Linux 5.15+), use memory.high 
  # to trigger proactive kernel page reclamation before reaching memory.max OOM threshold.

📝 Staff Architect & SRE Interview Scenarios