๐ง Round 1: Linux Internals, Systems & Low-Level Networking
In Staff Platform Engineer and SRE interviews at top Indian tech firms (Google IDC, Uber, Razorpay, PhonePe, Atlassian, Zerodha), Round 1 is the great filter.
Junior and mid-level candidates answer with surface-level tool commands ("I run netstat", "I check top"). Staff candidates answer with kernel subsystem mechanics, memory paging structures, TCP socket state machines, and eBPF kernel hooks.
๐ Question 1: Trace the Linux Kernel Packet Ingress Flow
๐ฏ The Scenario & Interviewerโs Trap
โA physical server with a 10GbE NIC receives an HTTP request destined for a container running in a Kubernetes pod. Trace the packet from the physical wire all the way into the applicationโs user-space socket buffer. What kernel subsystems are traversed, and where can packets be silently dropped before hitting user space?โ
Interviewerโs Intent: To determine if you understand the boundary between hardware interrupts, kernel SoftIRQs, the netfilter subsystem, and user-space socket buffers, or if you view networking as an opaque magic box.
๐ก Staff-Level Model Answer
flowchart TD
A["Physical Wire / Fiber"] --> B["NIC PHY & MAC Layer"]
B --> C["NIC RX Ring Buffer (DMA Transfer)"]
C --> D["Hard IRQ (MSI-X Interrupt to CPU)"]
D --> E["Interrupt Handler schedules NAPI Poll (SoftIRQ NET_RX)"]
E --> F["ksoftirqd polls NIC Ring Buffer"]
F --> G["Allocate sk_buff (Socket Buffer)"]
G --> H{"eBPF XDP Hook Attached?"}
H -- "Yes: Drop or Redirect" --> H1["Early Drop / Fast Route"]
H -- "No" --> I["Linux Network Stack: netif_receive_skb"]
I --> J["Traffic Control (tc) Qdisc & eBPF tc-hook"]
J --> K["Netfilter PREROUTING (conntrack & iptables/nftables)"]
K --> L["Routing Decision (Local Process vs Forward)"]
L --> M["Netfilter INPUT Chain"]
M --> N["TCP Layer: tcp_v4_rcv (Sequence Check, Window Sizing)"]
N --> O["Socket Receive Buffer (sk_rcvbuf)"]
O --> P["epoll_wait wakes up User-Space Process (read / recvfrom)"]
Step-by-Step Kernel Execution:
- DMA Transfer into Ring Buffer: The NIC receives electrical/optical frames. The NICโs Direct Memory Access (DMA) engine copies packet frames directly into host RAM mapped to the NICโs RX Ring Buffer without consuming CPU cycles.
- Hard IRQ: When the ring buffer receives packets, the NIC fires a hardware interrupt (MSI-X) to a designated CPU core.
- Interrupt Handler & NAPI: The CPUโs hard IRQ handler disables further NIC interrupts to avoid interrupt livelock and schedules a NAPI (New API) SoftIRQ (
NET_RX_SOFTIRQ). - SoftIRQ &
ksoftirqd: The kernel daemonksoftirqd/<cpu>executes the driverโs NAPI poll function, draining packets from the RX ring buffer in batches (controlled bynet.core.netdev_budget, default 300 packets). sk_buffAllocation: The kernel allocates ansk_buff(socket buffer descriptor) for each packet.- XDP & Traffic Control (
tc):- If an eBPF XDP program is loaded, it executes directly inside the driver before
sk_buffallocation, enabling ultra-fast DDoS dropping or packet redirection. - Otherwise, the packet passes into
netif_receive_skband traverses the Traffic Control (tc) queuing discipline (qdisc).
- If an eBPF XDP program is loaded, it executes directly inside the driver before
- Netfilter / Conntrack: The packet hits the Netfilter architecture (
PREROUTING), initializing the connection tracking engine (conntrack), creating state tuple records (ORIGINALandREPLY). - Routing & L4 TCP Processing: The routing subsystem checks if the destination IP is local or needs forwarding. For local pods, the packet traverses the Netfilter
INPUTchain totcp_v4_rcv(). - Socket Receive Buffer & User Space: The kernel verifies the TCP sequence number, sends an ACK, and queues the payload into the socketโs
sk_rcvbuf. An event is emitted on the socket file descriptor, waking up the applicationโsepoll_wait()orread()syscall.
Silent Drop Locations & CLI Diagnostic Commands:
- NIC RX Ring Buffer Overrun: Packet dropped before the OS sees it because
ksoftirqdis CPU-starved.ethtool -S eth0 | grep -E "rx_dropped|rx_missed_errors|rx_fifo_errors" - Netdev Backlog Queue Overflow: SoftIRQ queue (
net.core.netdev_max_backlog) exhausted.cat /proc/net/softnet_stat # Column 1 = Processed packets, Column 2 = Dropped due to full backlog - Conntrack Table Saturation: Netfilter drops new connections when the table is full.
cat /proc/sys/net/netfilter/nf_conntrack_count cat /proc/sys/net/netfilter/nf_conntrack_max - Socket Buffer Overflow: Application is too slow to read from the socket.
netstat -s | grep -i "buffer errors" # Or: ss -t -i -m
๐ก๏ธ Follow-Up Gotcha & Defense
Interviewer: โIf you see CPU 0 at 100% utilization while CPUs 1โ15 are at 5%, and your application is experiencing packet loss, what is happening and how do you fix it?โ
Staff Defense: โThis is IRQ Pinning Imbalance. All NIC hardware interrupts are hitting CPU 0. I would resolve this by checking
/proc/interrupts, enabling Receive Side Scaling (RSS) on the NIC with multi-queue support (ethtool -L eth0 combined 8), and configuringirqbalanceor steering SoftIRQs across cores using Receive Packet Steering (RPS) via/sys/class/net/eth0/queues/rx-*/rps_cpus.โ
๐ Question 2: CFS CPU Quota Throttling vs CPU Shares
๐ฏ The Scenario & Interviewerโs Trap
โYou deploy a Go microservice in Kubernetes with
resources.requests.cpu: 500mandresources.limits.cpu: 2. At peak traffic, the node CPU utilization is only 35%, and the podโs average CPU usage is only 0.8 cores. Yet, the serviceโs p99 latency spikes from 15ms to 650ms. What is causing this, and how do you prove it?โ
Interviewerโs Intent: To verify whether you understand the Linux Completely Fair Scheduler (CFS) bandwidth control and how multi-threaded runtimes cause micro-burst throttling within the 100ms CFS quota window.
๐ก Staff-Level Model Answer
flowchart LR
subgraph Period["100ms CFS Period (cpu.cfs_period_us)"]
direction TB
T1["0ms - 20ms: 8 Threads burst simultaneously"]
T2["Consumed Quota: 8 threads ร 20ms = 160ms CPU time"]
T3["Max Allowed Quota: 2 cores ร 100ms = 200ms"]
T4["At 25ms: 200ms quota completely exhausted!"]
T5["25ms - 100ms: ALL THREADS HARD-THROTTLED (Sleep for 75ms)"]
end
Period --> Result["Spike in p99 Latency (650ms) despite low average CPU!"]
The Kernel Mechanism:
- Kubernetes maps
resources.limits.cpu: 2to Linux cgroups CFS bandwidth controls:cpu.cfs_period_us=100000(100 milliseconds).cpu.cfs_quota_us=200000(200 milliseconds of aggregate CPU run time per 100ms period).
- When a multi-threaded application (like Go runtime with
GOMAXPROCS=8on an 8-core node, or Java JVM with thread pools) receives a burst of HTTP requests, 8 threads execute concurrently on 8 separate CPU cores. - In just 25 milliseconds of wall-clock time, 8 threads running at 100% consume: ๐ Total CPU Time = 8 ร 25ms = 200ms
- The container has completely exhausted its 200ms quota for that period. The Linux kernel deschedules all threads and places the cgroup on the throttle wait-queue for the remaining 75 milliseconds of the period.
- This cycle repeats every 100ms. Over a 1-minute Prometheus scraping interval, average CPU utilization appears low (200ms / 1000ms = 20%), but user requests spend up to 75ms waiting in the throttled state per period.
Diagnostic Verification:
# On the Kubernetes worker node (cgroups v1):
cat /sys/fs/cgroup/cpu/kubepods.slice/.../cpu.stat
# Output:
# nr_periods 12450
# nr_throttled 4320 <-- 34% of all periods throttled!
# throttled_time 285432190 <-- Nanoseconds spent throttled
In Prometheus / Grafana:
sum(rate(container_cpu_cfs_throttled_periods_total[5m]))
/
sum(rate(container_cpu_cfs_periods_total[5m])) * 100
Remediation:
- Set
GOMAXPROCSexplicitly: Useuber-go/automaxprocsin Go or-XX:ActiveProcessorCountin JVM to ensure the runtime does not spawn threads based on the host core count rather than container CPU limits. - Remove CPU Limits on Latency-Sensitive Workloads: Rely strictly on
resources.requests.cpu(which configurescpu.shares, a relative priority weight that allows burst without hard throttling), while using node-level alerting and pod autoscaling (HPA) to prevent noisy neighbor starvation.
๐ Question 3: Linux Memory Management & The OOMKiller Mystery
๐ฏ The Scenario & Interviewerโs Trap
โA Java service running in a pod with
limits.memory: 4Gihas its JVM heap configured with-Xmx2g. The application never uses more than 1.8GB of heap memory, but Kubernetes randomly terminates the pod withExit Code 137(OOMKilled). Why is this happening, and how does the kernel determine which process to kill?โ
Interviewerโs Intent: To verify whether you know that container memory is not just heapโit includes off-heap memory, thread stacks, JVM Metaspace, and Linux kernel Page Cache / Slab allocation.
๐ก Staff-Level Model Answer
flowchart TD
subgraph TotalMem["Container cgroup Memory Limit: 4.0 GB"]
Heap["JVM Heap (-Xmx2g): 1.8 GB"]
OffHeap["Off-Heap & Direct Byte Buffers: 0.8 GB"]
Metaspace["JVM Metaspace & CodeCache: 0.4 GB"]
Stacks["Thread Stacks (500 threads ร 1MB): 0.5 GB"]
PageCache["Linux Page Cache (File I/O, Logs): 0.8 GB"]
end
TotalMem --> Exceed["Sum = 4.3 GB > 4.0 GB Limit"]
Exceed --> OOM["Linux Kernel mem_cgroup_out_of_memory() Fires!"]
OOM --> Kill["SIGKILL (137) Sent to Java Process"]
Kernel Memory Breakdown:
A containerโs cgroup memory consumption (memory.usage_in_bytes) is calculated as:
๐ cgroup Memory = RSS (Resident Set Size) + Page Cache (file-backed) + Kernel Memory (Slab/Buffers)
- JVM Non-Heap Overhead:
- Thread Stacks: Each Java thread allocates a default 1MB native stack (
-Xss1m). 500 concurrent threads consume 500MB of native RAM outside the heap. - Direct Byte Buffers (NIO): Netty/gRPC allocate off-heap memory directly via
ByteBuffer.allocateDirect()for zero-copy I/O. - Metaspace & Native Libraries: Loaded classes, JIT compiler CodeCache, and glibc allocations.
- Thread Stacks: Each Java thread allocates a default 1MB native stack (
- Page Cache Dirty Pages: If the application writes heavy logs to stdout/stderr or local disk, Linux caches these writes in Page Cache. When total memory approaches the limit, the kernel invokes memory compaction and reclamation (
kswapd). If the kernel cannot reclaim dirty pages fast enough to satisfy newkmalloc()allocations, it triggersmem_cgroup_out_of_memory().
The oom_score_adj Calculation:
When the cgroup memory limit is breached, the kernel calculates the oom_score for each process:
๐ oom_score = (Memory Used by Process) / (Total Available Memory) ร 1000 + oom_score_adj
- In Kubernetes:
- Guaranteed Pods:
oom_score_adj = -997(Killed last). - Burstable Pods:
oom_score_adj = min(max(2, 1000 - (1000 ร memory.requests / node.memory)), 999). - BestEffort Pods:
oom_score_adj = 1000(Killed first).
- Guaranteed Pods:
How to Debug:
# Check dmesg on the node:
dmesg -T | grep -i "oom-killer"
# Check container memory breakdown from cgroup:
cat /sys/fs/cgroup/memory/kubepods.slice/.../memory.stat
# Look at: rss, inactive_file, active_file, mapped_file, unevictable
๐ Question 4: TCP TIME_WAIT Accumulation & Ephemeral Port Exhaustion
๐ฏ The Scenario & Interviewerโs Trap
โA high-throughput API gateway running on an EC2 instance handles 15,000 outgoing HTTPS requests per second to downstream microservices. Suddenly, new outbound connections fail with
Cannot assign requested address(EADDRNOTAVAIL).netstatreveals 60,000 connections inTIME_WAIT. Why doesTIME_WAITexist, why did you run out of ports, and why istcp_tw_recycledangerous?โ
Interviewerโs Intent: To test your command of the TCP state machine, the 4-tuple connection space, socket reuse flags, and why legacy optimization hacks cause disastrous bugs in modern NAT environments.
๐ก Staff-Level Model Answer
sequenceDiagram
autonumber
participant Client as API Gateway (Client)
participant Server as Downstream Service
Client->>Server: FIN (Active Close initiated by Client)
Server-->>Client: ACK
Server->>Client: FIN
Client-->>Server: ACK
Note over Client: Client enters TIME_WAIT (2 ร MSL = 60s)
Note over Client: Socket 4-tuple is locked to prevent delayed segment collision
1. Why TIME_WAIT Exists (2 ร MSL):
The standard Maximum Segment Lifetime (MSL) in Linux is 30 seconds, so TIME_WAIT lasts 60 seconds (2 * MSL). It serves two crucial purposes:
- Reliable Connection Termination: If the clientโs final
ACKis dropped in transit, the server re-transmits itsFIN. If the client closed immediately withoutTIME_WAIT, it would respond with anRST, causing the server to believe an ungraceful error occurred. - Preventing Delayed Segment Data Corruption: Ensures old duplicate packets still floating through intermediate routers are completely discarded before a new connection with the exact same 4-tuple (
SrcIP, SrcPort, DstIP, DstPort) is established.
2. The Root Cause of EADDRNOTAVAIL:
A client connecting to a specific downstream service has a fixed 3-tuple:
- Source IP = Host IP
- Destination IP = Downstream IP
- Destination Port = 443
Only the Source Port (Ephemeral Port) varies. The Linux ephemeral port range is defined in /proc/sys/net/ipv4/ip_local_port_range (typically 32768 - 60999, providing only 28,231 ports).
At 15,000 connections per second without HTTP connection pooling (opening and closing a new TCP socket per request), all 28,000 ports are consumed in less than 2 seconds and locked in TIME_WAIT for 60 seconds!
3. Why tcp_tw_recycle is Dangerous and Deprecated:
tcp_tw_recycle(removed in Linux kernel 4.12) attempted to recycleTIME_WAITsockets rapidly by checking the TCP timestamp option (PAWS- Protect Against Wrapped Sequence numbers).- The NAT Disaster: If multiple clients sit behind a common NAT Gateway, their outgoing packets share the same public IP but have different, desynchronized internal hardware timestamps. When Client A sends a packet with timestamp T_1, and Client B sends a packet with timestamp T_2 < T_1, the server with
tcp_tw_recycle=1drops Client Bโs packet as a delayed replay, silently blackholing legitimate users!
4. The Staff-Level Solution:
# 1. Expand ephemeral port range to 64,511 ports:
sysctl -w net.ipv4.ip_local_port_range="1024 65535"
# 2. Enable safe socket reuse for outbound connections:
sysctl -w net.ipv4.tcp_tw_reuse=1
# 3. Architecturally: ENFORCE HTTP Connection Pooling (Keep-Alive)
# In Go / Python / Node / Java HTTP clients:
# MaxIdleConnsPerHost = 500, IdleConnTimeout = 90s
๐ Question 5: CoreDNS UDP Conntrack Race Condition & ndots:5
๐ฏ The Scenario & Interviewerโs Trap
โYour Kubernetes cluster runs high-throughput microservices. Microservices randomly experience 5-second connection delays and intermittent
EAI_AGAINDNS resolution failures. When inspecting metrics, CoreDNS latency is under 2ms. What kernel race condition occurs between glibc and Linux netfilter, and how do you resolve it?โ
Interviewerโs Intent: To test if you understand the classic Linux netfilter conntrack UDP race condition and how the Kubernetes default /etc/resolv.conf ndots:5 setting multiplies DNS traffic by 5x.
๐ก Staff-Level Model Answer
flowchart TD
App["Application Pod (glibc)"] -->|Simultaneous UDP Queries| Conntrack["Linux Netfilter conntrack"]
App -->|Query 1: A Record| Conntrack
App -->|Query 2: AAAA Record| Conntrack
Conntrack -->|Insert Tuple 1| StateTable["State Table: src_ip, dst_ip, src_port, dst_port"]
Conntrack -->|Insert Tuple 2: RACE!| StateTable
StateTable -->|Race Condition| Drop["insert_failed: Second UDP reply silently dropped!"]
Drop --> Timeout["Application waits for 5.0s glibc DNS timeout!"]
1. The Under-the-Hood Race Condition:
- By default, the glibc resolver sends A (IPv4) and AAAA (IPv6) DNS queries simultaneously over two separate UDP sockets using the same source port or racing for connection tracking tuples.
- In Linux netfilter (
nf_conntrack), when two UDP packets with identical 5-tuples race to insert their entries into the conntrack hash table duringPREROUTING, the first packet succeeds, but the second packet encounters a lock contention collision:# Kernel counter increments: net.netfilter.nf_conntrack_insert_failed - The kernel drops the second packet. Because UDP has no built-in retransmission mechanism, the client hangs for the default glibc DNS timeout: exactly 5.0 seconds.
2. The ndots:5 Amplification Multiplier:
In Kubernetes, pods receive /etc/resolv.conf configured with:
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
- If your application resolves
api.stripe.com, it contains 2 dots (api.stripe.com). - Because
2 < 5(ndots:5), glibc assumes the domain is an internal cluster service and appends all search domains first! - Glibc executes 5 sequential DNS lookups:
api.stripe.com.default.svc.cluster.localโ NXDOMAINapi.stripe.com.svc.cluster.localโ NXDOMAINapi.stripe.com.cluster.localโ NXDOMAINapi.stripe.com.โ SUCCESS (Finally!)
- Every external API call results in 8 to 10 UDP queries (A + AAAA for each search path), maximizing conntrack churn!
3. The Enterprise Fixes:
- Fully Qualified Domain Names (FQDN): Append a trailing dot in application code:
https://api.stripe.com.(Bypasses all search domain lookups). - Deploy NodeLocal DNSCache:
Runs a CoreDNS caching agent on every node as a DaemonSet using a link-local IP (
169.254.20.10), terminating UDP queries locally and upgrading upstream cluster queries to persistent TCP connections. - Tune
resolv.confin Pod Spec:dnsConfig: options: - name: ndots value: "2" - name: single-request-reopensingle-request-reopendirects glibc to close and reopen the socket between A and AAAA queries, completely eliminating the conntrack race.
๐ Question 6: Path MTU Discovery (PMTUD) Blackhole over IPsec / WireGuard
๐ฏ The Scenario & Interviewerโs Trap
โYou set up a site-to-site IPsec / WireGuard tunnel between an on-premises datacenter and an AWS VPC.
pingworks with 0% packet loss.curl http://internal-service/healthreturns HTTP 200 immediately. However, when the frontend sends a largePOST /uploadrequest (10KB JSON payload), the HTTP connection freezes and eventually times out with zero bytes received. What is happening?โ
Interviewerโs Intent: To test if you understand Maximum Transmission Unit (MTU), Maximum Segment Size (MSS), packet fragmentation, and why ICMP drops create silent PMTUD blackholes.
๐ก Staff-Level Model Answer
flowchart TD
Client["Client (MTU 1500)"] -->|TCP SYN: MSS 1460| Router["VPN Gateway (MTU 1420)"]
Router -->|Tunnel Overhead: 60-80 Bytes| WireGuard["WireGuard Tunnel"]
Client --> Small["Small GET /health: 300B"] --> WireGuard --> Passes["Passes cleanly"] --> Server["Backend Service"]
Client -->|"Large POST: 1500B Packet (DF Flag Set)"| Router
Router -->|"Cannot Fragment: DF=1"| Drop["Packet Dropped by Router!"]
Router -.->|ICMP Type 3 Code 4: Frag Needed| Firewall{"Firewall / Security Group"}
Firewall -- "BLOCKED! (ICMP disabled)" --> DropICMP["ICMP Dropped!"]
DropICMP --> Blackhole["Client never receives notice: PMTUD BLACKHOLE!"]
The Architectural Root Cause:
- Standard Ethernet frames have an MTU of 1500 bytes. The TCP Maximum Segment Size (MSS) is: ๐ MSS = MTU - IP Header (20B) - TCP Header (20B) = 1460 bytes
- Encapsulation protocols (WireGuard, IPsec, VXLAN, GRE) add 60 to 80 bytes of encryption headers and outer IP headers. Therefore, the maximum MTU inside the VPN tunnel is reduced to 1420 or 1440 bytes.
- All modern TCP stacks set the Donโt Fragment (DF = 1) bit in the IP header to enable Path MTU Discovery (PMTUD).
- When a 1500-byte packet arrives at the VPN gateway, the gateway cannot fit the packet into a 1420-byte tunnel without fragmentation. Because
DF=1, the gateway drops the packet. - The gateway generates an ICMP Type 3, Code 4 packet (Destination Unreachable, Fragmentation Needed and DF Set) back to the sender, containing the acceptable MTU (1420).
- The Blackhole: Security engineers frequently block all ICMP traffic in firewalls or AWS Security Groups. The ICMP notification is dropped! The client never learns that 1500 bytes is too large, endlessly retransmitting the unfragmented packet until the connection times out.
The Staff-Level Fix:
- TCP MSS Clamping (Recommended): Configure the gateway router/firewall to rewrite the TCP SYN packet MSS on the fly:
# iptables MSS clamping rule: iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN \ -j TCPMSS --clamp-mss-to-pmtu - Explicit MTU Tuning on WireGuard Interface:
[Interface] PrivateKey = ... Address = 10.100.0.1/24 MTU = 1420 - Permit ICMP Type 3 Code 4: Ensure Security Groups and perimeter firewalls permit inbound ICMP fragmentation notifications.
๐ Question 7: PID 1, Zombie Processes & Signal Forwarding in Containers
๐ฏ The Scenario & Interviewerโs Trap
โYou deploy a Python microservice inside a Docker container where the
DockerfilespecifiesENTRYPOINT ["python", "app.py"]. When Kubernetes issues a pod termination during a deployment, the pod ignores the termination signal and continues running for exactly 30 seconds until Kubernetes issues a forcefulSIGKILL. In addition, inspectingps auxinside the container reveals dozens of<defunct>zombie processes. Why is this happening, and how do you fix both issues?โ
Interviewerโs Intent: To verify if you understand Linux PID Namespaces, the special behavior of PID 1 regarding signal handling, and parent process reap responsibilities.
๐ก Staff-Level Model Answer
1. Why PID 1 Ignores SIGTERM:
In Linux, the process with PID 1 (traditionally systemd or init) is treated with special kernel privileges:
- For all regular processes (PID > 1), if the process does not register a custom signal handler for
SIGTERM, the kernel invokes the default action: immediate termination. - For PID 1, the kernel disables default signal handlers. If PID 1 has not explicitly registered a signal handler for
SIGTERM, the kernel discards the signal entirely. - When Kubernetes sends
SIGTERMto initiate graceful shutdown, the Python runtime (running as PID 1) ignores it. Kubernetes waits forterminationGracePeriodSeconds(default 30 seconds) and then issuesSIGKILL(which cannot be ignored), abruptly aborting active user transactions.
2. Why <defunct> Zombie Processes Accumulate:
- When a child process terminates, it does not immediately disappear from the kernel process table. Its status code is preserved so its parent can read it using the
wait()orwaitpid()syscall. - While waiting for the parent to call
wait(), the child process remains in the Zombie (Z) state. - If the parent process dies before reaping the child, the child is orphaned and adopted by PID 1.
- A traditional
initsystem constantly runs a loop reaping orphaned child processes. A standard application binary (Python, Node.js, Go) does not implement an orphan reaping loop. The zombie processes remain in the kernel process table indefinitely, eventually exhausting the system PID limit (/proc/sys/kernel/pid_max).
3. The Architectural Solution:
Never run raw application binaries as PID 1 in a container. Use a lightweight init supervisor like tini or dumb-init:
# Best-practice Dockerfile with tini:
FROM python:3.12-slim
# Install tini
RUN apt-get update && apt-get install -y --no-install-recommends tini && rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY app.py .
# Use tini as PID 1 entrypoint
ENTRYPOINT ["/usr/bin/tini", "--"]
CMD ["python", "app.py"]
Or in Docker directly:
docker run --init my-app:latest
tiniregisters proper signal handlers, forwardsSIGTERMdown to child process trees, and continuously executeswaitpid(-1, ...)to reap zombie processes immediately.
๐ Summary: Quick-Fire Linux & Networking Drill
| Question / Symptom | Quick Root Cause | Fix / CLI Command |
|---|---|---|
EAI_AGAIN on CoreDNS |
glibc A/AAAA conntrack table race | Set single-request-reopen in resolv.conf or deploy NodeLocal DNSCache |
| High p99 latency, low CPU | CFS bandwidth throttling within 100ms window | Check /sys/fs/cgroup/cpu/cpu.stat, configure automaxprocs, remove CPU limits |
EADDRNOTAVAIL outbound |
Ephemeral port exhaustion (2 ร MSL TIME_WAIT) |
sysctl net.ipv4.tcp_tw_reuse=1 and enable HTTP keep-alives |
Exit Code 137 on low heap |
Off-heap / direct memory / page cache exceeding limit | Check dmesg, inspect memory.stat, configure -XX:MaxDirectMemorySize |
| VPN freezes on large POST | MTU mismatch + ICMP Type 3 Code 4 dropped (PMTUD blackhole) | Enable TCP MSS Clamping (iptables -j TCPMSS --clamp-mss-to-pmtu) |
| Pod ignores SIGTERM for 30s | Container PID 1 lacks signal handler and zombie reaper | Use tini or dumb-init as container entrypoint |
๐ง Complete Lesson-by-Lesson One-Line Memory Bank (Round 1 Scope)
๐ง Phase 0: The Primitives (All Lessons)
Lesson 1: Linux Primitives & systemd
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | What numeric value represents Read, Write, and Execute permissions in Linux octal notation? | Read (r) = 4, Write (w) = 2, Execute (x) = 1. |
| 2 | Why is the Execute (x) bit mandatory on a Linux directory for a user to cd into it? |
The x bit on a directory controls inode traversal (cd); without it, users can list file names (r) but cannot enter or stat files. |
| 3 | What is the difference between SIGTERM (15) and SIGKILL (9)? |
SIGTERM is a catchable signal allowing graceful cleanup, whereas SIGKILL is handled directly by the kernel to terminate the process immediately. |
| 4 | Why does a process in Uninterruptible Sleep (D state) ignore kill -9? |
Signals are only delivered when a process returns from kernel space to user space, and a D-state process is blocked inside a kernel hardware I/O syscall. |
| 5 | Why does df -h show 100% disk usage when du -sh shows only 40% used? |
An open process is holding a file descriptor to a deleted file, preventing the kernel from reclaiming disk blocks until the process closes the fd or restarts. |
| 6 | How do you find deleted files still held open by active processes? | Run lsof +L1 to list all open files with a link count less than 1. |
| 7 | Where should custom systemd unit files be placed to avoid being overwritten by OS package upgrades? | Place custom unit files in /etc/systemd/system/, which takes precedence over /lib/systemd/system/. |
| 8 | What systemd directive ensures a crashed daemon waits 5 seconds before restarting? | Set Restart=always (or on-failure) combined with RestartSec=5s under the [Service] block. |
| 9 | How do you stream live systemd logs for a specific service starting from the most recent entries? | Run journalctl -u <service-name> -f -n 50. |
| 10 | What file defines the maximum number of open file descriptors permitted per process at the kernel level? | /proc/sys/fs/file-max (system-wide) and LimitNOFILE= inside the systemd service unit (per-service). |
Lesson 2: Networking & CIDR Math
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | What is the mathematical formula to calculate total IPv4 addresses in a /N CIDR block? |
Total IPs = 2^(32 - N). |
| 2 | How many total IPs and AWS-usable IPs exist in a /24 subnet? |
256 total IPs, and 251 usable IPs because AWS reserves 5 IPs per subnet. |
| 3 | Which 5 IP addresses does AWS reserve in every VPC subnet (e.g., 10.0.1.0/24)? |
.0 (Network), .1 (VPC Router), .2 (AWS DNS), .3 (Future use), and .255 (Broadcast). |
| 4 | What are the three packets exchanged during a TCP 3-way handshake? | SYN (Client โ Server), SYN-ACK (Server โ Client), and ACK (Client โ Server). |
| 5 | What is the fundamental difference between AWS Security Groups and Network ACLs (NACLs)? | Security Groups are stateful (ENI-level, allow rules only), whereas NACLs are stateless (subnet-level, explicit allow/deny rules required for both directions). |
| 6 | Why must a stateless NACL allow inbound traffic on ports 1024โ65535? |
Outbound requests originate from random ephemeral source ports (1024โ65535), so return responses arrive targeting those ephemeral ports. |
| 7 | What command displays all listening TCP sockets along with their owning process PIDs on modern Linux? | ss -tulpn (replaces legacy netstat -tulpn). |
| 8 | What does a TCP socket stuck in SYN_SENT state indicate? |
The client sent a TCP SYN packet, but no SYN-ACK was receivedโtypically caused by a silent firewall drop or blackhole route. |
| 9 | What does a buildup of CLOSE_WAIT sockets on a server indicate? |
The remote peer closed the connection (FIN), but the local application has a code bug and failed to execute close() on the socket descriptor. |
| 10 | What port and protocol does standard DNS use, and when does it switch to TCP? | UDP Port 53 by default; it switches to TCP Port 53 for zone transfers (AXFR) or when responses exceed 512 bytes (or EDNS0 buffer size). |
Lesson 3: Git & GitOps Foundations
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | What are the 4 core principles of OpenGitOps? | Declarative state, Versioned & Immutable storage, Pulled automatically, and Continuously Reconciled. |
| 2 | What is the difference between git merge and git rebase? |
git merge preserves chronological history with a merge commit, whereas git rebase rewrites commit hashes onto a linear tip. |
| 3 | Why should you never run git rebase on a shared public branch like main? |
Rebasing generates brand-new SHA-1 hashes for existing commits, diverging history and breaking local clones for every other developer. |
| 4 | What is the exact difference between git reset --soft HEAD~1 and git reset --hard HEAD~1? |
--soft moves HEAD back one commit while keeping changes staged; --hard permanently destroys uncommitted changes in both the index and working tree. |
| 5 | Why is git revert <commit> preferred over git reset for production rollbacks in GitOps? |
git revert creates a new forward commit that cleanly inverses the bad change, preserving an immutable audit trail for ArgoCD/Flux. |
| 6 | How do you recover a commit that was accidentally deleted by a git reset --hard? |
Run git reflog to locate the orphaned commitโs SHA hash and execute git checkout or git reset --hard <sha>. |
| 7 | Why is --force-with-lease safer than git push --force? |
--force-with-lease aborts the push if another teammate has pushed new commits to the remote branch since your last fetch. |
| 8 | What command uses binary search across commit history to pinpoint the exact commit that introduced a bug? | git bisect start, followed by git bisect bad and git bisect good <sha>. |
| 9 | Why is Pull-based GitOps (ArgoCD inside cluster) more secure than Push-based CI (kubectl apply in GitHub Actions)? |
Pull-based GitOps keeps cluster admin credentials inside the cluster, eliminating the need to expose the Kubernetes API or store god-mode tokens in external CI runners. |
| 10 | How do you sign Git commits cryptographically to verify author identity? | Configure GPG or SSH signing keys and commit using git commit -S -m "message". |
Lesson 4: Docker & Container Primitives
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | Are containers lightweight virtual machines with their own kernel? | No, containers are isolated host Linux processes sharing the underlying host OS kernel via namespaces and cgroups. |
| 2 | What filesystem driver does Docker use to stack immutable image layers with a writable container layer? | OverlayFS (lowerdir for read-only image layers, upperdir for writable changes, and merged for the unified view). |
| 3 | How does OverlayFS handle deleting a file that exists in a read-only lower layer? | It creates a character 0/0 whiteout file in upperdir that masks the underlying file in the merged directory view. |
| 4 | What is the primary security and size benefit of a Multi-Stage Dockerfile? | It compiles code in a heavy builder stage and copies only the compiled binary into a minimal runtime stage, stripping compilers, source code, and build secrets. |
| 5 | Why is a distroless base image more secure than alpine or ubuntu? |
Distroless images contain zero shell (/bin/sh) and zero package managers (apt/apk), preventing attackers from spawning interactive reverse shells. |
| 6 | Why should COPY package.json . and RUN npm ci be placed before COPY . . in a Dockerfile? |
Docker caches layers sequentially; copying dependency manifests first prevents source code edits from invalidating the expensive dependency installation cache. |
| 7 | What is the difference between ENTRYPOINT and CMD in a Dockerfile? |
ENTRYPOINT defines the fixed executable binary run by the container, while CMD supplies default arguments that can be easily overridden at runtime. |
| 8 | Why should you always use exec form ["app", "--flag"] instead of shell form "app --flag" in ENTRYPOINT? |
Shell form wraps the process in /bin/sh -c as PID 1, which swallows SIGTERM signals and prevents graceful application shutdown. |
| 9 | How do you prevent a compromised container process from having root privileges on the host kernel? | Add a non-root user (USER 10001) in the Dockerfile and set allowPrivilegeEscalation: false at runtime. |
| 10 | How does docker run -p 8080:80 route host traffic into the containerโs isolated network namespace? |
It injects an iptables Destination NAT (DNAT) rule in the hostโs PREROUTING chain pointing to the containerโs virtual Ethernet (veth) IP. |
Lesson 5: Kernel Namespaces & cgroups
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | What is the exact division of responsibility between Linux Namespaces and Control Groups (cgroups)? |
Namespaces isolate visibility (what a process can see), while cgroups restrict resource consumption (how much CPU/RAM/I/O a process can use). |
| 2 | Name the 7 Linux Kernel Namespaces used by container runtimes. | pid (Process IDs), net (Networking), mnt (Mount points), uts (Hostname), ipc (Inter-process comms), user (UIDs), and cgroup (cgroup root). |
| 3 | What Linux system call creates a new process inside brand-new isolated namespaces? | clone() (passed with flags like `CLONE_NEWPID |
| 4 | What Linux command allows an SRE to enter the network namespace of a running container using its host PID? | nsenter -t <host-pid> -n ip addr (allows running host network debugging tools inside a distroless containerโs netns). |
| 5 | How does a container network namespace communicate with the host root network namespace? | Through a virtual Ethernet (veth) pipe pair, where one end sits inside the container netns and the other attaches to a host bridge (docker0 or cni0). |
| 6 | What is the difference between cpu.shares (CPU requests) and cpu.cfs_quota_us (CPU limits)? |
cpu.shares is a soft proportional weight enforced only during CPU contention, whereas cpu.cfs_quota_us is a hard ceiling enforced every 100ms regardless of idle host CPU. |
| 7 | What architectural upgrade did cgroups v2 introduce over cgroups v1? |
Unified single hierarchy design and the ability to safely restrict buffered Page Cache and writeback disk I/O alongside anonymous memory. |
| 8 | What is Linux Pressure Stall Information (PSI) in /proc/pressure/{cpu,memory,io}? |
A kernel metric measuring the exact percentage of wall-clock time tasks are stalled waiting for hardware resources (some vs full). |
| 9 | What happens when a containerโs memory usage (anon + non-reclaimable cache) breaches memory.max? |
The kernel invokes the cgroup OOM killer (mem_cgroup_out_of_memory), sending SIGKILL (Exit Code 137) to the highest-scoring process. |
| 10 | What system call changes the root directory of a container more securely than chroot? |
pivot_root(), which unmounts the old host root filesystem entirely so privileged processes cannot break out via file descriptors. |
Hands-On Labs: Phase 0 Exercises
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | How do you manually create an isolated network namespace named red using iproute2? |
Run sudo ip netns add red. |
| 2 | How do you create a virtual Ethernet (veth) cable connecting veth-red and veth-host? |
Run sudo ip link add veth-red type veth peer name veth-host. |
| 3 | How do you move one end of a veth interface into the red network namespace? |
Run sudo ip link set veth-red netns red. |
| 4 | Why does ping 127.0.0.1 fail inside a newly created network namespace? |
The loopback interface (lo) is down by default in a new netns and requires ip netns exec red ip link set lo up. |
| 5 | What host kernel sysctl must be set to 1 for Linux to route packets between interfaces like a router? |
sudo sysctl -w net.ipv4.ip_forward=1. |
๐ Phase 1: Enterprise Multi-Cloud Networking Lessons
Lesson 2: Transit Gateway & Egress
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | Why does VPC Peering fail at enterprise scale (N > 10 VPCs)? | VPC Peering is non-transitive and requires N(N-1)/2 full-mesh connections (e.g., 100 VPCs require 4,950 peerings). |
| 2 | What is the difference between a TGW Route Table Association and Propagation? | An Association binds an attachment to a single route table for its outbound routing decisions; a Propagation dynamically injects an attachmentโs CIDR into target route tables. |
| 3 | What single Transit Gateway setting prevents asymmetric routing across multi-AZ stateful inspection firewalls? | Enable Appliance Mode (ApplianceModeSupport = enable) on the Inspection VPCโs Transit Gateway attachment. |
| 4 | How many concurrent connections can a single AWS NAT Gateway support to a single destination IP:Port? | 55,000 connections per destination tuple before throwing port allocation errors. |
| 5 | How do you eliminate NAT Gateway data processing charges ($0.045/GB) for high-volume S3 and DynamoDB traffic? | Provision free Gateway VPC Endpoints for S3 and DynamoDB, which route traffic directly over the private AWS backbone via route table entries. |
Lesson 4: PrivateLink, BGP & Hybrid DNS
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | Can two VPCs with identical, overlapping 10.0.0.0/16 CIDRs communicate securely? |
Yes, by exposing the target application behind a Network Load Balancer (NLB) and connecting via an AWS PrivateLink Interface Endpoint. |
| 2 | In what order does a router evaluate BGP attributes to select the best outbound path? | Weight (highest) โ Local Preference (highest, outbound) โ AS_PATH (shortest) โ MED (lowest, inbound). |
| 3 | What protocol is paired with BGP to detect physical link failures in under 300ms instead of waiting 90s for BGP Hold timers? | BFD (Bidirectional Forwarding Detection). |
| 4 | What is the difference between a Route 53 Inbound Resolver Endpoint and an Outbound Resolver Endpoint? | Inbound allows on-prem DNS servers to resolve AWS Private Hosted Zones; Outbound forwards queries from AWS VPCs to on-prem DNS servers. |
| 5 | How do you connect AWS Direct Connect and Google Cloud Partner Interconnect privately without physical datacenter routers? | Provision a software-defined virtual router (MCR) on Megaport or Equinix Fabric to bridge BGP sessions across both cloud fabrics at Layer 3. |
Lesson 5: Cloud WAN, IPv6 & Network Forensics
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | How does AWS Cloud WAN improve upon managing multi-region Transit Gateways manually? | It replaces manual inter-region TGW peering and static route tables with a single declarative Global Core Network Policy written in JSON. |
| 2 | What device enables IPv6-only Kubernetes pods to communicate with legacy IPv4-only external APIs? | NAT64 paired with DNS64 (which synthesizes IPv6 64:ff9b::/96 records from IPv4 A records). |
| 3 | What is an Egress-Only Internet Gateway (EOIGW) in AWS? | The IPv6 equivalent of a NAT Gatewayโit allows outbound IPv6 traffic from private subnets while blocking unsolicited inbound IPv6 connections. |
| 4 | Why are standard VPC Flow Logs (srcaddr, dstaddr) insufficient when packets traverse an NLB or Transit Gateway? |
Standard fields only show the intermediate NLB/TGW ENI IP addresses, whereas v5 pkt-srcaddr and pkt-dstaddr reveal the original pod and final destination IPs. |
| 5 | What do the [S], [S.], [P.], and [R] flags mean in tcpdump output? |
[S] = SYN, [S.] = SYN-ACK, [P.] = PUSH-ACK (data payload), and [R] = RST (Connection Reset). |