🚨 Round 4: SRE Practices, Incident Management & War-Room Debugging

In Staff SRE and Principal Reliability Architect interviews at top Indian high-concurrency unicorns (Swiggy, PhonePe, Zomato, Razorpay, CRED, Zepto), Round 4 is the crucible.

Indian interviewers simulate high-stress live production war-rooms: “It’s Friday 8:00 PM during an IPL final or Diwali flash sale. Latency spiked 100x, 504 Gateway Timeouts are flooding users, executive leadership is on the bridge, and your pods are crashlooping. Walk me through your first 5 minutes.”

🧠 Beginner Mental Model & Step-by-Step War-Room Interview Playbook

Think Like an Airline Captain During an Engine Failure: In Round 4 War-Room interviews, interviewers test whether you panic and randomly restart servers, or follow a calm, structured Step-by-Step Incident Triage Playbook: 1. Measure Impact (SLI/Error Budget) → 2. Stop the Bleeding (Rollback / Shed Load / Failover) → 3. Isolate the Bottleneck (Metrics, Traces, Logs) → 4. Remediate Root Cause.

📌 Question 1: Mathematical SLOs & Multi-Window Multi-Burn-Rate Alerting

🎯 The Scenario & Interviewer’s Trap

“Your e-commerce checkout service has a 99.9% availability SLO over a 30-day rolling window. What is your total allowable downtime in minutes? If you configure a traditional alert: alert when error rate > 0.1% over 5 minutes, why is this alert completely broken in production? How does the Google SRE multi-window multi-burn-rate alerting algorithm fix this?”

Interviewer’s Intent: To verify whether you understand the mathematics of error budgets and how to construct production alerts that balance precision (no false positives) with recall (fast detection of fatal outages).

💡 Staff-Level Model Answer

1. Error Budget Mathematics:

For a 30-day rolling window: 📐 Total Window Minutes = 30 × 24 × 60 = 43,200 minutes 📐 Allowed Downtime for 99.9% = 43,200 × (1 - 0.999) = 43,200 × 0.001 = 43.2 minutes

• Your service can only experience 43.2 minutes of total downtime per month before breaching its SLA.

2. Why Naive Single-Window Alerts Fail:

• The 5-Minute Window Trap (error rate > 0.1% for 5m):
  • If a transient blip causes 0.2% errors for 5 minutes, an engineer is paged at 3:00 AM. But this blip consumed only 5 × 0.002 = 0.01 error minutes out of 43.2 minutes—less than 0.02% of your budget! Extreme alert fatigue.
• The 24-Hour Window Trap (error rate > 0.1% for 24h):
  • If the service is 100% down, the 24-hour average error rate will take hours to cross the threshold. By the time the alert fires, your entire monthly error budget is burned 10 times over!

3. The Google SRE Standard: Multi-Window Multi-Burn-Rate Alerts:

A Burn Rate of 1.0 means you will exhaust exactly 100% of your error budget over 30 days. A Burn Rate of 14.4 means you will exhaust 100% of your budget in 30 days / 14.4 = 2 days (or 2% of budget in 1 hour).

flowchart TD
    subgraph MultiBurn["Multi-Window Multi-Burn-Rate Algorithm"]
        Long["Long Window (1 hour): Burn Rate > 14.4"]
        Short["Short Window (5 mins): Burn Rate > 14.4"]
        AND{"Both Windows True? (Logical AND)"}
        Long --> AND
        Short --> AND
    end

    AND -- "YES: Severe Outage Active!" --> Page["PAGE ON-CALL SRE IMMEDIATELY!"]
    AND -- "NO: Transient Blip (Reset)" --> Silence["No Page / Auto-Resolved"]

Production Prometheus Alert Rule:

- alert: CheckoutHighErrorBudgetBurnRate
  expr: |
    (
      sum(rate(http_requests_total{job="checkout", status=~"5.."}[1h]))
      /
      sum(rate(http_requests_total{job="checkout"}[1h]))
    ) > (14.4 * 0.001)
    and
    (
      sum(rate(http_requests_total{job="checkout", status=~"5.."}[5m]))
      /
      sum(rate(http_requests_total{job="checkout"}[5m]))
    ) > (14.4 * 0.001)
  for: 2m
  labels:
    severity: page
  annotations:
    summary: "Checkout service burning 2% of 30-day error budget in 1 hour!"

• Why the Short Window is Essential: If the outage abruptly stops after 6 minutes, the 5-minute window immediately drops below the threshold, automatically resolving the alert and preventing spurious pages!

📌 Question 2: Prometheus TSDB High-Cardinality Outage & Memory Exhaustion

🎯 The Scenario & Interviewer’s Trap

“A developer merges a PR that adds the customer’s user_id and order_id as Prometheus metric labels on payment_transactions_total. Within 30 minutes, the central Prometheus server spikes to 128GB RAM, becomes unresponsive to PromQL queries, and terminates with OOMKilled. Upon restarting, Prometheus crashes again during WAL replay. Walk through the internal TSDB mechanics, explain why this happened, and provide the exact steps to recover Prometheus.”

Interviewer’s Intent: To test whether you understand the internal architecture of Prometheus TSDB (Head block, inverted index, postings tables, WAL replay) and how to sanitize high-cardinality label explosions.

💡 Staff-Level Model Answer

flowchart TD
    Metric["payment_transactions_total{user_id='1000001', order_id='998822'}"] --> Explode["1,000,000 unique users × 5,000,000 orders"]
    Explode --> TSDB["Prometheus TSDB Head Block"]
    TSDB --> Index["Inverted Index: Label Names to Postings Lists"]
    Index --> RAM["RAM Explodes to 128GB! (OOMKilled Exit 137)"]
    RAM --> Restart["Prometheus restarts & replays Write-Ahead Log (WAL)"]
    Restart --> Replay["WAL reads massive high-cardinality series back into RAM"]
    Replay --> Crash["CRASH-LOOP REPEAT!"]

1. Why High Cardinality Destroys Prometheus TSDB:

• In Prometheus, each unique combination of key-value label pairs constitutes a separate Time Series.
• If a metric includes user_id (10 million active users), Prometheus must create 10,000,000 independent time series in the in-memory Head Block.
• For each series, Prometheus creates:
  1. A chunk descriptor in memory.
  2. Entries in the Inverted Index (hash tables mapping label pairs to series IDs).
  3. A postings list on disk and RAM.
• Storing 10M active series requires gigabytes of pointer references. When Prometheus writes to the Write-Ahead Log (WAL), the log grows uncontrollably.
• The WAL Replay Trap: When Prometheus restarts after an OOMKill, its first task is to read the WAL from disk to restore the in-memory Head block state. It reloads the 10 million series back into RAM and crashes again before accepting any traffic!

2. The Surgical Emergency Recovery Procedure:

Step 1: Quarantine & Drop the Poisonous Label at Ingestion

Immediately modify the Prometheus scrape configuration using metric_relabel_configs:

scrape_configs:
  - job_name: 'payment-service'
    metric_relabel_configs:
      - action: labeldrop
        regex: "(user_id|order_id|device_ip)"

Step 2: Clear Corrupted WAL Segments (If in CrashLoop)

If Prometheus cannot boot due to WAL replay:

# Analyze TSDB memory usage using promtool:
promtool tsdb analyze /var/prometheus/data

# If Prometheus refuses to start, temporarily move the active WAL:
mv /var/prometheus/data/wal /var/prometheus/data/wal.bak

# Start Prometheus (it will start clean with empty Head block)
# Delete the rogue series via Prometheus Admin API:
curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]=payment_transactions_total{user_id=~".+"}'

# Clean up tombstones:
curl -X POST 'http://localhost:9090/api/v1/admin/tsdb/clean_tombstones'

Step 3: Implement Head Block Series Limits

Protect Prometheus against future developer errors:

scrape_configs:
  - job_name: 'payment-service'
    sample_limit: 50000        # Aborts scrape if target exposes > 50,000 metrics
    series_limit: 100000

📌 Question 3: The Live War-Room Scenario: Cascading Downstream Failure

🎯 The Scenario & Interviewer’s Trap

“It is Friday 8:30 PM. An Indian food-delivery super-app is handling 50,000 orders/minute. The order-placement service response time jumps from 45ms to 9,500ms. Ingress returns 504 Gateway Timeout. Worker node CPU is only 15%, but Kubernetes HPA has scaled the order-placement deployment to its maximum limit of 600 pods. Downstream Aurora PostgreSQL read replicas show 95% CPU and connection pool errors. What is your first 5-minute investigation protocol, what is the exact failure mechanism, and how do you stabilize the system?”

Interviewer’s Intent: To evaluate your calm, structured triage ability under live outage conditions and verify if you understand thread starvation, connection pool exhaustion, and the difference between symptom and root cause.

💡 Staff-Level Model Answer

flowchart TD
    subgraph Traffic["50,000 Orders/Min"]
        Clients["Mobile Apps"] --> ALB["AWS ALB"]
    end

    subgraph Service["order-placement Pods (Scaled to 600)"]
        ALB --> Pods["600 Pods × 50 DB Connections = 30,000 Connections!"]
    end

    subgraph DB["Aurora PostgreSQL Cluster"]
        Pods -->|Connection Storm| Aurora["Aurora Primary (max_connections: 5,000)"]
        Aurora --> Lock["Slow Query with Lock Contention"]
        Lock --> Hang["PostgreSQL worker processes stall"]
    end

    Hang --> ThreadStarve["Microservice Tomcat/Go worker threads stall waiting for DB socket"]
    ThreadStarve --> HPA["HPA sees high response time / scaling trigger -> SCALES MORE PODS!"]
    HPA --> Exacerbate["600 pods hammer DB with MORE connections: COMPLETE COLLAPSE!"]

1. First 5 Minutes: Rapid Triage Protocol (Do NOT Guess):

1. Minute 1: Mitigate Ingress Blast Radius:
   • Do not restart pods blindly (restarts cause cold-start stampedes).
   • Enable an emergency rate-limiting tier on Cloudflare / AWS WAF or temporarily return a cached degraded response (“We are experiencing high demand, please wait 30s”) to shed 40% of ingress load.
2. Minute 2: Check Thread & Connection States:
   • Why is CPU low (15%) while latency is high (9.5s)?
   • Low CPU + High Latency is the classic hallmark of I/O blocking / Thread Starvation (threads waiting on database sockets or external HTTP calls).
3. Minute 3: Inspect Database Active Queries:

   -- Run on Aurora PostgreSQL:
   SELECT pid, now() - query_start AS duration, state, query 
   FROM pg_stat_activity 
   WHERE state != 'idle' 
   ORDER BY duration DESC LIMIT 10;

4. Minute 4: Identify Lock Contention / Unindexed Table Scan:
   • Find queries stuck in idle in transaction or waiting on table/row exclusive locks (pg_locks).
5. Minute 5: Terminate Poison Queries:

   SELECT pg_terminate_backend(<pid>);

2. The Root Cause Mechanism (The Cascade):

• An unindexed query or row lock slowed query execution from 2ms to 2,000ms.
• Because the microservice lacked a tight connection timeout, client requests backed up inside the application server thread pool.
• Kubernetes HPA saw pod CPU or request counts rise and spawned more pods (scaling from 100 to 600 pods).
• Each new pod opened a pool of 50 connections: 📐 Total Attempted Connections = 600 × 50 = 30,000 connections
• PostgreSQL exceeded max_connections (5,000). The database kernel spent 90% of its CPU time context-switching between stalled backend processes rather than executing SQL!

3. Long-Term Architectural Immunization:

1. Deploy In-Cluster Database Connection Pooling (PgBouncer / AWS RDS Proxy):
   • Decouples application pod connection scaling from database process limits. 600 pods connect to RDS Proxy over 30,000 lightweight multiplexed connections; RDS Proxy maintains a clean, fixed pool of 300 real connections to PostgreSQL.
2. Enforce Strict Client-Side Timeouts:
   • Database socket timeout = 2000ms.
   • HTTP client downstream timeout = 1500ms.
3. Bulkhead Pattern & Circuit Breaking:
   • If the payment database is degrading, trip an Envoy / Resilience4j circuit breaker immediately, returning fast failure rather than letting threads stall.

📌 Question 4: The Dogpile Effect (Cache Stampede) & Mitigation

🎯 The Scenario & Interviewer’s Trap

“Your Redis cluster caches product catalog items for an e-commerce platform. A flash sale starts for an iPhone with 200,000 requests/sec hitting a single Redis key (product:iphone-16). At 12:00:00 AM, the key’s TTL expires. What happens to the underlying MySQL / Aurora database within the next 200 milliseconds, and how do you architecturally eliminate this without increasing cache TTL to infinity?”

Interviewer’s Intent: To verify whether you understand the Dogpile Effect (Cache Stampede) and know how to implement algorithmic cache defenses like distributed mutexes or probabilistic early recomputation (XFetch).

💡 Staff-Level Model Answer

flowchart TD
    subgraph Traffic["200,000 Req/sec for 'product:iphone-16'"]
        R1["Req 1"] & R2["Req 2"] & R3["Req 3..."] & R4["Req 200,000"]
    end

    Traffic --> Redis{"Redis: GET product:iphone-16"}
    Redis -- "TTL Expired at 12:00:00! (Cache Miss)" --> Miss["200,000 SIMULTANEOUS CACHE MISSES!"]
    
    subgraph Stampede["The Thundering Herd / Dogpile"]
        Miss --> DB["Aurora MySQL Database"]
        DB --> Crash["Database Connection Pool Collapses in 50ms!"]
    end

The 3 Architectural Solutions:

Strategy 1: Distributed Mutex Locking (Single-Flight Pattern)

When a cache miss occurs, the application process does not immediately query the database. It attempts to acquire a short-lived distributed lock in Redis:

# Acquire a 2-second lock using Redis SETNX:
if redis.set("lock:product:iphone-16", "1", nx=True, ex=2):
    try:
        # Only ONE worker fetches from DB:
        data = db.query("SELECT * FROM products WHERE id='iphone-16'")
        redis.set("product:iphone-16", data, ex=3600)
    finally:
        redis.delete("lock:product:iphone-16")
else:
    # All other 199,999 threads sleep for 50ms and re-check Redis:
    time.sleep(0.05)
    return redis.get("product:iphone-16")

• In Go microservices, use golang.org/x/sync/singleflight to collapse concurrent duplicate requests within the same pod into a single shared database call.

Strategy 2: Probabilistic Early Invalidation (The XFetch Algorithm)

Instead of waiting for the key to expire at t = expiry, workers probabilistically recompute the cache before it expires: 📐 Recompute If: -β × δ × ln(random()) > (expiry - currentTime)

• Where δ is the time taken to compute the value from the database, and β > 0 is an aggressive factor.
• As the remaining TTL decreases, the probability of a background request proactively refreshing the key approaches 1.0. The key never actually expires for real users!

Strategy 3: Asynchronous Background Refresh Daemon

Decouple client reads from cache updates entirely:

• Set key TTL to infinity.
• An asynchronous event-driven worker (Kafka / EventBridge) listens for product updates and writes to Redis, so user HTTP requests never perform read-through queries to the database.

📌 Summary: SRE Practices & Incident Triage Reference

🧠 Complete Lesson-by-Lesson One-Line Memory Bank (Round 4 Scope)

🏛️ Enterprise Observability, SRE & Database Lessons

Lesson 4: CloudFront Edge Delivery & Log Management

Lesson 5: RUM Specs, Telemetry & SRE Standards

Lesson 6: Securing Scalable Systems, KPIs, KRIs & Auditing

Lesson 8: DB Seeding, Migrations & Cyclic Dependencies

Lesson 12: War Room Incident Defense & RCAs

🚨 Incident Defense & Triage Modules (Modules 1–7 One-Line Recall)