🏗️ Round 5: Platform System Design & High-Concurrency Architectures

In Staff and Principal Platform Architect interviews (Uber IDC, Google Cloud, Razorpay, PhonePe, Swiggy, Atlassian), Round 5 is the whiteboard system design test.

Unlike application system design (which asks you to design Instagram or TinyURL), Platform System Design evaluates infrastructure architecture: designing internal developer platforms, edge ingress engines capable of handling 100,000 requests/second, and multi-tenant Kubernetes platforms governing thousands of engineers.

🧠 Beginner Mental Model & Step-by-Step System Design Playbook

Think Like a City Planner: Never start a System Design interview by drawing boxes randomly! Always follow this 5-Step Architecture Blueprint Playbook:

1. Step 1 (Clarify Scale & SLOs): Ask for QPS, concurrency (Little’s Law), RTO/RPO, and multi-tenancy blast-radius requirements.
2. Step 2 (Data & Network Foundation): Define the VPC/TGW topology, CIDR sizing, and stateful storage tier.
3. Step 3 (Control Plane & Compute): Layer on EKS, Karpenter autoscaling, Cilium eBPF isolation, and GitOps reconciliation.
4. Step 4 (Resilience & Failure Modes): Walk through how the architecture survives AZ loss, poison-pill deployments, and thundering herds.
5. Step 5 (FinOps Economics): Quantify the cost optimizations (Spot, Graviton, Shared ALBs, Gateway Endpoints).

📌 System Design 1: Multi-Tenant Kubernetes Platform for 500 Engineering Teams

🎯 The Challenge & Requirements

“Design a centralized Kubernetes platform for an enterprise with 500 engineering teams, 2,500 microservices, and strict PCI-DSS compliance requirements. How do you decide between cluster-per-team vs multi-tenancy? How do you guarantee noisy neighbor isolation, automated governance, secure egress, and multi-cluster GitOps orchestration?”

💡 Staff-Level Architectural Blueprint

flowchart TD
    subgraph Management["Central Management Cluster"]
        ArgoCD["ArgoCD Control Plane (ApplicationSets)"]
        Vault["HashiCorp Vault (AppRole / Raft)"]
        HNC["Hierarchical Namespace Controller (HNC)"]
    end

    subgraph Fleet["Workload Cluster Fleet (Per Business Unit / Region)"]
        subgraph ClusterA["Workload Cluster: Payments & Core (PCI-DSS)"]
            subgraph Tenant1["Namespace: team-checkout (Payment Gateway)"]
                Pod1["Checkout Pods"]
                RQ1["ResourceQuota + LimitRange"]
                NP1["Cilium L7 NetworkPolicy (Deny All by Default)"]
            end
            subgraph Tenant2["Namespace: team-billing"]
                Pod2["Billing Pods"]
            end
            Cilium["Cilium eBPF CNI & WireGuard Encryption"]
            Gatekeeper["Kyverno / OPA Gatekeeper Admission Controller"]
        end
    end

    ArgoCD -->|GitOps Synchronize| ClusterA
    HNC -->|Propagates Policies & RBAC| Tenant1
    HNC -->|Propagates Policies & RBAC| Tenant2

1. Tenancy Model Decision: Multi-Tenant Clusters with Hard Boundary Partitioning

• Why not Cluster-per-Team? 500 clusters would cost $100,000+/month in control plane fees alone, waste millions in idle compute overhead, and require managing 500 separate control plane upgrade schedules.
• The Hybrid Standard: Group teams by Risk & Compliance Domain (e.g., 3 Production Clusters: PCI-Regulated Core, General Workloads, and High-Memory ML/Analytics), then enforce Hard Multi-Tenancy inside each cluster via Namespaces.

2. The 4 Layers of Tenant Isolation:

1. Compute & Resource Isolation:
   • ResourceQuota: Hard ceilings on CPU, Memory, and Storage per namespace to prevent resource starvation.
   • LimitRange: Enforces mandatory requests and limits on every pod, defaulting to Burstable QoS.
   • Karpenter Node Pools per Tier: Critical payment services run on dedicated node pools isolated via Node Taints and Tolerations (dedicated=payments:NoSchedule).
2. Network Isolation (Zero-Trust with Cilium eBPF):
   • Default Deny-All ingress and egress policies in every tenant namespace.
   • Fine-grained L7 policies: Team Checkout can only speak to Team Billing on POST /v1/charge over mTLS.
   • Egress to external internet is restricted to specific FQDNs via Cilium DNS-aware network policies.
3. Control Plane & Governance Isolation:
   • Hierarchical Namespace Controller (HNC): Organizes namespaces into parent-child hierarchies (payments-root → payments-checkout, payments-fraud), automatically propagating RBAC roles, LimitRanges, and NetworkPolicies from parent to children.
   • Kyverno / OPA Gatekeeper: Enforces admission guardrails (prevents hostPath, privileged: true, and non-approved container registries).
4. GitOps Fleet Delivery:
   • ArgoCD ApplicationSets with Matrix Generators: Dynamically targets clusters and tenant namespaces based on Git repository folders and cluster labels.

📌 System Design 2: High-Throughput Edge Ingress for 100,000 Requests/Sec

🎯 The Challenge & Requirements

“Design the edge ingress network for an Indian payment gateway (like Razorpay / PhonePe) that processes 100,000 incoming HTTPS API requests per second at peak. Latency must be < 50ms p99 within India, with 99.999% availability, automated DDoS mitigation, TLS session resumption, and distributed rate limiting.”

💡 Staff-Level Architectural Blueprint

flowchart TD
    Users["100,000 Mobile Apps / Clients (India-wide)"] --> Route53["AWS Route 53 (Anycast Geolocation & Health Checks)"]
    Route53 --> Edge["Edge Tier: Cloudflare / AWS CloudFront (Anycast PoPs in BLR, BOM, DEL, MAA)"]
    
    subgraph EdgeLayer["Edge PoP Processing (<10ms)"]
        Edge --> WAF["Edge WAF (OWASP Top 10 + Bot Mitigation)"]
        WAF --> TLS["TLS 1.3 Termination & 0-RTT Resumption"]
        TLS --> Shield["AWS Shield Advanced (L3/L4 DDoS Mitigation)"]
    end

    Shield -->|Direct DirectConnect / AWS Backbone| ALB["AWS Application Load Balancer (Cross-Zone Enabled)"]
    
    subgraph IngressTier["Kubernetes Ingress Tier (EKS)"]
        ALB --> Envoy["Envoy Gateway / Cilium Gateway API (DaemonSet on Host Network)"]
        Envoy --> RateLimit["Distributed Rate Limiting (Redis Cluster - Sliding Window)"]
        RateLimit --> Microservices["Core Payment Microservices (mTLS via Cilium eBPF)"]
    end

1. Architectural Component Breakdown:

1. Edge Anycast Routing (Route 53 & Cloudflare/CloudFront):
   • Incoming traffic is routed to the nearest Point of Presence (PoP) in Mumbai (BOM), Bengaluru (BLR), Delhi (DEL), or Chennai (MAA) via BGP Anycast.
   • Edge PoPs absorb L3/L4 SYN floods and UDP reflection attacks before traffic touches AWS origin infrastructure.
2. TLS 1.3 Termination & Session Resumption:
   • Terminating TLS at the edge reduces round-trip handshake latency from 80ms to under 15ms.
   • Enforce TLS 1.3 with 0-RTT session resumption and Session Tickets (RFC 8446) for recurring mobile client calls.
3. AWS Application Load Balancer (ALB) Tuning:
   • Enable Cross-Zone Load Balancing to distribute traffic symmetrically across all 3 Availability Zones.
   • Pre-warm ALBs with AWS Support before anticipated flash events to avoid DNS lookup throttling during instant 10x traffic spikes.
4. Envoy / Cilium Gateway API Tier:
   • Run Envoy as a DaemonSet with hostNetwork: true on dedicated high-performance compute nodes (c7g.8xlarge with AWS Graviton3).
   • Bypasses kube-proxy iptables overhead, using eBPF sockops to route packets directly from NIC ring buffers to the target microservice pods.
5. Distributed Rate Limiting (Redis Sliding Window Counter):
   • Protects backend services against abusive API keys.
   • Uses Redis cluster with pipeline Lua scripts evaluating sliding window log counts over a 1-second interval: 📐 Allowed If: Count(Window_t - 1s, t) < Limit

📌 System Design 3: Internal Developer Platform (IDP) with Spotify Backstage

🎯 The Challenge & Requirements

“Your organization has 1,200 developers who currently spend 3 weeks submitting Jira tickets to provision a new microservice, configure S3 buckets, set up CI/CD pipelines, and request IAM roles. Design an Internal Developer Platform (IDP) powered by Spotify Backstage that reduces service creation time to under 5 minutes (‘Golden Path’).”

💡 Staff-Level Architectural Blueprint

sequenceDiagram
    autonumber
    participant Dev as Developer
    participant UI as Backstage UI (Portal)
    participant Scaffolder as Backstage Scaffolder Engine
    participant Git as GitHub Enterprise
    participant Crossplane as Crossplane / OpenTofu (K8s Control Plane)
    participant Argo as ArgoCD
    participant Vault as HashiCorp Vault

    Dev->>UI: Selects "Production Go Microservice" Template
    Dev->>UI: Fills Inputs: Service Name, Team, Cost Center, DB Requirements
    UI->>Scaffolder: Triggers Scaffolding Job
    Scaffolder->>Git: Creates Repository with boilerplate, Dockerfile & CI actions
    Scaffolder->>Vault: Provisions AppRole & Dynamic Secret Mount
    Scaffolder->>Git: Commits Infrastructure XR Manifest (Postgres, S3, IAM)
    Scaffolder->>Git: Registers Service in ArgoCD ApplicationSet directory
    Git->>Crossplane: Crossplane reconciles XR -> Provisions AWS RDS & S3
    Git->>Argo: ArgoCD detects new app -> Deploys to Staging Cluster
    Scaffolder-->>Dev: Ready! PR created, repo live, URL: https://service.dev.corp

1. Core Architecture Components:

1. Software Catalog (Entity Data Model):
   • Acts as the single source of truth for all software components, APIs (OpenAPI/gRPC), technical documentation (TechDocs), and team ownership (catalog-info.yaml stored in every repo).
2. Software Templates (The Scaffolder Engine):
   • Pre-packaged Golden Path templates for Go, Java, Python, and Node.js.
   • Eliminates boilerplate setup: injects standardized logging (JSON format), OpenTelemetry tracing headers, standard Dockerfiles with non-root users, and healthcheck endpoints.
3. Automated Infrastructure Provisioning via Crossplane:
   • Instead of waiting for platform team review on Terraform scripts, the Backstage Scaffolder commits a Kubernetes Custom Resource:

     apiVersion: platform.corp.com/v1alpha1
     kind: MicroserviceDatabase
     spec:
       engine: postgresql
       storageGB: 50
       environment: staging

   • Crossplane provisions the AWS RDS Aurora instance, configures KMS encryption, sets up Security Groups, and stores credentials directly into Vault.
4. Service Health Scorecards:
   • Automatically grades microservices (Grade A to F) based on:
     • Does the service have a valid PDB?
     • Does the service have CPU/RAM requests configured?
     • Is the container image signed with Cosign?
     • Does test code coverage exceed 80%?

📌 System Design 4: High-Performance LLMOps & GPU Cluster Serving

🎯 The Challenge & Requirements

“Design a private, high-concurrency LLM inference platform serving open-source models (Llama 3 70B, Mistral Large) using NVIDIA H100 GPU clusters on Kubernetes. Requirements: Sub-50ms Time to First Token (TTFT), continuous batching with vLLM, automated model caching, and multi-tenant GPU sharing.”

💡 Staff-Level Architectural Blueprint

flowchart TD
    Clients["AI Application Clients"] --> Gateway["AI Gateway (LiteLLM / Envoy)"]
    Gateway --> Route["Semantic Cache & Routing (Redis Vector Search)"]
    Route --> K8s["Kubernetes GPU Cluster (EKS / GKE)"]

    subgraph GPUCluster["GPU Node Pool (NVIDIA H100 80GB SXM5)"]
        subgraph Node1["Worker Node 1 (8x H100 GPUs via NVLink)"]
            vLLM1["vLLM Serving Worker (Tensor Parallelism: TP=4)"]
            vLLM2["vLLM Serving Worker (TP=4)"]
            Cache["Local NVMe Fast Cache (/models/llama-3-70b)"]
        end
        NvidiaDriver["NVIDIA GPU Operator & DCGM Exporter"]
    end

    Route --> vLLM1
    Route --> vLLM2

    subgraph Storage["High-Throughput Storage Fabric"]
        Lustre["Amazon FSx for Lustre / S3 Express OneZone"]
    end

    Lustre -->|Pre-loads 140GB weights in 12s via AWS EFA| Cache

1. Core Architectural Pillars:

1. Inference Engine: vLLM with PagedAttention:
   • Standard HuggingFace transformers allocate continuous blocks of GPU VRAM for the Key-Value (KV) cache, wasting 60–80% of memory due to fragmentation.
   • vLLM PagedAttention partitions the KV cache into non-contiguous virtual memory blocks (like OS paging), boosting serving throughput by 4× to 8×.
   • Continuous Batching: Dynamically injects incoming user prompts into active forward passes rather than waiting for entire sequence generations to complete.
2. GPU Slicing & Tensor Parallelism:
   • A 70-billion parameter model in FP16 requires ~ 140GB of VRAM. An 80GB H100 cannot hold it alone.
   • Configure Tensor Parallelism (TP = 4 or 8) across GPUs connected via NVIDIA NVLink (900 GB/s inter-GPU bandwidth), splitting attention heads across multiple GPUs on the same physical node.
3. High-Speed Model Caching Fabric:
   • Cold-starting a 140GB model from standard S3 takes 8 to 15 minutes!
   • Use Amazon FSx for Lustre or local instance-store NVMe SSDs pre-warmed via Kubernetes DaemonSets with direct kernel page caching, reducing cold-start load times to under 15 seconds.
4. GPU-Specific Autoscaling Metrics:
   • Traditional CPU/RAM HPA fails for LLM serving!
   • Scale worker pods based on vLLM-specific metrics scraped by Prometheus:
     • vllm:num_requests_waiting (Queue depth)
     • vllm:gpu_cache_usage_factor (KV Cache saturation > 85%)
     • vllm:time_to_first_token_seconds (TTFT SLA breach)

📌 Scenario 5: Designing an Active-Active Multi-Region Azure Platform (Central India + South India)

The Interview Prompt

“Design a zero-trust, active-active multi-region financial platform on Microsoft Azure across Central India (Pune) and South India (Chennai) capable of handling 50,000 TPS with < 30 seconds RTO, zero public IP exposure on backend clusters, and guaranteed transactional message processing.”

🎯 Interviewer’s Trap / Intent

In GCCs (Walmart, Target, Goldman Sachs, Morgan Stanley) and Microsoft IDC interviews, candidates who only know AWS terminology fail when asked to design with Azure’s native primitives. If you propose legacy Kubenet, aad-pod-identity, or AGIC (which takes 60s to update via ARM), or if you don’t explain Cosmos DB’s 5 Consistency Levels and the 10,000 RU/s Physical Partition Hot-Key Trap, you will be down-leveled.

✅ The Staff-Level Reference Architecture

flowchart TD
    Users["🌍 Global / Indian Banking Users"] -->|"Anycast TLS 1.3"| AFD["🛡️ <b>Azure Front Door Premium</b><br/>Global WAF + DDoS + 5s Health Probes"]

    AFD -->|"Private Link Origin (Weight 50%)"| AGC1["⚖️ <b>App Gateway for Containers (Pune)</b><br/>Sub-second Gateway API Data-Plane"]
    AFD -->|"Private Link Origin (Weight 50%)"| AGC2["⚖️ <b>App Gateway for Containers (Chennai)</b><br/>Sub-second Gateway API Data-Plane"]

    subgraph Region1 ["Central India (Pune) — Hub & Spoke"]
        AGC1 --> AKS1["☸️ <b>Private AKS (Cilium CNI Overlay)</b><br/>Ephemeral NVMe OS Disks + NAP Karpenter<br/>Entra Workload Identity + Key Vault CSI"]
        AKS1 --> SB1["📨 <b>Azure Service Bus Premium</b><br/>FIFO Sessions + KEDA Autoscaling"]
    end

    subgraph Region2 ["South India (Chennai) — Hub & Spoke"]
        AGC2 --> AKS2["☸️ <b>Private AKS (Cilium CNI Overlay)</b><br/>Ephemeral NVMe OS Disks + NAP Karpenter<br/>Entra Workload Identity + Key Vault CSI"]
        AKS2 --> SB2["📨 <b>Azure Service Bus Premium</b><br/>Geo-Disaster Recovery Pairing"]
    end

    AKS1 <-->|"Multi-Region Writes<br/>(Bounded Staleness / Session)"| Cosmos[("🪐 <b>Azure Cosmos DB</b><br/>Hierarchical Partition Keys<br/>[/merchantId, /userId, /orderId]")]
    AKS2 <-->|"Multi-Region Writes"| Cosmos

1. Core Architectural Pillars:

1. Global Edge without Public IPs (Azure Front Door + Private Link):
   • Terminate TLS and enforce OWASP WAF rules at the edge using Azure Front Door Premium.
   • Connect Front Door into the regional Application Gateway for Containers (ALB for AKS) via Azure Private Link Origins, ensuring zero public IPs exist inside the Corp Landing Zone VNets.
   • Use Application Gateway for Containers (Kubernetes Gateway API) rather than legacy AGIC so pod endpoint changes propagate via the fast data-plane API in < 1 second instead of 60-second ARM PUT locks.
2. Compute & CNI Dataplane (Private AKS + Cilium Overlay + Ephemeral NVMe):
   • Configure Azure CNI Overlay powered by Cilium (100.64.0.0/16 pod CIDR) to prevent VNet IP exhaustion and achieve O(1) eBPF kernel routing.
   • Enforce Ephemeral OS Disks (--node-osdisk-type Ephemeral) on local NVMe cache so heavy container image pulls never throttle OS disk IOPS or trigger PLEG is not healthy.
   • Attach an Azure NAT Gateway to the AKS subnet (64,512 dynamic SNAT ports per IP) to prevent the 1,024-port Load Balancer SNAT exhaustion trap on outbound payment calls.
3. Active-Active Data Tier (Cosmos DB Consistency & Hot Partition Mitigation):
   • Enable Cosmos DB Multi-Region Writes across Central India and South India.
   • Select Bounded Staleness (e.g., K = 100 versions or T = 5 seconds) for account ledgers requiring strict cross-region RPO bounds, and Session Consistency (x-ms-session-token) for user-facing order status so users always read their own writes at half the RU/s cost of Strong consistency.
   • Preventing the Hot Partition HTTP 429 Trap: Because each physical partition caps at 10,000 RU/s and 50 GB, configure Hierarchical Partition Keys ([/merchantId, /customerId, /orderId]) so a single large merchant never saturates a single physical partition while sibling partitions sit idle.
4. Transactional Messaging & Identity (Service Bus + KEDA + Entra Workload Identity):
   • Use Azure Service Bus Premium (with FIFO Message Sessions, Duplicate Detection, and Dead-Letter Queues) paired with KEDA to scale settlement pods from 2 to 200 replicas based on queue depth.
   • Authenticate pods using Entra Workload Identity OIDC federated tokens (azure.workload.identity/use: "true") and mount HSM-backed secrets into RAM via the Azure Key Vault Secret Store CSI Driver.

📌 Summary: Platform System Design Cheat Sheet

🧠 Complete Lesson-by-Lesson One-Line Memory Bank (Round 5 Scope)

🎭 Phase 7: Internal Developer Portals & Chaos Engineering (All 5 Lessons)

Lesson 1: Backstage Software Catalog

Lesson 2: Golden Path Templates

Lesson 3 & 4: TechDocs, Backstage RBAC & Scorecards

Lesson 5: Chaos Engineering (Chaos Mesh)

🧠 Phase 8: LLMOps Infrastructure (All 4 Lessons)

Lesson 1: Vector DBs & AI Gateways

Lesson 2: MLflow & Prompt Evaluation

Lesson 3 & Labs: GPU Slicing, vLLM & Guardrails

🏛️ Enterprise High-Concurrency Sizing, 3-Tier K8s & Capstone

Enterprise Lesson 3, Lesson 11 & Capstone