🧠 DevEx, Service Mesh & LLMOps Architect Assessment & Memory Bank
Welcome to the DevEx, eBPF Service Mesh, and LLMOps Infrastructure Diagnostic Matrix & Memory Bank. This module evaluates your mastery across Phase 6 (Cilium & eBPF), Phase 7 (Internal Developer Portals & Chaos Mesh), and Phase 8 (LLMOps & GPU Infrastructure)—followed by the complete Lesson-by-Lesson One-Line Memory Bank for every lesson in Phases 6, 7, and 8.
🧭 Beginner Mental Model & Step-by-Step Assessment Playbook
- Step 1 (Match the Core Primitive): Complete the 3 interactive Match-the-Pair tiers without looking at your notes.
- Step 2 (Active Recall Drill): Cover the right-hand column of the Lesson-by-Lesson One-Line Memory Bank below and answer each question out loud in one sentence.
- Step 3 (Re-Read Any Missed Lesson): If you miss more than 2 questions in a lesson section, click back into that lesson and walk through its Step-by-Step Operational Playbook.
🟢 Tier 1: Service Mesh, eBPF & Zero-Trust (Phase 6)
📌 Left Blocks: Primitives & Directives
A
`kubeProxyReplacement: true` in Cilium
B
Default Deny-All Egress without DNS Rule
C
Cilium `bpf_sockops` Socket Load Balancing
D
WireGuard ClusterMesh over 1500 MTU VPC
E
Gateway API Cross-Namespace `HTTPRoute`
🎯 Right Blocks: Behaviors & Mechanisms
1
Drops UDP/TCP Port 53 traffic to CoreDNS, breaking all pod hostname resolution.
2
Replaces linear O(N) `iptables` chains with O(1) kernel eBPF hash map lookups.
3
Requires explicit `ReferenceGrant` in the target namespace to prevent cross-tenant hijacking.
4
Rewrites Service ClusterIP to Pod IP at the `connect()` syscall before packets enter TCP/IP stack.
5
60-byte tunnel header drops large TLS handshakes unless inner MTU is clamped to `1420`.
💡 Reveal Tier 1 Solution Key & Architectural Rationale Click to Expand
Verified Architectural Pairings:
A → 2 B → 1 C → 4 D → 5 E → 3
A → 2 (`kubeProxyReplacement: true` in Cilium): Replaces linear O(N) `iptables` chains with O(1) kernel eBPF hash map lookups.
B → 1 (Default Deny-All Egress without DNS Rule): Drops UDP/TCP Port 53 traffic to CoreDNS, breaking all pod hostname resolution.
C → 4 (Cilium `bpf_sockops` Socket Load Balancing): Rewrites Service ClusterIP to Pod IP at the `connect()` syscall before packets enter TCP/IP stack.
D → 5 (WireGuard ClusterMesh over 1500 MTU VPC): 60-byte tunnel header drops large TLS handshakes unless inner MTU is clamped to `1420`.
E → 3 (Gateway API Cross-Namespace `HTTPRoute`): Requires explicit `ReferenceGrant` in the target namespace to prevent cross-tenant hijacking.
🟡 Tier 2: Internal Developer Portals & Chaos Engineering (Phase 7)
📌 Left Blocks: Primitives & Directives
A
`techdocs.builder: 'local'` in Production
B
Zombie 50% Packet Loss after Chaos Deleted
C
Backstage Scaffolder Partial Step Failure
D
Malformed `$ref` Loop in `swagger.yaml`
E
Gold-Tier Production Readiness Scorecard
🎯 Right Blocks: Behaviors & Mechanisms
1
`chaos-daemon` crashed before removing kernel `tc qdisc netem` rules from pod `netns`.
2
Spawns heavy Python `mkdocs` processes inside the Backstage pod; fix with `'external'` CI + S3.
3
Evaluates SLOs, PDBs, Cosign signatures, and CVEs to block Bronze services during freeze windows.
4
Requires idempotent actions or compensating Saga rollback steps to clean up orphaned Git repos.
5
Causes Backstage Catalog Processor to crash or reject the `kind: API` entity ingestion loop.
💡 Reveal Tier 2 Solution Key & Architectural Rationale Click to Expand
Verified Architectural Pairings:
A → 2 B → 1 C → 4 D → 5 E → 3
A → 2 (`techdocs.builder: 'local'` in Production): Spawns heavy Python `mkdocs` processes inside the Backstage pod; fix with `'external'` CI + S3.
B → 1 (Zombie 50% Packet Loss after Chaos Deleted): `chaos-daemon` crashed before removing kernel `tc qdisc netem` rules from pod `netns`.
C → 4 (Backstage Scaffolder Partial Step Failure): Requires idempotent actions or compensating Saga rollback steps to clean up orphaned Git repos.
D → 5 (Malformed `$ref` Loop in `swagger.yaml`): Causes Backstage Catalog Processor to crash or reject the `kind: API` entity ingestion loop.
E → 3 (Gold-Tier Production Readiness Scorecard): Evaluates SLOs, PDBs, Cosign signatures, and CVEs to block Bronze services during freeze windows.
🔴 Tier 3: LLMOps, Vector Databases & GPU Serving (Phase 8)
📌 Left Blocks: Primitives & Directives
A
Milvus `QueryNode` OOMKilled on 300GB Vectors
B
vLLM `PagedAttention` Memory Engine
C
NVIDIA MIG vs GPU Time-Slicing
D
LLM-as-a-Judge Flapping CI Pipeline Scores
E
HPA Scaling vLLM on GPU Memory Utilization %
🎯 Right Blocks: Behaviors & Mechanisms
1
Splits attention KV Cache into non-contiguous virtual blocks, eliminating 60–80% VRAM fragmentation.
2
Hardware-partitions an H100/A100 silicon die into up to 7 isolated GPU instances with dedicated VRAM.
3
HNSW in-memory graph index requires 1.5×–2× raw vector byte size in RAM; use `SQ8` or `DiskANN`.
4
vLLM pre-allocates 90% VRAM on boot; autoscale via KEDA on `vllm:num_requests_waiting` instead.
5
Non-zero temperature causes non-deterministic grading; enforce `temperature: 0.0` and pinned seed.
💡 Reveal Tier 3 Solution Key & Architectural Rationale Click to Expand
Verified Architectural Pairings:
A → 3 B → 1 C → 2 D → 5 E → 4
A → 3 (Milvus `QueryNode` OOMKilled on 300GB Vectors): HNSW in-memory graph index requires 1.5×–2× raw vector byte size in RAM; use `SQ8` or `DiskANN`.
B → 1 (vLLM `PagedAttention` Memory Engine): Splits attention KV Cache into non-contiguous virtual blocks, eliminating 60–80% VRAM fragmentation.
C → 2 (NVIDIA MIG vs GPU Time-Slicing): Hardware-partitions an H100/A100 silicon die into up to 7 isolated GPU instances with dedicated VRAM.
D → 5 (LLM-as-a-Judge Flapping CI Pipeline Scores): Non-zero temperature causes non-deterministic grading; enforce `temperature: 0.0` and pinned seed.
E → 4 (HPA Scaling vLLM on GPU Memory Utilization %): vLLM pre-allocates 90% VRAM on boot; autoscale via KEDA on `vllm:num_requests_waiting` instead.
🧠 Complete DevEx, Service Mesh & LLMOps One-Line Memory Bank (Lesson-by-Lesson)
Use these rapid-fire one-line questions and answers to lock every single lesson of Phase 6, Phase 7, and Phase 8 into long-term memory.
🐝 Phase 6: Service Mesh & eBPF 40 Rapid Q&As
Lesson 1: Cilium & eBPF
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | Why does legacy kube-proxy running in iptables mode degrade severely when a cluster scales past 5,000 Kubernetes Services? |
iptables evaluates rules sequentially in an O(N) linear list and must rewrite the entire multi-megabyte rule table atomically on every endpoint update. |
| 2 | How does Cilium (kubeProxyReplacement: true) achieve O(1) service load-balancing at any cluster scale? |
It attaches eBPF programs to kernel socket and TC hooks that look up Service-to-Pod backends directly inside O(1) eBPF Hash Maps in kernel memory. |
| 3 | What ensures an eBPF program loaded into the Linux kernel cannot crash the kernel or enter an infinite loop? | The In-Kernel eBPF Verifier, which performs static DAG code analysis, bounds-checking, and memory safety verification before JIT compilation. |
| 4 | How does Cilium Socket-Level Load Balancing (bpf_sockops) accelerate pod-to-Service communication? |
It intercepts the connect() syscall at the socket layer and rewrites the ClusterIP directly to the backend Pod IP before packets even enter the TCP/IP packet stack. |
| 5 | Why does traditional sidecar service mesh architecture (Istio Envoy per pod) add latency and double memory consumption? | Every packet traverses the Linux TCP/IP stack 4 times per hop (App → local Envoy → Wire → remote Envoy → remote App), and every pod runs a dedicated Envoy proxy container. |
| 6 | How does Cilium provide transparent node-to-node encryption across the cluster without sidecars? | By enabling Kernel-Mode WireGuard Encryption (encryption.enabled=true, encryption.type=wireguard), which encrypts all pod-to-pod traffic between worker nodes at the network interface layer. |
| 7 | What observability tool built into Cilium provides real-time eBPF network flow logs, DNS query visibility, and service dependency maps? | Hubble (hubble observe). |
| 8 | What happens to existing active pod network connections if the cilium-agent DaemonSet pod restarts or crashes? |
Zero packet loss—compiled eBPF programs and maps persist inside the Linux kernel datapath independently of the user-space cilium-agent control plane. |
| 9 | What is eBPF XDP (eXpress Data Path), and why is it the fastest packet processing hook in Linux? | XDP executes eBPF bytecode directly inside the NIC driver ring buffer before the kernel even allocates an sk_buff structure, dropping DDoS packets at line rate. |
| 10 | What CLI command inside a cilium-agent pod displays the live kernel eBPF load-balancing service map? |
cilium service list and cilium bpf lb list. |
Lesson 2: Zero-Trust L7 Policies
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | What is the default network security posture of a fresh Kubernetes cluster without NetworkPolicies? | Flat allow-all network—every pod in every namespace can freely send packets to every other pod across the entire cluster. |
| 2 | Why are Pod IP addresses unreliable for enforcing security firewall rules in Kubernetes? | Pod IPs are ephemeral and churn constantly during autoscaling; Cilium uses cryptographic Security Identities (numeric ID) derived from pod labels instead of IP addresses. |
| 3 | What layers of the OSI model can a standard Kubernetes NetworkPolicy filter vs a CiliumNetworkPolicy? |
Standard NetworkPolicy is strictly L3/L4 (IPs, Pod Labels, TCP/UDP ports), whereas CiliumNetworkPolicy filters L3, L4, and L7 (HTTP methods/paths, gRPC, Kafka topics, DNS FQDNs). |
| 4 | How do you configure a Zero-Trust Default Deny-All baseline for a namespace in Kubernetes? | Apply a NetworkPolicy with an empty podSelector: {} and policyTypes: ["Ingress", "Egress"] with no allow rules. |
| 5 | What critical traffic is accidentally broken if you apply a Default Deny-All Egress policy without an explicit DNS exception? | Pods immediately lose the ability to resolve DNS names because UDP/TCP Port 53 traffic to kube-dns / coredns in kube-system is blocked! |
| 6 | How does a CiliumNetworkPolicy allow a pod to egress only to https://api.stripe.com even when Stripe’s CDN rotates IP addresses every minute? |
Using an L7 DNS proxy rule (toFQDNs: [{matchName: "api.stripe.com"}]); Cilium intercepts the pod’s DNS lookup, learns the returned IPs dynamically, and updates the pod’s eBPF allow map in real time. |
| 7 | How does a CiliumNetworkPolicy allow frontend to call GET /api/v1/catalog on backend, while blocking DELETE /api/v1/catalog? |
By specifying an L7 HTTP rule (rules: http: [{method: "GET", path: "/api/v1/catalog"}]), which transparently routes matching flows through an embedded node-level Envoy listener. |
| 8 | What is the difference between a namespace-scoped CiliumNetworkPolicy and a cluster-wide CiliumClusterwideNetworkPolicy? |
CiliumClusterwideNetworkPolicy applies across all namespaces (and even host-network processes) to enforce non-bypassable enterprise security guardrails. |
| 9 | How do you test a new Zero-Trust CiliumNetworkPolicy in production without dropping real traffic before verifying rules? |
Enable Policy Audit Mode (policyAuditMode: true), which logs AUDIT verdicts in Hubble without dropping packets. |
| 10 | What command shows live packets being dropped by network policies across the cluster with the exact drop reason? | hubble observe --verdict DROPPED -f. |
Lesson 3: Mesh Resilience & Circuit Breaking
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | Why do naive unlimited retries during a downstream database slowdown trigger a catastrophic Retry Storm? | If 3 upstream tiers each configure 3 retries, a single failing request multiplies into 3 × 3 × 3 = 27× traffic amplification, DDoSing the recovering backend. |
| 2 | What two mathematical techniques must always be paired with retries to prevent synchronized retry waves? | Exponential Backoff (2^n × base_delay) combined with Random Jitter + a Retry Budget (capping retries at max 10–20% of total traffic). |
| 3 | Which HTTP status codes or methods should never be automatically retried without an explicit Idempotency Key? | Non-idempotent mutations like POST /charge-payment that timed out after already reaching the upstream server. |
| 4 | What is Circuit Breaking / Outlier Detection in Envoy and Cilium Service Mesh? | Automatically ejecting a degraded backend pod IP from the load-balancing pool for baseEjectionTime (e.g., 30s) when it returns consecutive 5xx errors or breaches max pending connections. |
| 5 | What are the 3 states of a Circuit Breaker? | Closed (normal traffic flows), Open (fast-fails immediately with 503 Overflow without calling backend), and Half-Open (allows a probe request to test recovery). |
| 6 | Why must upstream microservices always enforce shorter timeouts as requests travel deeper down a call chain (Timeout Budget Propagation)? | If API Gateway waits 2s, Service A waits 5s, and Service B waits 10s, Service B continues wasting CPU/DB connections for 8 seconds after the user already disconnected! |
| 7 | What Envoy response flag in access logs proves that a request was fast-failed by a circuit breaker (max_pending_requests overflow)? |
UO (Upstream Overflow). |
| 8 | What Envoy response flag indicates an upstream request was aborted because it breached the configured L7 per-try timeout? | UT (Upstream Request Timeout). |
| 9 | What is Mutual TLS (mTLS) in Cilium Service Mesh, and how does Cilium separate authentication from encryption? | Cilium uses SPIFFE/SPIRE x509 identities to mutually authenticate workload identities in control plane handshakes while encrypting data-plane packets at line rate via kernel WireGuard/IPsec. |
| 10 | How does Bulkhead Isolation (separate connection pools per downstream dependency) protect a microservice? | If Downstream Service B hangs, it only exhausts its dedicated 20-connection pool, leaving the remaining 80 connections free to serve healthy requests to Downstream Service A. |
Lesson 4: Cilium ClusterMesh & Gateway API
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | Why is the new Kubernetes Gateway API (gateway.networking.k8s.io) replacing the legacy Ingress resource? |
Legacy Ingress relied on vendor-specific annotation sprawl and lacked role separation, native traffic weighting, header routing, or gRPC/TCP support. |
| 2 | Name the 3 role-oriented CRDs of the Kubernetes Gateway API and who owns each. | GatewayClass (Infrastructure Provider), Gateway (Platform/SRE Team — listeners & TLS), and HTTPRoute / GRPCRoute (Application Developers — paths & weights). |
| 3 | How does HTTPRoute prevent developers in the marketing namespace from hijacking traffic meant for the payments namespace? |
The platform team’s Gateway controls allowed namespace bindings (allowedRoutes.namespaces), and cross-namespace backend references require an explicit ReferenceGrant in the target namespace. |
| 4 | What is Cilium ClusterMesh? | An eBPF multi-cluster networking technology that links multiple Kubernetes clusters across regions/clouds, enabling pod-to-pod routing and cross-cluster Service discovery without ingress gateways. |
| 5 | What prerequisite must be met by Pod CIDRs across all clusters participating in a Cilium ClusterMesh? | Each cluster must have a non-overlapping Pod CIDR and a unique cluster.id (1–255) so eBPF can route directly to remote pod IPs. |
| 6 | How do you turn a standard Kubernetes Service into a global multi-cluster service across Cilium ClusterMesh? |
Add the annotation service.cilium.io/global: "true". |
| 7 | How do you configure a global ClusterMesh service to prefer healthy pods in the local region and only fail over to the remote region if all local pods die? | Add the annotation service.cilium.io/affinity: "local" (or "remote" to prefer remote). |
| 8 | Why must Cilium MTU be reduced to 1420 when running ClusterMesh with WireGuard encryption over cloud VPC networks? |
Standard VPC MTU is 1500; WireGuard adds 60–80 bytes of encapsulation headers, which will drop large TLS handshake packets if inner MTU is not clamped. |
| 9 | How do you verify the health and connectivity of all remote clusters in a Cilium ClusterMesh? | Run cilium clustermesh status. |
| 10 | How does Gateway API perform a 90% / 10% canary split natively without third-party annotations? | By listing two backendRefs (app-v1 with weight: 90 and app-v2 with weight: 10) directly inside the HTTPRoute rule spec. |
🎭 Phase 7: Internal Developer Portals 46 Rapid Q&As
Lesson 1: Backstage Software Catalog
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | What core cognitive problem does an Internal Developer Portal (IDP) like Spotify Backstage solve? | It eliminates developer cognitive load and tribal knowledge silos by unifying service ownership, APIs, docs, CI/CD, and golden-path self-service into a single portal. |
| 2 | What YAML file placed at the root of a Git repository registers a microservice into the Backstage Software Catalog? | catalog-info.yaml. |
| 3 | Name the 3 Core Entity Kinds and 2 Grouping Kinds in the Backstage System Model. | Core: Component (service/website), API (OpenAPI/gRPC), Resource (DB/S3). Grouping: System (collection of components/resources) and Domain (business unit). |
| 4 | How does Backstage keep its PostgreSQL catalog database updated without requiring manual UI registrations? | Via Entity Providers (GitHub Org Discovery, GitHub Webhook processor, and Kubernetes/AWS Discovery processors) that continuously ingest and stitch entities into a graph. |
| 5 | What causes an entity in Backstage to show an Orphan Warning? | The catalog-info.yaml file was deleted or moved in the source Git repository, leaving a stale cached record in Backstage’s database. |
| 6 | What is the difference between spec.lifecycle: experimental and spec.lifecycle: production in catalog-info.yaml? |
It classifies the operational maturity of the service, driving automated production readiness scorecards and on-call paging rules. |
| 7 | How do two Backstage entities express a dependency relationship in the catalog graph? | Using spec.dependsOn: ["resource:default/orders-postgres-db"] or spec.providesApis / spec.consumesApis. |
| 8 | What causes the Backstage Catalog Processor to crash or reject an API entity during ingestion? | A malformed syntax error or unescaped $ref circular loop inside the referenced OpenAPI (swagger.yaml) or AsyncAPI specification file. |
| 9 | What is the canonical string format of a Backstage Entity Reference (EntityRef)? |
<kind>:<namespace>/<name> (e.g., component:default/payment-service or group:default/platform-team). |
| 10 | Which annotation in catalog-info.yaml links a Backstage Component to its live Kubernetes pods across clusters? |
backstage.io/kubernetes-id: payment-service (matching the backstage.io/kubernetes-id label on K8s Deployments). |
Lesson 2: Golden Path Templates
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | What is a “Golden Path” (Paved Road) in Platform Engineering? | An opinionated, automated self-service workflow that makes the secure, compliant, observable way of building and deploying software the easiest default path for developers. |
| 2 | What Backstage component executes multi-step infrastructure and code provisioning workflows (kind: Template)? |
The Backstage Scaffolder (@backstage/plugin-scaffolder-backend). |
| 3 | What are the 3 main sections of a Backstage Template YAML definition? |
spec.parameters (React JSON Schema UI form inputs), spec.steps (sequential actions: fetch, template, publish PR, register), and spec.output (links to created repo/ArgoCD). |
| 4 | Why should a Backstage Golden Path template create a Git Pull Request containing Crossplane/Terraform manifests instead of calling AWS APIs directly from Node.js? | Committing declarative manifests to Git preserves GitOps auditability, policy-as-code PR checks, and automatic drift reconciliation. |
| 5 | What templating engine does Backstage fetch:template use to inject user form parameters into skeleton source files? |
Nunjucks (${{ parameters.service_name }}). |
| 6 | What happens if Step 3 (publish:github) succeeds in creating a GitHub repo, but Step 4 (catalog:register) fails in a Scaffolder run? |
Without compensating cleanup or idempotency, retrying the template fails with Repository already exists; production custom actions must implement idempotent checks or Saga rollbacks. |
| 7 | How do you prevent a developer from typing an invalid Kubernetes service name (e.g., uppercase letters or spaces) into a Backstage Scaffolder form? | Add a regex pattern: '^[a-z0-9-]+$' validation rule inside the spec.parameters JSON Schema property. |
| 8 | What custom UI field extension in Backstage Scaffolder lets a user pick an existing Team or System from a live dropdown populated by the Software Catalog? | ui:field: OwnerPicker or ui:field: EntityPicker. |
| 9 | How do you test a Backstage Software Template dry-run before publishing it to 1,000 developers? | Use the built-in Backstage Template Dry-Run Editor (/create/edit) inside the portal. |
| 10 | How does a Golden Path template automatically provision an ArgoCD deployment without human intervention? | It uses publish:github:pull-request to commit an Application/Claim YAML into the central GitOps repository watched by an ArgoCD ApplicationSet. |
Lesson 3: TechDocs & Docs-as-Code
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | Why do traditional enterprise Confluence wikis fail for operational runbooks? | Documentation decoupled from source code rots immediately because developers rarely update a separate wiki when refactoring code or APIs. |
| 2 | How does Backstage TechDocs (Docs-as-Code) solve documentation drift? | Markdown docs (docs/index.md + mkdocs.yml) live inside the exact same Git repository as the microservice code and are reviewed in the same Pull Request. |
| 3 | What annotation in catalog-info.yaml tells Backstage where to find an entity’s TechDocs source files? |
backstage.io/techdocs-ref: dir:.. |
| 4 | Why must you never configure techdocs.builder: 'local' in a production Backstage deployment? |
'local' forces the Backstage Node.js backend pod to spawn heavy Python mkdocs subprocesses on the fly when users view pages, causing CPU starvation and OOMKills. |
| 5 | What is the recommended production architecture for Backstage TechDocs (techdocs.builder: 'external')? |
CI/CD Pre-Building—GitHub Actions runs techdocs-cli generate and techdocs-cli publish to push compiled static HTML/CSS to an Amazon S3 / GCS bucket, which Backstage serves instantly. |
| 6 | What MkDocs extension renders architecture-as-code diagrams (flowchart, sequenceDiagram) natively inside TechDocs pages? |
Mermaid.js (pymdownx.superfences with custom mermaid fence). |
| 7 | How do you enforce that a developer cannot change an API endpoint without updating the service’s TechDocs or Runbook? | Configure a GitHub Actions PR gate or CODEOWNERS check that flags changes to /api/* or /terraform/* lacking corresponding updates in /docs/*. |
| 8 | Where should every microservice’s operational runbook link so on-call SREs can find it at 3:00 AM? | Directly inside the Prometheus Alert annotations.runbook_url pointing to the service’s Backstage TechDocs URL. |
| 9 | How does the Backstage backend authenticate to the central TechDocs S3 bucket on AWS EKS? | Via an EKS IRSA Role attached to the Backstage Kubernetes ServiceAccount granting s3:GetObject and s3:ListBucket. |
| 10 | What search engine plugin indexes all TechDocs pages and Catalog entities for instant global search inside Backstage? | @backstage/plugin-search backed by OpenSearch / Elasticsearch or Postgres pg_trgm. |
Lesson 4: Backstage RBAC & Scorecards
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | How does Backstage authenticate users and automatically map them to their engineering teams (kind: Group)? |
Via OIDC / SAML SSO (Okta, Azure Entra ID, Google Workspace) paired with an automated Org Discovery Processor that syncs users and group memberships. |
| 2 | In the Backstage Permission Framework, what is the difference between a Basic Permission and a Resource Permission? | A Basic Permission controls global actions (catalog.entity.create), while a Resource Permission evaluates fine-grained conditions against a specific entity (e.g., isEntityOwner). |
| 3 | How do you restrict a destructive Scaffolder template (like "Decommission Production Database") so only Staff SREs can see or execute it? |
Tag the template with tags: ["restricted-sre"] and apply a Backstage RBAC Conditional Policy denying scaffolder.template.parameter.read and task.create unless the user is in group:default/sre-leads. |
| 4 | What is a Production Readiness Scorecard in an Internal Developer Portal? | An automated grading engine (Bronze / Silver / Gold) that continuously audits every microservice against security, reliability, observability, and documentation standards. |
| 5 | Name 5 automated checks typically evaluated by a Gold-Tier Production Readiness Scorecard. | (1) Valid PodDisruptionBudget, (2) SLO + Runbook URL defined, (3) Zero CRITICAL Trivy CVEs, (4) Cosign image signature verified, and (5) PagerDuty rotation attached. |
| 6 | How do you turn passive Backstage Scorecards into active Deployment Gates during major sales events (like IPL or Diwali)? | Query the Scorecard API from GitHub Actions or ArgoCD PreSync hooks and block production promotion for any service scoring below Silver/Gold. |
| 7 | How does Backstage prevent a developer in Team A from viewing sensitive runtime secrets or triggering deployments on Team B’s Kubernetes pods? | By applying the isEntityOwner RBAC condition on the Kubernetes and ArgoCD plugins, scoping actions strictly to spec.owner == user.claims.groups. |
| 8 | What audit trail must be enabled on the Backstage backend to satisfy SOC2 and ISO27001 compliance? | Structured Audit Logging capturing every Scaffolder task execution, RBAC permission decision (ALLOW/DENY), user identity, and timestamp sent to SIEM. |
Lesson 5: Chaos Engineering (Chaos Mesh)
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | What is the core discipline of Chaos Engineering? | Designing controlled, hypothesis-driven fault injection experiments to uncover systemic weaknesses before they cause unpredicted production outages. |
| 2 | What must always be defined before injecting any fault in a Chaos Engineering experiment? | The Steady-State Hypothesis (measurable business/technical SLIs such as "Checkout p99 < 250ms and HTTP 5xx < 0.1%"). |
| 3 | Name the 4 most common fault CRDs in Chaos Mesh. | PodChaos (pod-kill/container-kill), NetworkChaos (latency/loss/partition), StressChaos (CPU/memory burn via stress-ng), and IOChaos / HTTPChaos. |
| 4 | How does the Chaos Mesh chaos-daemon inject a 150ms network delay into a target pod without modifying the pod’s container image? |
The privileged chaos-daemon on the host uses nsenter to enter the target pod’s network namespace (netns) and configures a Linux kernel Traffic Control (tc) netem qdisc on its eth0 interface. |
| 5 | What happens if the chaos-daemon pod crashes or is OOMKilled while a NetworkChaos experiment is actively injecting 50% packet loss? |
The Linux kernel tc qdisc netem rule remains orphaned inside the target pod’s network namespace (Zombie Chaos Rule), causing permanent packet loss even after the NetworkChaos CRD is deleted! |
| 6 | How do you manually clear a Zombie NetworkChaos rule from an affected pod without restarting the pod? |
Enter the pod’s network namespace from the worker node and delete the root queuing discipline: nsenter -t <pod-pid> -n tc qdisc del dev eth0 root. |
| 7 | How do you prevent a Chaos Mesh experiment from accidentally targeting kube-system or critical control plane namespaces? |
Configure Chaos Mesh controller flags FILTER_NAMESPACE / annotation enforcement (chaos-mesh.org/inject: "true" required on target namespaces) and block system namespaces via Kyverno. |
| 8 | How do you automate an immediate Emergency Stop (Abort) of a Chaos Mesh Workflow if real customer error budgets start burning? |
Pair the Chaos Workflow with a Prometheus Alertmanager webhook (or Argo Rollouts Analysis) that automatically deletes the active Chaos CRD as soon as any production SLO alert fires. |
🧠 Phase 8: LLMOps Infrastructure 26 Rapid Q&As
Lesson 1: Vector DBs & AI Gateways
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | What is RAG (Retrieval-Augmented Generation), and why is it preferred over fine-tuning for dynamic enterprise data? | RAG retrieves relevant document chunks from a Vector DB at query time and injects them into the LLM prompt context, preventing hallucinations and enforcing real-time document ACLs without expensive weight retraining. |
| 2 | Why does a Milvus Vector Database QueryNode get OOMKilled when loading 50 million 1536-dimension float32 vectors (50M × 1536 × 4B ≈ 307GB)? |
In-memory HNSW (Hierarchical Navigable Small World) graph indices require 1.5× to 2× the raw vector size in RAM to store multi-layer graph adjacency pointers alongside raw vectors. |
| 3 | How do you reduce Vector Database memory consumption by 75% to 90% when scaling to billions of vectors? | Use Vector Quantization (SQ8 Scalar Quantization to 8-bit integers or PQ Product Quantization) or DiskANN (NVMe SSD-backed indexing). |
| 4 | Why should enterprise microservices never call OpenAI, Anthropic, or AWS Bedrock APIs directly from application code? | Direct calls leak API keys across dozens of repos, lack unified token cost tracking per team, and cannot automatically fail over when OpenAI returns HTTP 429 or 503. |
| 5 | What are the 4 core functions of an Enterprise AI Gateway (LiteLLM / Envoy AI Gateway)? | (1) Multi-provider automatic failover (OpenAI → Azure OpenAI → Bedrock), (2) Token-based rate limiting (TPM/RPM), (3) Per-tenant FinOps cost chargeback, and (4) Semantic Caching. |
| 6 | How does Semantic Caching (using Redis Vector Search) differ from traditional exact-match key-value caching? | Instead of requiring identical prompt strings, it embeds the user’s query into a vector and returns a cached response if Cosine Similarity exceeds a threshold (e.g., > 0.95), saving 30–50% of LLM spend. |
| 7 | Why do traditional HTTP load balancers and rate limiters (requests/sec) fail to protect LLM gateways? | One LLM request with a 100,000-token context window consumes 1,000× more GPU compute and quota than a 100-token request; LLM gateways must rate-limit by Tokens Per Minute (TPM). |
| 8 | What causes a LiteLLM Retry Storm during an upstream OpenAI outage, and how do you prevent it? | Setting high retry counts across multiple fallback regions; fix by setting num_retries: 2, configuring a cooldown_time: 60s circuit breaker on degraded deployments, and enforcing request timeouts. |
Lesson 2: MLflow & Prompt Evaluation
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | What are the 4 core components of an enterprise MLflow platform? | MLflow Tracking (experiments/params/metrics), MLflow Model Registry (versioning & aliases like @champion), MLflow Projects, and Artifact Store (S3) + Backend Store (PostgreSQL). |
| 2 | Why do traditional unit tests (assert output == "expected") fail when testing LLM applications in CI/CD? |
LLMs are probabilistic and non-deterministic—two semantically identical correct answers rarely have identical word-for-word string representations. |
| 3 | What are the 4 core metrics of the RAGAS (Retrieval-Augmented Generation Assessment) evaluation framework? | Context Precision (signal-to-noise of retrieved chunks), Context Recall (did retrieval find all facts?), Faithfulness (did LLM hallucinate outside context?), and Answer Relevance. |
| 4 | Which RAGAS metric specifically detects LLM Hallucinations (claims generated by the model that cannot be inferred from the retrieved context)? | Faithfulness (scored from 0.0 to 1.0; production CI gates typically require Faithfulness >= 0.90). |
| 5 | Why did an LLM-as-a-Judge CI evaluation pipeline pass a PR at 10:00 AM and fail the exact same code at 10:15 AM? | The judge LLM was invoked with temperature > 0 or an unpinned floating model name; LLM-as-a-Judge must always set temperature: 0.0, seed: 42, and pin exact model snapshots. |
| 6 | How do you version prompts in a GitOps workflow instead of hardcoding prompt strings inside Python application code? | Store prompts as versioned YAML templates in Git (or MLflow Prompt Registry), evaluate them against a Golden Dataset in CI using DeepEval/RAGAS, and promote via @production aliases. |
| 7 | What is Data Drift / Embedding Drift in a production RAG system? | When live user queries diverge semantically from the historical document corpus or when changing embedding models (text-embedding-ada-002 → text-embedding-3-large) invalidates existing vector distances. |
| 8 | Why can’t you query vectors generated by text-embedding-ada-002 using a query embedded with text-embedding-3-large? |
Different embedding models map semantics into completely incompatible vector spaces; switching embedding models requires re-indexing the entire document corpus into a new collection (Blue/Green Vector Re-indexing). |
Lesson 3 & Labs: GPU Slicing, vLLM & Guardrails
| # | One-Line Question | One-Line Answer |
|---|---|---|
| 1 | What is the structural difference between GPU Time-Slicing and NVIDIA MIG (Multi-Instance GPU) on an A100/H100 GPU? | Time-Slicing shares one GPU via context-switching with zero memory or fault isolation (one OOM crashes all pods); MIG physically partitions the silicon into up to 7 hardware-isolated GPU instances with dedicated VRAM and compute cores. |
| 2 | How much GPU VRAM is required just to load the model weights of a 70B parameter LLM in FP16 (16-bit = 2 bytes per parameter)? |
≈ 140 GB of VRAM (70B × 2 bytes), plus an additional 20–40 GB for the KV (Key-Value) Cache and activations—requiring at least 2× 80GB H100 GPUs or AWQ/GPTQ 4-bit quantization. |
| 3 | What breakthrough memory technology in vLLM eliminates 60–80% of GPU VRAM waste during concurrent LLM serving? | PagedAttention—it splits the attention KV Cache into non-contiguous virtual memory blocks (like OS virtual memory pages), eliminating contiguous memory fragmentation and enabling Continuous Batching. |
| 4 | What causes a vLLM CUDA Out-Of-Memory (OOM) crash during traffic bursts even though the model weights loaded cleanly at startup? | Setting --max-model-len too high (e.g., 128k context) or --gpu-memory-utilization to 0.98, leaving zero VRAM headroom for PyTorch activation spikes during large prefill batches. |
| 5 | How do you shard a 70B model across 4 GPUs on the same physical node in vLLM? |
Set Tensor Parallelism --tensor-parallel-size 4 (TP=4), which splits matrix multiplications across all 4 GPUs simultaneously over high-speed NVLink. |
| 6 | Why does autoscaling vLLM pods based on Kubernetes CPU or GPU Memory % fail completely? | vLLM pre-allocates 90% of GPU VRAM at startup (so memory is permanently constant at 90%), and CPU stays low; you must autoscale vLLM with KEDA on vllm:num_requests_waiting (queue depth) and vllm:gpu_cache_usage_perc. |
| 7 | Why do LLM GPU pods take 10+ minutes to start during scale-out, and how do you cut cold-start time to under 30 seconds? | Pulling 140GB of Safetensors weights over the internet or slow EBS throttles network I/O; store model weights on Amazon FSx for Lustre or pre-cached NVMe instance stores with --load-format safetensors. |
| 8 | What runtime security layer sits in front of an LLM to block Prompt Injection (Jailbreaks) and redact PII (Aadhaar, PAN, Credit Cards) before prompts leave the cluster? | LLM Guardrails (NVIDIA NeMo Guardrails, Llama Guard 3, and Microsoft Presidio PII redaction filters). |
| 9 | What Prometheus exporter deployed by the NVIDIA GPU Operator exposes hardware metrics like SM utilization (DCGM_FI_DEV_DEC_UTIL), VRAM temperature, and XID hardware errors? |
NVIDIA DCGM Exporter (dcgm-exporter). |
| 10 | What two core latency SLIs define user experience in streaming LLM inference? | TTFT (Time To First Token < 500ms) and ITL / TPOT (Inter-Token Latency / Time Per Output Token < 30ms/token). |