⚡
Zero to Platform Architect2026
War-Room Triage & 43 Traps
War-Room Triage & 43 Traps⏱️ 9 min read⚡ Staff SRE Standard
🎯 Operations Test Bank →
Quick Jump:🧭 CLI Playbook📝 Q&A Drill

🚀 GitOps, Control Planes & LLMOps Triage Guide

This guide covers complex, tricky production failure modes across modern GitOps engines (ArgoCD & Argo Rollouts), Kubernetes Control Planes (Crossplane), and AI/LLMOps infrastructure (Distributed Milvus, LiteLLM Gateways, and vLLM GPU inference).

🧭 Beginner-Friendly First-5-Minutes GitOps & AI Infrastructure Playbook (Step-by-Step SOP)

Mental Model (The Thermostat Loop): GitOps controllers (ArgoCD/Crossplane) work like a smart thermostat comparing Desired State (Git) against Actual State (Cluster/Cloud). When they get stuck, either two controllers are fighting over the same field, or a finalizer is refusing to let go of a deleted cloud resource.

Step 1: Identify ArgoCD Sync/Diff Loop & Pause Auto-Sync (Minute 0–1)

argocd app get <app-name> --hard-refresh
argocd app diff <app-name>

Step 2: Check Crossplane Composite & Managed Resource Conditions (Minute 1–2)

kubectl get composite,managed
kubectl describe managed <resource-name> | tail -n 25
  • What to look for: Synced: False (IAM permission or cloud API validation error) vs Ready: False (cloud resource still provisioning).

Step 3: Check GPU & Vector DB Memory Pressure for LLMOps Workloads (Minute 2–4)

kubectl top pods -n ai-platform
kubectl exec -n gpu-operator <pod> -- nvidia-smi

Step 4: Inspect AI Gateway Rate-Limit & Upstream Failover Metrics (Minute 4–5)

kubectl logs -n ai-platform -l app=litellm-gateway --tail=50 | grep -E "429|503|RateLimit"

🎯 High-Impact Real-World Scenarios

Scenario 1: ArgoCD Infinite Reconciliation Sync Loop

The Symptom

An ArgoCD Application is stuck in an infinite reconciliation cycle. Every few seconds, ArgoCD attempts to apply a manifest, marks the application OutOfSync, triggers another sync, and floods the Kubernetes API server with write requests:

ComparisonError: field 'spec.containers[0].resources' has diff: 
live state has limits, desired state has none.

Under-the-Hood Root Cause

A mutating admission webhook or a Kubernetes controller (such as Vertical Pod Autoscaler or an AWS/Istio sidecar injector) modified the live resource in the cluster after ArgoCD applied it. Because the live resource now differs from the Git source of truth, ArgoCD treats it as out-of-band drift. If selfHeal: true is configured in the sync policy, ArgoCD continuously overwrites the mutated field, which the webhook immediately re-injects, creating an infinite sync storm.

sequenceDiagram
    autonumber
    participant Git as Git Repo (ArgoCD)
    participant Argo as ArgoCD Controller
    participant API as K8s API Server
    participant Webhook as Mutating Webhook / VPA

    Argo->>API: 1. Applies Deployment (No resource limits specified)
    API->>Webhook: 2. Webhook intercepts creation
    Webhook-->>API: 3. Injects default limits (cpu: 500m, mem: 1Gi)
    API-->>Argo: 4. Reports live state updated
    Note over Argo: 5. ArgoCD detects drift! Live state != Git state!
    Argo->>API: 6. Re-applies Deployment without limits (Self-Healing)
    API->>Webhook: 7. Webhook re-injects limits again!
    Note over Argo,Webhook: 🚨 INFINITE RECONCILIATION STORM! API Server CPU spikes to 100%!

CLI Triage Commands

# 1. Compare live state against desired Git state to find the fighting fields
argocd app diff <app-name>

# 2. Inspect the application controller sync logs
kubectl logs -n argocd -l app.kubernetes.io/name=argocd-application-controller --tail=100 | grep -E "SyncFailed|OutOfSync"

Remediation & Architectural Fix

  • The Solution: Use ignoreDifferences in the ArgoCD Application manifest to instruct the diff engine to ignore fields managed dynamically by cluster operators:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: order-service
spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/template/spec/containers/0/resources
        - /spec/template/metadata/annotations/sidecar.istio.io~1status

Scenario 2: Canary Abort with Breaking Database Schema (The Dropped Column Trap)

The Symptom

Argo Rollouts begins a canary deployment for v2.0 of an e-commerce checkout service. During the 10% traffic phase, an AnalysisTemplate detects an elevated error rate and executes an automated abort. The rollout instantly shifts 100% of traffic back to the stable v1.0 pods. Immediately, all stable v1.0 pods begin crashing with HTTP 500 errors, resulting in a complete site outage!

Under-the-Hood Root Cause

Before the canary deployment commenced, a database migration Job executed ALTER TABLE orders DROP COLUMN legacy_discount_code;. The new v2.0 code was designed to operate without this column. However, when the canary failed and rolled back to v1.0, the stable pods were running code that actively executed SELECT legacy_discount_code FROM orders. Because the column was permanently deleted from PostgreSQL, all queries failed with column does not exist.

Architectural Remediation: The Two-Phase Expand-and-Contract Migration

Platform architects must enforce the Expand and Contract pattern for all automated GitOps pipelines:

  1. Phase 1 (Expand): Add the new column or table. Write to both old and new columns. Deprecate the old column in code, but never drop it.
  2. Phase 2 (Canary Rollout): Deploy v2.0. If aborted, v1.0 remains fully compatible because the legacy column still exists.
  3. Phase 3 (Contract): Only in a subsequent release after v2.0 is 100% stable, execute a cleanup migration to drop the obsolete column.

Scenario 3: Crossplane Composite Resource (XR) Finalizer Deadlock

The Symptom

A developer destroys a staging environment by running kubectl delete namespace staging or deleting a Crossplane Claim. The namespace remains stuck in the Terminating state for hours. Behind the scenes, the AWS resources (RDS databases, S3 buckets) continue running and accumulating cloud charges.

Under-the-Hood Root Cause

Crossplane resources utilize Kubernetes Finalizers (finalizer.managedresource.crossplane.io) to guarantee that cloud provider infrastructure is safely deleted via the AWS/Azure API before the Kubernetes CRD record is removed from etcd.

However, if:

  • The AWS IAM role credentials were deleted before the Crossplane claim, OR
  • An AWS resource had deletion protection enabled (e.g. deletion_protection = true on AWS RDS), the Crossplane provider fails to delete the cloud asset. The controller retries indefinitely, refusing to remove the finalizer.

CLI Triage Commands

# 1. Identify which Crossplane managed resource is blocking deletion
kubectl get managed -A | grep -v "True.*True"

# 2. Inspect the resource conditions and error messages
kubectl describe <resource-kind> <resource-name> | grep -A 8 Conditions:

# 3. Check finalizers on the blocking resource
kubectl get <resource-kind> <resource-name> -o jsonpath='{.metadata.finalizers}'

Remediation & Architectural Fix

  • Break-Glass Emergency Command: Disable deletion protection on AWS directly, or remove the finalizer if the cloud resource was already purged manually:
kubectl patch <resource-kind> <resource-name> \
  -p '{"metadata":{"finalizers":null}}' --type=merge
  • Architectural Hardening: In Crossplane Compositions, configure deletionPolicy: Delete for non-production environments and ensure control plane IAM roles have explicit AWS permissions to modify deletion protection before terminating resources.

Scenario 4: Milvus Distributed QueryNode OOMKilled During HNSW Index Load

The Symptom

A distributed Milvus cluster on Kubernetes crashes during high-throughput vector queries. Milvus QueryNode pods are repeatedly killed with Exit Code 137 (OOMKilled). Query latency spikes from 12ms to complete timeouts.

Under-the-Hood Root Cause

The platform team calculated vector storage using raw float vector sizes without factoring in the graph structure overhead of the HNSW (Hierarchical Navigable Small World) index.

For N = 10,000,000 vectors with dimension D = 1536: 📐 Raw Vectors = 10,000,000 × 1,536 × 4 bytes ≈ 61.44 GB However, an HNSW index with construction parameters M = 16 and efConstruction = 200 creates millions of graph neighbor pointers that must reside entirely in physical RAM. The actual memory required is: 📐 M_total = N × D × 4 × 1.5 ≈ 92.16 GB Because the QueryNode memory limits were capped at 64 GB, the Linux kernel OOM killer immediately executed SIGKILL as soon as the index was loaded into memory.

CLI Triage Commands

# 1. Verify OOM kill events on Milvus query nodes
kubectl get pods -l app.kubernetes.io/component=querynode \
  -o jsonpath='{.items[*].status.containerStatuses[*].lastState.terminated.reason}'

# 2. Check Milvus index loading metrics in Prometheus
# sum(milvus_querynode_memory_used_bytes) / sum(milvus_querynode_memory_total_bytes)

Remediation & Architectural Fix

  • Remediation: Provision memory matching the 1.5× formula across distributed QueryNodes.
  • Tuning: For massive vector collections where RAM is constrained, switch from in-memory HNSW to DiskANN (disk-backed vector indexing utilizing fast NVMe EBS io2 or instance-store SSDs), reducing memory footprint by 80%.

Scenario 5: LiteLLM Multi-Model Upstream HTTP 429 Retry Storm

The Symptom

During peak enterprise traffic, OpenAI API begins returning HTTP 429 Too Many Requests (Rate Limit Exceeded). LiteLLM AI Gateway starts returning HTTP 504 Gateway Timeout to all internal consumer microservices, and client applications freeze.

Under-the-Hood Root Cause

When OpenAI returned 429, LiteLLM’s retry policy was configured with aggressive, immediate retries (num_retries: 5 with no exponential backoff). With 50 microservices issuing requests, the gateway multiplied the request volume by 5×, exacerbating the rate-limit penalty. Furthermore, fallback routing to alternate providers (Azure OpenAI or Anthropic) was missing, causing a total blackout.

Architectural Remediation & LiteLLM Config

Configure LiteLLM Proxy with tiered fallbacks, Redis semantic caching, and exponential backoff with jitter:

model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY
      rpm: 10000
      tpm: 2000000
  - model_name: gpt-4o
    litellm_params:
      model: azure/gpt-4o-eastus
      api_key: os.environ/AZURE_API_KEY
      api_base: https://corp-eastus.openai.azure.com/

router_settings:
  routing_strategy: "latency-based-routing"
  num_retries: 3
  retry_after: 2
  allowed_fails: 2
  cooldown_time: 30 # Temporarily disable failing provider for 30s
  fallbacks:
    - "gpt-4o": ["azure/gpt-4o-eastus", "anthropic/claude-3-5-sonnet-20241022"]
  enable_redis_cache: true
  redis_host: "redis.cache.svc.cluster.local"
  redis_port: 6379

Scenario 6: vLLM PagedAttention GPU Memory Fragmentation (CUDA OOM)

The Symptom

A vLLM inference pod serving Llama-3-70B crashes with:

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 4.20 GiB. 
GPU 0 has 22.00 GiB total capacity; 1.10 GiB free.

Under-the-Hood Root Cause

By default, vLLM pre-allocates GPU VRAM for both the model weights and the KV-Cache (Key-Value cache) for context tokens. If gpu_memory_utilization is set too high (e.g. 0.98), or if max_model_len is configured to 8,192 tokens without accounting for activation memory during high concurrent batching, the PyTorch memory allocator runs out of unfragmented CUDA memory blocks during dynamic tensor generation.

CLI Triage Commands

# 1. Inspect GPU memory utilization via NVIDIA DCGM or nvidia-smi inside pod
kubectl exec -it <vllm-pod> -- nvidia-smi

# 2. Check vLLM server logs for KV cache allocation ratios
kubectl logs <vllm-pod> | grep -E "KV cache|memory utilization|CUDA"

Remediation & Architectural Fix

  • The Solution: Tune gpu_memory_utilization to 0.90 (leaving a 10% safety buffer for CUDA activations) and tune the swap space:
python3 -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Meta-Llama-3-70B-Instruct \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 4096 \
  --swap-space 16

This guarantees deterministic CUDA allocation and routes overflow tokens to CPU RAM swap during temporary traffic spikes instead of crashing the GPU engine.