🕸️ Service Mesh, eBPF & Chaos Engineering Triage Guide

This guide details the most common, tricky, and critical production failure modes encountered across eBPF Service Meshes (Cilium), Developer Portals (Spotify Backstage), and Chaos Engineering Injection (Chaos Mesh).

🎯 High-Impact Real-World Scenarios

Scenario 1: Cilium WireGuard / ClusterMesh MTU Silent Blackhole

The Symptom

Pods in Cluster A can successfully ping pods in Cluster B across Cilium ClusterMesh. Short HTTP requests (curl http://remote-pod:8080/healthz) return 200 OK instantly. However, HTTPS requests or requests with large JSON payloads (> 1,500 bytes) hang indefinitely during the TLS handshake:

* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* Server certificate: CN=payment-api.internal
* Client Hello sent...
* (Hangs here forever until timeout)

Under-the-Hood Root Cause

The cloud network fabric (AWS VPC / Azure VNet) enforces an MTU of 1,500 bytes. When Cilium enables WireGuard or Geneve/VXLAN overlay encryption, it appends tunnel headers:

• WireGuard encryption adds 60 bytes of overhead.
• Total packet size with a 1,500-byte IP payload becomes 1,560 bytes. Because the Don’t Fragment (DF) bit is set on modern TCP packets, the AWS VPC router silently drops packets exceeding 1,500 bytes without returning an ICMP Fragmentation Needed response (creating an MTU Blackhole).

sequenceDiagram
    autonumber
    participant Pod as Microservice (MTU: 1500)
    participant Cilium as Cilium eBPF + WireGuard
    participant VPC as Cloud VPC Fabric (Max MTU: 1500)
    participant Remote as Destination Pod

    Pod->>Cilium: TCP Packet with Large TLS Cert (1,450 bytes data + 40 bytes IP/TCP = 1,490B)
    Cilium->>Cilium: Encrypts packet via WireGuard (+60 bytes overhead!)
    Note over Cilium: Total packet size = 1,550 bytes!
    Cilium->>VPC: Transmits 1,550-byte encrypted packet
    Note over VPC: ❌ DROPPED! Packet exceeds physical MTU (1,500)!<br/>DF bit set -> No ICMP notification sent! 🚨
    VPC--xRemote: Packet never arrives! Client hangs forever!

CLI Triage Commands

# 1. Test ping with Don't Fragment (DF) flag set at various payload sizes
# -M do: set DF bit, -s 1420: packet size
ping -M do -s 1420 <remote-pod-ip>

# 2. Check Cilium detected MTU
kubectl exec -n kube-system ds/cilium -c cilium-agent -- cilium status | grep MTU

Remediation & Architectural Standard

• Clamp Cilium MTU in Helm values:

# Cilium Helm values.yaml
mtu: 1420 # 1500 physical MTU - 60 (WireGuard) - 20 (safety margin)
encryption:
  enabled: true
  type: wireguard

• Cilium automatically configures TCP Maximum Segment Size (MSS) clamping to force clients to send segments that fit safely within the tunnel envelope.

Scenario 2: Chaos Mesh Zombie tc Netem Rules Persisting After Deletion

The Symptom

A platform engineer runs an automated chaos engineering experiment (NetworkChaos) to test 20% packet loss on order-service. The test finishes, and the engineer runs kubectl delete networkchaos order-loss-test. However, the application continues to experience 20% packet loss, and error rates remain elevated for hours!

Under-the-Hood Root Cause

Chaos Mesh injects network latency and packet loss by manipulating Linux kernel Traffic Control (tc) queuing disciplines (netem) inside the container’s network namespace via the node-level daemon (chaos-daemon). If the chaos-daemon pod was evicted, restarted by OOM, or suffered a network partition while the test was being terminated, it never received the command to flush the tc netem rules. The kernel continues dropping 20% of packets indefinitely!

CLI Triage Commands

# 1. Find the target container process ID on the host node
CONTAINER_PID=$(crictl inspect --output go-template --template '{{.info.pid}}' <container-id>)

# 2. Inspect active traffic control queuing disciplines inside the container namespace
sudo nsenter -t $CONTAINER_PID -n tc qdisc show dev eth0
# Output will reveal the lingering zombie netem rule:
# qdisc netem 1: root refcnt 2 limit 1000 loss 20%

Emergency Remediation Runbook

# 1. Manually flush the netem queuing discipline from the container interface
sudo nsenter -t $CONTAINER_PID -n tc qdisc del dev eth0 root

# 2. Alternatively, perform a rolling restart on the affected application pods
kubectl rollout restart deployment/order-service

# 3. Restart the chaos-daemon to re-sync active experiment states
kubectl rollout restart ds/chaos-daemon -n chaos-mesh

Scenario 3: Backstage Catalog Processor Crash on Malformed OpenAPI Spec

The Symptom

The Spotify Backstage software catalog stops updating. Entities pushed by developers do not appear in the portal, and the catalog backend logs report:

CatalogProcessingError: Processor EntityFilterProcessor failed to process 
component:default/customer-api: YAMLSyntaxError: Map keys must be unique at line 482

The entire catalog worker ingestion loop crashes, stalling synchronization for all 2,000 components in the enterprise!

Under-the-Hood Root Cause

A developer committed an invalid or duplicated YAML key in their apis/openapi.yaml referenced by a catalog-info.yaml entity. Because Backstage’s ingestion processor parsed the file synchronously within the main database ingestion transaction, the unhandled parser exception aborted the batch transaction, starving subsequent entities.

CLI Triage Commands

# 1. Inspect Backstage catalog backend logs for failed entities
kubectl logs -n backstage deploy/backstage -c backstage-backend --tail=100 | grep -E "CatalogProcessingError|YAMLSyntaxError"

# 2. Validate catalog-info.yaml locally using Backstage CLI before pushing
npx @backstage/cli repo lint

Remediation & Architectural Standard

• Client-Side Pre-Commit Gate: Mandate catalog-info.yaml and OpenAPI validation in PR pipelines using the @backstage/cli repo lint action.
• Resilient Processor Configuration: In app-config.yaml, configure catalog worker batch sizes and enable isolated error boundaries so that malformed entities are quarantined without halting the global refresh loop.