Enterprise Lesson 4: CloudFront Edge Delivery & Enterprise Log Management Pipelines

🧠 The Concept (Explain Like I’m 5)

Imagine a global postal service delivering packages and letters:

• CloudFront Edge Delivery: Instead of forcing a customer in Tokyo to order a book shipped across the Pacific Ocean from a warehouse in Virginia (Origin ALB/S3: 250ms roundtrip):
  • You stock copies of the book in local neighborhood postal kiosks across Tokyo (Edge PoP: 5ms).
  • If the local kiosk runs out, it asks the Tokyo Central Post Office (Regional Edge Cache & Origin Shield) before bothering the Virginia factory.
  • Security guards at the front gate check IDs and throw out junk mail (Edge WAF & Shield Advanced).
• Enterprise Log Management: Every machine in the company produces thousands of receipts every minute (Application Logs):
  • The Local Shredder / Marker (Vector DaemonSet on Node): Before receipts leave the room, a trusted clerk blacks out credit card numbers and passwords with a marker (Edge PII Redaction). It glues multi-page torn receipts back together (Multiline Stack Trace Parsing).
  • The Storage Tiers: Active receipts stay in the front office filing cabinet for 7 days (Hot NVMe). Older receipts move to a secure underground concrete bunker locked in titanium safes that cannot be destroyed or edited for 7 years (S3 Glacier WORM Object Lock for SEC/FINRA Compliance).

🌐 CloudFront Multi-Tier Caching Architecture

Modern enterprise edge architectures reduce load on origins by up to 98% through a four-tier caching hierarchy:

flowchart TD
    User["🌍 Global End Users"] --> PoP["1. CloudFront Edge PoP (600+ Locations)<br/>• Sub-10ms TLS Termination<br/>• CloudFront Functions (URL Rewriting)<br/>• AWS WAF Rate Limiting"]

    PoP -->|"Cache Miss"| REC["2. Regional Edge Cache - REC (13 Locations)<br/>• Massive multi-gigabyte cache pools<br/>• Absorbs traffic across geographical regions"]

    REC -->|"Cache Miss"| OriginShield["3. Origin Shield (Centralized Cache)<br/>• Single consolidated caching layer in origin AWS region<br/>• Collapses thousands of concurrent cache misses into 1 request (Request Collapsing)"]

    OriginShield -->|"Consolidated Request"| Origin["4. Origin Workloads<br/>• Application Load Balancer (EKS)<br/>• Amazon S3 (Origin Access Control - OAC)"]

⚖️ CloudFront Functions vs. Lambda@Edge Decision Matrix

Enterprise edge architectures utilize two tiers of serverless compute:

🔒 Origin Access Control (OAC) vs. Legacy OAI

Legacy CloudFront Origin Access Identity (OAI) is deprecated. Origin Access Control (OAC) provides enterprise-grade S3 origin protection:

sequenceDiagram
    autonumber
    participant Client as 🌍 Web Browser
    participant CF as ⚡ CloudFront Distribution (us-east-1)
    participant S3 as 🪣 Private Amazon S3 Bucket (SSE-KMS Encrypted)

    Client->>CF: GET /assets/app.c12d4a.js
    Note over CF: Viewer validates TLS 1.3 & AWS WAF
    CF->>CF: Sign request with AWS SigV4 using Service Principal: cloudfront.amazonaws.com
    CF->>S3: GET /assets/app.c12d4a.js with SigV4 Signature
    Note over S3: Bucket Policy evaluates Condition: StringEquals AWS:SourceArn
    S3->>S3: KMS decrypts object using Customer Managed Key (CMK)
    S3-->>CF: HTTP 200 OK (Encrypted Stream)
    CF-->>Client: HTTP 200 OK (Cached for 1 Year)

Enterprise S3 Bucket Policy Enforcing OAC

{
  "Version": "2012-10-17",
  "Statement": {
    "Sid": "AllowCloudFrontServicePrincipalReadOnly",
    "Effect": "Allow",
    "Principal": {
      "Service": "cloudfront.amazonaws.com"
    },
    "Action": "s3:GetObject",
    "Resource": "arn:aws:s3:::enterprise-app-assets/*",
    "Condition": {
      "StringEquals": {
        "AWS:SourceArn": "arn:aws:cloudfront::123456789012:distribution/EDFDVBD6EXAMPLE"
      }
    }
  }
}

📊 Enterprise Log Ingestion: Vector vs. Fluent Bit vs. Fluentd

Logging DaemonSets run on every node in a 500-node cluster. An inefficient logging agent wastes gigabytes of RAM and thousands of vCPU cores across the fleet:

🗺️ Visual Architecture: High-Performance Log Pipeline with Edge PII Redaction

flowchart TD
    subgraph EKSNode ["EKS Worker Node"]
        AppPod["App Pod (Emits JSON & Multi-line Java Stack Traces)"]
        LogFile["/var/log/pods/*/*.log"]
        AppPod --> LogFile

        subgraph VectorDaemon ["Vector DaemonSet (Rust)"]
            Source["File Source (Tail Log)"]
            MultiLine["Multiline Parser (Join Java Stack Traces)"]
            VRLRemap["<b>VRL Transformation Engine</b><br/>• Extract Kubernetes Metadata<br/>• Redact SSN: <code>\d{3}-\d{2}-\d{4}</code><br/>• Mask Bearer Tokens: <code>ey[A-Za-z0-9_-]+</code>"]
            
            Source --> MultiLine --> VRLRemap
        end
        LogFile --> Source
    end

    subgraph KafkaBuffer ["High-Throughput Buffer"]
        Kafka["Apache Kafka / AWS Kinesis Stream<br/>(Decouples Logging Spikes from Storage Engines)"]
    end

    subgraph IngestionEngines ["Log Search & Analytics"]
        OpenSearch["Amazon OpenSearch / Elasticsearch"]
        Loki["Grafana Loki (Cost-Effective Object Storage)"]
    end

    subgraph LongTermCompliance ["Regulatory WORM Compliance (7 Years)"]
        S3WORM["Amazon S3 Glacier Flexible<br/><b>Object Lock: Compliance Mode</b><br/>• SEC Rule 17a-4 / FINRA / SOC 2<br/>• Tamper-proof: Cannot be deleted even by root!"]
    end

    VRLRemap -->|"Clean Stream (TLS)"| Kafka
    Kafka --> OpenSearch
    Kafka --> Loki
    Kafka --> S3WORM

🔬 Under the Hood: Vector VRL Script for PII Masking & Structuring

# /etc/vector/vector.toml
[transforms.mask_and_enrich]
type = "remap"
inputs = ["kubernetes_logs"]
source = '''
  # 1. Parse JSON log payload safely
  structured, err = parse_json(.message)
  if err == null {
    . = merge(., structured)
  }

  # 2. Redact Social Security Numbers (SSN)
  .message = replace(.message, r'\b\d{3}-\d{2}-\d{4}\b', "[REDACTED_SSN]")

  # 3. Mask Authorization Bearer Tokens
  .message = replace(.message, r'(?i)bearer\s+[a-z0-9_\-\.]+', "Bearer [REDACTED_TOKEN]")

  # 4. Mask 16-Digit Credit Card Numbers
  .message = replace(.message, r'\b(?:\d{4}[- ]?){3}\d{4}\b', "[REDACTED_CC]")

  # 5. Enrich with Enterprise Cluster Tags
  .cluster_name = "eks-prod-us-east-1"
  .environment = "production"
  .compliance_audited = true
'''

Complete Production Vector DaemonSet Manifest (vector-daemonset.yaml)

This Kubernetes manifest deploys Vector across all worker nodes, mounts host log paths, parses container runtime logs, scrubs sensitive data in memory, and fans out to both Amazon S3 (for 7-year WORM compliance) and OpenSearch (for real-time queries):

# vector-daemonset.yaml - High-Throughput Kubernetes Logging DaemonSet
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: vector-agent
  namespace: logging
  labels:
    app.kubernetes.io/name: vector
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: vector
  template:
    metadata:
      labels:
        app.kubernetes.io/name: vector
    spec:
      serviceAccountName: vector-service-account
      tolerations:
        - operator: Exists # Run on all master/worker nodes
      containers:
        - name: vector
          image: timberio/vector:0.38.0-distroless-libc
          args: ["--config", "/etc/vector/vector.yaml"]
          resources:
            requests:
              cpu: "100m"
              memory: "128Mi"
            limits:
              cpu: "1000m"
              memory: "512Mi" # Rust footprint is 1/6th of Ruby-based Fluentd
          volumeMounts:
            - name: config
              mountPath: /etc/vector
              readOnly: true
            - name: var-log
              mountPath: /var/log
              readOnly: true
            - name: var-lib
              mountPath: /var/lib
              readOnly: true
            - name: data
              mountPath: /vector-data-dir
      volumes:
        - name: config
          configMap:
            name: vector-config
        - name: var-log
          hostPath:
            path: /var/log
        - name: var-lib
          hostPath:
            path: /var/lib
        - name: data
          hostPath:
            path: /var/lib/vector-data # Persist checkpoint cursors across pod restarts
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: vector-config
  namespace: logging
data:
  vector.yaml: |
    data_dir: /vector-data-dir

    sources:
      kubernetes_logs:
        type: kubernetes_logs
        auto_partial_merge: true # Automatically merges multiline stack traces

    transforms:
      pii_scrubber:
        type: remap
        inputs: ["kubernetes_logs"]
        source: |
          # Redact Social Security Numbers & Credit Cards
          .message = replace(.message, r'\b\d{3}-\d{2}-\d{4}\b', "[REDACTED_SSN]")
          .message = replace(.message, r'\b(?:\d{4}[- ]?){3}\d{4}\b', "[REDACTED_CC]")
          .message = replace(.message, r'(?i)bearer\s+[a-z0-9_\-\.]+', "Bearer [REDACTED_TOKEN]")
          .cluster = "eks-prod-us-east-1"

    sinks:
      # Sink 1: Regulatory S3 Glacier Compliance WORM Vault
      s3_worm_archive:
        type: aws_s3
        inputs: ["pii_scrubber"]
        bucket: "corp-audit-logs-compliance-vault"
        region: "us-east-1"
        key_prefix: "year=%Y/month=%m/day=%d/"
        encoding:
          codec: ndjson
        compression: gzip
        batch:
          max_bytes: 10485760 # 10 MB chunks before uploading
          timeout_secs: 60

      # Sink 2: Real-Time OpenSearch Search Cluster
      opensearch_cluster:
        type: opensearch
        inputs: ["pii_scrubber"]
        endpoint: "https://vpc-logging-opensearch.us-east-1.es.amazonaws.com"
        mode: bulk
        index: "k8s-logs-%Y.%m.%d"

🗄️ Storage Tiering & Immutable WORM Compliance Retention

Enterprise regulations (SEC Rule 17a-4, HIPAA, PCI-DSS Requirement 10.5) dictate that audit and operational logs must be WORM (Write Once, Read Many):

flowchart LR
    Hot["<b>Hot Tier: NVMe / SSD</b><br/>Retention: 7 Days<br/>Fast search (p95 &lt; 500ms)<br/>High Cost ($$$)"] --> Warm["<b>Warm Tier: S3 Standard</b><br/>Retention: 30 Days<br/>OpenSearch UltraWarm<br/>Medium Cost ($$)"]
    Warm --> Cold["<b>Cold Tier: S3 Glacier Instant</b><br/>Retention: 90 Days<br/>Queryable via Athena<br/>Low Cost ($)"]
    Cold --> Archive["<b>Compliance Archive: S3 Glacier Flexible</b><br/>Retention: 7 Years (2,555 Days)<br/><b>S3 Object Lock (Compliance Mode)</b><br/>Lowest Cost ($0.0036/GB)"]

[!CAUTION] S3 Object Lock Compliance Mode: In Compliance mode, no user—including the AWS Account Root User—can delete or overwrite an object or decrease the retention period until the retention timer expires. This provides absolute legal proof against ransomware or malicious insider tampering during legal discovery.

⚠️ Real-World Failure Modes & Exception Playbooks

Failure Mode 1: The CloudFront Invalidation Storm FinOps Bill

• The Disaster: A CI/CD deployment script runs aws cloudfront create-invalidation --paths "/*" on every Git commit. At 80 deployments a day across 10 distributions, the company incurs thousands of dollars in surprise CloudFront invalidation fees.
• The Root Cause: CloudFront charges $0.005 per path after the first 1,000 paths per month. Furthermore, invalidating /* flushes the cache worldwide, sending millions of requests directly to the origin and crashing backend pods!
• The SRE Playbook:
  1. Never invalidate /* in production CI/CD.
  2. Implement Content-Hashed Asset Naming (app.a8f23c.js). Hashed files can be cached for 1 year (Cache-Control: public, max-age=31536000, immutable).
  3. Only invalidate index.html on new releases, which costs pennies and completes in under 15 seconds.

Failure Mode 2: The Vector OOM Crash on Runaway Java Exception

• The Disaster: A backend developer deploys a broken microservice that throws a recursive Java exception producing a single 50 MB log line. The node logging agent attempts to buffer the line into RAM, crashes with an Out-Of-Memory (OOM) error, and halts logging for all other pods on that node.
• The Root Cause: Vector/Fluent Bit buffered multiline entries without an upper byte limit.
• The SRE Playbook:
  1. Set strict multiline length limits: max_bytes = 65536 (64 KB). Any single log entry exceeding 64 KB is automatically truncated.
  2. Configure memory buffers with disk backpressure: buffer.type = "disk" with a maximum queue size of 5 GB on node NVMe.

🎤 Staff/Principal Architect Interview Questions

Q1: Why is CloudFront Origin Shield placed between Regional Edge Caches and the origin?

Model Answer:
In a globally distributed CDN, cache misses from 13 Regional Edge Caches (RECs) across Europe, Asia, and the Americas hit the origin concurrently. If a viral news event or product drop occurs, 13 independent RECs will fetch the same asset from the origin simultaneously.
Origin Shield acts as a centralized cache placed in the exact AWS region of the origin. It introduces Request Collapsing: all 13 RECs query Origin Shield, and Origin Shield executes only 1 single request to the origin ALB/S3, eliminating origin thundering herd problems and slashing cross-region data transfer costs.

Q2: How do you achieve compliance-grade log immutability against malicious AWS root credentials?

Model Answer:
We deploy S3 Object Lock in Compliance Mode with an S3 Bucket Lock Policy within a dedicated, isolated Log Archive AWS Account managed by AWS Organizations.
Cross-account IAM roles are strictly limited to s3:PutObject. In Compliance Mode, the retention period cannot be reduced or disabled by any IAM identity, including the AWS Root Account. Even if an attacker gains root credentials to the AWS management console, AWS API guardrails strictly reject DeleteObject and DeleteBucket calls until the retention timestamp matures.