Enterprise Lesson 10: Brownfield Cloud Migration Architecture (Zero-Downtime Modernization)

🧠 The Concept (Explain Like I’m 5)

Imagine moving an international mega-airport (like Mumbai or London Heathrow) to a brand-new location 30 kilometers away:

• The Rookie Way (Big Bang Cutover): You shut down the old airport at midnight on Friday, tell all airplanes to land at the new airport on Saturday morning, and hope the baggage conveyors and air traffic radar work. If a radar screen glitches, 500 airplanes run out of fuel in the sky, passengers riot, and the airline collapses (Total Business Disaster).
• The Enterprise Architect Way (Dual-Run Phased Migration):
  1. You build a private bullet train between the old and new airports (AWS Direct Connect with BGP Failover).
  2. You mirror passenger records continuously in real time so both control towers see identical luggage and passenger data (AWS DMS with Continuous Change Data Capture - CDC).
  3. You route 1 cargo flight to the new airport (Canary Traffic Shift). If baggage arrives cleanly, you shift 10% of domestic flights, then 50%, and finally all international flights.
  4. Only when 100% of flights operate flawlessly at the new airport for 30 consecutive days do you shut down the old runways. Zero canceled flights, zero lost luggage, zero downtime.

🧭 The 6 Rs of Cloud Migration: Decision Matrix

In enterprise brownfield environments, every legacy application must be evaluated against the AWS 6 Rs Migration Framework:

flowchart TD
    App["Legacy Application Portfolio<br/>(e.g., 200 On-Prem Workloads)"] --> Eval{"Evaluate Business Value,<br/>Technical Debt & Urgency"}

    Eval -->|"No longer needed / redundant"| Retire["<b>1. RETIRE (Decommission)</b><br/>Archive data to S3 Glacier WORM.<br/>Turn off VMs. Save 100% compute cost."]

    Eval -->|"Third-party off-the-shelf software"| Repurchase["<b>2. REPURCHASE (Drop & Shop)</b><br/>Replace legacy self-hosted tool with SaaS<br/>(e.g. self-hosted Jira ➔ Atlassian Cloud)."]

    Eval -->|"High business value, but tightly coupled to on-prem hardware / mainframes"| Retain["<b>3. RETAIN (Keep On-Prem)</b><br/>Keep in local datacenter.<br/>Revisit during next hardware lifecycle."]

    Eval -->|"Urgent datacenter exit deadline (e.g. lease expiring in 90 days)"| Rehost["<b>4. REHOST (Lift & Shift)</b><br/>Replicate VMs byte-for-byte via AWS Application Migration Service (MGN).<br/>Zero code changes."]

    Eval -->|"Modernize OS/DB without core code rewrite"| Replatform["<b>5. REPLATFORM (Lift & Reshape)</b><br/>Move app to container (EKS); replace self-hosted Oracle with Amazon Aurora RDS."]

    Eval -->|"Strategic core competitive advantage requiring agility & scale"| Refactor["<b>6. REFACTOR (Cloud-Native Re-Architecture)</b><br/>Decompose monolithic code into microservices, event-driven Kafka pipelines, and serverless."]

The 6 Rs Architectural Comparison Matrix

🌐 Hybrid Network Transition Architecture (Direct Connect + VPN Failover)

During a 6 to 12-month migration phase, on-premises applications and cloud workloads must communicate with predictable sub-10ms latency and high-throughput reliability:

flowchart TD
    subgraph OnPremDC ["Enterprise On-Premises Data Center"]
        LegacyApp["Legacy Microservices & Core DB<br/>CIDR: 192.168.0.0/16"]
        CustomerGateway["Customer Gateway (Edge BGP Routers)"]
        LegacyApp --- CustomerGateway
    end

    subgraph AWSCloud ["AWS Multi-Account Cloud (us-east-1)"]
        TGW["AWS Transit Gateway (Central Network Account)"]
        AppVPC["Target EKS Workload VPC<br/>CIDR: 10.100.0.0/16"]
        TGW --- AppVPC
    end

    CustomerGateway ==>|"Primary: Dedicated 10 Gbps AWS Direct Connect (Transit VIF + MACsec)"| DXLocation["AWS Direct Connect Location"]
    DXLocation ==> TGW

    CustomerGateway -.->|"Backup: IPsec Site-to-Site VPN over Public Internet (BGP AS-Path Prepending)"| TGW

    NoteLink["BGP Dynamic Routing: If Direct Connect fiber is severed,<br/>traffic fails over to IPsec VPN in &lt; 3 seconds automatically!"]

Critical Network Architecture Details:

1. Dynamic BGP Routing with AS-Path Prepending: On-premises BGP routers advertise the datacenter CIDR (192.168.0.0/16) over both Direct Connect and the backup IPsec VPN. By prepending the on-prem Autonomous System Number (ASN) three times on the VPN route, AWS Transit Gateway prefers Direct Connect as the primary path, falling back to the VPN tunnel in under 3 seconds only if BGP keepalive packets fail.
2. Transit VIF (Virtual Interface): A single Transit VIF attaches the Direct Connect connection directly to the Transit Gateway, enabling seamless routing to all 700+ VPCs across the enterprise.

🗄️ Database Modernization: Heterogeneous Migration with AWS DMS & SCT

Migrating an active production database from on-premise proprietary engines (Oracle Database or Microsoft SQL Server) to open-source cloud-native engines (Amazon Aurora PostgreSQL) is the most challenging phase of enterprise migration.

sequenceDiagram
    autonumber
    participant OnPremDB as 🗄️ On-Premises Oracle Primary (Active OLTP)
    participant SCT as 🔧 AWS Schema Conversion Tool (SCT)
    participant DMS as 🔄 AWS Database Migration Service (DMS Replication Instance)
    participant Aurora as 🚀 Amazon Aurora PostgreSQL (Target Cluster)
    participant App as 💻 Cloud Microservice (EKS)

    Note over OnPremDB,Aurora: PHASE 1: SCHEMA CONVERSION & FULL LOAD
    SCT->>OnPremDB: Analyze PL/SQL packages, stored procedures, triggers, data types
    SCT->>Aurora: Apply converted PostgreSQL DDL schemas & tables (without indexes)
    DMS->>OnPremDB: Read full historical table data (Full Load)
    DMS->>Aurora: Write initial baseline rows into Aurora PostgreSQL

    Note over OnPremDB,Aurora: PHASE 2: CONTINUOUS CHANGE DATA CAPTURE (CDC)
    DMS->>OnPremDB: Continuously stream Redo / Archive Logs (CDC) over Direct Connect
    DMS->>Aurora: Replay transactions in Aurora (Replication Lag < 1 second)
    Note over DMS: Dual-run parity maintained for 30 days of validation!

    Note over OnPremDB,Aurora: PHASE 3: ZERO-DOWNTIME CUTOVER (< 5 MINUTES)
    App->>App: Put on-prem application into brief read-only maintenance mode (2 minutes)
    DMS->>Aurora: Drain remaining in-flight CDC transaction queue (Lag reaches 0 seconds)
    Aurora->>Aurora: Create target secondary indexes & foreign keys
    App->>Aurora: Update application JDBC connection string to Aurora PostgreSQL Endpoint
    App->>Aurora: Resume live read/write traffic ✅
    Note over Aurora: Cutover complete! Zero data loss! On-prem database decommissioned!

🚦 Dual-Run & Traffic Phasing Patterns (The Reverse Proxy Router)

To avoid an all-or-nothing cutover, enterprise platform architects implement a Weighted Reverse Proxy Traffic Router:

flowchart TD
    Client["🌍 Customer Requests: api.company.com"] --> GlobalRouter["Route 53 Weighted DNS / CloudFront Edge"]

    subgraph LegacyOnPrem ["On-Premises Data Center"]
        OnPremLB["F5 BIG-IP Load Balancer"]
        OnPremMonolith["Legacy Monolith (v1.8)"]
        OnPremLB --> OnPremMonolith
    end

    subgraph CloudTarget ["AWS Production Landing Zone"]
        ALB["Application Load Balancer"]
        CloudEKS["Modern EKS Microservice (v2.0)"]
        ALB --> CloudEKS
    end

    GlobalRouter -->|"Phase 1: 90% Traffic"| OnPremLB
    GlobalRouter -->|"Phase 1: 10% Canary Traffic"| ALB

    OnPremMonolith -.->|"Reverse Sync (Fall-Back Safety Net)"| CloudEKS

The 4 Phased Cutover Steps

• Phase 1 (Internal Canary - 0% Public): Route internal corporate IP addresses and employee testing accounts to the AWS EKS cluster. Validate telemetry, authentication tokens, and database read latency.
• Phase 2 (10% Public Canary): Shift 10% of real internet traffic via Route 53 Weighted Routing. Observe Prometheus error rates and Aurora database connection pool saturation for 48 hours.
• Phase 3 (50% Equilibrium Shift): Shift traffic to 50% across on-prem and AWS. Both platforms operate concurrently with synchronized databases.
• Phase 4 (100% Full Promotion & Cutover): Shift 100% of traffic to AWS. Leave the on-premises database in Reverse CDC replication mode (streaming changes from Aurora back to on-prem Oracle) for 7 days. If a critical architectural flaw emerges on Day 3, you can fail back to the on-premise datacenter with zero lost transactions.

⚠️ Real-World Failure Modes & Exception Playbooks

Failure Mode 1: The CDC Replication Lag Memory Spill

• The Disaster: During a high-volume batch job on the on-premise Oracle database, the AWS DMS replication instance runs out of memory (ReplicationInstanceOutofMemory), crashes, and restarts. CDC replication falls 6 hours behind, postponing the scheduled production cutover.
• The Root Cause: AWS DMS buffers in-flight change transactions in memory. When a massive SQL batch updates 5 million rows in a single transaction, the DMS memory buffer overflows.
• The SRE Playbook:
  1. For large batch migrations, configure DMS settings:

     {
       "TargetTablePrepMode": "DO_NOTHING",
       "MemoryLimitTotal": 8192,
       "MemoryKeepTime": 120,
       "StreamBufferSettings": {
         "StreamBufferCount": 16,
         "StreamBufferSizeInMB": 32
       }
     }

  2. Size the DMS replication instance with adequate memory (e.g. dms.r6i.2xlarge with 64 GB RAM).

Failure Mode 2: The Direct Connect MTU Packet Drop Blackhole

• The Disaster: After shifting 20% of traffic to AWS, large API POST requests containing JSON payloads greater than 1,500 bytes hang indefinitely and terminate with client HTTP 504 gateway timeouts, while small GET requests succeed.
• The Root Cause: The on-premise network team configured the edge router interface with an MTU of 9000 (Jumbo Frames), but the AWS Transit VIF had the default MTU of 1500 bytes. Routers on the path had Path MTU Discovery (PMTUD) blocked by corporate firewall rules that dropped ICMP Type 3 Code 4 packets, silently blackholing large packets!
• The SRE Playbook:
  1. Standardize MTU across the entire hybrid path: explicitly set MTU to 1500 bytes on all Transit VIFs and customer gateways, or ensure end-to-end Jumbo Frame (9001 MTU) support.
  2. Verify that firewalls allow ICMP Type 3 Code 4 (Fragmentation Needed) packets to pass freely.

Failure Mode 3: The Oracle LOB (Large Object) Truncation Trap

• The Disaster: After cutover, customer documents and profile photos stored as Oracle BLOB / CLOB columns are truncated to exactly 32 KB, corrupting user data.
• The Root Cause: AWS DMS defaults to Limited LOB Mode with a max size of 32 KB for performance optimization. Any LOB exceeding 32 KB is silently truncated!
• The SRE Playbook:
  1. Identify max LOB sizes in the source database:

     SELECT MAX(DBMS_LOB.GETLENGTH(document_data)) FROM customer_documents;

  2. Configure DMS task settings with Inline LOB Mode sized to the 99th percentile LOB size, or use Full LOB Mode for tables containing multi-megabyte payloads.

🎤 Staff/Principal Architect Interview Questions

Q1: When migrating from on-premises Oracle to Amazon Aurora PostgreSQL, why do we disable secondary indexes and foreign keys during the DMS Full Load phase?

Model Answer:
During the initial Full Load phase, AWS DMS inserts hundreds of millions of rows into target tables concurrently. If secondary indexes and foreign key constraints are active:

1. Massive Disk I/O Write Penalty: Every row insert requires the database engine to update multiple B-Tree index pages and perform foreign key reference lookups against parent tables, reducing full load throughput by 80% to 90%.
2. Constraint Order Violations: DMS loads tables in parallel; child table rows may be inserted before parent table rows exist, triggering constraint validation errors and halting the migration.
   The standard architectural procedure is to load schema tables with only primary keys, execute the DMS Full Load at maximum wire speed, start CDC replication to capture ongoing changes, and build secondary indexes and foreign keys in the target Aurora database after the full load completes.

Q2: How do you design a migration cutover strategy that allows instant failback to the on-premise datacenter without losing cloud transactions?

Model Answer:
We implement a Bi-Directional Change Data Capture (CDC) Architecture with Loopback Prevention:

1. Ahead of the cutover, DMS streams data forward from on-premise Oracle to AWS Aurora PostgreSQL.
2. At cutover, we switch application traffic to Aurora in AWS.
3. Immediately upon cutover, we launch a Reverse DMS Task that captures changes from Aurora PostgreSQL and replicates them back to on-premise Oracle.
4. We configure loopback suppression filters (ignoring transactions originating from the DMS user) to prevent replication echo loops.
   If an unrecoverable failure occurs in AWS 48 hours later, we can point traffic back to the on-premises datacenter with zero data loss, because every transaction processed in the cloud was continuously synchronized back to the legacy database.