Multi-Cloud Disaster Recovery Guide

Multi-Cloud Disaster Recovery Overview

Disaster recovery (DR) ensures business continuity when infrastructure fails. While single-cloud DR protects against regional outages within one provider, multi-cloud DR protects against the most catastrophic scenario: a complete cloud provider outage or a provider-level incident that degrades services across all regions. Though rare, provider-wide incidents have occurred. AWS experienced a multi-hour us-east-1 outage in December 2021 that affected thousands of services, and Azure had a global Active Directory incident in March 2021 that impacted authentication worldwide.

Multi-cloud DR is not just about surviving provider outages. It also addresses compliance requirements for geographic data sovereignty, regulatory mandates for provider diversity (common in financial services and government), and strategic goals around avoiding single-vendor dependency. However, multi-cloud DR is significantly more complex and expensive than single-cloud DR. It requires cross-provider networking, data replication, identity federation, and application-layer portability.

This guide covers cross-cloud DR architecture patterns, data replication strategies, DNS-based failover, identity management during DR events, application-layer failover, compliance considerations, and testing procedures. Every section provides practical CLI commands and configuration examples across AWS, Azure, and GCP.

RPO & RTO Across Providers

Recovery Point Objective (RPO) and Recovery Time Objective (RTO) are the two fundamental metrics for DR planning. RPO defines how much data loss is acceptable (measured in time). RTO defines how quickly services must be restored. Multi-cloud DR introduces additional complexity because data replication across providers is inherently slower than within a single provider's backbone network.

DR Tier Definitions

Cross-Provider Replication Latency

Data replication between cloud providers traverses the public internet or dedicated interconnects, adding latency compared to intra-provider replication:

• Intra-provider (same region): Sub-millisecond to single-digit milliseconds
• Intra-provider (cross-region, same continent): 10–50 ms
• Cross-provider (dedicated interconnect): 5–20 ms (AWS Direct Connect to Azure ExpressRoute via colocation)
• Cross-provider (public internet): 20–100 ms (variable, not guaranteed)

Cross-Cloud Architecture Patterns

Multi-cloud DR architecture must balance cost, complexity, and recovery capability. The four primary patterns correspond to the DR tiers defined above, each with increasing investment and decreasing recovery times.

Pattern 1: Active-Active Multi-Cloud

In an active-active configuration, your application runs simultaneously on two (or more) cloud providers, with traffic distributed between them via DNS-based or anycast-based load balancing. Both environments serve production traffic at all times. If one provider fails, the other absorbs the full load.

This is the most resilient but most complex and expensive pattern. It requires:

• Bidirectional data replication between providers with conflict resolution
• Application code that is cloud-agnostic or uses abstraction layers
• DNS or global load balancing that can detect provider failures and redirect traffic
• Identity and authentication that works across both providers
• Monitoring and alerting that spans both environments

Pattern 2: Warm Standby

A warm standby maintains a scaled-down but functional copy of your application in a secondary cloud. Core databases replicate continuously, and application servers run at minimal capacity. During a DR event, you scale up the secondary environment and redirect traffic. Recovery time is 15–30 minutes.

Pattern 3: Pilot Light

A pilot light keeps only the most critical infrastructure components running in the secondary cloud: database replicas, DNS configurations, and container images in a registry. Compute resources (VMs, containers, functions) are not running. During DR, you provision compute, restore configurations, and scale services. Recovery time is 1–4 hours.

Pattern 4: Backup & Restore

The simplest pattern: back up data to the secondary cloud's object storage (S3 to Azure Blob Storage, or GCS to S3) on a scheduled basis. No infrastructure runs in the secondary cloud. During DR, you provision everything from scratch using Infrastructure as Code and restore data from backups. Recovery time is 4–24 hours.

# Terraform multi-cloud pilot light configuration
# Primary: AWS | Secondary: Azure (pilot light)

# AWS Primary Infrastructure
module "aws_primary" {
  source = "./modules/aws"

  vpc_cidr        = "10.0.0.0/16"
  cluster_name    = "prod-cluster"
  db_instance     = "db.r6g.xlarge"
  node_count      = 5
  environment     = "production"
}

# Azure Pilot Light (minimal infrastructure)
module "azure_pilot_light" {
  source = "./modules/azure"

  vnet_cidr       = "10.1.0.0/16"
  cluster_name    = "dr-cluster"
  db_sku          = "GP_Gen5_2"  # Minimal SKU for replication target
  node_count      = 0            # No application nodes until DR activation
  environment     = "dr-standby"

  # Database replication from AWS via logical replication
  replication_source = module.aws_primary.db_endpoint
}

# Cross-cloud networking via Megaport
module "interconnect" {
  source = "./modules/megaport"

  aws_dx_connection_id    = module.aws_primary.dx_connection_id
  azure_er_circuit_id     = module.azure_pilot_light.er_circuit_id
  bandwidth_mbps          = 1000
}

# Cloudflare DNS for failover (provider-neutral)
module "dns_failover" {
  source = "./modules/cloudflare"

  domain     = "app.example.com"
  primary    = module.aws_primary.alb_dns_name
  secondary  = module.azure_pilot_light.ag_dns_name
  health_check_path = "/health"
  failover_ttl      = 60
}

Data Replication Strategies

Data replication across cloud providers is the most challenging aspect of multi-cloud DR. Unlike intra-provider replication (which uses the provider's high-speed backbone network), cross-provider replication traverses external networks and must handle different database engines, APIs, and consistency models.

Replication Approaches

# Cross-cloud database replication using PostgreSQL logical replication
# Source: AWS RDS PostgreSQL | Target: GCP Cloud SQL PostgreSQL

# 1. Enable logical replication on AWS RDS
aws rds modify-db-parameter-group \
  --db-parameter-group-name prod-pg-params \
  --parameters "ParameterName=rds.logical_replication,ParameterValue=1,ApplyMethod=pending-reboot"

# 2. Create a publication on the AWS source
psql -h prod-aurora.cluster-xyz.us-east-1.rds.amazonaws.com -U admin -d appdb -c "
  CREATE PUBLICATION dr_publication FOR ALL TABLES;
"

# 3. Create the subscription on GCP Cloud SQL target
psql -h /cloudsql/my-project:us-central1:dr-postgres -U admin -d appdb -c "
  CREATE SUBSCRIPTION dr_subscription
    CONNECTION 'host=prod-aurora.cluster-xyz.us-east-1.rds.amazonaws.com port=5432 dbname=appdb user=replication_user password=xxx sslmode=require'
    PUBLICATION dr_publication
    WITH (copy_data = true, create_slot = true);
"

# Cross-cloud object storage sync using rclone
# Configure rclone with both providers
rclone sync s3:my-backup-bucket gcs:my-dr-bucket \
  --transfers 16 \
  --checkers 8 \
  --s3-provider AWS \
  --gcs-project-number 123456789 \
  --log-file /var/log/rclone-sync.log \
  --log-level INFO

# Automate with a cron job or Cloud Scheduler
# 0 */6 * * * rclone sync s3:backups gcs:dr-backups --transfers 16

DNS-Based Failover

DNS is the primary mechanism for directing user traffic between cloud providers during a DR event. By updating DNS records to point to the secondary cloud's endpoints, you can redirect traffic without changing application URLs. However, DNS-based failover has inherent limitations: TTL propagation delays, client-side caching, and the need for health checking to trigger automatic failover.

DNS Failover Options

# AWS Route 53: Configure failover routing with health checks
# 1. Create health checks for both providers
aws route53 create-health-check --caller-reference "aws-primary-$(date +%s)" \
  --health-check-config '{
    "IPAddress": "203.0.113.10",
    "Port": 443,
    "Type": "HTTPS",
    "ResourcePath": "/health",
    "RequestInterval": 10,
    "FailureThreshold": 3
  }'

aws route53 create-health-check --caller-reference "azure-secondary-$(date +%s)" \
  --health-check-config '{
    "IPAddress": "198.51.100.20",
    "Port": 443,
    "Type": "HTTPS",
    "ResourcePath": "/health",
    "RequestInterval": 10,
    "FailureThreshold": 3
  }'

# 2. Create failover DNS records
aws route53 change-resource-record-sets --hosted-zone-id Z1234567890 \
  --change-batch '{
    "Changes": [
      {
        "Action": "CREATE",
        "ResourceRecordSet": {
          "Name": "app.example.com",
          "Type": "A",
          "SetIdentifier": "primary-aws",
          "Failover": "PRIMARY",
          "TTL": 60,
          "ResourceRecords": [{"Value": "203.0.113.10"}],
          "HealthCheckId": "hc-aws-primary-id"
        }
      },
      {
        "Action": "CREATE",
        "ResourceRecordSet": {
          "Name": "app.example.com",
          "Type": "A",
          "SetIdentifier": "secondary-azure",
          "Failover": "SECONDARY",
          "TTL": 60,
          "ResourceRecords": [{"Value": "198.51.100.20"}],
          "HealthCheckId": "hc-azure-secondary-id"
        }
      }
    ]
  }'

# Cloudflare: Configure load balancing with health checks (provider-neutral)
# Using Cloudflare API
curl -X POST "https://api.cloudflare.com/client/v4/zones/ZONE_ID/load_balancers" \
  -H "Authorization: Bearer $CF_TOKEN" \
  -H "Content-Type: application/json" \
  --data '{
    "name": "app.example.com",
    "default_pools": ["aws-pool-id"],
    "fallback_pool": "azure-pool-id",
    "proxied": true,
    "steering_policy": "failover",
    "session_affinity": "cookie"
  }'

Identity & Access During DR

Identity management is one of the most overlooked aspects of multi-cloud DR. Your application needs to authenticate users and authorize access in the secondary cloud environment, which may use a completely different identity provider. There are several strategies:

Identity Federation Approaches

• External IdP as source of truth: Use a cloud-agnostic identity provider (Okta, Auth0, PingIdentity, Keycloak) as the primary authentication source. Both cloud environments federate with the same external IdP. If the IdP itself fails, this becomes a single point of failure.
• Cross-cloud federation: Configure OIDC federation between providers. For example, Azure AD (Entra ID) can act as the IdP for both Azure-native and AWS/GCP workloads. AWS IAM can assume roles via web identity federation with Azure AD tokens.
• Replicated user store: Maintain a user database in both environments, synchronized via CDC or application-level replication. This ensures authentication works independently in each cloud.

# Configure AWS to trust Azure AD (Entra ID) tokens via OIDC federation
# This allows workloads authenticated by Azure AD to assume AWS IAM roles

# 1. Create an OIDC identity provider in AWS IAM
aws iam create-open-id-connect-provider \
  --url https://login.microsoftonline.com/TENANT_ID/v2.0 \
  --client-id-list api://aws-dr-access \
  --thumbprint-list 1234567890abcdef1234567890abcdef12345678

# 2. Create an IAM role that trusts the Azure AD OIDC provider
aws iam create-role \
  --role-name AzureADFederatedAccess \
  --assume-role-policy-document '{
    "Version": "2012-10-17",
    "Statement": [{
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::123456789012:oidc-provider/login.microsoftonline.com/TENANT_ID/v2.0"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "login.microsoftonline.com/TENANT_ID/v2.0:aud": "api://aws-dr-access"
        }
      }
    }]
  }'

# GCP: Configure workload identity federation with Azure AD
gcloud iam workload-identity-pools create azure-dr-pool \
  --location=global \
  --display-name="Azure AD DR Federation"

gcloud iam workload-identity-pools providers create-oidc azure-ad-provider \
  --location=global \
  --workload-identity-pool=azure-dr-pool \
  --issuer-uri="https://login.microsoftonline.com/TENANT_ID/v2.0" \
  --allowed-audiences="api://gcp-dr-access" \
  --attribute-mapping="google.subject=assertion.sub,attribute.groups=assertion.groups"

Application-Layer Failover

DNS-based failover redirects traffic, but the application itself must be ready to serve requests in the secondary environment. This requires application-layer considerations that go beyond infrastructure:

Configuration Management

Applications need provider-specific configuration (database endpoints, queue URLs, storage bucket names, API keys) that differs between primary and DR environments. Store environment-specific configuration in a secrets manager or configuration service that is available in both environments:

• HashiCorp Vault: Deploy Vault in both clouds with replication. Cloud-agnostic secrets management.
• Environment variables: Inject configuration via Kubernetes ConfigMaps/Secrets or container environment variables, managed by IaC.
• Feature flags: Use a feature flag service (LaunchDarkly, Unleash) to toggle provider-specific behavior at runtime.

Stateful Services

Stateful services (databases, caches, file storage) are the hardest to fail over because state must be synchronized between environments. For each stateful component:

• Databases: Use cross-cloud replication (logical replication, DMS, Debezium) with promotion scripts that switch the replica to a primary role.
• Caches: Accept cache cold start in the DR environment. Pre-warm critical cache keys as part of the failover runbook.
• Object storage: Replicate objects between providers using rclone, cross-cloud replication rules, or application-level dual-write.
• Message queues: Accept message loss for non-critical queues. For critical queues, use a cross-cloud broker (Confluent Cloud Kafka) as the source of truth.

Network Connectivity & Routing

Multi-cloud DR requires reliable network connectivity between providers for data replication, health checking, and management traffic. There are three primary connectivity models:

Connectivity Options

# AWS: Create a VPN connection to Azure
# 1. Create a Virtual Private Gateway on AWS
aws ec2 create-vpn-gateway --type ipsec.1 --amazon-side-asn 65001
aws ec2 attach-vpn-gateway --vpn-gateway-id vgw-abc123 --vpc-id vpc-xyz789

# 2. Create a Customer Gateway pointing to Azure VPN Gateway
aws ec2 create-customer-gateway \
  --type ipsec.1 \
  --bgp-asn 65002 \
  --public-ip <AZURE_VPN_GATEWAY_IP>

# 3. Create the Site-to-Site VPN connection
aws ec2 create-vpn-connection \
  --type ipsec.1 \
  --vpn-gateway-id vgw-abc123 \
  --customer-gateway-id cgw-def456 \
  --options '{
    "TunnelOptions": [
      {"PreSharedKey": "super_secret_psk_1", "TunnelInsideCidr": "169.254.10.0/30"},
      {"PreSharedKey": "super_secret_psk_2", "TunnelInsideCidr": "169.254.10.4/30"}
    ]
  }'

# Azure: Create a VPN Gateway and connection
az network vnet-gateway create \
  --name azure-vpn-gw \
  --resource-group rg-networking \
  --vnet vnet-dr \
  --gateway-type Vpn \
  --vpn-type RouteBased \
  --sku VpnGw2 \
  --asn 65002

az network local-gateway create \
  --name aws-local-gw \
  --resource-group rg-networking \
  --gateway-ip-address <AWS_VPN_GATEWAY_IP> \
  --local-address-prefixes 10.0.0.0/16

az network vpn-connection create \
  --name aws-to-azure \
  --resource-group rg-networking \
  --vnet-gateway1 azure-vpn-gw \
  --local-gateway2 aws-local-gw \
  --shared-key "super_secret_psk_1" \
  --enable-bgp true

Compliance & Data Sovereignty

Multi-cloud DR introduces complex compliance considerations, particularly around data sovereignty, cross-border data transfer, and regulatory requirements for specific industries.

Key Compliance Considerations

• Data residency: Some regulations (GDPR, LGPD, data localization laws) require data to remain within specific geographic boundaries. Ensure your DR target region complies with applicable data residency requirements. Replicating EU customer data to a US-based DR site may violate GDPR.
• Encryption in transit: All cross-cloud replication must use encrypted channels (TLS 1.2+, IPsec VPN, or dedicated interconnects with encryption). Ensure encryption keys are managed according to your compliance framework.
• Encryption at rest: Data in both primary and DR environments must be encrypted at rest using provider-managed or customer-managed keys. If using customer-managed keys, ensure key material is accessible in the DR environment without depending on the primary provider.
• Audit logging: Maintain audit logs for data access and replication activities in both environments. CloudTrail (AWS), Azure Activity Log, and Cloud Audit Logs (GCP) should all feed into a centralized SIEM that survives a provider outage.
• Financial services: Regulations like DORA (EU), OCC guidance (US), and MAS TRM (Singapore) may mandate multi-cloud or multi-provider DR for critical financial systems.

Testing & Operational Readiness

A DR plan that has never been tested is not a DR plan; it is a wish. Multi-cloud DR testing is more complex than single-cloud testing because it involves coordinating across provider consoles, CLI tools, and monitoring systems. Establish a regular testing cadence and document every test outcome.

Testing Types

DR Runbook Template

Every DR plan should include a detailed runbook with the following sections:

• Detection: How do you determine that a DR event has occurred? Define the monitoring signals, thresholds, and escalation procedures.
• Decision: Who authorizes failover? Define the decision tree, including partial vs. full failover options and rollback criteria.
• Execution: Step-by-step commands for each failover action: DNS switch, database promotion, compute scaling, identity configuration verification.
• Validation: How do you confirm the DR environment is functioning correctly? Define smoke tests, health checks, and data integrity verification steps.
• Communication: Who is notified, and how? Define status page updates, customer communication templates, and internal escalation channels.
• Failback: Procedures for returning to the primary environment after the incident is resolved, including data reconciliation and verification.

# DR failover execution script (example - AWS to Azure)
#!/bin/bash
set -euo pipefail

echo "=== MULTI-CLOUD DR FAILOVER: AWS -> Azure ==="
echo "Started at: $(date -u +%Y-%m-%dT%H:%M:%SZ)"

# Step 1: Verify Azure DR environment health
echo "[1/6] Verifying Azure DR environment..."
az aks get-credentials -g rg-dr -n dr-cluster
kubectl get nodes -o wide
kubectl get pods -n app --field-selector status.phase!=Running

# Step 2: Promote Azure database replica to primary
echo "[2/6] Promoting Azure database to primary..."
az sql db replica set-partner \
  --resource-group rg-dr \
  --server sql-dr-server \
  --name appdb \
  --partner-server sql-prod-server \
  --partner-resource-group rg-database

# Step 3: Scale up Azure AKS node pool
echo "[3/6] Scaling Azure AKS nodes..."
az aks nodepool scale \
  --resource-group rg-dr \
  --cluster-name dr-cluster \
  --name userpool \
  --node-count 5

# Step 4: Update application configuration
echo "[4/6] Updating application configuration..."
kubectl set env deployment/app-api \
  -n app \
  DATABASE_URL="postgresql://admin:xxx@sql-dr-server.database.windows.net:5432/appdb" \
  CLOUD_PROVIDER="azure"

# Step 5: Switch DNS to Azure endpoint
echo "[5/6] Switching DNS to Azure..."
aws route53 change-resource-record-sets --hosted-zone-id Z1234567890 \
  --change-batch '{
    "Changes": [{
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "app.example.com",
        "Type": "A",
        "TTL": 60,
        "ResourceRecords": [{"Value": "198.51.100.20"}]
      }
    }]
  }'

# Step 6: Validate
echo "[6/6] Running validation checks..."
sleep 120  # Wait for DNS propagation
curl -sf https://app.example.com/health || echo "HEALTH CHECK FAILED"
echo "=== Failover complete at: $(date -u +%Y-%m-%dT%H:%M:%SZ) ==="

Related Resources

Explore provider-specific disaster recovery guides for deeper coverage:

• AWS Disaster Recovery Strategies Guide
• Azure Disaster Recovery & Site Recovery Guide
• GCP Disaster Recovery & Backup Guide