For modern enterprise enterprises, a single 15-minute maintenance outage can cause thousands in lost revenue and irreversible client trust degradation. Achieving genuine 99.99% high availability mandates a declarative, multi-region Infrastructure-as-Code architecture paired with automated progressive Canary rollouts. Here is our engineering playbook.
1. The Myth of Single-Region High Availability
Deploying multi-AZ (Availability Zone) clusters within a single AWS, GCP, or Azure region protects against single rack or data center hardware failures. However, it provides zero protection against regional routing fiber cuts, regional control-plane outages, or localized cloud provider service degradation.
True fault resilience requires an active-active or active-standby topology distributed across geographically distinct regions (e.g., us-east-1 and eu-west-1), bound together by global Anycast DNS.
2. Declarative Multi-Region Infrastructure with Terraform
Managing multi-region cloud topology manually via cloud web consoles creates configuration drift and introduces human error. Terraform ensures complete reproducibility through composable modules:
source = "terraform-aws-modules/eks/aws"
cluster_name = "${var.environment}-${var.region}-cluster"
cluster_version = "1.30"
vpc_id = module.vpc.vpc_id
subnet_ids = module.vpc.private_subnets
eks_managed_node_groups = {
workloads = {
min_size = 3
max_size = 50
desired_size = 6
instance_types = ["c6i.2xlarge", "c6a.2xlarge"]
}
}
}
3. Kubernetes Pod Lifecycle: Solving In-Flight Dropped Connections
When a deployment updates, Kubernetes sends a SIGTERM signal to the old container pod. If your ingress controller or cloud load balancer has not finished deregistering the pod IP from its iptables/endpoint slice, new incoming requests will continue reaching a terminating pod, resulting in dropped TCP handshakes and 502 Bad Gateway errors.
We resolve this using a mandatory preStop lifecycle hook and fine-tuned probes:
preStop:
exec:
command: ["/bin/sh", "-c", "sleep 15"]
readinessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 5
periodSeconds: 3
terminationGracePeriodSeconds: 45
4. Automated Canary Deployments with Argo Rollouts
Instead of swapping 100% of your production traffic instantaneously, our CI/CD pipelines leverage Argo Rollouts:
- Phase 1: Route 5% of global traffic to the Canary release.
- Phase 2: Monitor real-time Prometheus metrics for 10 minutes. If HTTP 5xx responses rise by more than 0.1% or P99 response time increases by 25ms, trigger an automated instant rollback.
- Phase 3: Gradually step traffic to 25%, then 50%, and finally 100% once stability criteria are verified.
5. Zero-Lock Database Migrations: The Expand/Contract Pattern
The biggest blocker to zero-downtime releases is database schema migration. Running ALTER TABLE on a table with 50 million rows locks the table and halts the application.
We strictly follow the Expand and Contract pattern:
- Expand: Add the new nullable column without touching existing production code.
- Dual Write: Deploy code that reads from the old column but writes to both old and new columns simultaneously.
- Backfill: Run asynchronous batch jobs to migrate historical records in the background.
- Contract: Switch reads entirely to the new column and decommission the legacy column in a subsequent release.
