Atlas Platform

AWS path: infrastructure/terraform/bootstrap.sh provisions VPC, EKS 1.31, and IAM/IRSA modules in ap-south-1. The node group uses four t3.large Spot instances with t3a.large as capacity-type fallback (ADR-008).

Local path: kind (three nodes) on Colima, platform Helm values under platform/, and Makefile targets verify, up, platform, bootstrap-gitops, status, down. LOCAL_SETUP.md documents the Mac workflow.

Teardown: infrastructure/terraform/destroy.sh and scripts/pre-destroy-cleanup.sh delete Argo CD applications, CNPG clusters, StatefulSets, PVCs, and EBS volumes in dependency order before Terraform destroy. This addresses the EBS orphan issue captured in the Week 6 Day 2 sprint notes.

gitops/apps/root-app-of-apps.yaml bootstraps the app-of-apps pattern. Platform components (ingress-nginx, kube-prometheus-stack, Loki, Argo Rollouts, CloudNativePG operator, metrics-server) sync in wave 0 after wave -1 operators.

workloads-non-prod.yaml ApplicationSet generates three-tier-dev (sync wave 1) and three-tier-staging (wave 2). workloads-prod.yaml generates three-tier-prod (wave 3) with manual sync only.

Measured end state on a fresh four-node EKS cluster (2026-06-10): eleven Argo CD Applications Synced and Healthy, including three-tier-dev (13 pods), three-tier-staging (13 pods), and three-tier-prod (14 pods), 82 pods total at roughly 21% memory utilization across nodes.

Manifests live under gitops/workloads/three-tier-app/base/ (env-neutral) and overlays/dev, overlays/staging, overlays/prod (ADR-010). Built-in types (Deployment, ConfigMap) use strategic-merge patches. CRDs (Rollout, AnalysisTemplate, Ingress host edits, CronJob traffic generator) use JSON 6902 patches with explicit target blocks so Kustomize does not drop sibling fields on CRD strategic merge.

Replica and resource tiers differ by overlay: for example dev runs four backend Rollout replicas for parallel canary experiments; staging runs two; prod runs three with higher CPU and memory limits. Ingress hosts are backend.atlas.local, backend-staging.atlas.local, and backend-prod.atlas.local.

Verification before commit: kustomize build per overlay plus a twelve-check script (image, probes, replicas, ingress host, AnalysisTemplate namespace, ServiceMonitor selector, and related fields). Documented in docs/learning/week-6-eks/multi-env-gitops-day-2.md.

ADR-011 defines three controls per environment: unique Ingress host, namespace filter in AnalysisTemplate PromQL (namespace="three-tier-{env}"), and HOST_HEADER on the traffic-generator CronJob curl loop.

Without the namespace filter, http_requests_total{job="backend-svc-canary"} aggregates all environments. A failing prod canary could still pass analysis if dev traffic is healthy.

Matrix tests (multi-env-canary-isolation-demo.md): dev success canary with LATENCY_MS=5 left staging and prod pod template hashes unchanged across snapshots; staging abort with FAIL_RATE=0.5 under k6 load left dev and prod unchanged. SLO recording rules stayed at 1.0 for all environments during both runs.

The backend is an argoproj.io/v1alpha1 Rollout, not a Deployment (ADR-005). NGINX Ingress implements canary traffic split. Steps start at 25% weight, run an AnalysisRun against AnalysisTemplate metrics (success rate and p95 latency), then advance through 50%, 75%, and 100% on success.

Analysis failure threshold: success rate below 0.99 for two failed measurements triggers RolloutAborted and rollback to the stable ReplicaSet. Under injected failure (staging FAIL_RATE=0.5) and k6 load (load-tests/k6/, parameterized NAMESPACE and HOST_HEADER), abort completes in about ninety seconds.

Operational procedure: docs/runbooks/canary-failure-response.md covers Degraded rollouts, paused canaries, and forensics collection before scaleDownDelaySeconds removes failed canary pods.

Each environment runs a CloudNativePG Cluster CR with three instances (one primary, two replicas), anti-affinity, PgBouncer, and MinIO for S3-compatible WAL archive and scheduled base backups (ADR-003).

Runbooks under docs/runbooks/ cover PITR and manual failover. INC-001 and INC-002 postmortems document failover GameDays. Primary pod deletion recovers in about fifteen seconds in exercised scenarios.

kube-prometheus-stack provides Prometheus, Alertmanager, and Grafana. Loki and Promtail cover logs (ADR-004). Per-environment ServiceMonitors scrape the backend metrics port.

Recording rules and alerts live in gitops/platform/kube-prometheus/rules/three-tier-slo-rules.yaml and three-tier-slo-alerts.yaml. Alerts use multi-window burn-rate pattern on SLIs scoped with namespace=~"three-tier-.*" so one rule group covers all environments.

Week 4 targets documented in ROADMAP.md: database availability 99.5%, latency SLI 99%, backup freshness 99.9%. INC-003 records an availability GameDay against those signals.

docs/adr/ holds twelve ADRs (cluster runtime, GitOps engine, Postgres operator, Loki, Argo Rollouts, Terraform layout, NAT and Spot choices, canary validation app, Kustomize overlays, canary isolation, node sizing).

docs/incidents/ holds five postmortems. docs/runbooks/ covers Argo CD sync issues, SLO breach response, and canary failure. docs/learning/ captures sprint notes, including the Kustomize strategic-merge CRD trap that motivated JSON 6902 for Rollout patches.

Demo recording: README links a four-minute Vimeo walkthrough of multi-env GitOps, canary abort, k6 load, and teardown.