Auto-Healing SRE Pipeline
GitOps Drift Shield
Keep AWS aligned with Terraform: scheduled drift checks, Slack and Prometheus signals, optional Jenkins apply with S3 audit JSON in ap-south-1.
<0s
incident remediation
0%
audit coverage
0/7
drift detection
Goal
GitOps Drift Shield keeps AWS resources in terraform/environments/dev aligned with Git. Manual console or CLI edits create drift. The repo detects that drift, notifies operators, and can apply the saved Terraform plan through Jenkins.
The loop is: provision with Terraform, configure EC2 with Ansible, detect with drift-detector/detector.py, schedule checks in GitHub Actions, remediate in jenkins/Jenkinsfile, record audits in S3.
Terraform and remote state
Modules under terraform/modules/ cover vpc (10.0.0.0/16, public subnets, IGW), ec2 (Ubuntu 22.04, security group for SSH 22, app 8080, node_exporter 9100 from VPC CIDR), and s3 (versioned audit bucket gitops-drift-shield-audit-log-dev).
Remote state uses S3 bucket gitops-drift-shield-tfstate-prashant-31306, key dev/terraform.tfstate, region ap-south-1, DynamoDB table drift-shield-tfstate-lock. Bootstrap the bucket and lock table before the first terraform init.
CI runs terraform fmt -check, init, and validate on pull requests via .github/workflows/ci.yml.
Configuration on EC2
scripts/generate-inventory.sh reads Terraform outputs and writes ansible/inventories/dev/hosts.ini (gitignored).
ansible/site.yml targets group app. Role common installs packages and appuser. Role monitoring installs node_exporter v1.7.0 on port 9100. Role app deploys the Flask demo with GET /health on port 8080.
Drift detection
drift-detector/detector.py accepts --env dev and calls parser.run_terraform_plan on terraform/environments/{env}. parser.py parses JSON plan lines for planned_change events.
terraform plan -detailed-exitcode meanings: 0 no changes, 1 error, 2 drift pending. On drift, notifier.py pushes infra_drift_detected and infra_drift_resource_count to Pushgateway when reachable and sends Slack webhook alerts (30 minute cooldown).
.github/workflows/drift-check.yml runs on cron */30 * * * * and workflow_dispatch with AWS secrets and optional SLACK_WEBHOOK_URL.
Jenkins auto-apply
jenkins/Jenkinsfile runs terraform plan -detailed-exitcode -out=tfplan.binary. Exit 0 stops. Exit 2 runs terraform apply -auto-approve tfplan.binary and sets REMEDIATION_DONE.
After apply, the pipeline uploads audit JSON to s3://{audit_bucket}/audit-logs/{timestamp}-remediation.json using terraform output -raw audit_bucket.
jenkins/Dockerfile ships Jenkins LTS JDK 17, Terraform 1.5.7, and AWS CLI v2 (linux/arm64 binaries for Apple Silicon agents).
Observability
node_exporter on EC2 feeds host metrics. kubernetes/ manifests deploy Prometheus, Pushgateway, Grafana, and optional Jenkins agent into namespace monitoring.
Prometheus scrapes Pushgateway and node targets. Grafana loads bundled dashboard JSON. Drift gauges appear when Pushgateway is reachable from the detector run.
Repository layout
Top-level paths: drift-detector/, terraform/, ansible/, jenkins/, kubernetes/, scripts/, .github/workflows/, docs/assets/architecture.png.
Makefile wraps common local tasks. Known limitations in README cover partial K8s stack requirements and Loom demo placeholder.