Reviving drifted production Terraform: 40% to 100% under code, zero changes

TL;DR

• Problem: the company had Terraform, but it had quietly stopped being the source of truth. Only about 40% of resources were managed, code was years behind what had been applied, and the main branch would have produced 5 harmful plans in production, including destroying live CDN resources, turning off the network setting that carries VPN traffic, and downgrading Kubernetes by six minor versions.
• Approach: a phased revival, map → triage → reconcile → adopt, run as human-governed, AI-assisted infrastructure as code. An AI agent did the investigation, code and plans inside guardrails I designed. I made every decision, reviewed every plan and ran every apply.
• Result: 100% of resources are now managed by Terraform and in sync. 135 resources were imported across 11 applies with 0 added, 0 changed and 0 destroyed in Azure. New environments now provision in about 20 minutes, and a GitHub Actions pipeline with scheduled drift detection keeps code and Azure in sync.

The lesson in one line: in a drifted estate, the most valuable thing Terraform can do on day one is nothing. Prove that, then build from it.

Context

The company runs its production platform on Microsoft Azure across separate subscriptions for production, UAT and development: hub-and-spoke networking with Azure Firewall, two Kubernetes (AKS) clusters, core virtual machines and managed MySQL, PostgreSQL and Redis databases.

Terraform had been introduced years earlier. Since then, the people and practices around it had changed, and changes were made through a mix of Terraform runs and the Azure portal. Nobody could say with confidence what an apply from the main branch would do.

The problem

Most of the estate wasn’t in Terraform. Around 60% of resources, including core networking, the firewall’s rules, a Kubernetes cluster, core VMs and the databases, had never been imported.

What was in Terraform didn’t match the code. Mapping every production code folder to its state file turned up several problems:

• 8 of 12 production states had been written by Terraform 1.9.4, while their code pinned versions 1.1 to 1.3. Changes had been applied from newer tooling, and sometimes from feature branches that were never merged.
• One production-labelled folder pointed at UAT state. One state had no code at all.
• Provider lock files were gitignored, so provider versions weren’t pinned.
• An inherited ignore_changes was written in a way that made Terraform blind to an entire Kubernetes node pool.

The result was a main branch that was actively dangerous. Planned against production, it would have:

Component	What the main branch would have done
CDN	Destroyed 3 live resources that existed only in an unmerged branch
Networking	Reverted 9 settings, including gateway transit on the peering that carries VPN traffic
Production AKS	Attempted a 6-minor-version Kubernetes downgrade and reverted autoscaling and maintenance windows
UAT AKS	Recreated a deliberately deleted node pool, re-enabled a removed service mesh and attempted a downgrade
Shared modules	Broke one environment’s plans after being rewritten for another environment’s newer provider

This is the most dangerous state an IaC codebase can be in: it looks like a source of truth, so someone will eventually trust it.

Constraints

• Production could not be disrupted or recreated. Every change had to be import-only or refresh-only. Any planned replace or destroy meant stop and report.
• No version upgrades during the revival. Each component had to run on the Terraform version recorded in its state, so upgrades didn’t mix with reconciliation.
• Shared code across environments. Module changes had to leave UAT, development and shared environments exactly as they were.
• Credentials with real power. Plans and applies ran under an identity with production write access, so the safety net had to be built, not assumed.

How I ran it: human-governed, AI-assisted

I used an AI coding agent (Claude Code) for the hands-on investigation, code and documentation. That’s only safe if the agent can’t do the dangerous parts, so I designed the guardrails first:

• Operating rules in the repo: the source-of-truth order (live cloud > state > code), a four-phase plan, and hard rules. The agent may plan but never apply, destroy, import via the CLI or edit state. Plans always run with -lock=false. No version changes, no hard-coded secrets, every ignore_changes must be recorded with a reason, and the agent stops on any replace or destroy. Humans decide every drift.
• Enforcement, not trust. The first version used command deny rules, but they were prefix-based and could be bypassed (for example terraform -chdir=<dir> apply). I replaced them with a pre-tool-use hook that blocks any destructive Terraform verb anywhere in a command, blocks the cloud CLI and direct API calls, enforces -lock=false, protects its own files from edits and fails closed. It was tested against 29 cases before use.
• A paper trail. A component map, drift report, decision log and follow-up list were kept in the repo, so every accepted drift and every exception can be audited.

flowchart LR
  accTitle: Human-governed, AI-assisted change loop
  agent[AI agent<br/>investigate, code, plan] --> review[Plan + decision log]
  review --> me[I review and decide<br/>accept or revert]
  me -->|approved| apply[I run apply]
  apply --> verify[Re-plan:<br/>no changes, no drift]
  verify --> agent
  guard[Guard hook<br/>blocks apply, destroy,<br/>state edits, cloud CLI] -.-> agent

Approach and decisions

Decision	Alternative considered	Why I chose this
Revive existing state	Start a fresh codebase	Existing state was the only reliable map from Terraform addresses to real resources. Starting over risked duplicate ownership and forced replacements.
import {} blocks + plan -generate-config-out	terraform import CLI, or hand-building state from exported JSON	Import blocks are reviewable in a pull request, visible in the plan and applied by a human. Hand-built state is fragile: only Terraform should write state.
Generate config with Terraform itself	aztfexport	aztfexport targets the latest provider. Components were pinned to older providers, so it would have created needless churn.
Plan on each state’s exact Terraform version	Upgrade everything first	Mixing upgrades with reconciliation makes every diff ambiguous. Matching the state’s version turned “phantom” diffs into clean plans.
New components on a current provider for VMs, hub networking and databases	Add fresh imports to 2022-era components	Fresh imports shouldn’t inherit an old provider’s limits. The newest provider was needed anyway for current database versions.
One resource per NSG rule	Inline rules on the security group	Inline rules would fight Microsoft Defender’s just-in-time access rules, which Azure manages itself.
Frozen copies of the Kubernetes modules for production	Edit the shared modules	Production was rebuilt from frozen copies, so UAT kept its newer modules untouched.
removed {} with destroy = false	terraform state rm	Removing deliberately deleted resources from state stays declarative, reviewed and in Git.

Implementation

Phase 1: map. I mapped every code folder to its remote state key and read each state’s resource count and recorded Terraform version, using read-only, backend-only probes.

Phase 2: triage. I ran refresh-only and normal plans for every component and labelled each one Clean, Drift, Code behind, Dangerous or Error in a drift report.

Phase 3: reconcile. For each drift, I decided whether to accept Azure’s value into code or revert it:

• Out-of-band changes I accepted were codified, and shared modules got optional inputs with neutral defaults ({} / null), so other environments’ plans stayed identical.
• CDN and Kubernetes code was rebuilt using the unmerged branches as templates, verified line by line against state, not trusted.
• Over-broad ignore_changes rules were narrowed to the attributes Azure really owns, such as auto-upgraded Kubernetes versions and autoscaled node counts.
• A storage account kept showing an “update” with no value changes. Comparing the plan JSON showed only sensitivity markers differed: the state had been written by a newer Terraform. Planning on that exact version made it clean.

Deliberately deleted resources were dropped from state without touching Azure:

removed {
  from = module.<node_pool_module>
  lifecycle {
    destroy = false
  }
}

Phase 4: adopt. Unmanaged resources came in through the same loop each time: import blocks → generated config → tidy (remove read-only and zero values, de-duplicate inline blocks) → re-plan until it shows “N to import, 0 to add, 0 to change, 0 to destroy” → commit → human apply → re-verify with both normal and refresh-only plans.

import {
  to = azurerm_virtual_network.hub
  id = "/subscriptions/${var.subscription_id}/resourceGroups/<hub-rg>/providers/Microsoft.Network/virtualNetworks/<hub-vnet>"
}

The order ran from the foundations outward: hub networking, the production VNet, both Kubernetes clusters, core VMs, VM security rules, firewall rule collections and the databases. Critical resources such as networks, VMs and database servers were given prevent_destroy, and database admin passwords never entered code or state.

Real-world data didn’t always fit the provider. Firewall address ranges stored with leading spaces, protocol casing, a VM built from an attached disk that needed the legacy resource type, and a registry in a different subscription each needed a specific, documented fix. Where an unrelated pending change would otherwise have ridden along with an import, I added a temporary, documented ignore so the apply stayed import-only:

lifecycle {
  # TEMPORARY: provider rejects values stored with leading spaces; remove once
  # fixed at source. See DECISIONS.md.
  ignore_changes = [private_ip_ranges]
}

Before merging, every non-production component using a changed shared module was planned twice, once with the main branch and once with the revival branch, and the outputs compared: 15 of 15 identical. The diff review confirmed no secrets, every ignore_changes recorded, and no state or plan files committed.

Keeping it in sync. A revival only lasts if drift can’t creep back in, so the final step was a GitHub Actions pipeline. Every change now goes through a pull request: the Terraform plan runs and is reviewed before anything is applied, automated scanning checks the code, and a scheduled drift check compares Terraform with live Azure, so any out-of-band change is caught quickly, not years later.

Results

Measure	Before	After
Resources managed by Terraform and in sync	~40%	100%
Resources imported	—	135, every import 0 changed
Azure changes caused by the revival	—	0 added / 0 changed / 0 destroyed across 11 applies (7 import, 4 refresh-only)
Harmful plans on the main branch	5	0
Non-production plans changed by shared-module edits	—	0 of 15
Environment provisioning	Manual	~20 minutes from code
Drift detection	None	Scheduled, in CI
Infrastructure changes	Portal or ad-hoc runs	Pull request → reviewed plan → apply

Platform-managed items, such as Microsoft Defender’s just-in-time access rules, are deliberately left to Azure so Terraform never fights them.

Security findings and how they were fixed

Bringing everything under code also made the estate reviewable. Weaknesses that had been invisible while resources were unmanaged showed up in plain text during the inventory, and each one became a tracked fix:

What the revival surfaced	How it was fixed
Remote-management ports reachable from the internet	Removed the public exposure. Admin access now goes through Azure Bastion or time-limited just-in-time access.
Caches and databases with broad network access	Narrowed allow-lists to known sources, moved to private endpoints or private subnets where possible, and disabled non-TLS ports.
Terraform state storage with weak defaults	Enforced TLS 1.2, disabled public blob access and restricted network access. State holds sensitive values, so it’s treated as a secret store.
A credential committed to the repository	Rotated it, moved secrets to Azure Key Vault as sensitive variables with no defaults, and added secret scanning so it can’t happen again.
An end-of-life database engine version	Planned and carried out the major-version upgrade, now a reviewed Terraform change rather than a portal click.

None of these were visible as risks while the resources lived only in the portal. Code turned them into a reviewable list.

What I’d do differently, and next phases

• Read every state’s Terraform version on day one. The single most useful fact in the project, that 8 of 12 states were written by a newer Terraform than their code, came in the middle of reconciliation. It should have been the first query.
• Inventory every subscription before concluding anything is gone. A container registry that looked deleted actually lived in another subscription.
• Commit before every apply. Once, an import was applied before its code was committed. Nothing broke, but the rule is now strict: the apply command is handed over only after the commit.
• Build the pipeline first, not last. The CI came after the adoption. Standing it up first would have enforced the review discipline from the very first import, instead of relying on people to follow it.

Next phases:

• Standardise versions. Every component is now clean on its own Terraform and provider version. The next step is upgrading them, one reviewed component at a time, to a single current version.
• Refactor into modules. Generated code is correct but verbose. Moving it into reusable modules with moved {} blocks keeps the state intact while making new environments even faster to build.
• Harden critical resources. Add delete locks and geo-redundant backups to the databases and other critical resources, declared in Terraform, so protection can’t be removed by accident.
• Tidy the state layout. Move the state that lives in another environment’s storage account to where it belongs, and drop state for resources that no longer exist.
• Policy as code in the pipeline. Add rules (for example required tags, no public endpoints, approved SKUs) that fail a pull request before a risky change can reach a plan.