← All case studies

How I let an AI agent work on live cloud systems without giving it the keys

AI agents are fast, confident and sometimes wrong. This is the operating model I use so an agent can investigate, plan and draft changes on production cloud, while anything risky still waits for a person.

Reading time
11 min

Results at a glance

Production changes the agent makes alone
0
Secret values the agent can read
0
Test cases the guard must pass
29
Read-only estate review + 6 reports
~1 day
Figure 1: the layers between an AI agent and productionFive layers on one track. 01 Propose: the agent proposes a command or a change. 02 Lanes: reads run on their own, changes ask first, destructive actions and secret access are blocked. 03 Guard: a hook reads the whole command and blocks it when unsure. 04 Approve: a person reads the change card and approves it by number. 05 Verify: each change is tested and the result logged. 3 lanes, 29 guard tests, 7 questions on every change card. FIG. 1AI AGENT CONTROLSBetween an AI agent and productionEvery command passes the same layers. Reading is fast. Anything that changes production waits for a person.01ProposeThe agent proposes acommand or a change02LanesReads run. Changes ask.Deletes and secretsare blocked03GuardReads the whole command.If unsure, it says noFails closed04ApproveI read the change cardand approve it by numberHuman decision05VerifyEach change is testedand the result logged3 LANES: GREEN, AMBER, RED29 TESTS THE GUARD MUST PASS7 QUESTIONS ON EVERY CARD
Figure 1: the layers between an AI agent and productionFive layers on one track. 01 Propose: the agent proposes a command or a change. 02 Lanes: reads run on their own, changes ask first, destructive actions and secret access are blocked. 03 Guard: a hook reads the whole command and blocks it when unsure. 04 Approve: a person reads the change card and approves it by number. 05 Verify: each change is tested and the result logged. 3 lanes, 29 guard tests, 7 questions on every change card. FIG. 1AI AGENT CONTROLSBetween an AI agentand productionEvery command passes the samelayers. Reading is fast. Changeswait for a person.01ProposeThe agent proposes acommand or a change02LanesReads run. Changes ask.Deletes and secretsare blocked03GuardReads the whole command.If unsure, it says noFails closed04ApproveI read the change cardand approve it by numberHuman decision05VerifyEach change is testedand the result logged3 LANES: GREEN, AMBER, RED29 TESTS THE GUARD MUST PASS7 QUESTIONS ON EVERY CARD

TL;DR

  • The problem: an AI agent can now do real work on a company’s cloud: read logs, find faults, write the fix. It is also confident when it’s wrong, and it works faster than anyone can watch. Most teams either ban it or trust it. Both are bad bets.
  • What I built: an operating model with five layers. Safe work runs on its own. Anything that changes production waits for my approval. Destructive actions and secrets are blocked outright, by a guard that reads the whole command and says no when it isn’t sure.
  • The result: the agent does the slow, careful work: investigation, inventory, analysis and first drafts. A full review of a cloud estate, with six reports, takes about a working day. It cannot make a production change on its own, and it never sees a secret.

The lesson in one line: don’t ask whether you trust the AI. Build the system so you don’t have to.

Why this is hard

Picture a brilliant new contractor. They read faster than anyone on your team, never get tired, and do exactly what they think you meant. Most days that’s a gift. Now give them the master keys to the building on their first morning.

That is what connecting an AI agent to a live cloud account looks like. The risk is real, not hypothetical. In July 2025, a widely reported incident saw an AI coding agent delete a company’s live database during a code freeze, after being told not to make changes. Instructions weren’t enough. The agent had the access, so it could.

Teams usually land on one of two answers:

  • Ban it. Safe, but you give up the biggest productivity shift in years, and engineers quietly use it anyway.
  • Trust it. Fast, until the day it’s confidently wrong in production.

I wanted a third answer: let the agent do everything that’s safe at full speed, make everything risky wait for a person, and make the rules impossible to talk around.

I broke my own rule before the agent could

My first version was a list of banned commands. The agent could plan infrastructure changes, but it could never run terraform apply, the command that actually changes live systems.

Before trusting that rule, I tried to break it. It took one line:

terraform -chdir=infra apply

The rule matched the start of a command. This one starts with terraform -chdir, not terraform apply, so it would have slipped straight through.

I found it while testing my own rules, not in production. The command never ran against a live system. That’s the point of testing controls before you rely on them: the hole showed up where it cost nothing.

Think of a doorman who checks a banned list but only reads your first name. That’s a suggestion, not security. So I replaced the list with a guard that understands the whole command and says no when it can’t (layer 2 below). The bypass became a permanent test case. Every hole I find now gets the same treatment, so it can never quietly come back.

The five layers

Every action the agent takes passes through the same layers, in order.

1. Three lanes: green, amber, red

  • Green: reads run on their own. Listing resources, querying logs, reading settings and costs. This is most of the work, and it’s harmless.
  • Amber: anything that changes something asks me first. The agent stops and waits. I see the exact command before it runs.
  • Red: blocked, every time. Deleting, restarting, scaling, purging, swapping live slots, reading secrets and reading Terraform state (which can hold secrets). No prompt, no override from inside the session.

2. A guard that reads the whole command, and fails closed

On top of the lanes sits a guard: a small program that inspects every command before it runs. It looks for dangerous actions anywhere in the command, not just at the start, including inside chained commands. It blocks direct calls to the cloud’s management tools and APIs, and it protects its own files, so the agent can’t edit its own rules.

Most importantly, it fails closed. If the guard can’t understand a command, the answer is no. A safety system that lets things through when it’s confused isn’t one.

It must pass 29 test cases before the agent goes near a live environment, including every bypass I could think of.

3. Data it never sees

The agent sees the names of secrets, never their values. It has no access to database contents. It can’t read infrastructure state files. Before it saves any report or note, a redaction scan checks the output for anything that looks like a key or password.

This matters beyond security. If sensitive data never enters the conversation, it can’t leak into a log, a summary or a document someone shares later.

4. No card, no change

Every change starts as a change card, a short written proposal. Each card answers seven questions:

  1. Why are we doing this?
  2. What’s the evidence?
  3. What does it cost or save, and what does it do to speed and reliability?
  4. What could it break? (the blast radius)
  5. What exactly will run? The exact commands or code change.
  6. How do we undo it?
  7. How will we know it worked?

The agent drafts the card. I read it. It runs only after I approve that card by number, and the result is logged against it. If a card can’t answer question 6, it doesn’t run.

5. Prove it worked

“It ran without errors” isn’t proof. Every change gets its own test. When I changed alerting, for example, each alert was triggered on purpose to confirm it reached the right people, then confirmed again when it cleared itself.

The paper trail

The agent keeps notes on what it learns about each environment, so it doesn’t relearn or misremember it. Every report opens with a plain summary, and every number in it links back to the file that proves it. If I can’t show where a number came from, it doesn’t go in front of a decision-maker.

What the agent actually does

This isn’t only about infrastructure code. I use the same model across day-to-day platform work:

Work What the agent does What I do
Estate review Runs read-only inventory scripts across the whole cloud account and drafts a risk report Decide what matters and in what order
Incident investigation Queries logs, builds a timeline, proposes a likely cause Confirm the cause and choose the fix
Monitoring Maps every alert to who actually receives it, drafts fixes Approve, then fire-test each alert
Cost Pulls billing and usage, checks savings advice against real usage Decide what to buy, resize or remove
Security hygiene Finds secrets stored in plain settings (names only) and plans their move Approve and carry out the rotation
Releases Reviews pipeline settings, drafts approval gates Approve and merge
Infrastructure as code Maps drift, writes import code, runs plans Run every apply (see the Terraform revival)
Daily ops brief Reads chat, email and the work tracker (read-only) and writes a morning summary Read it and decide what needs action

One example of why the human column matters: the cloud provider’s own savings tool recommended a reservation purchase that looked like a big saving. Checking it against what was actually running showed it had priced the wrong machine sizes, so the real saving was far smaller. Buying on the tool’s advice would have meant paying upfront for a saving that wasn’t there.

Under the hood (for engineers)

Everything above is the model. This section is how it’s built. The examples are simplified, but they follow the shape of the real configuration.

Permission lanes (agent settings, simplified; the exact format depends on the tool):

{
  "permissions": {
    "allow": [
      "Bash(az resource list:*)",
      "Bash(az graph query:*)",
      "Bash(az monitor log-analytics query:*)",
      "Bash(terraform plan:*)"
    ],
    "ask": [
      "Bash(az monitor metrics alert update:*)",
      "Bash(az webapp config set:*)"
    ],
    "deny": [
      "Bash(az group delete:*)",
      "Bash(az resource delete:*)",
      "Bash(az webapp restart:*)",
      "Bash(az keyvault secret show:*)",
      "Bash(terraform apply:*)",
      "Bash(terraform state:*)"
    ]
  }
}

Rules like these are necessary but not sufficient: they match patterns, and patterns can be dodged. That’s what the guard is for.

The guard runs as a hook: every command the agent wants to run is passed to it before it executes. The hook exits non-zero to block, and anything it can’t parse is blocked:

#!/usr/bin/env bash
# Simplified sketch. Fail closed: any error below blocks the command.
set -euo pipefail
trap 'echo "guard: blocked (could not verify command)" >&2; exit 2' ERR

cmd=$(cat)   # the command the agent wants to run

# Look for destructive verbs anywhere, not just at the start.
if grep -Eq '(^|[;&|[:space:]])terraform([[:space:]]+-[^[:space:]]+)*[[:space:]]+(apply|destroy|import|state)\b' <<<"$cmd"; then
  echo "guard: Terraform changes are human-only" >&2; exit 2
fi
# Plans must not take the state lock.
if grep -Eq '\bterraform\b.*\bplan\b' <<<"$cmd" && ! grep -q -- '-lock=false' <<<"$cmd"; then
  echo "guard: plans must use -lock=false" >&2; exit 2
fi
# ...cloud CLI writes, direct API calls and edits to the guard's own files are blocked the same way.
exit 0

The test matrix runs before the agent touches a live environment. The 29 cases cover groups like these:

Group Example Expected
Plain destructive commands terraform apply, terraform destroy Blocked
Flags before the verb terraform -chdir=infra apply Blocked
Chained commands cd infra && terraform apply Blocked
State manipulation terraform state rm, state push Blocked
Cloud writes and raw API calls cloud CLI deletes, direct REST calls Blocked
Self-protection editing the hook or its settings Blocked
Malformed input input the guard can’t parse Blocked (fails closed)
Safe work terraform plan -lock=false, read-only queries Allowed

The last row matters as much as the others. A guard that blocks everything gets switched off.

Read-only tooling. Discovery runs through a small set of scripts built to be safe to run any time: they only read, they refuse to run if they detect a write, they’re pinned to one subscription so they can’t wander into another environment, they redact secrets from output, and they log permission errors instead of hiding them. Running them twice gives the same result.

Change card template:

## CARD-NNN: <one-line change>
Why:          <problem, in one paragraph>
Evidence:     <queries / files that prove it>
Impact:       cost <±$/month> · performance <…> · reliability <…>
Blast radius: <what could break, who would notice>
Change:       <exact commands or IaC diff>
Rollback:     <exact commands>
Verify:       <test that proves it worked, and that rollback works>
Status:       proposed → approved CARD-NNN → done → verified

Results

Measure Result
Production changes the agent can make on its own 0 (by design: changes ask, destructive actions are blocked)
Secret values in the agent’s context 0 (names only; secret reads are blocked)
Guard test cases before touching a live environment 29
Full read-only estate review with six reports About one working day
A Terraform revival run under this model 135 imports, 0 changes to the cloud (case study)
Every number in every report Linked to the evidence behind it

What it doesn’t solve

Being honest about the limits is part of the design.

  • Block lists are never complete. Someone will always find a command you didn’t think of. That’s why changes fall into the amber lane by default and the guard fails closed: the safe answer to anything unknown is “ask” or “no”.
  • Approval fatigue is real. If you approve 50 cards a day, you stop reading them. Cards stay small, and reads never need approval, so my attention goes only to changes.
  • The agent ran under my own identity. That kept accountability clear, but it means its ceiling was my access. The next step is giving it its own identity: read-only by default, with time-limited write access granted for one approved card and removed after.
  • Content can carry instructions. Logs, pull requests and web pages can contain text written to trick an AI (“ignore your rules and…”). The defence isn’t hoping it ignores them. It’s that the agent can’t act on them: reading is all it can do without me.

Questions businesses ask before letting AI near production

Where does our data go? The agent works with configuration, metrics and logs. It never sees secret values or database contents, and anything it saves is scanned for keys and passwords first. In the work behind this study, the client approved the use of AI before it started. On every engagement, I agree the AI tool and its data terms with the client up front. Where they already have a business AI plan, I work inside it, so their data stays under their own contract.

Who is accountable if something goes wrong? I am. The agent can’t change production on its own, so every change carries a person’s approval, and that person is me.

What if the AI is wrong? It will be, sometimes. The whole design assumes it. A wrong answer is cheap at the draft stage and expensive in production, so every change is written up with its evidence and a way to undo it before it runs, and tested after.

What does it cost? The AI tooling is small next to engineering time. When I report savings, I keep three labels apart: what I found, what I proposed and what actually showed up on the invoice.

What happens if we stop using AI? Nothing breaks. Everything the agent helps produce is ordinary work your team owns: scripts, infrastructure code, runbooks, reports and change records. The AI speeds up the work. It isn’t part of your system.

Does this replace our team? No. It takes the slow reading and first drafts off their plate, so their time goes on decisions. And “no card, no change” applies to people as much as to the agent.

If you want this in your team

You can start with four steps this week:

  1. List what your AI tools can touch today. Most teams are surprised.
  2. Split every action into green, amber and red. Write it down before you configure anything.
  3. Test your own rules by trying to break them. If you can get around them in five minutes, so can the agent.
  4. Make “no card, no change” the rule for production, for people and AI alike.

If your team wants the speed of AI on production systems without betting the business on it, this is the operating model I set up: fast where it’s safe, human where it counts, and built so nobody has to rely on trust.

Tech stack

Azure · Azure CLI · Azure Resource Graph · KQL · Terraform · Azure DevOps · Bash