The 3am Test: Autonomy Levels for AI Agents in Kubernetes Operations

While you slept last night, your cluster healed itself several times. A kubelet restarted a container that failed its liveness probe. A ReplicaSet replaced a pod that died with its node. The HPA added two replicas during a traffic spike and took them back an hour later. Nobody approved any of it, and nobody minds.

So why does "an AI agent restarted the pod at 3am" land so differently from "the kubelet restarted the container at 3am"?

That question is basically the whole debate about AI agents in production operations, and right now a lot of tools answer it with a config flag.

TL;DR. Agents don't add automation to a cluster - Kubernetes has plenty. They add judgment, and you have to decide how much of it to delegate. This post proposes six levels for that decision, L0 to L5. The line that matters runs between L3, where the agent acts after you say yes, and L4, where it acts and you read about it at 9am. Two rules keep the levels meaningful: a level belongs to a verb in an environment ("restart pods, in staging"), not to an agent as a whole, and a level only counts if the write path enforces it, because the prompt won't. Most teams are at L1-L2 today. Too many tools ship L1 plus a config flag that jumps straight to L4.

Rules are not judgment

The kubelet that restarted your container was executing a contract you can read: restartPolicy: Always, a liveness probe, a documented backoff curve. Same inputs, same behavior, every time. Kubernetes is a pile of these contracts reconciling toward desired state, and that pile is why we trust it. "Should automation touch production at night" stopped being a live question the day you adopted it.

An agent is a different kind of thing. Before it restarts a pod, it diagnosed a situation - read logs, correlated events, formed a theory - and then picked that action out of everything else it could have done. Give it the same symptoms tomorrow and you might get a different theory and a different action. A controller that surprises you means you misread a rule somewhere. An agent that surprises you is working as designed.

Delegating rules is engineering. Delegating judgment is trust, and you would never take a new engineer from "can read the runbook" straight to "unsupervised prod changes at night." That's exactly the jump most AI-for-Kubernetes tooling offers, though: a read-only mode, a fully-autonomous mode, and not much in between. Driving had the same vocabulary problem and fixed it with SAE J3016's six levels. Operations could use the same.

The six levels

This isn't the first ladder of its kind, and it's worth saying so: the Google SRE book sketched an automation hierarchy a decade ago, Google published AI autonomy levels for its own operations this spring, and Rootly maintains a maturity model for AI SRE products. What none of them pin down is what a small platform team actually negotiates with an agent: which cluster verbs, in which environments, on whose approval. So here's the ladder in those terms.

L0 - Manual. kubectl, a dashboard, a runbook. Still where every break-glass procedure should bottom out.

L1 - Explain. The agent reads cluster state and answers questions; the diagnosis and the typing are still yours. This is where most teams genuinely are in mid-2026, and it's the most value for the least risk anywhere on the ladder.

L2 - Propose. The artifact changes from prose to a diff: a root cause plus a specific remediation, which you review like a colleague's change and run yourself.

L3 - Act on approval. The agent executes its own proposal through a bounded set of verbs, after an explicit yes. It feels like a small step from L2, but the agent now holds write credentials, and the whole thing turns on the gate: what exactly did you approve, and is that exactly what runs?

L4 - Bounded autonomy. The agent acts without waking anyone, inside an envelope you wrote down in advance: "may restart crashlooping pods in staging; may roll back deploys under 30 minutes old; may never touch a PVC." You review at 9am, from a report.

L5 - Full autonomy. "Keep production healthy," any verb, any cluster. Nobody ships this responsibly - driving, with a decade head start and standardized roads, hasn't shipped its L5 either. When a vendor says "fully autonomous," they mean L4 with an envelope they haven't shown you.

The 3am test, and the 9am question

The ladder turns into a tool when you apply it per action. For every verb an agent could run, ask two questions.

The 3am test: would I rather be woken up to approve this, or read about it at 9am? That sorts the verb into L3 or L4. It's a question about you rather than about the model, which is why the answers tend to be honest. Nobody wants to be paged to approve a staging pod restart. Nearly everyone wants to be awake when something drains a prod node.

The 9am question: if it chose wrong, is 9am too late to find out? That caps how high the verb can ever go, and the cap comes down to reversibility and blast radius. Note that a smarter model does not raise the ceiling - no future model makes deleting a PVC reversible. The ceiling is a property of the verb.

For what it's worth, I'd hand an agent most of the staging column tomorrow - and nothing drains a production node while I sleep, no matter whose model it runs.

"What level is your agent?" is the wrong question, and the table shows why: one agent should be running L4 for staging restarts, L3 for prod rollbacks, L2 for anything near a security boundary, and L1 for whatever it can't classify. The level lives in the (verb, environment) cell.

Promotion between levels works the way it does for people: on track record. A verb earns L4 when, say, its last 50 proposals were approved unmodified and none got reverted - the same bar as a new hire's path to unsupervised deploys. The ratchet turns both ways, too. An agent-caused incident demotes the verb, immediately.

Where the tools actually are in mid-2026

Mapping real tools onto the ladder is clarifying. I went through the docs of the tools people actually deploy; this isn't a dunk, and most of these defaults are reasonable choices.

The read side has gone autonomous without much debate. Nobody gates "the agent may look" anymore. The contested zone is the L3/L4 boundary, which is some validation that the 3am line is where the real question lives.

The graduated middle is thin, especially in open source. Most open-source tools give you two settings: a safe read-only mode, and a flag like --skip-permissions or autoRemediation.enabled: true that skips straight from L1 to L4. The rungs in between - approving individual verbs, writing an envelope, promoting a verb once it has a track record - mostly show up in commercial products, when they show up at all.

The coding world already ran this experiment. Coding agents went from "YOLO mode or nothing" to graduated permission modes - plan modes, per-action approval, sandboxed bypass - in roughly 18 months, helped along by public failures. The most famous failure came in July 2025, when Replit's agent deleted a founder's production database during an explicit code freeze; Replit added dev/prod separation and a planning-only mode afterward. Kubernetes tooling is earlier on the same curve, and a cluster usually holds more than one database. It would be good to converge faster than the coding tools did.

The level lives in the write path, not the prompt

Telling the model to "always ask before making changes" is a suggestion. Models drift, prompts get long, and one poisoned log line can argue back. A level counts when it's enforced where the writes happen - in the tool surface:

• Bounded verbs, not shell. The agent gets restart, scale, rollback, sync - not kubectl anything. You can only put a ceiling on a verb if the verbs are a short list you can point to, not whatever a shell happens to allow.
• The agent acts as somebody. While a human is in the loop, the natural identity is that human: calls go through RBAC as you, and the agent's ceiling can't exceed yours. An unattended L4 agent can't borrow a sleeping person's identity, so it gets its own scoped account - robot accounts are nothing new - and that account's permissions become the envelope. What you don't want is one shared, anonymous account that's more privileged than anyone on the team.
• Destructive verbs are marked as destructive in the tool definition itself, so the client has to show a confirmation. The L3 gate lives in the protocol, not in the model's manners.
• Investigation and execution don't share a session.

That last one is where prompt injection lives, and it's why we built Diagnose in Radar v1.8.3 the way we did: the investigation runs against a read-only MCP mount - write tools aren't just disabled, they're not discoverable, and a regression test fails if one leaks in. When you approve the proposed fix, the apply runs in a fresh session bound to exactly the confirmed fix, so nothing the agent read along the way - logs, events, annotations, all of it untrusted - can steer what gets written.

And if you run GitOps, you already own this infrastructure. Flux's docs are blunt: change the cluster with kubectl and the change will be "promptly reverted." Argo CD with selfHeal reverts live drift within seconds (the three-minute figure you sometimes hear is the Git polling interval, a different mechanism). An agent that patches a managed resource hasn't fixed anything; it has started an argument with a reconciler, and the reconciler wins. On a GitOps cluster the durable write path is Git itself: the fix arrives as a PR, review is the L3 gate, and git log is the audit trail. GitOps teams built graduated agent autonomy years ago - for humans. (Radar's Diagnose currently gates applies to managed resources behind an explicit acknowledgment; proposing the PR instead is where the arrow points.)

What unattended autonomy costs

There's a 1983 paper I keep coming back to, Lisanne Bainbridge's "Ironies of Automation". Automate the routine work, she argued, and the human is left holding exactly the tasks the designer couldn't automate - while losing the practice that made them good at those tasks. That's on-call with an L4 agent: it eats the crashloops, so the 3am page you still get is the one it couldn't handle, and you arrive at it with fewer recent reps than you've ever had.

She also warned that humans are bad at monitoring mostly-correct automation. Waymo learned this on camera and walked away from Level 3 driving entirely - its 2013 highway tests filmed drivers sleeping and doing makeup at 56mph, because once the system handles almost everything, the human stops being a real fallback. The ops version is my real worry about L3 in practice: an approval gate you always click through is an autonomy level you haven't admitted to. If the agent files thirty approvals a day and you approve all thirty, you're at L4 without an envelope. Write the envelope for the verbs you rubber-stamp, and keep the gates rare enough that a human actually reads them.

One more thing worth stealing from the big-cloud postmortems: when automation takes down production, it usually did exactly what it was configured to do, just with too much authority. When DynamoDB's DNS automation deleted its own active plan last October, AWS's response was to disable that automation worldwide while they rebuilt the safeguards. No AI involved, but that's the move you want available for agents too: a per-verb off switch that works at 3am without shipping anything.

I'll spare you the survey tour - the short version is that executives trust agents to act far more than the people on call do (PagerDuty found 81% of executives trust AI agents in a crisis, while 84% of the same companies have already had an AI-related outage). Draw the 3am line with the people who hold the pager, not the people who bought the tool.

Where the framework runs out

Levels don't fix diagnosis quality. They just decide how expensive a wrong diagnosis is. An agent that's wrong 15% of the time is mildly annoying at L2 and dangerous at L4; autonomy amplifies whatever accuracy you have.

The driving analogy leaks. Roads are standardized and clusters are anything but: your CRDs, your operators, your admission chain. Driving is one action space, while operations is hundreds of verbs with different ceilings. And there's no DMV - every vendor grades its own homework, so expect level inflation. When you hear "autonomous," it's worth asking which verbs that covers, in which environments, and based on what track record.

The ladder is a policy, not an org chart. It won't tell you who reads the 9am report or who owns an agent-caused incident. A good envelope document is where those answers end up.

Where Radar sits on its own ladder

Radar ships L1 through L3 and stops there. This is the part of the post about our own product, so read it with that in mind.

L1 - ask the cluster anything. Radar's MCP server is on by default in the OSS binary. It's read-only by construction, it scrubs secrets from everything it serves, and it answers in structures instead of YAML dumps: get_topology hands back the ownership and routing graph as one object, get_changes answers "what changed in the last 20 minutes," and issues returns a ranked queue with causally-related failures already grouped. On a 52-fault benchmark, the same model diagnosing through this surface beat raw kubectl on accuracy while making 4x fewer tool calls.

L2 - Diagnose. This one shipped in v1.8.3. Hit Diagnose on a misbehaving resource and your own agent CLI (Claude Code, Codex, or Cursor - local, keyless, no account) investigates over the read-only mount, then hands back a root cause, a specific proposed fix, and a coarse confidence band. It will also conclude "healthy" or "inconclusive" when that's the truth. We spent real effort on the inconclusive path - an engine that can't say "I don't know" shouldn't be trusted with a fix.

L3 - the gated apply. One click, behind a confirmation that previews exactly what will change. GitOps-managed resources get an extra acknowledgment, and after applying, Radar re-checks health instead of declaring victory. Every action runs under your RBAC identity via impersonation, and Radar Cloud adds the org view: a list of every agent and token that can reach your clusters, each one revocable in one click, with an audit trail behind it. That's the red button.

We have not shipped L4, and by this post's own rules we shouldn't yet: bounded autonomy needs envelopes you can actually write down, per-verb track records to justify promotions, and reports worth reading at 9am. That's what we're building toward, in the open.

Ask your team the question

Here's the exercise. It takes half an hour, and it's worth doing before you flip on any tool's write mode. List the five verbs an agent would most plausibly run against your clusters. For each one, ask both questions: woken to approve, or read at 9am? And if it chose wrong, is 9am too late? Disagreements are the interesting part - when the platform lead says L4 and last week's on-call says L2, that conversation was overdue anyway.

You end up with a one-page envelope document. The verbs everyone agreed on are your L4 candidates, everything contested stays behind an approval, and the irreversible ones go on the never list. It's the page most "autonomous AI SRE" rollouts skip.

If you want an L1-L3 substrate to run the exercise against:

curl -fsSL https://get.radarhq.io | sh && kubectl radar

Point your agent at the MCP endpoint, break something in staging on purpose, and see where your own 3am line falls.