Benchmark · SREGym · 54 live faults

One broken Kubernetes cluster, two agents: raw kubectl vs Radar's MCP. Which finds the root cause first?

We gave the same model the same fault twice: once with a shell and kubectl, once reading Radar's live model of the cluster with kubectl blocked. 54 paired fault scenarios on a live EKS cluster. Pick any of them below and watch both agents troubleshoot it, call by call.

41s
Radar median, faults both arms got right
vs 154s on kubectl — 3.8× at the median
43 of 44
scenarios where Radar answered sooner
counting only faults both arms got right
4 min
Radar's slowest correct answer
kubectl's was 24 min

Replay any of the 54 Kubernetes faults

Nothing here is hand-picked. Every paired scenario from the run is in the dropdown, including the ones Radar got wrong. Each solid mark is one tool call, drawn at the moment it was issued and as wide as it blocked for; the hatched ground between marks is the model thinking. Hover any mark to see the call.

Accuracy you can actually wait for

Given unlimited time the two arms land close together: 91% vs 87%. But nobody lets an agent grind for half an hour while production is down. Cap it at a budget you would really grant, and the gap opens up, because Radar has already finished.

Time budgetRadar MCPkubectl
2 min83%30%
5 min91%56%
10 min91%72%
15 min91%83%
30 min91%87%

Not every Kubernetes MCP server would do this

MCP is how an agent reaches a cluster now, and it is the right interface. But the protocol is a connector, and Kubernetes MCP servers differ enormously in what they send back through it. One that proxies kubectl hands the model the same wall of YAML with an extra hop in front of it; we would not expect that to beat a shell. What moved this benchmark is what came back: a model of the cluster Radar keeps current, so the agent asks one question instead of rebuilding the resource graph from a dozen serial commands. The agent didn't get smarter. It stopped waiting.

Radar MCP41s median to diagnosis
38s
3s blocked · 38s model

6% of the incident blocked on the cluster

kubectl154s median to diagnosis
161s
161s blocked · -7s model

105% of the incident blocked on the cluster

blocked on a toolmodel workingMedians across 54 paired scenarios, both bars on one scale.

With a shell, the agent spends nearly three quarters of the incident waiting on the cluster: rollout status, log tails, sleep-and-retry loops, re-listing resources to rebuild context it already lost. Radar answers from a model of the cluster it is already maintaining, so almost none of its time is spent blocked.

That flips diagnosis from I/O-bound to model-bound. Radar's floor is how fast the model thinks; kubectl's floor is how fast a cluster answers serial commands. Faster models widen this gap rather than closing it.

Which is why “does a Kubernetes MCP server help?” is the wrong question to ask of the category. The useful question is what a given server sends back: raw resources the model still has to assemble, or state that has already been correlated. We are writing that comparison up separately; this page is the evidence for one server, measured rather than asserted.

All 54 scenarios, not the flattering ones

One row per paired fault, sorted by how long kubectl took. The question isn't whether Radar was faster; it's how much, and where the shell blew past ten minutes.

Where Radar's MCP server loses to kubectl

Radar loses scenarios, and they are all in the dropdown. Two patterns account for most of it.

Evidence inside a running process. When a database grant is revoked inside the database and no Kubernetes object changes, the kubectl agent can shell in and read it. Radar has no exec, so it gets the component right and the mechanism shallow. Closing that safely, with typed runtime probes rather than free-form shell access, is work in progress.

Detectors that have not fired yet. The benchmark injects a fault and starts the agent seconds later. Some of Radar's signals are time-based and are still warming up, so the agent asks, gets a short and confident-looking answer, and stops. That one is squarely our bug, and the fix is for Radar to say “I have not watched long enough yet” instead of looking done.

Worth being blunt about the other side too: the kubectl arm's distinctive wins rest on giving an autonomous agent free-form exec on your cluster. Almost no security team grants that, and they are right not to.

How we ran the Kubernetes agent benchmark

  • Benchmark: SREGym (Microsoft + UIUC), faults injected into OpenTelemetry Astronomy Shop, DeathStarBench Hotel Reservation, and Social Network.
  • Cluster: live 3-node EKS, us-east-1. Same cluster, same faults for both arms.
  • Agent: Claude Code on claude-sonnet-5, both arms.
  • Arms are disjoint: the Radar arm cannot fall back to a shell — kubectl is blocked outright. That models how Radar actually deploys, and it attributes results to the tool surface rather than to whichever tool the agent felt like reaching for.
  • Grading: SREGym's own LLM judge at temperature 0. The graded artifact is the first diagnosis each agent submits.
  • Timing: agent start to submitted diagnosis, read from the raw transcripts. Not session wall-clock, which also counts the separate remediation stage the benchmark runs afterwards.
FAQ

AI agents on Kubernetes: common questions

Should I give an AI agent kubectl access to my Kubernetes cluster?
Measured against the alternative, a shell is both slower and riskier. On 54 injected faults the same model driving raw kubectl took a median 154 seconds to a correct diagnosis against 41 seconds through a Kubernetes MCP server, and needed the full thirty minutes to reach its accuracy ceiling. It is also ungoverned: the agent inherits everything the shell can do, every call is a potential mutation, and this benchmark contains a fault whose most natural diagnostic command prints a production password straight into an LLM context window. What this benchmark compared was an unrestricted shell against Radar's correlated, read-only MCP surface; it did not test least-privilege read-only kubectl or other MCP servers, so the defensible conclusion is narrower than never use kubectl: do not hand an agent unrestricted shell access by default, and prefer least-privilege RBAC-scoped read access.
Does a Kubernetes MCP server actually make an AI agent better, or is it hype?
MCP by itself is plumbing and will not move a number. It is a protocol for connecting an agent to a tool, and an MCP server that simply proxies kubectl hands the model the same YAML firehose with an extra hop in front of it - there is no reason to expect that to beat a shell, and good reason to expect it to be slower. What moved this benchmark was not the protocol but what came back through it: a cluster model Radar maintains continuously, so the agent asks one question instead of reconstructing the graph from a dozen serial commands. Accuracy barely moves given unlimited time, 91% against 87%. Time to an answer moves about fourfold, because driving raw kubectl the agent spends 73% of the incident blocked on the cluster and here that drops to 6%. The agent did not get smarter, it stopped waiting.
How much faster is an AI agent with an MCP server than with kubectl?
Roughly four times faster to a correct diagnosis: a median 41 seconds against 154 seconds, measured across the 44 faults both arms answered correctly so that neither side's failures move its own number. The Radar arm answered sooner in 43 of the 44. Read it as a result about pre-correlated cluster context rather than about MCP: the protocol is only the connector, and a server that merely wraps kubectl would hand back the same raw output the shell already gives you. The advantage is larger under a realistic time budget: held to five minutes, accuracy is 91% against 56%.
What is an AI SRE and can it actually debug production Kubernetes?
In practice it is an ordinary coding agent, Claude Code or Cursor or Codex, pointed at live cluster state. On 54 real injected faults it reached the correct root cause 91% of the time when given structured access, against 87% with a raw shell. The honest limits: it fails on evidence living inside a running process where no Kubernetes object changed, and it can answer too early when time-based signals have not warmed up. Both failure modes are visible in the replay on this page.
Is a Kubernetes MCP server that wraps kubectl any better than kubectl?
We did not measure that configuration, so treat this as reasoning rather than a result: there is little reason to expect it to help and some to expect it to hurt. The cost driving this benchmark is not the transport, it is what the agent receives. A single kubectl get -o yaml returns hundreds of lines the agent re-reads every turn, and tracing a cross-cutting fault means rebuilding the graph by hand over dozens of serial commands. Wrapping those same commands in an MCP server changes how the call is made, not what comes back, and adds a hop. The gain here came from answering out of a continuously maintained model of the cluster, which turns diagnosis from I/O-bound into model-bound - and is why the gap widens as models get faster rather than closing.
Where does Radar's MCP server lose to raw kubectl?
Five of the 54 scenarios. Two patterns account for most of it. Evidence that lives inside a running process, such as a database grant revoked inside the database with no Kubernetes object changed, where the kubectl agent can shell in and read it and Radar has no exec to reach it. And time-based detectors that had not warmed up when the agent asked, so it got a confident-looking answer too early. Every losing scenario is replayable on this page.
How was this Kubernetes MCP benchmark run?
SREGym, the fault-injection benchmark from Microsoft and UIUC, on a live 3-node EKS cluster in us-east-1. Both arms ran claude-sonnet-5 against identical faults on the same cluster. The MCP arm had kubectl blocked outright so it could not fall back to a shell, which attributes the result to the tool surface rather than to whichever tool the agent preferred. Grading used SREGym's own LLM judge at temperature 0, on the first diagnosis each agent submitted. Per-trial data is downloadable.

More on AI agents and Kubernetes

  • The full write-up — methodology, per-scenario analysis, and what we changed in Radar because of it.
  • Radar's MCP server — the 22 read tools and 6 annotated write tools the Radar arm was driving, and how to point your own agent at them.
  • Diagnose with AI — the same correlation engine the benchmark measured, driven from Radar's own UI.

Bring your first cluster online in 60 seconds.

Install the Helm chart, paste a token, see your cluster. No credit card required.

Install with HelmTry hostedLet's Talk

Apache 2.0 OSS · Unlimited clusters with Radar OSS · Hosted free tier for up to 3 clusters