Your AI Agent Should Never Get Cluster Admin by Default

We’ve seen teams do something that feels clever for about 18 minutes: wire an AI agent to Kubernetes, hand it broad kubectl access, and call it “autonomous operations.”

That’s not autonomy. That’s giving a sleep-deprived intern the master keys to your data center.

And yes, this is bad.

The problem isn’t that AI agents are useless. We build agent systems, and they can be wildly effective when the workflow is messy, the context is scattered, and the number of steps can’t be hardcoded in advance. The problem is that people keep confusing “can call tools” with “should have production authority.” Those are not the same thing, and Kubernetes is exactly where that confusion gets expensive.

Mike Anderson made the same core point in a useful post: your AI agent should not have direct kubectl access, especially not with broad privileges, because the blast radius is absurdly high for a system that can misinterpret intent, chain tools incorrectly, or act on incomplete context Source.

We agree. Strongly.

If your agent can delete namespaces, patch deployments, read secrets, exec into pods, and “figure it out,” you haven’t built a platform feature. You’ve built a very enthusiastic outage generator.

Key Takeaways

• Your ai agent should operate through narrow, purpose-built tools, not raw `kubectl` with broad RBAC.
• Least privilege beats “smart enough” every time. Intelligence does not replace access control.
• Approval gates belong on mutating actions, especially in production.
• Every agent action should be auditable: who asked, what context was used, what tool ran, and what changed.
• If a workflow can’t be safely scoped, don’t agentify it yet. Not everything needs an AI agent.

The seduction of direct cluster access

We understand why teams do it.

You’ve got an LLM. You’ve got a backlog full of repetitive ops tasks. Restart this deployment. Check failing pods. Scale a worker. Roll back the release. It’s tempting to connect the model to the cluster, toss in a service account, and let it cook.

That works right up until the model sees “fix the service” and decides the most statistically likely fix is to replace half your deployment spec.

Here’s the part people miss: Kubernetes is not one tool. It’s a loaded warehouse of tools with weird defaults, subtle side effects, and enough power to turn a small misunderstanding into a very public postmortem.

One bad patch can be worse than ten hallucinated emails.

Here’s how the risky version usually looks:

It feels efficient because there’s no middleware. No approval queue. No policy layer. No scoped action catalog.

That’s exactly why it’s dangerous.

Why raw kubectl is the wrong abstraction

kubectl is a human operator interface. It’s broad, flexible, and full of footguns. Humans use it because we can stop, squint, reread the namespace, and realize we’re about to do something dumb.

Agents don’t “squint.” They predict.

That’s fine when drafting a status summary. It’s not fine when mutating production state.

The dev.to piece above makes the core case well: don’t hand agents direct shell or kubectl access when what you actually need is a constrained action layer Source. We’d go one step further: even “read-only except sometimes write” setups are often badly designed if the model still gets to compose arbitrary commands.

Because arbitrary command composition is where safety goes to die.

A safer pattern is boring on purpose:

• the model decides intent
• a policy layer maps that intent to approved operations
• the system executes a predefined tool
• mutating actions require approval or policy checks
• everything gets logged with full context

Boring is good here. Boring keeps your Friday night intact.

Your agent doesn’t need power. It needs a toolbox with child locks.

This is the hot take: most “AI ops agents” are overbuilt in the wrong direction.

Teams obsess over reasoning loops, memory, and multi-step planning. Fine. Useful sometimes. But if the execution layer is sloppy, all that cleverness is lipstick on a forklift.

Your ai agent should be allowed to do a tiny number of very specific things:

• list pods in approved namespaces
• fetch deployment health
• compare desired vs current replica counts
• propose a restart plan
• open a change request
• trigger a rollout restart through a controlled API
• rollback only to known-good revisions
• never read secrets unless there is an extremely specific, audited reason

That last one shouldn’t be controversial, yet here we are.

We’ve found that the safest agent systems look less like “an engineer in a box” and more like “a dispatcher with a laminated checklist.” That sounds less sexy. It’s also how you stay employed.

The architecture that actually works

Here’s the pattern we recommend for production Kubernetes workflows.

The model never talks to the cluster directly. It talks to an orchestration service. That service exposes a small set of task-specific tools. Each tool validates inputs, enforces namespace and resource restrictions, checks environment policy, and emits an audit event. For mutating actions, the tool either requires human approval or follows a pre-approved policy path.

Here’s how the pipeline works:

flowchart TD
  U[User or Alert] --> A[AI Agent]
  A --> O[Orchestration API]
  O --> P[Policy & RBAC Checks]
  P --> T[Scoped Kubernetes Tool]
  T --> K[Kubernetes API]
  T --> L[Audit Log]
  P --> H[Human Approval Gate]
  H --> T

That extra orchestration layer is not bureaucracy. It’s the difference between “helpful automation” and “why is the ingress controller gone?”

A good scoped tool might be restart_deployment(namespace, name) with hardcoded allowlists, dry-run validation, rate limits, and environment rules. A bad tool is run_kubectl(command).

We shouldn’t even have to say that. But apparently we do.

Least privilege is not optional theater

A lot of teams treat least privilege like flossing. They know they should do it, they nod earnestly in meetings, and then they don’t.

Kubernetes punishes that laziness.

If your agent only needs to inspect pod status in staging-payments, don’t give it write access in prod, don’t give it access to other namespaces, and absolutely don’t hand it a role that can touch secrets, clusterroles, or admission configs. Separate service accounts by environment and by action class. Read-only identity for diagnostics. Different identity for controlled mutations. Different approval path for production.

The principle is simple: permissions should match the smallest useful unit of work.

Not the biggest thing the agent might someday maybe possibly need.

We’ve seen teams bundle everything into one service account because “it’s easier for the prototype.” That prototype has a nasty habit of becoming production with a nicer dashboard.

And then everyone acts surprised.

Approval gates are not a sign of failure

There’s a weird macho streak in agent design right now. People want “fully autonomous” because it demos well.

We think that’s overrated.

For Kubernetes, approval gates are often the right design. Not because the model is dumb, but because the cost of a wrong action is asymmetric. Reading deployment status? Fine, automate it. Restarting a deployment in staging? Maybe fully automate with policy. Scaling a production workload or rolling back a release? Put a human in the loop.

Fine-tuning this is like seasoning food. Too little autonomy and the system is bland and annoying. Too much and you’ve dumped a cup of salt into dinner.

A practical approval setup looks like this:

• No approval: read-only diagnostics, log summaries, health snapshots
• Auto-approved by policy: low-risk staging actions, known runbooks, reversible actions with guard conditions
• Manual approval required: production mutations, scaling changes, rollbacks, anything affecting ingress or networking
• Never allowed: arbitrary command execution, secret reads, RBAC changes, CRD modifications unless explicitly designed for a separate workflow

If you need “full autonomy” for cluster-admin-level actions, what you actually need is better runbooks and better engineering judgment.

Auditability is where trust comes from

If the agent touched the cluster, you should be able to answer four questions immediately:

1. Who or what initiated the action?
2. What context did the agent use?
3. Which scoped tool actually executed it?
4. What changed in the cluster?

If you can’t answer those, you don’t have an AI operations system. You have vibes.

This is where a lot of no-code business automation tooling falls down for infrastructure use cases. Those tools can be great for CRM updates, routing tickets, or internal workflows. But Kubernetes is not a lead form. The bar for traceability is much higher when the system can affect uptime, security boundaries, and customer traffic.

That’s one reason we push teams toward purpose-built agent architecture instead of generic “connect anything to anything” automation when the target is production infrastructure. Different domain. Different risk. Different rules.

The real surprise came when we tested it.

The hardest part wasn’t getting the agent to choose a likely action. It was making sure every action was explainable enough that an SRE would trust it at 2:13 a.m. while half-awake and staring at an incident channel.

That’s the standard. Not “the demo looked smooth.”

Design tools around intents, not commands

This is the implementation detail that changes everything.

Don’t expose command surfaces. Expose intents.

Bad:

• kubectl apply -f ...
• kubectl patch deployment ...
• kubectl exec ...
• bash

Better:

• get_workload_health(service, environment)
• propose_restart(service, environment)
• execute_restart(change_request_id)
• rollback_to_previous_stable(service, environment)
• compare_release_state(service, environment)

Intent-based tools make validation possible. They also make policy understandable. You can write rules around “restart service in staging” much more safely than around “allow arbitrary patch operations.”

This is the same reason we prefer narrow interfaces in production AI systems generally, whether it’s AI agents, custom models, or voice AI. Give the model a well-lit hallway, not a warehouse full of unmarked doors.

A practical policy model for Kubernetes agents

If you’re building this now, start with a matrix. Not a giant one. Just enough to stop bad ideas.

Define permissions by:

• environment: dev, staging, prod
• resource type: pods, deployments, services, ingresses, secrets
• action type: read, restart, scale, rollback, patch
• approval mode: none, policy, human
• identity: service account per action class

For example:

• staging deployment restart: allowed by policy
• prod deployment restart: human approval
• prod rollback to previous revision: human approval + change ticket
• secret read: denied
• arbitrary patch: denied
• pod logs read: allowed with namespace scope
• scale deployment in prod: approval + max replica delta guard

That “max replica delta guard” matters. A lot.

Without guardrails, “scale to handle traffic” can become “why are we running 300 pods?”

Don’t confuse observability with authority

A strong pattern is to let the agent see more than it can do.

That sounds backwards, but it works.

Your ai agent should have broad enough observability to diagnose problems well: metrics summaries, events, deployment status, pod health, recent rollouts, maybe log snippets with redaction. But execution authority should stay narrow. Diagnosis can be expansive. Mutation should be constrained.

Think of it like a junior doctor with full access to scans and charts, but not unrestricted authority to wheel patients into surgery.

Same principle. Fewer lawsuits.

Where on-device and voice agents fit in

This gets especially interesting when the interface is conversational.

We’ve seen this in voice-driven workflows too. If you’re building a system where an operator can speak a request — the sort of thing we think about in products like RunHotel or broader on-device AI systems — the risk goes up because human language is messy. “Can you fix the booking service?” is not an operation. It’s a vague wish wrapped in urgency.

That means the backend toolchain has to be even stricter. The interface can be natural language. The execution path cannot be.

Voice is where ambiguity goes to the gym.

What to build first if you’re serious

If you’re early, don’t start with autonomous remediation. Start with these three:

1. Read-only cluster diagnostics
2. Change proposal generation
3. Human-approved execution for one or two safe actions

That alone is useful. It cuts toil without gambling with production.

Then add:

• dry-run previews
• policy simulation
• rollback safety checks
• immutable audit trails
• environment-specific service accounts
• alert-triggered recommendations, not direct actions

And if you’re trying to estimate whether the workflow is even worth automating, do the boring math first. We made an AI cost estimator for exactly this kind of reality check. A lot of “autonomous ops” ideas sound brilliant until you price the engineering, review burden, and incident risk.

Funny how that works.

The line we wouldn’t cross

Here’s our hard line: we would not give a general-purpose LLM agent default cluster-admin access in production.

Not temporarily. Not “just for the pilot.” Not behind a cheerful admin dashboard.

If a workflow truly requires cluster-admin, then build a separate, tightly controlled system for that exact workflow, with explicit approvals, policy checks, and a paper trail that would survive an angry security review. Don’t hide god-mode behind the word “agent.”

That’s not architecture. That’s denial.

What you should do next

Audit your current agent-to-Kubernetes path this week.

List every action the agent can take, every credential it can use, every namespace it can touch, and every mutation that can happen without approval. If that list makes you a little uncomfortable, good. That feeling is your platform engineering instincts trying to save you.

Then redesign around:

• scoped intent-based tools
• least-privilege RBAC
• approval gates for mutations
• full auditability
• read-more, do-less defaults

If you want help designing that system without turning your cluster into a science experiment, talk to us about AI consulting or contact us here.

Your agent can absolutely help run operations.

It just shouldn’t be allowed to “wing it” with cluster admin and a dream.

Sources

• Mike Anderson, “Your AI Agent Should Not Have Direct kubectl Access” — https://dev.to/mike_anderson_d01f52129fb/your-ai-agent-should-not-have-direct-kubectl-access-b1o