Every platform vendor now ships an agent that watches your cluster and offers to fix it. The pitch is consistent across all of them: the agent observes, reasons and acts, with a human approving anything meaningful. SUSE describes a crew of specialised agents behind an intelligent router, with human validation for approvals and support for external Model Context Protocol servers. Other vendors describe the same shape in different words.
The pitch skips a question that matters more than any of the capabilities in it. If your cluster is reconciled from Git, the agent's write does not last. The Argo CD documentation is unambiguous about this: with self-heal enabled, automatic sync runs when the live cluster's state deviates from the state defined in Git, and the sync is attempted again after a self-heal timeout that defaults to five seconds. An agent that patches a live object has made a change with a five-second half-life.
Agents are still worth having. The useful question is where they connect. We have spent the last year putting them in front of estates we run, and the integration point that survives contact with a reconciler is the pull request.
The reconciler already decided who changes what
Before an agent arrives, a GitOps estate has already answered the permissions question for every human in the organisation. Nobody edits live objects, because the controller undoes it. We made this argument about delivery coordination in Argo CD syncs desired state, it doesn't coordinate delivery, and it applies to a non-human actor without modification.
What self-heal actually does
Four settings carry the whole policy, and each one is worth setting deliberately rather than inheriting from a template.
Setting | Default | What it means for an agent |
|---|---|---|
syncPolicy.automated | Off | Without it, nothing reconciles on its own and drift persists until someone syncs |
prune | false | With it on, resources that disappear from Git are deleted from the cluster. It does not touch objects the Application never tracked |
selfHeal | false | With it on, any live change is reverted roughly five seconds after the controller notices |
timeout.reconciliation | 120 s plus jitter | The upper bound on how long a drifted resource stays drifted |
The asymmetry worth noticing lives in the last column. Self-heal governs the resources the Application tracks, and nothing else. An agent that edits a managed object is reverted within seconds. An agent that creates a stray object with no tracking metadata has made something the reconciler cannot see and will never remove. The edit is undone, the mess is not.
The same intent, two integration points. Only one of them is still in the cluster an hour later.
Why this changes the question
The vendor framing asks how much authority to grant an agent. In a reconciled estate that question has a mechanical answer rather than a policy answer. Authority over live objects is already spoken for. What remains to be decided is whether the agent may write to the repository, and under what review.
Three places an agent can touch the system
Read, write to the API server, write to Git. The three have very different half-lives.
Reads are not free
A read-only service account feels like the safe default, and it is the one most teams start with. It is safer than write access and it is not free. Everything the agent reads goes into a context window: config maps, annotations, environment blocks, error strings with connection details in them, and the output of whatever diagnostic command seemed useful at the time.
This is a reason to keep secret material out of the objects an agent can read in the first place, which is the same discipline we argued for in secrets in GitOps. A sealed secret in the repository is ciphertext. The decrypted value lives in a Secret object, and an agent with cluster-wide read on Secrets has a copy of your estate in a transcript somewhere.
Direct writes to the API server
We do not grant them, and durability is only half the reason. The other half is what a live write costs you during the seconds it does exist: a diff and an author, which are the two things an incident review needs a week later.
The restart case makes this concrete. An agent that notices a crash-looping pod and restarts it has removed the evidence. The next engineer to look sees a healthy pod, no diff, and an audit entry attributing the change to a service account. The pod will crash again, and the loop will have cost one full diagnostic cycle.
Writes to Git
The pull request is the one write path that behaves like every other change in the estate. Nothing has to be invented to make it reviewable or reversible, because the same mechanism already carries every change a human makes.
What the pull request actually contains
The useful output of an agent in this setup is small and specific. It is a branch with a manifest change, a title that names the symptom, and a body that shows the evidence it used.
Read what that artifact gives a reviewer. The claim is checkable, the evidence is attached, the blast radius is one value file, and the change arrives on the path every other change uses. Compare it to an agent that quietly raised the limit on the live object at 02:41 and left nothing behind.
The latency objection
The obvious complaint is that a pull request is slower than an action. It is, by the length of a review plus up to one reconciliation interval, which the Argo CD defaults put at 120 seconds plus up to 60 seconds of jitter.
That cost is real and it is usually quoted against the wrong baseline. The alternative is not an instant correct fix, it is an instant unreviewed change that the controller reverts, followed by a human investigating why the symptom came back. Measured end to end, the slower path finishes first more often than it loses.
What agents are genuinely good at here
The left column is where the value is. The right column is why review stays.
Correlation across tools
Logs in one system, events in the API server, metrics in a third place, the manifest in Git and the deploy history in the pipeline. Assembling those five into one timeline is tedious, mechanical and exactly the kind of work that gets skipped at three in the morning. An agent does it in seconds and does it the same way every time.
A first draft of the change
Drafting a values override, a network policy or a probe adjustment is fast work for a model and slow work for a person who has to look up the field names. The draft is worth having even when it is wrong, because reviewing a concrete proposal is easier than starting from an empty file.
Explaining a failed sync
Sync failures are a well-defined domain with structured error output, which makes them the most reliable thing we have seen an agent handle. The same applies to hook failures, where the ordering rules we described in database migrations in GitOps are easy for a model to apply correctly once it can see the hook annotations and the job logs.
What we do not let an agent do
Restart, delete or scale a live workload. The exception people ask for is a restart during an incident, which is the moment losing the evidence costs the most.
Hold a long-lived token. Agent credentials are short-lived and issued per task, the same as any other automation in the estate.
Write to a repository without a review. An agent with merge rights is an agent with direct cluster access on a delay.
Touch anything with a migration in it. Schema changes have an ordering contract, and a model that has not read your data has no way to honour it.
Operate during an incident without being asked. The value is in correlation, and correlation offered unprompted during a page is noise.
The MCP server is the access point, and that is the argument for it
Most of these integrations arrive over the Model Context Protocol, and the case for them is stronger than the usual framing allows.
The alternative is giving every agent its own credentials, which scatters authority across as many places as you have agents. A server running in the cluster puts that authority in one reviewable place instead: one tool surface, one policy, one thing to revoke. The specification is also direct about what a tool is. Tools represent arbitrary code execution, descriptions of tool behaviour should be treated as untrusted unless they come from a trusted server, and the protocol cannot enforce any of this at the protocol level.
Read those two things together rather than as a warning. Enforcement sits with the host application and with you, and the MCP server is precisely where you get to do it. That is the argument for putting it in the cluster rather than beside it: the server holds the access, not the agent, and the set of tools it exposes is the complete list of what any agent connected to it can do. Scope that list once and you have scoped every agent at once.
Two things follow from that, and both are about the server rather than about the agent. It now carries the authority a token used to carry, so it deserves the review you would give a new controller rather than the glance you would give a configuration line. And a server somebody else wrote is a different proposition from one you run, because the specification’s caution about untrusted tool descriptions is aimed squarely at that case. A kubectl-shaped tool on a server you do not control is still a write path into your cluster.
Where the industry numbers actually point
The 2025 CNCF annual survey found that 82% of container users now run Kubernetes in production, up from 66% in 2023, and that 66% of organisations hosting generative AI models use Kubernetes to manage some or all of their inference workloads. The same survey reports that 58% of cloud native innovators use GitOps principles extensively against 23% of adopters.
Put those two findings next to each other. The organisations most likely to put AI workloads on Kubernetes are also the ones most likely to reconcile their clusters from Git. For that group, an agent that writes to the API server fails on correctness before anyone reaches the security argument, and the security argument then follows from it.
The 2025 DORA research frames AI as an amplifier that magnifies an organisation's existing strengths and weaknesses. An estate with a clean reconciliation boundary gets a faster diagnostic loop. An estate where people already patch live objects by hand gets the same habit at machine speed.
The audit trail is the whole argument
Agent writes to the API server | Agent opens a pull request | |
|---|---|---|
What exists afterwards | An audit log line attributed to a service account | A branch, a diff, a review and a merge commit |
Who approved it | Nobody, or a chat message | A named reviewer in the repository |
How you reproduce it | Reconstruct from the audit log | Check out the commit |
How you revert it | Nothing to revert, the controller already did | Revert the commit |
Where the reasoning lives | In a conversation nobody kept | In the pull request body |
Kubernetes audit logging is a real facility and worth enabling, and the documentation describes what it records. It answers what happened and when. It does not answer why anyone thought it was a good idea, which is the question that matters during a review a week later.
When this is premature
Three situations where we would not introduce an agent at all.
The estate is not reconciled. If people still apply manifests by hand, adding an agent adds a second uncontrolled writer to a system that already has too many.
The estate is small enough that one person already knows why everything broke. Correlation is the value on offer, and correlation across six services is something a human does in their head.
Change control has to stay manual for a regulatory reason. In that case the agent can read and summarise, and nothing else, which is a much smaller project than the one being sold.
None of the three is a reason to dismiss the idea permanently. They are reasons the first version should be a great deal smaller than the one in the pitch.
How we would introduce one
Narrow, read-only, and connected to the repository rather than to the cluster.
The third step is the one that decides whether to keep going, and it answers what the agent actually gives you. In this setup the agent makes a periodic pass over the cluster and its usage statistics and proposes improvements. Counting the proposals only shows activity. What matters is how many get merged and what they changed afterwards: fewer OOM kills, fewer restarts, less throttling. A proposal stream with a low merge rate, or merged changes that move nothing, is a tool producing work for reviewers, and the honest response is to narrow its scope or switch it off.
We make the same argument about measuring a platform in our golden path article, and it transfers without modification. An agent whose pull requests nobody merges has exactly the problem a paved road nobody takes has, and the diagnosis is the same: find out what the people who went around it needed.
Closing
The question is not how much authority an AI agent should have over your cluster. If you reconcile from Git, the reconciler has already answered it, and the answer is that live writes do not survive. What is left to decide is whether an agent may propose changes to the repository, and that is a question about review capacity rather than about trust in a model.
Our position in the estates we run is settled. The agent reads a scoped slice, correlates signals faster than a person can, and writes exactly one artifact: a pull request with the evidence attached. Everything after that is the delivery path that was already there.
Thinking about where an agent fits in your delivery path
If you are weighing one of these integrations against an estate you already reconcile, we are happy to talk through the boundary before anything gets installed.
SHARE ON SOCIAL MEDIA
