A developer on your platform needs to run an agent. They do what the platform tells them to do: they ask the service template for a new service, and it gives them a Deployment, the default container runtime, and whatever egress the namespace already allows. The agent ships, it works, and nobody files a ticket.
That is the default answer on most platforms today, and for this workload it is the wrong one. The process inside that container may execute code a model wrote thirty seconds ago. It shares a kernel with every other pod on the node. Its network rules were written for a service that calls two internal APIs, and the agent calls a model endpoint, a package registry and whatever tool server someone wired up last week.
Two upstream projects now exist to make isolation the default for this kind of workload. They agree on the boundary and disagree about nearly everything else, including what counts as one unit. Which one fits turns out to depend on a question that has nothing to do with Kubernetes: whether a person is working inside the agent, or your application is dispatching work to it.
Isolation by default is a path problem
What a developer does when the path has no answer
We wrote the argument in designing a golden path that developers actually use and it holds here without modification: a path competes for adoption, and a team that hits a requirement the path cannot meet goes around the path rather than abandoning the requirement. The policy layer is where that bites, because controls that are not provisioned by default end up applied inconsistently and audited afterwards.
An agent runtime is exactly that kind of requirement. It needs a stronger boundary than a web service, a narrower egress list than a web service, and a lifecycle that no chart in your repository currently describes. A developer who needs one today will build it out of the parts the path does provide, which means an ordinary pod.
Why an ordinary pod is the wrong default here
The kernel is shared. A normal container boundary is sized for code you wrote and reviewed. Model-generated code executed at request time is a different threat model on the same node as everything else.
The egress is wrong by construction. Service templates grant what a service needs. An agent reaches further by design, and the gap gets closed by opening the namespace rather than by listing what the agent may call.
The shape is wrong. A Deployment replicates interchangeable pods. An agent session holds state that belongs to one caller, so it lands in a shape that was built to throw its instances away.
None of these is a reason to stop developers running agents.
Two projects now exist upstream to be that something better, and they arrive from opposite directions. Agent Sandbox is a Kubernetes SIG Apps project that gives each agent session a pod of its own with a stable identity. Agent Substrate is a runtime that does the reverse: it packs many sessions onto a smaller pool of worker pods and snapshots the idle ones.
They agree on the part that matters most. Both put a real isolation boundary around code the agent did not write, and both reach for the same two technologies to do it. Where they disagree is what counts as one unit, and that single choice decides what your quotas, your network policies and your dashboards are describing once agents are running.
We take them in that order below, Agent Sandbox first, because it is the one that keeps the pod as the unit. Everything you already know about running workloads on Kubernetes still holds for it, which makes it the cheaper thing to adopt and the clearer thing to reason about. Substrate is the more interesting engineering and the larger change to your platform, so it comes after the costs of the first are on the table.
What Agent Sandbox actually is
The four kinds
Kind | API group | What it does |
|---|---|---|
Sandbox | agents.x-k8s.io/v1beta1 | A single stateful pod with a stable hostname, network identity and optional persistent storage |
SandboxTemplate | extensions.agents.x-k8s.io | A reusable definition other objects create sandboxes from |
SandboxClaim | extensions.agents.x-k8s.io | Creates a sandbox by taking one from a warm pool |
SandboxWarmPool | extensions.agents.x-k8s.io | Keeps pre-warmed sandboxes ready so allocation is fast |
The smallest useful object is short enough to read in one go.
Why not a Deployment
A Deployment manages interchangeable replicas. Agent runtimes are not interchangeable: each one holds a session, a working directory and a conversation state that belongs to one caller. The project README makes the distinction in the same terms, noting that Deployments suit stateless replicated applications and StatefulSets suit numbered stable sets, while Sandbox targets a single stateful pod with a stable identity.
Why not a size-one StatefulSet
This is the honest objection, and the project answers it directly. You can approximate a sandbox with a StatefulSet of one replica plus a Service and a volume claim template. The README calls that approach cumbersome and missing lifecycle features such as hibernation. Two practical gaps follow from that.
A StatefulSet has no concept of being claimed. Allocation from a pre-warmed pool is the mechanism that makes sandbox startup fast, and there is nowhere to express it.
A StatefulSet has no hibernate and resume. An idle agent session either holds a running pod or loses its state, and neither is what you want for something a user may come back to in twenty minutes.
The third column is the shape an agent session actually has. Nothing in a standard chart produces it.
Stable identity has consequences downstream
A stable hostname per sandbox is the feature, and it is also the part that leaks into everything around it. Service discovery, certificate issuance, log labels and any allow-list keyed on a name all have to cope with a set of hosts that changes shape during the day. A fixed service name in front of a replica set is a simpler object to reason about than a population of named singletons, and most of the surrounding tooling was built for the simpler one.
Isolation is delegated to a runtime class
This is the part most summaries get wrong. Agent Sandbox provides no isolation of its own. It manages pods and delegates the boundary to a RuntimeClass, naming gVisor and Kata Containers as the runtimes it is designed to work with, with vendor neutrality as a stated goal.
Three boundaries, three costs. The right one depends on whose code the agent runs.
The question that picks the runtime
Does the agent execute code you did not write? That single question settles it more reliably than any feature comparison.
Situation | What we would run |
|---|---|
The agent calls your own APIs with a fixed tool set | The default runtime. The boundary you need is network policy and RBAC, not a second kernel |
The agent runs model-generated code against your data | gVisor, for a smaller kernel attack surface at a modest syscall cost |
The agent runs code submitted by your customers | Kata Containers, for a hardware boundary, accepting a slower start and a second kernel to maintain |
Both runtimes are real projects with real operational weight. gVisor intercepts syscalls in user space and has known compatibility gaps for workloads that reach for unusual kernel features. Kata Containers gives a hardware boundary through KVM and asks you to run and patch a guest kernel per pod. Neither is a checkbox.
Warm pools are capacity you pay for in advance
SandboxWarmPool keeps sandboxes pre-warmed so a SandboxClaim can allocate one immediately. The mechanism is sensible and its cost lands somewhere your current model probably does not look.
A ResourceQuota counts requests, not usage, so a warm pool consumes namespace quota from the moment it exists. If your capacity planning is built on request totals, and most of it is, a pool of ten changes the arithmetic before a single agent has done any work.
Picking a pool size is a product decision
Pool size trades money for start latency, and neither side of that trade belongs to the platform team alone. A pool of zero means every session pays a cold start, including the start of whatever isolation runtime you chose. A pool large enough that nobody ever waits is a standing reservation that shows up in the same budget line as production capacity.
The version worth writing down is a floor during working hours and zero outside them, with the numbers in the repository where somebody can argue with them. A pool size buried in a Helm value nobody reads becomes a permanent cost that no review ever revisits.
A second stack is solving the same problem differently
Agent Sandbox is not the only project in this space, and the other one arrives from a different direction. Agent Substrate is an Apache-2.0 runtime for running large numbers of agent sandboxes on Kubernetes, built on the observation that agent workloads sit idle most of the time. Rather than giving each session a pod, it maps many actors onto a smaller pool of worker pods, snapshots an actor when it goes idle and resumes it when a request arrives.
What Substrate claims and what it is
The repository states roughly ten times the density of standard container runtimes, sub-500ms resume and more than 500 suspend and resume activations per second, with a published demo multiplexing about 250 stateful actors onto 8 pods, a ratio near thirty to one. Isolation uses the same two technologies as Agent Sandbox, gVisor and microVMs, so the difference between the projects is not the boundary. It is what sits inside it.
The components are a gRPC control plane, a node-level daemon that supervises worker pods and coordinates snapshots, a controller reconciling worker pools, and a networking layer that routes callers to an actor by a stable identity rather than by whichever pod currently holds it.
And a control plane on top of it
Above Substrate sits AX, or Agent Executor, a declarative control plane started at Google and published under Apache-2.0 with the API group ax.io/v1alpha1. It adds four primitives: Task for the sandboxed run, Workspace for the Git repositories, MCP servers and skills the agent needs, Gateway for an egress allowlist with credential injection, and Model for model configuration and keys in one place.
The Gateway primitive is worth noting for anyone who read the policy section above. The egress problem is real enough that this stack answers it with a first-class object rather than leaving it to a network policy somebody has to write by hand.
How the two stacks line up
Agent Sandbox | Agent Substrate | AX | |
|---|---|---|---|
Layer | Workload API | Compute runtime | Task control plane |
Home | Kubernetes SIG Apps | Independent project | Started at Google |
The unit | One pod per sandbox | Many actors per worker pod | A Task |
Isolation | Delegated to RuntimeClass | gVisor and microVMs | Inherited from Substrate |
Idle sessions | Hold a pod, or a warm pool does | Snapshotted and resumed | Snapshotted and resumed |
Maturity | v1beta1 | Pre-1.0, no compatibility guarantees | v1alpha1 |
The trade worth understanding
Density is bought by giving up the pod as the unit of accounting. In the Sandbox model every Kubernetes object you already rely on keeps meaning what it meant: a ResourceQuota counts sandboxes, a NetworkPolicy applies to one agent, a dashboard panel describes one session. In the Substrate model those same objects describe a worker that holds thirty of them.
What a per-actor limit, a per-actor network rule or a per-actor cost figure looks like in that world is a question neither project documents yet, and the honest answer today is that you would build it. For a team that runs tens of thousands of short agent sessions, the density is worth that work. For a team running four, it is a second control plane to operate in exchange for capacity it does not need.
The ecosystem is converging on the lower layer regardless. kagent, a CNCF sandbox project, already documents running on Substrate, which suggests the runtime will show up underneath tools teams adopt for other reasons.
Which one you want depends on who starts the agent
The project pages do not frame it this way, and after looking at both we think this is the question that actually settles it. Two shapes of demand show up on a platform, and each project fits one of them cleanly.
A harness a person works inside
Somebody wants an environment to experiment in: a coding agent, a research harness, a notebook with tools wired up. The session is long-lived, there are not many of them at once, and the person comes back to it tomorrow expecting their working directory to still be there.
Inside that environment you want to be permissive. The whole value of a harness is that it can read the repository, run the test suite, install a package and call a tool server without asking permission for each step. Agent Sandbox fits because it gives that environment a stable identity, its own storage and a hibernate and resume cycle, and because the boundary around it is what makes the permissiveness inside it affordable.
The fence is the point. Broad rights inside a boundary that reaches nothing of yours is a sound arrangement. Broad rights in an ordinary pod on a shared kernel is the same agent with the fence missing.
Tasks your application dispatches
The other shape is an application that needs to run agents as part of what it does, and the first instinct is to run them in the application's own container. That is the arrangement to avoid: the agent now shares a process space, a filesystem and a set of credentials with the service that called it. AX exists for this case. The application declares a Task, AX sandboxes it, assembles the Workspace from the repositories, MCP servers and skills it needs, fences egress through the Gateway and injects credentials there rather than handing them to the agent.
These tasks are short, numerous and bursty, which is also why the runtime underneath them multiplexes. A pod per task would spend most of its life idle, and that is the bill the Substrate layer is built to remove.
The two cases side by side
A harness a person works inside | Tasks an application dispatches | |
|---|---|---|
Who starts it | A developer or an end user | Your application, programmatically |
How many at once | Few, and long-lived | Many, short and bursty |
State | A working directory somebody returns to | Per task, discarded after |
What you want inside | Broad rights, fenced from outside | Only the tools the task declares |
Fits | Agent Sandbox | AX on Agent Substrate |
A platform that serves both will end up running both, and that is a reasonable outcome rather than a failure to standardise. They are answers to different questions that happen to share an isolation technology.
Worth noting what neither case includes. An agent that needs to change your infrastructure does not get rights to do it, however good the sandbox is. The boundary makes broad rights safe inside the environment, and it says nothing about what that environment is allowed to reach. Those are separate decisions, and the second one belongs to your delivery path rather than to a runtime class.
What the path has to emit instead
Whichever runtime you pick, the path has to emit it, and that is five separate pieces of work rather than one. Each layer below is a place where the default today produces an ordinary pod, and where somebody has to decide what the agent variant looks like instead.
Five layers, five gaps. None of them is hard on its own, and none of them exists today.
Scaffolding
The service template emits a Deployment, a Service and a chart. It has no parameter that produces a Sandbox or a SandboxTemplate, and adding one is not a cosmetic change: the generated repository layout assumes a replicated workload with a horizontal scaling story.
Pipeline and infrastructure
The pipeline changes least, and it does change. The image has to run under the runtime class you picked, which is a real constraint with gVisor, where a workload reaching for an unusual kernel feature fails at runtime rather than at build time. That belongs in a pipeline check, not in an incident.
Infrastructure defaults move further. A service template that provisions a database claim per service has to learn to provision storage per sandbox instead, with a lifecycle that matches the session rather than the service. The pattern is the one we use for ephemeral environments, where every piece is a manifest keyed on the thing it belongs to, described in how to build and optimize dynamic preview environments.
Policy
A default-deny NetworkPolicy is the right starting point and it is also the layer that breaks first. An agent runtime exists to call things: your APIs, a model endpoint, possibly a package registry. Each of those is an egress rule that someone has to write down, review and keep current.
The temptation is to open egress broadly and move on. That converts a workload chosen for its isolation into the least constrained thing in the namespace.
Observability
Dashboards keyed on a deployment name aggregate across replicas. Sandboxes are singletons with distinct names, so the same panel either shows one line per session or collapses into something meaningless. Per-session cardinality is a design decision to make before the first pool, rather than a surprise on the metrics bill.
Storage and backups
Stable identity with optional persistent storage means a volume per sandbox. That is a different backup conversation from the one most teams have had, because the data is small, numerous and short-lived, which is the opposite profile to the databases we discussed in why we use CNPG instead of a generic provisioner. Deciding that sandbox volumes are explicitly not backed up is a legitimate answer, and it should be written down rather than assumed.
The comparison worth making is with preview environments
A sandbox and a preview environment are both ephemeral, both provisioned on demand and both destroyed when their reason for existing goes away. If you have already built the second one, as we did for Greenstep, the instinct is to reuse that machinery. Most of it does transfer. One row in the table below does not, and it is the row that decides how much policy work a sandbox needs.
Preview environment | Agent sandbox | |
|---|---|---|
Lifetime | Tied to a change request, hours to days | Tied to a session, minutes to hours |
Created by | A pull request opening | A claim, often from a warm pool |
Teardown | Cascade delete of the manifests keyed on the change request | The sandbox controller, on delete or hibernate |
Who reviews it | A human looking at the preview | Nobody. The session is the user |
Concurrency control | A cap the platform team sets | Pool size and namespace quota |
The row that matters is the fourth one. A preview exists so a person can look at something, which is why we argued in preview environments beyond a namespace per pull request that seed data decides whether previews get used at all. A sandbox has no reviewer. Nobody is going to notice that it came up wrong, which puts the entire burden on policy and on the template rather than on observation.
When this is premature
For most teams it is, and saying so is more useful than a roadmap.
You have no agent workloads. A sandbox is a runtime for something. If nothing in your product runs model-driven sessions, there is nothing to put in it.
Your agents call fixed APIs with a fixed tool set. That is an ordinary service. Network policy and a scoped service account do the job, and a second kernel buys you nothing.
You run one cluster and a handful of services. The overhead of a new CRD, a runtime class and a pool is larger than the problem it solves.
The API is at v1beta1 and parts of the roadmap, including deep hibernation and cross-sandbox memory sharing, are still described as goals. Build against it when you need it, not before.
Two stacks are competing for the same job, one at v1beta1 and one pre-1.0 with no compatibility guarantees. Picking now means picking before the question has settled.
This is the same reasoning we applied to a universal control plane in when a Terraform module library beats a Kubernetes control plane. A new abstraction earns its place by removing more work than it adds, and the honest version of a recommendation names the case against.
Why this stops being optional
The workloads are arriving whether or not the path has a shape for them. The CNCF annual survey found that 66% of organisations hosting generative AI models use Kubernetes for some or all of their inference workloads, and CNCF added an AI inference and agentic track to KubeCon North America this year, which is the usual sign that a category has stopped being experimental.
The practical consequence is that the decision gets made either way. If the platform team does not make it, it gets made by whichever developer needs an agent first, in a pull request nobody reads as an isolation decision.
Coverage of the project has moved quickly, including the claim that the sandbox is becoming the unit of agent deployment. We would treat the stronger market claims in that genre with the scepticism they deserve, and treat the upstream API as what it is: a well-shaped primitive at v1beta1 that fills a real gap.
Closing
The upstream work here is good and it is the easy part. Agent Sandbox names a workload shape Kubernetes did not have, Substrate answers the bill that shape creates, and both put a real boundary around code you did not write. What neither of them does is decide that agents on your platform get that boundary by default. That decision belongs to the path, and until it is made the default stands: an ordinary pod, a shared kernel and whatever egress the namespace already allowed.
The framing that matters here is not a restriction. Platform teams are not in the business of telling developers they may not run agents, and a rule to that effect would produce a workaround and a worse estate. The job is to make the isolated option the one that is already there when somebody asks, so the crutch never gets built and nobody has to go looking for it later. That only works while the paved version stays easier than the alternative, which is the same bar every other part of the path has to clear.
The cheap move is to answer three questions before anyone asks them: which runtime class you would standardise on, what the egress allowlist contains, and whether sandbox storage is backed up. Those three answers are an afternoon of work now, and they are what turns the next agent request into a template parameter rather than an exception.
Picking between the two is easier than the comparison tables suggest, because the question is not which runtime is better. It is whether a person works inside the agent or your application dispatches work to it. Long-lived environments somebody returns to want a pod each and the identity that comes with it. Short tasks your code submits want multiplexing, a declared workspace and an egress list, and they want to run anywhere other than inside your application container.
If agent workloads are starting to appear on a platform you already run, we are happy to look at what the path offers them today and what it would take to make isolation the default.
Deciding what your path offers an agent
We are happy to look at what the path offers them today and what it would take to make isolation the default.
SHARE ON SOCIAL MEDIA
