Sooner or later a platform team gets the request: every product team wants its own cluster. Sometimes the reason is security, sometimes autonomy, and quite often the real reason is that someone once deleted someone else's deployment and nobody wants to repeat that week.

Before choosing a design, ask a blunter question: isolated against what? A team that accidentally deploys into the wrong namespace and a tenant who is actively trying to read another tenant's secrets are two different problems. They justify very different amounts of infrastructure. Most of the cluster-per-team requests we see are the first problem, asked for with the tooling of the second.

This article walks through the layers Argo CD and Kubernetes give you, from AppProject to separate clusters, what each one really enforces, and where we have landed for our own clients.

The question behind the question

Autonomy versus a blast radius you can name

Teams want to ship without filing tickets. Platform teams want to be able to say, when something goes wrong, exactly how far it could have spread. Both are reasonable, and a separate cluster delivers both in the crudest possible way. The cost shows up later, in upgrades and add-ons nobody budgeted for.

Define isolation before the design

Write down what a tenant must not be able to do. Typical answers fall into three levels:

  1. Accidental damage: a team must not deploy into, or delete from, another team's namespace by mistake.

  2. Curious or careless access: a developer must not be able to read another team's secrets or logs, even if they try.

  3. Hostile tenant: a workload must not be able to affect others even if its code is malicious, including attempts to escape the container.

Each level below needs a different layer. Most internal platforms need the first two. Very few need the third, and those usually know it already because a regulator or a contract told them.

What AppProject actually does

AppProject gets criticised a lot, usually by people describing it inaccurately, so first the accurate version. It is the grouping Argo CD uses for Applications, and it restricts several things:

  • Sources: sourceRepos limits which Git repositories an Application in the project may deploy from, with allow and deny patterns.

  • Destinations: destinations limits which clusters and namespaces it may deploy to.

  • Resource kinds: clusterResourceWhitelist controls which cluster-scoped kinds are allowed at all, and namespaceResourceBlacklist removes namespace-scoped kinds a team should not create.

  • Who may act: project roles, together with Argo CD RBAC, decide which users and groups can view, sync or edit the project's Applications.

  • Which clusters exist for the project: project-scoped repositories and clusters, plus permitOnlyProjectScopedClusters, keep one team's credentials out of another team's projects.

A typical tenant project looks like this:

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: team-a
  namespace: argocd
spec:
  sourceRepos:
    - https://github.com/acme/team-a-*
  destinations:
    - server: https://kubernetes.default.svc
      namespace: team-a-*
  clusterResourceWhitelist: []      # no cluster-scoped objects
  namespaceResourceBlacklist:
    - group: ''
      kind: ResourceQuota
    - group: networking.k8s.io
      kind

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: team-a
  namespace: argocd
spec:
  sourceRepos:
    - https://github.com/acme/team-a-*
  destinations:
    - server: https://kubernetes.default.svc
      namespace: team-a-*
  clusterResourceWhitelist: []      # no cluster-scoped objects
  namespaceResourceBlacklist:
    - group: ''
      kind: ResourceQuota
    - group: networking.k8s.io
      kind

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: team-a
  namespace: argocd
spec:
  sourceRepos:
    - https://github.com/acme/team-a-*
  destinations:
    - server: https://kubernetes.default.svc
      namespace: team-a-*
  clusterResourceWhitelist: []      # no cluster-scoped objects
  namespaceResourceBlacklist:
    - group: ''
      kind: ResourceQuota
    - group: networking.k8s.io
      kind

That prevents a large class of real mistakes: the copy-pasted Application that still points at another team's namespace, the chart that quietly ships a ClusterRole, the team that edits its own quota away. For level 1 isolation it does most of the work. The default project, which allows any source, any destination and any kind, is what you should get rid of first.

Where the boundary ends

All of those checks are made by Argo CD, against the Application object, before the sync. The sync itself is a different matter.

The credential problem


When an Application passes the project checks, the Argo CD application controller applies the manifests. It does that with its own service account. In a default installation that service account can do practically anything in the cluster, because Argo CD has to be able to deploy whatever any project might allow.

So AppProject constrains what a tenant may declare. It does not constrain what the controller does once the declaration is accepted. The Kubernetes API server never learns that this particular sync was for team A. It sees the controller, and the controller is allowed. A bug, a misconfigured project or a gap in your patterns is then checked by nothing else.

AppProject is a policy boundary: Argo CD's own rule check in front of a powerful identity. A security boundary is one that Kubernetes enforces regardless of which tool sends the request, and that means RBAC applied to the identity doing the work. That distinction drives everything below.

Impersonation, for syncs and for people

Kubernetes has a built-in mechanism, user impersonation, for an authenticated client to act as another user or service account, with RBAC then evaluated for that impersonated identity. It shows up in multi-tenant setups in two places.

Syncs that run as the tenant

Argo CD can sync using impersonation. With it enabled, the controller performs the sync as a service account in the destination cluster that you name per project and namespace. As of Argo CD 3.5 the documentation marks the feature as beta. It is switched on system-wide in argocd-cm, and each project then maps destinations to deployer service accounts:

# argocd-cm
application.sync.impersonation.enabled: 'true'
 
# AppProject team-a
spec:
  destinationServiceAccounts:
    - server: https://kubernetes.default.svc
      namespace: team-a-*
      defaultServiceAccount

# argocd-cm
application.sync.impersonation.enabled: 'true'
 
# AppProject team-a
spec:
  destinationServiceAccounts:
    - server: https://kubernetes.default.svc
      namespace: team-a-*
      defaultServiceAccount

# argocd-cm
application.sync.impersonation.enabled: 'true'
 
# AppProject team-a
spec:
  destinationServiceAccounts:
    - server: https://kubernetes.default.svc
      namespace: team-a-*
      defaultServiceAccount

Now the sync can only do what team-a-deployer is allowed to do by Roles in that namespace. If the project configuration is wrong, Kubernetes still says no. Enforcement is on by default, so a project without a matching service account fails its sync instead of quietly falling back to the controller's identity.

What it does not solve:

  • It covers the lifecycle of objects an Application manages: sync, prune, deletion and actions from the UI. Read the documentation for your version before assuming anything else runs as the tenant.

  • You now maintain a deployer service account and its Roles for every tenant namespace. That is the point, and it is also work.

  • Nothing changes on the node layer. A namespace with perfect RBAC still shares nodes, kernel and network with its neighbours.

People who log in as themselves

The second place matters at least as much in practice: developers looking at their workloads with kubectl. In the clusters we operate, access goes through the company's identity provider. We have used kube-oidc-proxy for this: the developer signs in with OIDC, the proxy authenticates them and forwards the request to the API server, impersonating that user. Namespace Roles then decide what each person can read or change, and teams cannot see what is in other teams' namespaces.

It is a convenient pattern, because access follows the identity provider and nobody hands out shared kubeconfigs. One thing to know before you copy it: the original Jetstack repository was archived in May 2024 and points to a fork maintained by Tremolo Security, so check which build you deploy.

Both kinds of impersonation move the decision to RBAC in the destination cluster, which is where Kubernetes can enforce it.

The layout we actually run

Here is where most of our clients sit today, because it answers the question this article started with.


  • Development: one shared cluster for the development teams, separated by namespaces. Teams cannot see what runs in another team's namespace.

  • Production: two separate clusters per project, one for prod and one for beta, again with namespaces inside each.

  • People: OIDC login and namespace Roles. When developers need more access, we describe the role they need and add it, instead of widening access for everyone.

Clusters split by environment and project. Teams split by namespace and RBAC. The reason is practical: the environment boundary is the one where a mistake costs real money, and it is also the boundary where lifecycles differ, since prod and beta upgrade on different schedules. Inside an environment, the teams of one client are cooperative tenants, and namespaces handle them well.

Everything in those clusters arrives through Argo CD from Git, including the Roles and the AppProjects themselves, so a new tenant is a pull request. The same principle is why an AI agent gets a pull request instead of cluster access, and why Sealed Secrets fit this layout: by default a sealed secret only decrypts in the namespace and under the name it was sealed for, so a team cannot reuse another team's encrypted value.

A cluster per tenant, honestly costed

It does isolate

Separate clusters give each tenant its own API server, its own etcd, its own nodes and its own network. There is no argument about whether that isolates. The argument is about price.

Everything multiplied

Each cluster is a control plane to patch and upgrade, plus the full set of add-ons that make it usable: ingress, DNS, certificates, monitoring, logging, secrets delivery, policy, backups, and the Argo CD registration for all of it. For three clusters that is manageable. For thirty, keeping versions aligned and upgrades current turns into a second full-time job, and the clusters that fall behind are the ones nobody looks at.

We also see the opposite arrangement work. When a client gives a team its own cluster, it is usually simpler for that team to run it themselves: their cluster, their upgrades, their on-call. When we operate the cluster, we slice it ourselves and restrict access per namespace. Mixing the two, a dedicated cluster per team that a central platform team still has to maintain, is the expensive combination.

When it is the right answer

  • The threat model is hostile: external customers, untrusted code, anything at level 3.

  • A regulator or contract requires hard separation of data or infrastructure.

  • The workloads have genuinely independent lifecycles: different Kubernetes versions, different cloud accounts, different owners who will run them.

vCluster as the middle path

vCluster gives each tenant a virtual cluster: its own API server and control plane, running as a pod in a namespace of a host cluster. Tenants see what looks like a full cluster. A syncer translates their workloads, such as Pods, Services, ConfigMaps and Secrets, into a namespace on the host, where they actually run.

What it isolates and what still leaks

It solves the problem namespaces cannot: tenants get their own CRDs, cluster-scoped objects and RBAC. A team that needs to install an operator, or wants cluster-admin inside its own world, can have it without touching anyone else's API.

The node layer is another matter. In the shared-nodes model, which is the only one vCluster OSS provides, the vCluster architecture page says it directly: tenants share the same kernel and worker nodes, and shared nodes are not a security boundary for untrusted tenants. Anything that escapes the container escapes the tenant. For untrusted tenants vCluster points to its private-nodes model, which needs a license.

The operational surface

A virtual control plane per tenant is still a control plane per tenant: a backing store (embedded SQLite by default, etcd or an external database as options), versions to upgrade, a syncer whose behaviour you need to understand when something does not show up on the host.

To be transparent: we have not run vCluster in production for a client. So far there has been no case where tenants needed their own cluster-scoped objects badly enough to justify it, and namespaces with RBAC covered what they did need. What we write here comes from the documentation, and that is the honest place to start your own evaluation.

Choosing

Start from the threat model. A new department does not need a new cluster. A new threat might.


Option

What enforces isolation

What stays shared

Use it when

Namespaces + AppProject

Argo CD policy checks; namespace RBAC for people

Controller identity, nodes, network

Cooperative teams, accidental damage is the risk

+ sync impersonation

Kubernetes RBAC for each tenant's deployer

Nodes, network

Syncs must be limited by rules the cluster enforces

vCluster

Separate API server, CRDs and RBAC

Nodes and kernel (shared-nodes model)

Tenants need their own cluster-scoped objects

Separate clusters

Everything separate

Only your platform tooling

Hostile tenants, regulation, independent lifecycles

Most internal platforms we work with sit in the first row, with sync impersonation as the next step when deployments need a limit the cluster itself enforces. Our own layout adds one deliberate exception: separate clusters per environment and project, because there the lifecycles really are independent.

Name what you are isolating against, then pick

If the answer is mistakes, a well-scoped AppProject, namespace Roles and OIDC access will carry you far, and adding sync impersonation moves the remaining trust from Argo CD's checks to Kubernetes RBAC. If the answer is tenants who need their own APIs, look at vCluster and budget for its control planes. If the answer is someone hostile, separate clusters are worth their bill.

If you are deciding how to split clusters and teams on your own platform, talk to our engineers.

Oleksandr Simonov

Founder and CEO @ Amoniac OÜ

Working with Linux since 1998. I've seen every layer of the stack break in every possible way — so when you describe your problem, I usually know where to look before the call even starts.

Oleksandr Simonov

Founder and CEO @ Amoniac OÜ

Working with Linux since 1998. I've seen every layer of the stack break in every possible way — so when you describe your problem, I usually know where to look before the call even starts.

Oleksandr Simonov

Founder and CEO @ Amoniac OÜ

Working with Linux since 1998. I've seen every layer of the stack break in every possible way — so when you describe your problem, I usually know where to look before the call even starts.

SHARE ON SOCIAL MEDIA

Start with a focused platform assessment

Our first engagement is a paid technical assessment. We map your delivery workflow, infrastructure gaps, platform debt, and implementation priorities before anything is built.

This is a paid first engagement, designed to produce a roadmap your team can act on.