The Argo CD pullRequest generator watches a repository, creates an Application for every open pull request, and removes it when the pull request closes. It is reliable, it is about fifteen lines of YAML, and it solves the deployment step completely.
The deployment step is not the hard part. For a reviewer to actually exercise the change, that environment needs data worth clicking on, credentials that work, a certificate that validates, and isolation from every other open pull request. Each of those is a separate design decision, and each one has a wrong answer that only shows up two months in, usually as a cloud bill or a preview nobody trusts.
Worth putting a number on why this is worth the effort. Our QA team, having worked both ways, with deployments to fixed staging servers and with dynamic preview environments, puts the difference at roughly two to three times less time spent, provided there are enough resources to spin the previews up. That last clause matters: the saving is real, and it is conditional on the infrastructure actually being there, which is what the rest of this article is about.
This article walks through the decisions in the order you hit them, with the trade-offs and the option we pick in each case. Where something is a genuine trade-off rather than a rule, we say which way we go and why.
What a preview environment has to do
Start from the acceptance test, because it decides everything downstream. A preview environment works when a reviewer can reach the running change, usually at a URL, exercise it against realistic data, in isolation from every other open pull request, without asking anyone for anything, and the environment then disappears on its own when the pull request is done.
Six requirements fall out of that sentence.

The pullRequest generator delivers the deployment. The six requirements above sit on top of it, and each one is an independent decision.
The rest of this article is those six, in build order. Isolation first, because it constrains the others. Then data, which is the requirement that decides whether anyone bothers using previews at all. After that come secrets, reachability, lifecycle, and cost.
Isolation: namespace per pull request, or a virtual cluster
A namespace per pull request is the default for good reasons. It provisions in seconds, it costs nothing beyond the workloads inside it, and it is a boundary Kubernetes already understands. For most changes it is the right answer and you should not reach past it.
What matters is knowing precisely what it does and does not contain.

A namespace contains namespaced objects. Cluster-scoped objects stay shared, which is where two concurrent previews can collide.
Namespaced objects are contained: Deployments, Services, ConfigMaps, Secrets, PersistentVolumeClaims, Ingress objects, RoleBindings, NetworkPolicies, ResourceQuotas. Two pull requests each get their own copy and never see each other's.
A namespace does not isolate cluster-scoped objects, and in day-to-day product work that rarely matters, because a feature branch almost never touches them. The exception is the one case where a virtual cluster earns its place: a pull request that changes something already installed cluster-wide, most commonly a new version of an operator or its CustomResourceDefinitions. A namespace cannot help you, because the CRD and the operator are shared by every environment on the cluster at once, and testing the upgrade in a namespace would mean changing them for everyone. That is the signal to reach for a virtual cluster, and close to the only one.
That distinction gives you the rule. If the change under review only touches namespaced objects, which is the large majority of application changes, a namespace isolates it properly. If the pull request bumps a CRD version, changes an operator, adds an admission webhook, or modifies cluster RBAC, then two concurrent previews are writing to the same cluster-scoped object, and the second one to sync wins. No amount of namespace configuration fixes that, because the object being changed does not live in a namespace.
When the answer is a virtual cluster
That case is what vCluster exists for. A virtual cluster runs its own API server, controller manager and data store inside a host cluster. Cluster-scoped objects created inside it are real to workloads inside it and invisible outside, so a reviewer can be granted cluster-admin within the preview without holding any privilege on the host. Two previews can install conflicting CRD versions and neither notices the other.
The cost is a control plane per environment. That is memory, CPU, and startup time per preview instead of a namespace that appears instantly, plus a syncer reconciling objects down to the host cluster, plus one more component in your platform to upgrade and debug. For a repository where most pull requests change application code, you would be paying that on every preview to serve the small percentage that needs it.
Namespace per PR | Virtual cluster per PR | |
|---|---|---|
Provisioning time | Seconds | Tens of seconds, plus control plane startup |
Isolates namespaced objects | Yes | Yes |
Isolates CRDs, cluster RBAC, webhooks | No, these stay shared | Yes, they live inside the virtual cluster |
Reviewer can hold cluster-admin | No, that would be on the host cluster | Yes, scoped to the virtual cluster |
Marginal cost per preview | The workloads only | The workloads plus a control plane |
Components to operate | None beyond Kubernetes | The virtual cluster and its syncer |
Where we land on thisNamespace per pull request as the default, with a virtual cluster reserved for changes that touch cluster-scoped objects. In practice that means the platform repository and the operator repositories get virtual clusters, and application repositories do not. Splitting the decision by repository rather than by pull request keeps it predictable, because a developer should not have to know which isolation model their change triggered. Whichever model you pick, put a ResourceQuota and a LimitRange on every preview namespace and a default-deny NetworkPolicy between them. Previews are the environment where somebody will deploy an accidental infinite loop, and a quota is what stops that from becoming everyone's problem. |
Data: the requirement that decides whether previews get used
A preview environment with no data is a screenshot with extra steps. A reviewer reaches it, finds empty lists and nothing to click through, and goes back to reading the diff. This is the most common reason preview environments get built and then quietly abandoned, and it deserves more design attention than the deployment mechanics that usually get all of it.
The failure mode to design against is a shared database. If every preview points at one staging database, a schema migration on one pull request changes the schema under every other open pull request, and seed data written by one test run corrupts the assertions of the next. That is not an isolation problem you can configure around. It is the absence of isolation.
Three mechanisms give a pull request its own data, and they differ in provisioning time, cost, and fidelity.

Approach | Isolation | Verdict |
|---|---|---|
Shared database, one for all previews | None | Bad practice. One PR's migration or data change breaks every other preview. Listed only so it can be ruled out. |
Schema or database per PR on one shared instance | Namespaces only, not resources | Bad practice for us. The instance is shared, so you cannot test anything that depends on the instance, query-plan or index changes read the shared server's state, not the PR's. |
Dedicated instance per PR, restored from a QA or staging snapshot | Full: own instance, own data | What we run. Realistic anonymized data and true isolation, so performance and migration behaviour match production. Costs an instance per preview, which the concurrency cap below bounds. |
Our default is a database per pull request on one shared instance, seeded from a size-capped anonymized dataset that is maintained deliberately rather than copied from production. That combination provisions in seconds, costs one instance regardless of how many previews are open, and tears down with a single statement. The seed is the part worth investing in, because the quality of that dataset is what determines whether reviewers use previews or ignore them.
Running the preview database inside the cluster, through an operator such as CloudNativePG, has a property that matters more than it first appears: the database becomes a manifest inside the same Application as the workload. That means it is created by the same sync and deleted by the same deletion, with no out-of-band provisioning step and nothing left behind. We come back to why that matters in the lifecycle section, because it is the single decision that removes most teardown problems before they exist.
One rule sits above all of these. A preview environment never points at production data. Anonymized or synthetic only. Preview URLs are widely shared, short-lived, and lightly guarded by design, which is exactly the wrong place for real customer records.
Migrations: only the preview-specific part
How we run migrations through Helm hooks is its own topic, and we cover the mechanics in a separate article. Here only one preview-specific consequence matters.
Forward migrations are add-only: a migration adds a column or a table, and anything destructive happens later through its own deliberate release. A preview gets rebuilt constantly, on every force-push, reopen, or recreate, and each rebuild replays the same ordered migration history on top of the same base. Because every step only adds and is safe to re-run, those rebuilds converge: the environment you get on the fourth rebuild matches the first. If migrations dropped things or were not safe to re-run, rebuilds would drift, and you would end up debugging the environment instead of the change it was built to show. That reproducibility is the only migration property this article needs.
Secrets: sealed for the cluster they land on
Git is the source of truth and secrets cannot sit in it in plaintext, so something has to bridge that. We use sealed-secrets: encrypt with the cluster's public key, commit the SealedSecret manifest, and let the in-cluster controller decrypt it into an ordinary Secret. Ciphertext lives in Git, nothing external sits in the deploy path, and there is no store to authenticate against at sync time.
Previews add one constraint to that model. The sealing key belongs to the cluster. A SealedSecret sealed for cluster A does not decrypt on cluster B, which is exactly the isolation you want and also an operational fact you have to plan around.
Same cluster is the easier half of that, but it is not the whole problem, and the part people miss turns into real debugging time the first time they meet it. What you actually lose the time to is this: a preview deploys, the sync goes green, and the pod sits in CrashLoopBackOff on a missing environment variable. The SealedSecret is right there in Git and applied to the cluster, so nothing looks wrong until you check the controller logs and find it refused to decrypt. The reason is scoping. The default scope is strict: the secret's name and its namespace are both folded into the ciphertext, which exists for a good reason, because otherwise anyone who can create objects in one namespace could take a SealedSecret meant for another, change the namespace field, and read the decrypted value out of their own.
A per-PR preview breaks that assumption directly, because the namespace does not exist when the secret is sealed. A SealedSecret sealed for preview-pr-811 will not decrypt in preview-pr-812. Same cluster, same controller, same key, and it still fails, with the controller logging that no key could decrypt the secret. The sync is green, the Secret never appears, and the pod sits in CrashLoopBackOff on a missing environment variable.
There are three ways out, and they are not equally good.
Generate rather than inherit. This is the right answer for almost everything. A preview needs credentials that work, not the credentials your other environments use. The database password, the signing key, the internal service token: generate them at provisioning time, straight into the namespace. Nothing is sealed, nothing is committed, nothing can leak from Git, and the dynamic namespace stops being a problem because no ciphertext was ever bound to a namespace name.
Widen the scope, deliberately and narrowly. For the few values a preview genuinely has to share, a sandbox API key for a third-party service, seal them with sealedsecrets.bitnami.com/cluster-wide: "true" so they decrypt into any namespace. Understand what that trades: you are switching off the protection described above, so it is defensible on a cluster that holds only preview environments and indefensible on a cluster that also holds anything you would mind losing.
Re-seal per environment. The pipeline runs kubeseal with the new namespace at creation time, keeping strict scope. It works, and it requires the plaintext to be available to the pipeline, which reintroduces the handling problem sealed-secrets was adopted to avoid. Worth it only when a value must be shared and must stay strictly scoped.
In practice the first option covers most of a preview environment, the second covers a short list you can name, and the third is rare. The useful question when designing this is not which tool to use but how many secrets a preview actually has to inherit rather than generate. For most services the honest answer is one or two, and once that is true the whole problem shrinks to something a provisioning step handles.
If previews run on their own cluster, that cluster needs its own sealing pass. Either the pipeline seals preview secrets against the preview cluster's public key at creation time, or you restore the controller's key to it from backup. Both work. The one that fails is assuming a SealedSecret is portable between clusters, and it fails at the least convenient moment.
The second rule is stricter and non-negotiable: a preview never receives production secret material. Preview environments get their own credential set, scoped to preview resources, sealed separately. The blast radius of a leaked preview credential should be a preview.
Reachability: DNS, TLS, and who can open the URL
A preview needs a predictable address. The pattern is a per-PR hostname under a dedicated subdomain, for example the pull request number under preview.example.com, so that the URL can be posted into the pull request automatically and nobody has to look anything up.
DNS
A wildcard record pointed at the ingress controller covers every preview with no per-PR DNS work, which is the simplest thing that works. Where records need to be real, for example because certificates are issued per host or because a wildcard is not acceptable in that zone, external-dns creates and removes records from Ingress objects automatically. We maintain the DigitalOcean webhook provider for external-dns, so this part is close to home for us: the important property is that record lifecycle follows object lifecycle, which means a deleted preview takes its DNS record with it instead of leaving a dangling name behind.
TLS
Do not issue a certificate per pull request. Let's Encrypt rate limits allow 50 certificates per registered domain per week and 300 new orders per account per three hours. A busy repository with a dozen previews a day will reach the weekly limit and then every new preview arrives with a browser warning, which is the kind of failure that makes people stop trusting the whole system.
Issue one wildcard certificate for the preview subdomain, once, and reference it from every preview Ingress. With that is a single Certificate resource using a DNS-01 solver, renewed automatically, and it removes per-PR certificate issuance from the critical path entirely. Use the DNS-01 challenge to issue it, not HTTP-01. It is the more secure option and, unlike HTTP-01, it does not need the preview to be reachable from the public internet, so it still works when previews are closed to the world and reachable only from specific IPs, which for internal previews they usually should be.
Who can open it
A preview URL is reachable and contains data, so it needs authentication in front of it by default rather than per application. An auth proxy tied to your identity provider, sitting in front of the preview ingress, gives you that: every preview is protected by construction and no individual team has to remember. The part an off-the-shelf proxy does not solve is the redirect URL. Most identity providers expect a fixed callback, and a preview hostname is different for every pull request, so a plain auth proxy authenticates but cannot make one registered redirect work across all of them.
That piece we handle with our own small proxy in front. It does not take authentication away from the application: the app keeps doing its own auth exactly as it always did. What the proxy does is rewrite the authorization requests internally, so that one registered redirect URL keeps working no matter which preview hostname the request came from.
The alternative, protecting previews one at a time, means the one that gets forgotten is the one that gets indexed.
Lifecycle: what gets deleted, and when
Teardown has to be automatic and safe, and those two requirements pull in different directions. Here is how we keep both.

What gets deleted
Closing the pull request removes the Application, and Argo CD deletes the resources it applied, following cascade deletion through the resource finalizer. Anything Argo CD applied is handled correctly and needs no further thought.
The thing that makes this clean is that we do not create preview resources out of band. Everything a preview needs is a manifest inside the Application: the namespace, the database as a custom resource through its operator, the secrets, the services, the ingress. Because Argo CD created all of it, closing the pull request and removing the Application deletes all of it too, the database and its storage, the DNS record, the certificate, the namespace, in one cascade. There is no separate cleanup step to forget, because there was no separate creation step to begin with.
That is the whole rule: if a preview needs it, it is a manifest in the Application. Anything created by a side script becomes something teardown cannot see, which is exactly the orphaned-bill problem, so we do not create anything that way.
When it gets deleted
The generator keys on pull request state, so a closed or merged pull request tears its environment down automatically. A pull request that stays open for weeks keeps its environment, and that is usually correct, not a leak: a long-lived preview is often one that several people are actively testing against, sometimes with test data set up specifically for a feature. Deleting that on a timer is how you erase someone's afternoon of setup, which is why we do not run an automatic reaper.
When an environment genuinely is finished before its pull request closes, the person who owns it removes a label from the pull request and the environment comes down. Teardown stays a deliberate act, by the pull request closing or by someone deciding it is done, never a scheduled job that deletes environments other people may still be using.
Because teardown is tied to the pull request, an environment removed by mistake is cheap to get back: reopen or re-sync and the Application rebuilds the preview from the same manifests and the same base, converging to the same state. Recoverability comes from everything being declarative, not from a trash can.
Cost: where it goes and how to cap it
The honest lever on preview cost is not squeezing each environment, it is not running environments you do not need. A dedicated instance per preview is the right call for isolation, and the way you keep that affordable is to bound how many previews exist at once, rather than trying to make an unbounded number of them cheap.
In order of how much they actually pay back:
Cap the number of concurrent previews. This is the one that matters. A limit on how many previews can run at once puts a ceiling on spend directly, and it has a second effect that is arguably more valuable: it pushes teams to finish testing one change and free the slot instead of leaving environments parked. Fewer, faster-moving previews beats many idle ones, for both the bill and the review cycle.
Scale idle previews down. Previews spend most of the day with nobody looking at them. Scaling their workloads down between uses, within whatever your platform supports, trims the cost of the ones that are alive without touching their data or their URL.
Quota each preview and place them on cheaper capacity. A ResourceQuota per preview caps the damage from a misconfigured workload, and a dedicated node pool or spot capacity lowers the unit cost of the environments you do run.
Underneath all of it sits attribution. Labelling each preview with its team, repository, and pull request at creation is what lets you answer who a given environment belongs to when the bill arrives, and what turns the concurrency cap from a blunt global number into per-team limits if you need them.
The reference shape
In practice we split this across two ApplicationSets keyed on the same pull request. The first sets up what the application depends on: the namespace, the database as a custom resource through its operator, the sealed or generated secrets, and the backing services. The second deploys the application itself, with its own ingress, and the application checks at startup that its database and dependencies are reachable rather than relying on the delivery layer to sequence them. Everything in both is a manifest, so closing the pull request removes all of it in one cascade.
The one thing that deliberately stays out of the delivery layer is readiness. The application decides when its dependencies are available and waits for them itself, which keeps startup ordering inside the app where it belongs and out of the platform.
Build order
Building all of it before shipping any of it is how these projects stall. The sequence below delivers something usable at every step, and each stage removes a specific complaint about the stage before it.
Stage | What you build | What it makes possible |
|---|---|---|
1 | ApplicationSet with the pullRequest generator, namespace per PR, quota and NetworkPolicy | A deployed branch on a URL. Reviewers can see the change running. |
2 | Wildcard DNS, one wildcard certificate, auth in front of the ingress class | A URL that is safe to post publicly in the pull request. |
3 | Database per PR as a manifest, schema migration PreSync, seed PostSync | A preview with data, which is the point at which people start using it. |
4 | A cap on concurrent previews, plus labels for team, repo, and PR | Cost stops growing with the number of open PRs, and every environment is attributable to a team. |
5 | Scale idle previews down, spot or cheap node pool | The remaining cost tracks actual use instead of the number of environments that exist. |
6 | Virtual clusters for repositories that change cluster-scoped objects | Platform and operator changes get previews too. |
Most teams get real value from stages one through three and can stop for a while. Stage four is what stops the finance conversation from arriving unannounced. Stage six is only worth doing if the repositories in question genuinely change cluster-scoped objects, and for a lot of teams they do not.
Closing
Preview environments fail for unglamorous reasons. The data was empty so nobody used them. The certificate expired so people learned to click through the warning. A secret did not decrypt and the sync went green anyway. The bill grew and no label existed to explain it. Every one of those has a concrete fix, and the fixes reinforce each other: making everything the preview needs a manifest solves teardown in one cascade, a cap on concurrent previews solves cost, and one wildcard certificate solves both the rate limit and the browser warning.
The generator gives you the deployment for fifteen lines of YAML. The remaining decisions are the ones worth making deliberately, in one place, written down, rather than accumulating as a set of pipeline scripts that two people understand and nobody owns.
Working on this now?We map what your preview path actually provisions, which parts are already manifests and which are scripts with no owner, and what the TTL and cost policy should be for your repository volume. Talk to an engineer. A technical conversation about your setup, not a sales call. |
SHARE ON SOCIAL MEDIA




