Argo CD Syncs Desired State. It Doesn't Coordinate Delivery

Argo CD does one job and does it well. It takes the desired state declared in Git and makes the cluster match it. That job is narrow, well defined, and the project has been consistent about keeping it that way.

The trouble starts when a team assumes that job is the same as delivery. Reconciliation answers one question: does the cluster match Git? Delivery answers a different one: is this change safely in production, in the right order, with its data, its secrets, and the infrastructure it depends on? Argo CD answers the first and stays silent on the second, by design.

This article walks that boundary: what Argo CD actually reconciles, what it deliberately leaves out, the failure modes teams hit when they assume otherwise, and, most of the point, how we close that gap in practice without asking Argo CD to be something it is not. The short version of the answer is that the missing coordination gets modeled as manifests Argo CD applies, not as scripts bolted on beside it. The rest of the article earns that sentence.

What Argo CD actually does, and does well

The common mental model is that Argo CD watches Git continuously. It does not. By default the repo-server polls the repository on a timer (timeout.reconciliation, 180 seconds out of the box). Webhooks exist to skip the wait and trigger reconciliation the moment a commit lands. Inside the cluster the application-controller does watch resources through the Kubernetes API and reacts to drift.

So the cluster side is watch-based and the Git side is poll-based, with webhooks as the optimization. This is worth stating plainly because "GitOps is instant" is a claim about your webhook configuration, not about Argo CD.

The loop itself is simple. Render the manifests from the source, compare the result to live state, mark the Application OutOfSync if they differ, and on sync apply the difference. Drift in the cluster is detected the same way and, with self-heal enabled, corrected. That is the whole product, and its narrowness is the reason it is dependable.

The primitives teams rely on

Four primitives carry almost every real setup:

• Sync waves. The argocd.argoproj.io/sync-wave annotation orders resources inside a single Application. Negative waves run first. Argo CD applies a wave, waits for the resources in it to report Healthy, then moves to the next one.

• Hooks. PreSync, Sync, PostSync, SyncFail and PostDelete run Jobs at defined points in the sync operation.

• Health checks. Built in for core types such as Deployment, Service and Ingress, and extensible with Lua for custom resources.

• ApplicationSets. Generate Applications from a template using list, cluster, git, pullRequest, matrix or merge generators.

Those four cover a great deal of ground. They also draw the ceiling, and the rest of this article is about where that ceiling sits. Everything below is checked against the Argo CD documentation.

Argo CD reconciles the middle box. Everything above it, below it, and after it is delivery, and none of that is in Argo CD's state model.

Where reconciliation ends and delivery begins

Sync succeeded means two things: the manifests rendered from Git were applied, and the resources Argo CD knows how to health-check reported Healthy. That is a statement about convergence.

It does not mean the artifact running in production is the one that passed staging. It does not mean the database has the schema this code expects. It does not mean the service this one calls has been deployed, or that the secrets are the right ones for this environment, or that the previous version can be rolled back to safely. Those are all release properties, and Argo CD has no opinion about any of them because none of them are cluster state.

Where the rest of the lifecycle currently lives

If Argo CD does not own those stages, something else does. In most teams the answer is a mix of CI jobs, shell scripts, and knowledge that lives in two people's heads.

Read that table as a whole and the shape of the problem appears. Argo CD occupies one row cleanly. The rows above and below it are owned by processes with no shared model, no shared history, and no single place to ask what happened.

Failure modes teams actually hit

Argo CD syncs whatever is at the configured path and revision: spec.source.path and spec.source.targetRevision. Promotion, therefore, means changing what is at that path. There are four common ways to do it and each one has a specific cost.

Four promotion mechanisms in common use. Each works. None of them models the artifact as something with an identity that moves through stages.

CI edits the manifest. A pipeline bumps the image digest and commits, or opens a pull request. This works, and it is the most honest of the four because the change is visible in Git. What is missing is the reason: the commit records that a digest changed, not that this artifact passed staging and was cleared for production. The promotion decision lives in a CI job's logs, which have a retention policy.

Moving a Git tag. The Application points at a tag such as prod-stable and the pipeline moves the tag forward on promotion. This is the one that hurts later. The tag stops mapping to a stable history, so rollback means moving the tag back, which is a rewrite of what production meant rather than a revert. Argo CD's own sync history still shows what it did, but Git no longer tells you what was in production last Tuesday. The long-running community discussion on promotion is largely a catalogue of teams discovering this.

A branch per environment. Merge conflicts become promotion conflicts, and environment-specific configuration drifts into the branch where it is invisible until the next merge. The mechanism is simple to explain and expensive to operate.

A values file per environment. The pipeline writes the version into values-prod.yaml. Functionally the same as editing the manifest, with more files that can disagree with each other.

The shared gap underneath all four is the same. None of them treats the artifact as an object with an identity that moves through stages under conditions. Argo CD has a path and a revision. It does not have a concept of "this exact digest passed staging on this date, approved by this person, and is now cleared for production."

A promotion model needs five things: artifact identity based on a digest rather than a mutable tag, an explicit stage definition, a gate describing what must be true to advance, a promotion history you can query, and rollback as a first-class operation instead of a Git rewrite. None of those are Argo CD objects, and asking for them to be would be asking Argo CD to become a different product.

App and infrastructure coordination: readiness, not just apply-order

This one is widely described and usually described wrong, so it is worth being precise.

Sync waves do wait for health. Argo CD applies a wave, then blocks until the resources in it are Healthy before starting the next. The gap is not that waves ignore readiness. The gap is in what Healthy means. Argo CD's health model is per resource type: it ships checks for built-in types and supports Lua health checks for custom ones. When no health check is registered for a type, the resource is assumed to be Healthy.

There are two places to put the logic that closes this, and the choice is architectural rather than cosmetic.

Whichever side you choose, the point holds. Argo CD does not coordinate this. It hands you a wave number and a health check interface. The dependency logic is yours either way, and the only real mistake is not deciding on purpose.

Database migrations: what makes hook-based migrations safe

The standard warning is that a PreSync hook running a migration Job fires on every sync rather than on first deploy, and will therefore either slow every deployment or re-run a migration and corrupt the database. That warning is half right, and the half that is wrong matters. It fires on every sync. That is safe, and it is worth walking through exactly why, because the reasons are the same ones that decide whether your own setup is safe.

Start with the mechanics. Argo CD translates Helm hooks into its own hooks, and the mapping is not one to one:

So a Job annotated helm.sh/hook: pre-upgrade,pre-install runs on every sync. Argo CD has no install-versus-upgrade distinction to work with; it has a sync. Both annotations collapse into PreSync. In practice you pin the order and the cleanup with two more annotations: hook-weight: "-1" runs the migration ahead of the rest of the wave, and hook-delete-policy: before-hook-creation removes the previous migration Job before the next one is created, so identically named Jobs never collide.

And that is fine, for a reason that has nothing to do with luck. The delivery layer does not know what changed in the application or what the new code needs. It cannot know, so it runs the migration on every sync by default. What makes that unconditional re-run safe is that any migration tool worth using tracks what it has already applied, in a history table of its own. The Job starts, connects, finds nothing new, exits zero. The cost is a few seconds, and in exchange the migration is guaranteed to have run before the new code starts. The re-run is a no-op, not a hazard.

Two migrations, two phases

The detail that most treatments of this topic miss is that there is not one migration, there are two, and they belong in different phases. Schema changes and data changes have opposite safety requirements, so running them at the same point is what actually causes the trouble people blame on the hook.

Schema migration runs before the new code as PreSync, so the schema has room for what is coming. Data migration runs after it as PostSync, so backfills never block the deploy. Every migration still runs on every sync; the tool's own history makes the re-run a no-op.

Schema migration runs before the new code, as PreSync. Annotated pre-install,pre-upgrade, it adds the columns and tables the new version expects and finishes before the rollout begins. It is small and fast by design, because all it does is make room.

Data migration runs after the new code, as PostSync. Annotated post-install,post-upgrade, the backfills and transforms run once the new code is already live and serving traffic. This is what keeps a heavy data job off the critical path: a backfill that takes twenty minutes takes them after the deploy is done, not while it is blocked waiting.

That split is most of the answer to the classic objection, "won't a slow migration block every sync?" A schema change that only adds is quick. The slow work is data, and data runs afterward, where its duration costs nothing to the deploy.

Why rollback stays safe: add-only

Argo CD can roll an Application back to a previous revision. That rolls back manifests. It does not roll back the schema, because the schema is not in the manifests. So in the general case a rollback leaves old code running against a newer schema, and whether that works is entirely a property of how the migration was written.

The rule that makes this a non-issue is to keep forward migrations add-only. A migration may add a column or a table; it does not drop or rename one. Anything destructive is done deliberately, later, through its own release, once nothing depends on the old shape any more. Because the schema has only ever grown, rolling the code back never lands on a schema that removed something the old code still needs. This is the expand-and-contract pattern stated as a rule a team can actually hold to: expand in the migration, contract only through a separate, intentional release.

Parity follows from the same discipline. The worry is that staging and production drift apart because migrations run per environment at different times. In practice the migration tool records which migrations ran in each environment's own database, so each environment converges to the same schema by replaying the same ordered history, not by hoping. Argo CD guarantees the same manifests; the migration tool, not Argo CD, is what guarantees the same schema.

So the honest conclusion is narrow. The hook is not the risk. A migration that is idempotent, split into schema-before and data-after, and add-only makes the hook a solved problem, and you should stop worrying about it. Where it does break, non-idempotent raw DDL or a destructive change smuggled into a forward migration, the hook is simply where a design decision that was already wrong becomes visible. The wider catalogue of Argo CD anti-patterns is worth reading with that lens: some entries describe genuine traps, and some describe tools being used exactly as intended.

Secrets: the state that cannot live in Git

Git is the source of truth, and secrets cannot sit in Git in plaintext, so something has to bridge that gap. We use sealed-secrets. The model is simple: encrypt with the cluster's public key, commit the SealedSecret manifest, and let the in-cluster controller decrypt it into a regular Secret. Ciphertext lives in Git, nothing external sits in the deploy path, and there is no store to authenticate against at sync time. For static secrets that is a good trade and we would make it again.

It also has a consequence that matters here, and it is a coordination consequence rather than a security one. The sealing key is per cluster. A SealedSecret sealed for cluster A does not decrypt on cluster B. That is exactly the security property you want, and it is also an operational obligation that Argo CD knows nothing about:

• A new cluster needs its secrets re-sealed, or the key restored from a backup you must have thought about in advance.

• Preview environments on a separate cluster need their own sealing pass, which is one more step in a lifecycle that is already mostly scripts.

• Rotating the sealing key means re-sealing everything that was sealed with the old one.

• Losing the key means losing every secret whose plaintext does not exist somewhere else.

Now watch how that interacts with reconciliation. Argo CD applies a SealedSecret the way it applies any other manifest. Whether the controller in that cluster can actually decrypt it is not something Argo CD checks, reports, or has an opinion about. If it cannot, you get a SealedSecret resource that synced successfully, a Secret that never appears, and a pod that crash-loops on a missing environment variable.

Sync succeeded. The release did not.

Cross-Application dependencies

Sync waves order resources inside one Application. Between Applications you have App-of-Apps and ApplicationSets, and neither is a dependency model.

With App-of-Apps, the parent's manifests are child Application resources, so you can annotate them with sync waves and get coarse ordering. Argo CD does have a health check for the Application type, so the parent will wait. That is genuinely useful and it is where most teams stop.

What you do not get is anything that expresses intent. There is no conditional dependency, no way to say deploy B only once A has been healthy for five minutes. There is no failure propagation: if A fails, nothing stops B, and nothing rolls back what already went out. And there is no dependency graph you can look at and reason about, only a directory tree that happens to imply an order.

ApplicationSets are further from a dependency model, deliberately. A generator produces Applications from a template and there is no ordering between the generated Applications at all. That is correct for a fleet of similar services across clusters, which is what the generators were built for. It is not a substitute for expressing that one thing needs another.

The practical question this leaves open is blast radius. When a shared chart in an App-of-Apps tree changes and something breaks, what else went out with it? The answer is derivable from the tree, if you know the tree. It is not stored anywhere as intent, so it lives with whoever built it.

Preview environments and their full lifecycle

The pullRequest generator watches a repository and creates an Application per open pull request, then deletes it when the PR closes. That works, it is genuinely good, and it is the smallest part of a preview environment.

A preview environment that a developer can actually use needs its own database, migrated and seeded with data that makes the application do something. It needs its secrets sealed for whichever cluster it landed on, or generated there. It needs DNS, a certificate, an ingress, and usually some auth in front of it. And when it goes away, all of that has to go away with it. None of this is exotic, and none of it has to live in a script off to the side.

The seeding question alone is a design decision most teams make by accident. Fixtures are fast and unrealistic. An anonymized production dump is realistic and slow, and it is a compliance question the moment it exists. Synthetic data is the middle ground, and somebody has to maintain the generator. Argo CD does not make that choice for you, but where the choice gets expressed matters: as a manifest the delivery layer applies, not as a script bolted on beside it. That is the difference between a seeding step teardown can see and one it cannot.

The failure mode is not that these steps are hard. It is where they live. When each of them is a hand-rolled CI script that provisions something out of band, teardown cannot see it, the logic is understood by two people, and the setup quietly rots when those two move on. When instead each of them is expressed declaratively, as part of the application's own Helm chart and as helper resources that Argo CD applies, they are configuration like everything else: versioned, reviewed, and deleted by the same cascade that removes the environment. The difference between a preview setup that survives a team change and one that does not is almost entirely whether these steps are manifests or scripts.

Concretely, the pattern that works is to let the delivery layer generate the setup rather than script it. An ApplicationSet produces, alongside the application's own Application, a helper Application per preview whose job is to stand up exactly those dependencies: the database, the mail and messaging configuration, the seeded state, the per-environment secrets. It is all manifests, keyed on the same pull request, so it is created and destroyed as one unit. This is a technique worth knowing precisely because it turns the list above from a pile of glue into ordinary, reviewable configuration.

Add up what we have walked through. To ship one change, a developer's work passes through a repository for code, a repository or path for manifests, a CI pipeline that builds and bumps a digest, a promotion mechanism that is a pipeline step or a moved ref, a migration that runs as a hook, a secret that must be sealed for the right cluster, and a preview environment made of Argo CD plus three scripts.

Argo CD is one box in that chain. It is also the box with a user interface, which is why it gets blamed when something upstream or downstream of it fails. The question "who owns this step" has a clear answer for maybe half of those items. That ambiguity is the coordination gap in the form the whole organization can feel, and it shows up as delivery friction long before anyone identifies it as an architecture problem.

Why this reflects Argo CD's design, not a shortcoming

Everything listed above requires state that is not in the cluster. Which artifact passed which test. What the schema looked like before this deploy. Which environment is downstream of which. Who approved, when, and on what evidence.

For Argo CD to model those, it would need to own a database of delivery history and a set of opinions about how software should move through stages. It would stop being a reconciler and become a pipeline engine. The project has consistently declined to do that, and that refusal is precisely why it is reliable, why it composes with whatever else you run, and why it has outlasted several tools that tried to own the whole lifecycle.

Deployment engine versus delivery control plane

A deployment engine answers one question: make the cluster look like this. A delivery control plane answers a different set: what should be where, in what order, under which conditions, and what actually happened.

Those are different jobs with different state models and different failure characteristics. Conflating them is what produces the CI job that moves a tag, which is a pipeline engine implemented in twelve lines of bash and owned by nobody.

What coordination looks like when it is modeled

There is a simple test for whether something is modeled rather than implied. Can you query it?

• Which digest is in production, and which stage did it last pass?

• What depends on this service, and what breaks if it goes out first?

• Which preview environments exist right now, who owns them, and when do they expire?

• When did the schema in staging last diverge from production, and why?

If the answer to any of those is "grep the CI logs" or "ask Dmytro", that thing is not modeled. It is a side effect of a process, and side effects cannot be queried, audited, or handed to a new engineer.

Modeling means those are objects with identity, state and history, and in an Argo CD world the most direct way to get that is to keep them inside the thing Argo CD already manages. Concretely: an ApplicationSet generates, per environment, a helper Application whose only job is to stand up and record the dependencies that environment needs, the database, the messaging and mail configuration, the seeded data, the per-environment secrets. Because it is a real Application, its existence, state and sync history are queryable the same way any other Application is. You did not buy a coordination tool; you expressed the coordination as manifests and let the reconciler you already run make it observable. Git stays the source of truth, and the answer to "which environments exist and what did they provision" stops being a question you grep the CI logs for.

Git stays the source of truth; the coordination becomes observable

The tempting mistake, once a team accepts there is a coordination layer, is to start moving state out of Git and into it. That trades one opaque system for another.

Git stays the source of truth for desired state. The coordination layer records decisions and history: what was promoted, by whom, against what evidence, what it depended on, and what happened next. Argo CD keeps reconciling exactly as before. The layer above explains why the desired state is what it is, which is the question Git alone has never been able to answer.

What stays with Argo CD, and what sits above and below it

That third row is the one most discussions of this topic leave out. Not every gap should be closed by moving logic up into the platform. Some of it belongs in the application, and putting it there deliberately is what keeps the delivery layer thin enough to stay reliable. A platform that absorbs every responsibility becomes exactly as fragile as the scripts it replaced, with worse documentation.

Where to draw the boundary

Argo CD is not the problem here, and replacing it is not the answer to anything. It is a good reconciler that has stayed a good reconciler by refusing to grow into a pipeline engine, and the discipline behind that refusal is why it is worth building on.

The actual work is deciding, explicitly, where each part of delivery lives: in Argo CD, in the application, or in a coordination layer that exists whether or not anyone designed it. Most teams have never made that decision as a decision. They accumulated it, one reasonable CI job at a time, and ended up with a delivery path that works, that nobody can draw on a whiteboard, and that has a single point of failure wearing a hoodie.

Drawing that boundary on purpose is usually the first genuinely useful thing a platform team can do. Everything after it, including whether you need anything new at all, gets easier once the map exists.