A sync goes green. The pull request is merged, the Application reports Synced, and the resource is there in the cluster when you go looking for it. Twenty minutes later someone posts a screenshot of a pod that has restarted forty times, and the conversation starts with the image tag, moves to the deployment spec, and gets to the actual cause about half an hour in. A SealedSecret was applied successfully and never decrypted, so the Secret it was supposed to produce does not exist.
That failure is not a bug in anything. It is the shape of the trade you accept when you put ciphertext in Git, and it is worth understanding before you have a hundred manifests depending on it.
Every GitOps setup hits the same wall in its first week. The model rests on one idea: the desired state lives in version control and a reconciler makes the cluster match it. Then someone needs a database password in a namespace, and the one category of state you cannot commit in plaintext is the one every workload needs.
We use bitnami/sealed-secrets as the default on the platforms we build. This article covers the mechanism in enough detail to debug it, the reasons we lead with it, the limits we accept, and the failure mode above in full.
The problem, stated once
Why plaintext in Git fails
A private repository feels like a safe place to put a secret. It is not, for three reasons that compound.
Git history is permanent in practice. Rewriting history removes a value from the tip of your branch, not from every clone, fork, mirror, CI cache and local checkout that already pulled it. A value committed once should be treated as a value that exists forever somewhere you do not control.
Read access is broader than the mental model. The people who can read the repository are not the people who should know production credentials. Add CI runners, contractors on a two-week engagement, backup systems and repository mirrors, and the audience is much larger than the team that approved it.
A leak is unrecoverable in a specific sense. You can rotate the value, and you should, but you cannot unpublish it. Every system that authenticated with the old value has to be updated before the exposure is closed, and that is a coordinated operation under time pressure. GitGuardian's State of Secrets Sprawl tracks how routinely this happens in public repositories, and the pattern in private ones is the same shape with a smaller audience.
One clarification that saves an argument later: a Kubernetes Secret is not encrypted either. It is base64-encoded in etcd unless you have configured encryption at rest. Everything below is about how the value travels and where it is stored outside the cluster, not about making the in-cluster Secret something it is not.
The four common answers
Four tools solve this in four different places. All of them are defensible, and the choice is about where you want the complexity to sit.
Approach | Where the ciphertext lives | In the deploy path at sync time | Dynamic credentials | Main operational cost |
|---|---|---|---|---|
sealed-secrets | In Git, as a SealedSecret | Nothing external | No | Private key lifecycle, per cluster |
SOPS | In Git, encrypted with an external KMS | The KMS or key backend | No | Key access management, decryption inside the delivery tool |
External Secrets Operator | Not in Git at all | The external secret manager | Partly, depending on the backend | Another operator plus an external store to run and secure |
Vault | Not in Git at all | Vault, on every fetch | Yes, this is the point | Vault becomes production infrastructure you own |
Read the third column first. It decides how much of your delivery path depends on a system being reachable at the moment a sync runs.
How sealed-secrets works
The mechanism is small enough to describe precisely, which is part of the appeal. A controller runs in the cluster and holds a key pair. The public half is not secret. You encrypt a value against it with kubeseal, producing a SealedSecret custom resource, and you commit that resource like any other manifest. Argo CD applies it. The controller decrypts it with the private half and creates an ordinary Secret in the target namespace. The workload mounts that Secret and knows nothing about any of this.
The value's source of truth stays in a managed vault. Git carries only ciphertext, and the private key never leaves the cluster.
The envelope, in one paragraph
It helps to know what is actually inside the encrypted blob, because two of the failure modes below follow directly from it. Per the crypto documentation, each value is encrypted with AES-256-GCM under a randomly generated single-use 32-byte session key, and that session key is then encrypted to the controller's public key with RSA-OAEP using SHA-256. The sealed value is the length of the encrypted session key, the RSA output and the AES output, concatenated. The controller generates a 4096-bit RSA key pair on first start and persists it in a regular Secret in its own namespace.
The interesting part is the OAEP label. RSA-OAEP takes an additional input that has to match on decryption, and sealed-secrets puts the identity of the target Secret into it. In the default scope, the label is the namespace and the name. That is not metadata sitting next to the ciphertext, it is part of what the decryption depends on, which is why a resource sealed for one namespace cannot be moved to another and read there.
The resource, and what it templates
A SealedSecret is a recipe for a Secret rather than a copy of one. The template block is what the controller puts into the object it creates, so labels, annotations and the Secret type belong there and not on the SealedSecret itself.
Sealing a value looks like this. The intermediate Secret never reaches the cluster; it is a local file that goes straight into kubeseal.
The offline certificate matters more than it looks. It means the person or pipeline that seals a value does not need cluster credentials, only the public certificate, which the controller also serves at /v1/cert.pem and prints to its log at startup. Certificates are renewed every 30 days, so a stored copy should be refreshed on a schedule, or fetched from a URL that your automation keeps current.
One property of the generated Secret is easy to miss and pays off at teardown: the controller sets an ownerReference on it. The Secret is a dependent object of the SealedSecret, so deleting the SealedSecret deletes the Secret with it. In a preview environment that is deleted as a cascade when the pull request closes, that is exactly the behaviour you want, and it is the reason we do not create secrets out of band.
Scope: the decision that decides where a secret can land
Scope is the setting teams meet by accident, usually while debugging, and it deserves a deliberate choice. It controls what goes into that OAEP label, which controls where the ciphertext is allowed to decrypt. The scopes section of the README is the reference; the practical summary is below.
Scope is not a convenience setting. It decides how much of a secret's identity is cryptographically fixed at sealing time.
The default is strict, and it exists for a good reason. Without it, anyone who can create objects in one namespace could take a SealedSecret meant for another, change the namespace field, apply it where they can read Secrets, and get the plaintext out. Strict scope makes that fail.
The same request can be made with annotations on the input Secret, sealedsecrets.bitnami.com/namespace-wide or sealedsecrets.bitnami.com/cluster-wide, which is the form that survives being checked into a repository next to the manifest it belongs to.
Where this bites is per-pull-request preview environments, because the namespace does not exist when the secret is sealed. A SealedSecret sealed for preview-pr-811 will not decrypt in preview-pr-812. Same cluster, same controller, same key, and it still fails. Our answer is to generate rather than inherit: a preview needs credentials that work, not the credentials the other environments use, so the database password and internal tokens are generated at provisioning time into the namespace. Nothing is sealed, nothing is committed, and the dynamic namespace stops being a problem. For the short list of values a preview genuinely has to share, such as a sandbox key for a third-party service, cluster-wide scope is defensible on a cluster that holds only previews and indefensible on one that holds anything you would mind losing.
Why this is our default
One system in the deploy path
At sync time Argo CD needs exactly one thing: the Git repository. No external secret store has to be reachable, no token has to be valid, no network path has to be open between the delivery tooling and a vault. A sync during an incident does not fail because a second system is also having a bad day.
To be precise about what this does not mean: we do keep a second system. The plaintext has to live somewhere a human can retrieve it, and for us that is a managed vault the client already uses, usually Keeper or 1Password. The vault is the source of truth for the value and the audit trail around it, while Git is the record of what is deployed. Only one of the two is in the path when the cluster reconciles.
Static secrets are most secrets
A database URL, an API key for a payment provider, a webhook signing secret, an SMTP password. These change when someone decides to change them, which is on the order of months. For that category the trade is clean: the value gets into the cluster with no runtime dependency and no extra moving parts.
Access separation comes free
Developers do not need to hold production credentials to ship to production. They can write, review and merge a manifest that references a secret without ever seeing its value, because the value entered through the vault and left as ciphertext. That property is easy to state and hard to retrofit, which is one of the reasons we lead with it rather than adding it later.
The limits we accept
This is the part most write-ups skip, and it is where the actual work lives.
The key is per cluster
A secret sealed for cluster A does not decrypt on cluster B. Every cluster we run has its own key pair, and we treat that as correct rather than inconvenient: compromising one cluster does not hand over the secrets of another.
The obligation it creates is real. A new cluster needs its secrets re-sealed against the new key. If you plan to rebuild a cluster from Git, the sealing key has to be part of the restore procedure or the rebuild produces a cluster full of manifests that cannot decrypt. Sharing one key pair across clusters is possible, since sealing keys are ordinary Secrets labelled sealedsecrets.bitnami.com/sealed-secrets-key=active, and we do not do it. It trades the isolation property for convenience and the convenience is small.
Renewal is scheduled, and backups go stale
The controller renews its sealing key every 30 days by default, configurable with --key-renew-period. Renewal appends a new key; old keys are retained, so existing SealedSecret resources keep decrypting and nothing appears to change on the day it happens. New seals use the newest key, which is also the one --fetch-cert returns.
The trap is on the backup side. A key backup taken once captures the keys that existed that day. After a renewal it is missing the current key, and it will keep looking fine until the day you restore from it. The upstream FAQ says this plainly in the backup instructions: recreate the backup after renewal. Backup and renewal have to be one process, or the backup quietly stops being a backup.
That file contains private keys. It goes into the client's managed vault, per cluster, with access granted on demand rather than standing. In normal operation nobody needs it, since it is required only for restore and for key rotation.
The other half of key hygiene is re-encryption. Old keys are never garbage collected on their own, so if you want to retire one, the existing resources have to be re-sealed with the current key first. kubeseal does that without the plaintext leaving the cluster.
Note what re-encryption does not do. It does not update the object in the cluster, it produces a new file for you to commit, and it does nothing about the fact that the old ciphertext is still in your Git history and still decryptable by the old key. Re-encryption is key hygiene. It is not rotation.
Renewal is not rotation
These two words get used interchangeably and they are separate jobs on separate schedules.
Sealing key renewal | Secret value rotation | |
|---|---|---|
What changes | The key pair the controller uses | The credential itself |
Who does it | The controller, automatically | You, deliberately |
Effect on existing manifests | None, old keys are retained | The SealedSecret is regenerated and committed |
What it protects against | Long-lived key exposure | Credential exposure, staff turnover, policy |
Happens on its own | Yes | No |
Automatic renewal creates a comfortable feeling that secret hygiene is handled. It is not. When a value itself needs rotating, our process is to obtain the new value through the managed vault from whoever can produce it, or generate it ourselves when we can, then re-seal and commit. That work does not happen unless someone schedules it.
Losing the private key is the worst case
If the private key is gone and no backup exists, every secret whose plaintext lives nowhere else is gone with it. Not inaccessible, gone. There is no backdoor and the project says so directly. This is the single failure that turns an inconvenience into an incident, and it is why the vault side of the setup is not optional. Because the plaintext values also live in the managed vault, a lost key means re-sealing everything, which is tedious and recoverable. Without that second copy it is neither.
If you do hold a key backup and the cluster is unavailable, values can be recovered offline rather than by standing up a controller to do it.
No dynamic credentials
Sealed-secrets encrypts a static string. It has no concept of a lease, an expiry, or a credential generated on demand for a specific consumer. If the requirement is short-lived database credentials issued per pod and revoked after an hour, that is a Vault-shaped problem and sealed-secrets is the wrong tool. We would rather say that plainly than stretch one tool across a requirement it was not built for.
How it interacts with Argo CD, and where it bites
Argo CD applies a SealedSecret the way it applies any other manifest. It does not check that the controller in that cluster can decrypt it, because from Argo CD's point of view the resource was created successfully. That is correct behaviour, and it produces a diagnosis problem.
The symptom surfaces several steps away from the cause. Nothing lies to you, but nothing points at the real problem either.
Walk the sequence, because the time cost is in the middle of it. The SealedSecret is applied without error. The controller tries to decrypt it and fails. No Secret appears in the namespace. The pod that mounts that Secret cannot start and enters a crash loop. The Application does not go green, it stalls on health, so you are not being told everything is fine. You are being told a pod is unhealthy, which sends people to inspect the workload, the image and the deployment spec, none of which are the problem.
In our experience there are only two causes.
The first is a wrong-cluster or wrong-namespace seal. The resource was encrypted against a different cluster's public key, or against a namespace name that does not match where it landed. Preview environments are the usual source, for the reason described above.
The second is that the Secret already existed. The controller will not take ownership of a Secret it did not create, so a manually created one from an earlier debugging session blocks the managed one from appearing. That behaviour is deliberate and it has an explicit opt-out: annotate the existing Secret with sealedsecrets.bitnami.com/managed: "true" and the controller will overwrite and adopt it. Two related annotations are worth knowing before you need them: sealedsecrets.bitnami.com/patch merges keys into an existing Secret instead of replacing it, and sealedsecrets.bitnami.com/skip-set-owner-references keeps the Secret alive after the SealedSecret is deleted.
Both causes are visible in one place. Check the controller, not the workload.
The ownerReferences field is the tell for the second case. A Secret with no owner reference back to a SealedSecret was created by something else, and the controller is deliberately leaving it alone. There is also a cheap pre-merge check: kubeseal --validate sends a resource to the controller and reports whether it can be decrypted, without applying anything. In a repository where previews and multiple clusters are in play, that check in CI removes most of this class of incident before it reaches a cluster.
Where we draw the line to Vault and ESO
The line is not purely static versus dynamic, although that is the clearest part of it.
Running Vault or External Secrets Operator alongside sealed-secrets means two systems answering the same question, each with its own audit trail, access model and failure modes. The complexity is not the sum of the two, it is the product, because now every secret has a location and every engineer has to know which system holds which value. A single place for access separation is worth more than the marginal capability of the second system.
Vault earns its complexity when you genuinely need leased credentials, per-consumer issuance, or revocation as a first-class operation. For a team with a hundred static secrets and no such requirement, it is a large piece of production infrastructure to patch, upgrade, back up and secure in exchange for a capability nobody is using.
External Secrets Operator earns it when an external secret manager is already the mandated system of record, often for compliance reasons outside the platform team's control. In that case the value is already in the store and syncing it into the cluster is the shortest honest path.
What we actually run
The setup that survives a cluster rebuild is a short list, and none of it is exotic.
One key pair per cluster, never shared between clusters.
The sealing keys backed up into the client's managed vault, with the backup refreshed as part of the same process that renews the key.
The plaintext values in that same vault, so a lost key is a re-sealing exercise rather than an incident.
Access to the key granted on demand, not standing, because it is needed only for restore and rotation.
The public certificate stored where CI can reach it, so sealing never requires cluster credentials.
Preview environments generating their own credentials rather than inheriting sealed ones.
kubeseal --validate in CI on every changed SealedSecret.
A named owner and a schedule for rotating the values themselves, because nothing in the automation does that for you.
Closing
The tool is easy. kubeseal, a controller, a custom resource, and an afternoon to wire it into a repository. The discipline around the private key is the actual work, and it is what separates a setup that survives a cluster rebuild from one that quietly cannot.
If your secrets are currently spread across sealed secrets in one repository, a bit of SOPS somewhere else, and a value in a CI variable nobody remembers adding, bringing that back to one default and a short documented list of exceptions is a specific and very doable piece of work. It is usually smaller than it looks from the inside.
Working on this now?
We map where your secrets actually live, which of them are in the deploy path, and what the key backup and rotation policy should be for the number of clusters you run.
SHARE ON SOCIAL MEDIA
