Major Postgres Upgrades on Kubernetes with CNPG: What the Image Tag Change Actually Does

On a CloudNativePG cluster, moving from PostgreSQL 17 to 18 can be a one-line diff: the major version in the image tag goes up, someone approves the pull request, Argo CD syncs it. Since CNPG 1.26 the operator takes it from there and runs the whole upgrade declaratively.

That is a real achievement. It is also why people merge that line without knowing what happens next. The diff looks exactly like a minor version bump, and a minor bump is a rolling restart nobody notices. A major bump stops the entire cluster. If you run CNPG and have a major upgrade coming, this is the walkthrough we give our own engineers before they open the pull request: what the operator does, how long the database is gone, when the other upgrade route is worth its cost, and what rollback looks like once it is too late to revert.

We are not going to argue here why we run CNPG at all. That case, along with how its backups and failover work, is in why we use CNPG instead of a generic provisioner.

What changing the image tag actually triggers

The CNPG documentation on major upgrades describes the in-place route as an offline operation. When the cluster spec asks for an operand image with a higher major version, the operator runs a fixed sequence, and each step has a consequence you should plan for.

Every pod stops

The operator shuts down all instances, the primary and every replica. There is no rolling phase and no read-only phase. Before it starts, it records the current image and version in .status.pgDataImageInfo, which is what makes a declarative rollback possible later.

For the application this means connection errors for the whole duration. Whatever your services do when Postgres is unreachable for several minutes is what they will do during the upgrade. If nobody knows the answer, find out on a staging copy first: retries with backoff are fine; a crash loop that pages someone at 02:00 is what you want to find early.

A job runs pg_upgrade

The operator creates a Kubernetes Job named after the primary with a -major-upgrade suffix. It checks that the binaries and the data files on the volume match the upgrade that was requested, initialises a new data directory for the target version and runs pg_upgrade with --link.

Most failures surface here, before anything is swapped. Ours did. On one client cluster the new image carried a different TimescaleDB version from the one installed in the database. The check stage stopped the upgrade, the old data was never touched, and the cluster came back on the old version. We rebuilt the image with the matching extension version and ran the upgrade again. Annoying, harmless, and a good argument for testing the image before production does it for you.

New directories for PGDATA and WAL

The job sets up new directories for PGDATA, and for WAL and tablespaces when they live on separate volumes. Only when pg_upgrade succeeds do the new directories replace the originals. Until that moment the old cluster is intact on disk, which is the whole basis of the rollback story further down.

Replicas are rebuilt, not upgraded

This is the step people miss. CNPG does not run pg_upgrade on the replicas. It deletes their PersistentVolumeClaims and clones fresh replicas from the upgraded primary. The cluster is serving again at that point, but with one copy of the data until the clones finish. On a small database that is minutes. On a large one it is however long a full copy over your network and storage takes, and during that time a primary failure leaves you restoring from backup.

Two more things come after. Up to PostgreSQL 17, pg_upgrade does not carry planner statistics over, so run ANALYZE or expect bad query plans for a while. PostgreSQL 18 preserves most optimizer statistics during pg_upgrade, extended statistics excepted. And take a fresh base backup as soon as the primary is up, for reasons that become very concrete in the rollback section.

Why offline is the honest word for it

The documentation says offline and we will not soften it. For the duration of the shutdown and the job, there is no Postgres: no reads, no writes, no read replicas. You need a maintenance window, and you need to announce it.

Downtime follows objects, not size

What keeps the window short is --link. Instead of copying data files into the new directory, pg_upgrade creates hard links to them. The cost of the operation then depends mainly on how many tables, indexes and other objects it has to rewrite in the catalog, and barely on how many gigabytes sit behind them.

Gabriele Bartolini's CNPG Recipe 17 makes the point with a 2.2 TB pgbench database upgraded from 16 to 17 in 33 seconds. Keep in mind what that example is: a single instance and the handful of tables pgbench creates. On the client schemas we have upgraded, with real numbers of tables, indexes and extensions, plus the shutdown and restart around the job, 5 to 10 minutes of downtime in link mode is the realistic figure.

What hard links buy you, and what they cost

Hard links mean no second copy of the data and almost no extra disk I/O, which is why the upgrade does not need double storage. The trade-off is in the PostgreSQL docs: once the new cluster has started on linked files, the old cluster can no longer be safely started. In CNPG terms, after the directory swap there is no going back to the old data directory. Before the swap, there is.

A fleet of small clusters versus one large one

Because the offline time barely grows with data volume, in-place behaves well across a fleet. Twenty small and medium clusters are twenty pull requests with the same shape, each with a short window you can schedule per team. The size-sensitive part is the replica rebuild after the upgrade, and that is where one large database starts to look different from many small ones.

The other route: logical replication into a new cluster

CNPG supports three major upgrade methods: a dump and restore into a new cluster, logical replication into a new cluster, and the in-place upgrade above. Dump and restore is offline and slow on anything sizable, so the real alternative is logical replication.

Build alongside, replicate, cut over

You create a second Cluster on the new major version next to the old one. The CNPG logical replication guide walks through it with declarative resources:

• a Publication on the old cluster, usually allTables: true;

• a new cluster bootstrapped with initdb.import of type microservice and schemaOnly: true, so the tables exist before data arrives;

• a Subscription on the new cluster that points at the old one through an externalClusters entry;

• after the initial copy, a continuous stream of row changes from old to new.

Everything above is a manifest in Git, which suits how we work: the second cluster, the publication and the subscription arrive the same way as any other change, through a pull request and an Argo CD sync.

Cutover time no longer depends on size

The only write outage is the cutover itself. You stop writes to the old cluster, wait for the subscription to catch up, copy sequence values across, and move the application. With a CNPG Pooler in front, setting it to paused makes PgBouncer finish running queries and hold new ones, which turns the cutover into a pause of seconds to minutes. The pooler's cluster reference is immutable, so the application moves through a changed connection target in Git.

What it costs

Logical replication leaves gaps, and each one is a step in your runbook:

• Schema changes are not replicated. Any migration that runs during the replication window breaks the subscription or silently diverges. Freeze schema migrations until cutover; with add-only migrations as PreSync hooks that simply means holding the merge.

• Sequences are not kept in sync. Run kubectl cnpg subscription sync-sequences after writes stop, or the first insert on the new cluster collides with an existing key.

• Large objects are not replicated. If the application stores them, this route needs a separate plan.

• Tables need a replica identity, normally a primary key. Tables without one need attention before you start.

On top of that you run two clusters, with two sets of storage and backups, for days or weeks. More resources, more moving parts, and more places for something to go wrong between the first sync and the cutover.

Choosing between them

Our default is in-place. In the client upgrades we have run, it is the route we used in most cases, because it is predictable and has fewer points of failure. Logical replication has more moving parts, and every one of them can fail on the night. We pick it only when the numbers call for it.

The comparison we use:

A fleet of small clusters: in place

Per-team databases, internal tools, the clusters behind preview environments: a short scheduled window is acceptable, and the same procedure repeated twenty times beats twenty bespoke migrations. The simplicity compounds.

One critical database: logical replication

A single database where 10 minutes without writes costs real money, or where nobody can agree on a window, justifies the second cluster and the longer runbook.

Where the calculus flips

Not at a fixed number of gigabytes, because in-place downtime barely depends on gigabytes. It flips on two questions:

1. Can the business accept the measured in-place window at any time slot you can schedule? If yes, size is mostly irrelevant.

2. How long does the replica rebuild take at your size, and are you willing to run on a single instance for that long? The same number tells you how long a restore from backup would take if the upgrade went wrong.

Both numbers come out of one rehearsal: restore the latest backup into a scratch cluster, run the upgrade on it, time the job and the clone. If either number is unacceptable, use logical replication. If you cannot get the numbers, you are not ready for either route.

Constraints that will stop you

Most failed upgrades fail before pg_upgrade does any work, on something that was knowable in advance.

Same OS base on both sides

CNPG supports major upgrades only between images built on the same operating system distribution. Going from a bullseye-based image to a bookworm-based one needs a different procedure. If you need both, change the OS base as its own step and do not combine it with the version change. An OS move can also change the C library's collation data, which can quietly affect indexes on text columns; the PostgreSQL wiki on locale data changes explains the mechanism.

Extensions in the target image

Every extension the database uses must exist in the target image, at a version pg_upgrade can carry over, or the upgrade stops. For TimescaleDB that means the exact version already installed in the database. The TimescaleDB case above is the typical shape. CNPG states plainly that extension compatibility is your responsibility. If you load extensions as image volume extensions, the upgrade job mounts the target versions next to the source ones; check the mapping in the manifest before you merge. After the upgrade, look for an update_extensions.sql file that pg_upgrade may leave in PGDATA and run it.

This is also why we keep Postgres images as part of the golden path instead of letting each team pick its own: one image per major version, with the extension set tested once.

Known version issues

Check the current CNPG and PostgreSQL release notes rather than a blog post, this one included. One example worth knowing: PostgreSQL 17.0 to 17.5 fails to upgrade when max_slot_wal_keep_size is set to anything other than -1. The fix is in 17.6. Either move to 17.6 or later first, or set the parameter to -1 for the upgrade.

The backup archive path

pg_upgrade gives the cluster a new system ID and resets the timeline to 1. With the Barman Cloud plugin, archives are not separated automatically, so the new cluster can collide with the old one's WAL in the object store. The fix is to change serverName in the same commit as the image:

Treat the pull request that bumps a Postgres major as a planned change. If update automation proposes image bumps on your repositories, as updatecli does on ours, a major Postgres bump in that stream deserves its own review, a scheduled merge time and a named person watching the sync.

The rollback path

This is the section people skip, so here is the short version: reverting the commit covers one failure case, and a restored backup covers the other only partially.

Rolling back a failed job

If the upgrade job fails, revert the image to the previous major version. In GitOps that means reverting the commit, serverName included, and letting Argo CD sync it. The operator sees the change, deletes the failed job and starts the cluster on the original version. Because the old directories were never swapped, no restore is needed. This is exactly what happened in our TimescaleDB case.

When the upgrade succeeded and the app did not

The harder case: the job succeeds, the cluster is healthy, and three hours later the application hits a query that behaves differently on the new version. Reverting the image now achieves nothing, because the data directory is already on the new format.

Point-in-time recovery is the safety net here, with a limit that has to be said out loud: PITR cannot cross a major version boundary. Pre-upgrade backups cannot recover to a point after the upgrade. What you can get is a new Cluster restored from the pre-upgrade backup on the old major version, at a point before the shutdown. Everything written after the upgrade is not in it, and you would have to recover those writes some other way.

In practice that makes the restore a last resort and fixing forward the usual answer. It is also why we run the application's tests against the new major version on a restored copy before the production merge.

A backup you have not restored

Before any operation on a Postgres cluster, we confirm that a specific, recent backup exists. For most upgrades that has been enough. But an existing backup only proves that a backup job finished. It does not prove that the backup restores, that the WAL archive is complete, or how long the restore takes.

We do not run a full PITR rehearsal before every upgrade. We do it when the database is one where losing a day would hurt, and for those it happens before the upgrade. A rehearsal after the incident is just a restore under pressure. Conveniently, the rehearsal is the same scratch cluster that gives you the downtime and clone numbers, so one restore answers three questions.

The tag change is the easy part

CNPG made the mechanics of a major upgrade declarative and repeatable, which is the right place for them. What it cannot decide for you is whether your application tolerates the window, whether the image has the right extensions, and whether the backup you are counting on actually restores. Settle those three before you open the pull request, and the one-line diff is as boring as it looks.

If you are planning major upgrades across a fleet of CNPG clusters and want a second pair of eyes on the runbook, talk to our engineers.