Recover a re-created database

Use this runbook when a CloudNativePG Cluster is created again under its old name, after a Kubernetes cluster rebuild or a lost volume. The barman-cloud plugin archives each Cluster under s3://cnpg/<namespace>/<serverName>/, and serverName defaults to the Cluster name. A new Cluster refuses to archive into a prefix that holds WAL, so it recovers from the old prefix and archives under a new one. The charts in packages/charts/ render both from backup.recoverFrom and backup.serverName. For recovery from another namespace, packages/charts/kthx-engine/templates/backup.yaml renders a separate source store from backup.recoverFromPath. Restore a database covers a restore into a Cluster that still runs.

warning

This procedure changes live state by hand. It is an exception to the GitOps rule because git holds no data, only the declaration of the database.

Before you start#

Get kubectl access, as Get cluster admin access describes.
Install kubectl-cnpg, as Operate Postgres describes.
Find the declaration: the backup values of the HelmRelease in clusters/<site>/apps/<app>/, or a plain Cluster manifest such as clusters/folly/apps/tronbyt/04-database.yaml.

<site> is folly or offsite. <old> is the prefix the Cluster archives under today: backup.serverName, or the Cluster name if that is empty. <new> is a prefix no Cluster has used, such as <cluster>-2.

Take a last backup#

If the old Cluster still runs, stop the writers of the database first. The backup switches the WAL file, so the archive then holds every write.

Read the prefix in use and the archive state.
```
kubectl cnpg status <cluster> -n <namespace> --context <site>
```
Result: ObjectStore / Server name: garage/<old> and Working WAL archiving: OK.

Take a base backup.

kubectl cnpg backup <cluster> -n <namespace> --method plugin --plugin-name barman-cloud.cloudnative-pg.io --context <site>

Result: backup/<cluster>-<timestamp> created.

Wait until the backup completes.
```
kubectl get backup -n <namespace> --context <site>
```
Result: The new row has the phase completed.

Declare the new Cluster#

caution

Merge after the old Cluster stops and before Flux creates the new one. A running Cluster takes <new> and fills it with WAL, and the new Cluster then stays in Setting up primary.

Set backup.serverName: <new> and backup.recoverFrom: <old> in the release's values.
If the source is in another namespace, set backup.recoverFromPath to s3://cnpg/<source-namespace>/.
For a plain manifest, copy the serverName, bootstrap.recovery and externalClusters from packages/charts/app/templates/database.yaml.
If the Cluster is offsite's kthx-db or folly's tronbyt, set serverName: <new> in clusters/<site>/apps/restore-drill/cluster.yaml.
Open a PR and merge it, as Apply a Kubernetes change describes.

warning

Deleting a Cluster deletes its data claim. Make sure that the source archive holds the completed backup before you delete it.

If the new Cluster cannot recover, delete it after the source path is correct. Then reconcile the release or the plain manifest's Kustomization.
```
kubectl delete cluster.postgresql.cnpg.io/<cluster> -n <namespace> --context <site>
flux reconcile helmrelease <release> -n <namespace> --force --context <site>
```
Result: cluster.postgresql.cnpg.io "<cluster>" deleted, then HelmRelease reconciliation completed.

Check the recovery#

Wait until the Cluster is ready.

kubectl wait --for=condition=Ready cluster.postgresql.cnpg.io/<cluster> -n <namespace> --timeout=30m --context <site>

Result: cluster.postgresql.cnpg.io/<cluster> condition met.

Count the rows of a table that the app always fills.

kubectl cnpg psql <cluster> -n <namespace> --context <site> -- -d <database> -At -c 'select count(*) from <table>'

Result: A number above zero.

Make sure the Cluster archives under the new prefix.
```
kubectl cnpg status <cluster> -n <namespace> --context <site>
```
Result: ObjectStore / Server name: garage/<new> and Working WAL archiving: OK.
Take a base backup under the new prefix, as in Take a last backup.
Open a PR that removes backup.recoverFrom. CloudNativePG reads bootstrap only at creation, so the change is inert.

Remove the old prefix#

The plugin prunes only the prefix it archives under. Wait until <new> holds 30 days of backups.

Set the Garage remote, as Restore from the GCS copy describes, with the cnpg key.

Delete the old prefix.

rclone purge garage:cnpg/<namespace>/<old>/

Result: No output.

If something goes wrong#

Symptom	Cause	Action
Helm fails with `backup.recoverFrom ... is the prefix this Cluster archives to`.	`<new>` is empty or equals `<old>`.	Set `backup.serverName` to a prefix no `Cluster` has used.
The pod logs print `Expected empty archive`.	`<new>` holds WAL.	Choose another `<new>`, then do step 6 of Declare the new Cluster.
The recovery log says `no target backup found`.	The source store path or `<old>` has no completed backup.	Compare `backup.recoverFromPath` with the source archive namespace, then read the `<cluster>-1-full-recovery` pod logs.
The `Cluster` stays in `Setting up primary` with no such log line.	The recovery cannot read the `garage` ObjectStore, or `<old>` holds no backup.	Read the logs of the `<cluster>-1-full-recovery` pod.
`restore-drill` fails after you remove the old prefix.	The drill still reads `<old>`.	Set `serverName: <new>` in `clusters/<site>/apps/restore-drill/cluster.yaml`.