infra_wiki

Restore etcd on Talos

Restore the etcd of a Talos control plane from a snapshot, when the etcd data is lost or corrupt.

Use this runbook when the etcd data of a control plane that runs Talos Linux is lost or corrupt. etcd is the database of the Kubernetes API server, and each cluster has one member, on its control plane. The control plane keeps etcd on its own ETCD partition, so the restore wipes only that partition. The image cache in EPHEMERAL and the data user volume stay. The restore returns every Kubernetes object to its state at the snapshot time, and Flux then applies main again.

warning

This procedure changes live state by hand. It is an exception to the GitOps rule because git does not hold the etcd data.

Before you start#

  • Get the cluster's admin context in ~/.talos/config, as Get cluster admin access describes.
  • Sign in to op, and install talosctl and rclone.

<site> is folly or offsite. <cp> is API_SERVER_IP in clusters/<site>/config/cluster-topology.json.

Get the snapshot#

note

The etcd-snapshot CronJob in clusters/<site>/etcd-snapshot/ writes a snapshot under etcd/<site>/ each night and keeps 14 days. A file under etcd/<hostname>/ is from NixOS and does not restore on Talos.

  1. Put the Garage remote in the environment. Use the etcd key of the site.

    export RCLONE_CONFIG_GARAGE_TYPE=s3 RCLONE_CONFIG_GARAGE_PROVIDER=Other RCLONE_CONFIG_GARAGE_REGION=garage
export RCLONE_CONFIG_GARAGE_ENDPOINT=<GARAGE_S3_ENDPOINT>
export RCLONE_CONFIG_GARAGE_ACCESS_KEY_ID="$(op read 'op://homelab/garage-<site>/etcd-access-key-id')"
export RCLONE_CONFIG_GARAGE_SECRET_ACCESS_KEY="$(op read 'op://homelab/garage-<site>/etcd-secret-access-key')"

    <GARAGE_S3_ENDPOINT> is the value in clusters/<site>/config/cluster-settings.yaml.

  2. If Garage has lost the etcd bucket, follow Restore from the GCS copy.

  3. Copy the newest snapshot of the cluster to your machine.

    rclone lsl garage:etcd/<site>/
rclone copyto garage:etcd/<site>/<file>.db ./etcd-snapshot.db

    Result: One <UTC timestamp>.db file for each snapshot, then a local ./etcd-snapshot.db.

Restore etcd#

caution

The API server is down from step 3 until step 5. Pods on the workers keep running, but nothing changes them.

  1. Read the state of etcd.

    talosctl --context <site> -n <cp> service etcd
talosctl --context <site> -n <cp> etcd status

    Result: The state and health of the service, and the database size when etcd answers. If etcd is healthy, stop. A restore loses every change since the snapshot.

  2. Keep a copy of the current data. If etcd does not answer, use the second command.

    talosctl --context <site> -n <cp> etcd snapshot ./before-restore.db
talosctl --context <site> -n <cp> cp /var/lib/etcd/member/snap/db ./before-restore

caution

Wipe ETCD, not EPHEMERAL. A wipe of EPHEMERAL leaves etcd as it is and deletes the image cache.

  1. Wipe the ETCD volume and reboot the control plane. --graceful=false is required, because a single member cannot leave etcd.

    talosctl --context <site> -n <cp> reset --system-labels-to-wipe=ETCD --reboot --graceful=false

    Result: post check passed. After the reboot, talosctl -n <cp> service etcd shows Preparing.

  2. Bootstrap etcd from the snapshot.

    talosctl --context <site> -n <cp> bootstrap --recover-from=./etcd-snapshot.db

    Result: recovering from snapshot, with the hash, revision and key count of the snapshot.

  3. Make sure that the API server answers.

    kubectl --context <site> get --raw /readyz

    Result: ok, about 20 seconds after the bootstrap.

  4. Make sure that each node is Ready and that Flux applies main.

    kubectl --context <site> get nodes
flux --context <site> get kustomizations

    Result: Each node Ready, and each Kustomization True at the newest revision of main.

  5. When the cluster works, delete ./etcd-snapshot.db and ./before-restore*.

If something goes wrong#

SymptomCauseAction
snapshot file integrity check failedThe file is a copy of the database file, with no hash.Add --recover-skip-hash-check to the bootstrap command.
etcd is Running after step 3, with the old data.The reset wiped another volume.Do step 3 again with --system-labels-to-wipe=ETCD.
etcd stays Preparing after step 4.The bootstrap did not reach etcd.Read talosctl -n <cp> logs etcd, and make sure that talosctl -n <cp> get machinetype is controlplane.
The cluster does not recover.The snapshot is bad, or older than the damage.Do steps 3 and 4 again with an older snapshot, or with ./before-restore.db.

Edit on GitHub

Linked from 5
  • BackupsHow Garage, Velero, CloudNativePG and etcd snapshots back up the lab's databases, volumes and etcd, and how each site copies them to GCS.
  • Rebuild a cluster on TalosReplace a cluster's NixOS nodes with Talos Linux, bootstrap a fresh etcd, install Cilium and Flux, and restore the data that has a backup.
  • Reprovision a nodeReinstall, attest again or wipe an installed folly node by setting its NetBox device's desired_action.
  • Restore a databaseRecover a Postgres database from its barman backups in Garage into a scratch CloudNativePG cluster, copy the data back, and run the restore drill.
  • Restore from the GCS copyCopy a Garage bucket back from the encrypted GCS copy of its site, when Garage has lost the bucket.

↑↓ move ↵ open esc close