infra_wiki

Backups

How Garage, Velero, CloudNativePG and etcd snapshots back up the lab's databases, volumes and etcd, and how each site copies them to GCS.

Backups write to a Garage S3 store at each site. A CronJob copies each store to GCS, encrypted.

Parts#

PartJobWhere it runs
GarageStores the buckets velero, cnpg and etcd, and folly's netboxA NixOS service on spore, capsule and forge for folly, a StatefulSet on oldschool's data volume for offsite
VeleroBacks up objects, and the volumes of pods with the annotation backup.velero.io/backup-volumesThe velero namespace
barman-cloud pluginArchives each database's base backups and WAL, through an ObjectStore named garageThe cloudnative-pg namespace
etcd-snapshotSnapshots the cluster's etcd through the Talos API into etcd/<site>/The backups namespace on each site
backup-pushSyncs each bucket to GCS through an rclone crypt remoteThe backups namespace
restore-drillRecovers the kthx database on offsite, and the tronbyt database on folly, into a throwaway clusterThe restore-drill namespace on each site

Schedule#

StepWhen
Velero Schedule daily02:00 Halifax
CNPG ScheduledBackupDaily, 01:00 to 04:00 Halifax
etcd snapshot02:30 Halifax
backup-pushBACKUP_PUSH_SCHEDULE: weekly on folly, because of Starlink, and daily on offsite
restore-drill06:00 Halifax: the 1st of the month on offsite, the 15th on folly

Retention#

  • Velero keeps a backup 720 hours.
  • Each ObjectStore keeps 30 days.
  • The etcd jobs delete snapshots older than 14 days.
  • GCS has object versioning, and keeps a deleted or replaced object 30 days.

Garage cluster#

folly's Garage runs at replication factor 3 on the anchors: spore, capsule and forge. Each node stores every object and is its own zone. The cluster holds as much as the smallest node, so the layout gives each node 80 GB. RPC uses port 3901 on the Anchor network, and the firewall opens it to the other anchors only.

Every anchor serves S3, and clients name s3.lolwtf.ca, the S3 VIP (S3_VIP in clusters/folly/config/anchor-topology.json). keepalived from nix/services/anchor-vips.nix holds the VIP on capsule, then forge, then spore, and moves it off an anchor whose Garage loses its quorum or stops answering S3. AnchorS3VipDown fires when no anchor holds the VIP and serves.

A node with homelab.garage.clustered unset is a standalone store with replication factor 1. spore keeps its old standalone store in /nfs/data/garage as the rollback of Migrate Garage to three nodes.

Credentials#

1Password holds the Garage keys in garage-<site>, and the crypt and Velero repository passwords in backup-crypt. The NixOS hosts read theirs from SOPS under garage/. The three anchors share one garage/rpc-secret in their own SOPS files. backup-push reaches GCS through workload identity, as backups:backup-push.

Velero permissions#

Velero's server runs as velero-server, which the chart binds to cluster-admin. The node agents, one on every node, run as velero-node-agent instead. Its role in clusters/base/platform/velero/node-agent-rbac.yaml reads pods, claims and volumes, and works only on Velero's own objects in the velero namespace, so a token read from a node is not a cluster-admin token. The chart has no value for a second account, so a post-renderer on the HelmRelease sets the DaemonSet's serviceAccountName. The server keeps cluster-admin because the chart offers no narrower binding.

Volumes#

Every claim on folly uses local-path, a local volume on one node's data volume. No pod mounts storage from a host outside the cluster, so a stopped anchor does not hang a pod. A claim lives on one node, so Velero is the copy of a claim that has no second instance. Restore a volume recovers it on another node.

Velero skips a PVC bound to a hostPath volume. Migrate any such claim to a local volume with Migrate a local-path volume before relying on its backup.

Volume coverage#

Velero runs in opt-in mode: defaultVolumesToFsBackup is false, so it copies a volume with kopia only when the running pod names it in backup.velero.io/backup-volumes. Every claim on folly has one of three answers.

AnswerClaimsWhy
Velerojellyfin-config, prom-stack-grafana, smiirl-data, tronbyt-data, data-vault-openbao-0The state is not in the repo and cannot be rebuilt. Grafana keeps what the UI adds, such as public dashboard links; its dashboards and data sources come from the repo.
CloudNativePGtally-db-*, tronbyt-*The barman-cloud plugin archives base backups and WAL, so Velero skips them.
ExcludedPrometheus, VictoriaLogs and Tempo volumesMetrics, logs and traces history. Losing it costs history only. Rules, alerts and dashboards come from the repo.
Excludedvalkey-applol-cache-0-0-dataA cache that the kthx engine owns.
Excludedhajimari-config, dump-data-dump-nfs-0Empty. Hajimari reads its settings from a ConfigMap, and dump is scratch space.
Excludedpvc-nfs-provisioner-nfs-subdir-external-provisionerA mount handle for the NFS export. The data lives on spore.

Velero skips a pod that is not running when the backup starts, and records a warning. A volume that the warnings name goes uncovered for that night.

mise run backup:coverage [context] lists each claim with its answer and fails when a claim has none. scripts/backup-coverage.exclusions holds each excluded claim and its reason.

Rules#

  • Give a new PVC a pod annotation backup.velero.io/backup-volumes, or a line in scripts/backup-coverage.exclusions that says why it needs no backup. Run mise run backup:coverage before you merge.

  • Keep backup-crypt in 1Password. Without it the GCS copy cannot be read.

  • Label each backup CronJob lolwtf.ca/backup: "true". BackupJobStale and BackupJobFailed watch only those.

  • Give a new database an ObjectStore and a ScheduledBackup. It has no backup without them.

  • Give a re-created Cluster a new backup.serverName. The kthx engine does this for a Datastore: each generation archives under its own prefix, <name>-<8 hex characters of the row id>. The plugin refuses a prefix that holds WAL; Recover a re-created database has the steps.

Where it lives#

  • nix/services/garage.nix: Garage on the anchors
  • clusters/<site>/etcd-snapshot/: the etcd snapshots
  • clusters/offsite/garage/: offsite's Garage
  • scripts/backup-coverage.sh, scripts/backup-coverage.exclusions: the volume coverage check
  • clusters/base/platform/velero/, barman-cloud/, backups/: Velero, the plugin and backup-push
  • clusters/base/platform/kthx-engine-target/backup/: the store the kthx Datastores archive to
  • clusters/offsite/apps/restore-drill/, clusters/folly/apps/restore-drill/: the restore drill
  • clusters/<site>/config/cluster-settings.yaml: GARAGE_S3_ENDPOINT, BACKUP_PUSH_SCHEDULE
  • terraform/gcp/projects/homelab-ng/backups.tf: the buckets
  • clusters/base/monitoring/backup-rules.yaml, clusters/base/monitoring/etcd-snapshot-rules.yaml: the alerts

Edit on GitHub

Linked from 15
  • capsuleA Raspberry Pi 5 on NVMe that is one of the three lab DNS and time anchors, and runs NetBox.
  • forgeA Raspberry Pi 5 on NVMe that is the lab's native arm64 build host.
  • oldschoolAn HP EliteDesk 800 G3 mini PC that is the offsite cluster's worker node, on Talos Linux.
  • sporeA Raspberry Pi 5 on NVMe that serves x86 netboot, forge's HTTP boot image, the Talos boot image and Tinkerbell, lab DNS and time, and the SPIRE server.
  • PlatformThe shared systems that the apps and hosts depend on, and what applies each layer from git.
  • NetBoxThe record of every machine in the lab, on capsule, reachable only over the tailnet.
  • PKIThe FML certificate chain, the Kubernetes cluster CAs and token signers, the cluster OIDC issuers, workload identity federation and the hardware DevID CAs.
  • Migrate a local-path volumeMove a PersistentVolumeClaim from a hostPath volume to a local volume under the same claim name, so that Velero can back it up.
  • Migrate Garage to three nodesMove folly's backup store from the one-node Garage on spore to a replication factor 3 cluster on spore, capsule and forge.
  • Operate PostgresFind, connect to, inspect and restart the CloudNativePG Postgres databases with the kubectl cnpg plugin, and check which ones have a backup.
  • Recover a re-created databaseBring a Postgres database back from its barman backups when its CloudNativePG Cluster is created again under the same name, such as after a Kubernetes cluster rebuild.
  • Restore a databaseRecover a Postgres database from its barman backups in Garage into a scratch CloudNativePG cluster, copy the data back, and run the restore drill.
  • Restore a volumeRestore the files of a volume from a Velero backup in Garage, or from the encrypted GCS copy, when the volume is lost or damaged.
  • Restore etcd on TalosRestore the etcd of a Talos control plane from a snapshot, when the etcd data is lost or corrupt.
  • Restore from the GCS copyCopy a Garage bucket back from the encrypted GCS copy of its site, when Garage has lost the bucket.

↑↓ move ↵ open esc close