infra_wiki

Restore a volume

Restore the files of a volume from a Velero backup in Garage, or from the encrypted GCS copy, when the volume is lost or damaged.

Use this runbook to restore a PersistentVolumeClaim from a Velero backup. Velero restores the claim, its pod and the files that the node agent backed up with kopia. Only a pod with the annotation backup.velero.io/backup-volumes has a copy of its volumes. Backups describes the schedule and the retention.

warning

This procedure changes live state by hand. It is an exception to the GitOps rule because git holds no data, only the declaration of the volume.

Before you start#

  • Get kubectl access, as Get cluster admin access describes.
  • Use the velero CLI from the mise toolchain. Pass --kubecontext <site> to each velero command.
  • If Garage has lost the velero bucket, follow Restore from the GCS copy. Then wait one minute for Velero to read the bucket.

<site> is folly or offsite.

Restore the volume#

  1. Find the newest completed backup of the schedule daily.

    velero backup get --selector velero.io/schedule-name=daily --kubecontext <site>

    Result: Backup names of the form daily-<timestamp>.

  2. Make sure that the backup holds the volume.

    velero backup describe <backup> --details --kubecontext <site>

    Result: A Pod Volume Backups list with the volume and the phase Completed.

  3. Stop the app that mounts the volume. Suspend its HelmRelease or Kustomization, then scale it to zero.

    flux suspend helmrelease <release> -n <namespace> --context <site>
kubectl scale deployment <app> -n <namespace> --replicas 0 --context <site>

  4. Delete the claim. Velero skips an object that exists.

    kubectl delete pvc <pvc> -n <namespace> --context <site>

    Result: persistentvolumeclaim "<pvc>" deleted.

  5. Keep the scaled-down owner from adopting the restored pod. Velero strips the pod's ownerReferences, but the pod keeps its selector label, so the owner adopts and deletes it, the PodVolumeRestore hangs and the restore queue blocks. Put a resource modifier ConfigMap in the velero namespace. Its rule is a JSON patch on pods in <namespace> that tests the label, then removes it. The label is pod-template-hash for a Deployment, and controller-revision-hash and statefulset.kubernetes.io/pod-name for a StatefulSet. A test that fails leaves the pod unmodified, so make sure the label is gone after step 6.

  6. Restore the pod, the claim and the volume. Do not use --selector. Include persistentvolumes, or the claim stays Pending.

    velero restore create --from-backup <backup> --include-namespaces <namespace> --include-resources pods,persistentvolumeclaims,persistentvolumes --resource-modifier-configmap <modifiers> --wait --kubecontext <site>

    Result: Restore completed with status: Completed.

  7. Make sure that the restore filled the volume.

    velero restore describe <restore> --details --kubecontext <site>

    Result: A Pod Volume Restores list with the phase Completed.

  8. Delete every bare pod in the namespace, so that no owner shares a claim with one. The restore recreates each pod in the backup that has no live namesake, not only the one with the annotation. Then resume the release and scale the owner back up: flux resume and a reconcile leave the replicas at zero.

    kubectl get pods -n <namespace> --context <site> -o json | jq -r '.items[] | select(.metadata.ownerReferences == null) | .metadata.name' | xargs -r kubectl delete pod -n <namespace> --context <site>
flux resume helmrelease <release> -n <namespace> --context <site>
kubectl scale deployment <app> -n <namespace> --replicas 1 --context <site>

  9. Make sure that the app's pod is Running and has the files.

    kubectl exec <pod> -n <namespace> --context <site> -- ls <mount-path>

If something goes wrong#

SymptomCauseAction
velero backup get lists no backups.The BackupStorageLocation is unavailable.Run velero backup-location get --kubecontext <site>. Make sure that Garage answers.
The backup has no Pod Volume Backups.The pod has no annotation, or its PVC is bound to a hostPath volume.Follow Migrate a local-path volume. Restore an older backup that has the volume.
The restore is PartiallyFailed.An object exists, or a plugin is missing.Run velero restore logs <restore> --kubecontext <site>. Delete the object that exists, then restore again.
The restore stays InProgress, and velero restore delete waits.The restored pod does not run, so its PodVolumeRestore never ends.Run kubectl -n velero patch podvolumerestore <pvr> --type merge -p '{"spec":{"cancel":true}}'.
The restored pod is Init for a long time.The init container restores the volume.Wait. Read kubectl logs <pod> -c restore-wait -n <namespace>.

Restore beside the live volume#

Use this procedure to test a backup while the app runs. It leaves the live volume as it is.

warning

A resource modifier patch that fails does not block the object. Velero creates the object unmodified and reports only a namespace error at the end. Use test operations. Make sure that the pod spec is right after creation.

  1. Restore into a new namespace. Include the same three resources as step 6.

    velero restore create --from-backup <backup> --include-namespaces <namespace> --namespace-mappings <namespace>:drill-<namespace> --include-resources pods,persistentvolumeclaims,persistentvolumes --resource-modifier-configmap <modifiers> --wait --kubecontext <site>

  2. Put a resource modifier ConfigMap in the velero namespace first. The rules drop scarce resources such as gpu.intel.com/i915 from limits and requests. They drop hostPath mounts of live data. They set the container command to sleep infinity.

  3. Do not patch /metadata/ownerReferences. Velero removes it, and the patch fails.

  4. Make sure that the restored claim is Bound to a new volume. Compare the files with the live pod.

  5. Delete the namespace drill-<namespace>. If a restore stays InProgress, use the troubleshooting row above.

Edit on GitHub

Linked from 9
  • Tidbyt appsSix Pixlet apps, rackstat, tempest, wishin, callerid, burnsafe and flameboss, that the Tronbyt server on folly renders for the lab's Tidbyt pixel displays.
  • BackupsHow Garage, Velero, CloudNativePG and etcd snapshots back up the lab's databases, volumes and etcd, and how each site copies them to GCS.
  • RunbooksStep-by-step procedures for the hosts, clusters, network, secrets and apps, grouped by area.
  • Migrate a local-path volumeMove a PersistentVolumeClaim from a hostPath volume to a local volume under the same claim name, so that Velero can back it up.
  • Move a cluster's data through a Talos rebuildFreeze a cluster's writers and take its last backups before a Talos rebuild, then restore its databases, volumes and kthx Apps on the new cluster.
  • Reprovision a nodeReinstall, attest again or wipe an installed folly node by setting its NetBox device's desired_action.
  • Restore a databaseRecover a Postgres database from its barman backups in Garage into a scratch CloudNativePG cluster, copy the data back, and run the restore drill.
  • Restore from the GCS copyCopy a Garage bucket back from the encrypted GCS copy of its site, when Garage has lost the bucket.
  • Retire a nodeDrain a folly worker, back up its volumes, and set its NetBox device to decommissioning, so that fleetd wipes it, then remove its entry from the nodes file.

↑↓ move ↵ open esc close