Reprovision a node · infra wiki

Use this runbook to run an installed folly node through Provision a node's path again. You set the device's desired_action in NetBox, and fleetd starts the run.

`desired_action`	The run	After it
`provision`	HookOS attests, the install action wipes the disk and runs the installer with the code in META, and the disk boots	The node fetches its config by the code, and the controller sets `active`
`attest`	Attests, and boots the installed disk	The status stays, and the node has a new LDevID
`wipe`	Wipes the system disk, with no attestation	The node stays in HookOS until a Workflow runs

provision and wipe move the device back to planned, which the Talos root applies no config to, until the node fetches its config. A planned device attests only against the approved EK binding in its nodes-file entry, so a provision of a node without one waits in HookOS; Approve a machine's EKs adds it. attest leaves the status alone, and an active node without a binding attests against its ek_hash pins. A class that the controller does not render, such as the control plane, stays in HookOS after a provision.

warning

This procedure changes live state by hand. It is an exception to the GitOps rule because NetBox holds each machine's lifecycle.

Before you start#

You need the folly context with cluster admin, and the velero CLI.
You can sign in to NetBox at https://capsule.<tailnet>, and you can see the node's console.
<node> is the host name.

warning

provision and wipe erase every partition of the system disk: META, STATE, EPHEMERAL, ETCD and the data volume. Velero copies only the volumes of a running pod with the annotation backup.velero.io/backup-volumes, so take the backup before the drain.

warning

optiplex is folly's only control-plane node. Its wipe destroys folly's etcd. Take a snapshot first, as Restore etcd on Talos describes.

Run it#

If the run wipes the disk, take a Velero backup.

velero backup create <node>-pre-reprovision --from-schedule daily --wait --kubecontext folly
velero backup describe <node>-pre-reprovision --details --kubecontext folly

Result: a Pod Volume Backups list with each volume on the node, in the phase Completed.

Drain the node. Every run reboots it at least once.

kubectl --context folly drain <node> --ignore-daemonsets --delete-emptydir-data

Result: node/<node> drained.

In NetBox, set the device's desired_action.

Result: within two minutes, last_workflow names <action>-<node>-<unix seconds>, and desired_action is none.
If the run wipes the disk, delete the node's Node object once HookOS runs. Before the wipe, the old kubelet registers it again.
```
kubectl --context folly delete node <node>
```
Watch the run, as Watch the run describes. An attest run boots HookOS, which shows attest: PASSED, then the installed disk. A provision run ends when NetBox shows active.
Make sure that the node is Ready, then uncordon it.
```
kubectl --context folly get node <node>
kubectl --context folly uncordon <node>
```
Result: Ready, then node/<node> uncordoned.
If the run wiped the disk, restore the node's volumes, as Restore a volume describes.

If something goes wrong#

Symptom	Cause	Action
The `Workflow` ends `FAILED` within seconds with `BootJobFailed`, and the journal says `ended FAILED before any action ran: its BMC job netboot-<workflow> failed: <message>`	The BMC refused a power action or did not answer, so no action ran	Fix the BMC: the message is the BMC's answer, and `kubectl -n tinkerbell get machine <node> -o yaml` on spore shows whether Rufio reaches it. Then set `desired_action` again
A new `desired_action` starts nothing, and the journal says `<action> waits: Workflow <run> (<its action>) is still running`	A run of another action is unfinished	Wait for it. `desired_action` stays set, and the first pass after the run ends starts it. A run that never reports its first action ends after its global timeout with a journal line that says `never reported its first action`
The journal says `is already running; the request is that run`	You asked for the action of the run that is going	Nothing. The running `Workflow` is the request
The journal says `its record was late`	NetBox refused fleetd's write that names the run, so the request stayed set	Nothing. fleetd adopts the run it started, finished or not, and starts no second one
The power fails during a run	The run stops where the power cut it	Find the symptom in Provision a node: `never reported its first action`, `resumed in a new boot after wipe-disk`, `passed its global timeout`, or a 404 for the config
HP shows 3F0 Boot Device Not Found after HookOS	The firmware has no IPv4 network boot entry	Boot `IPV4 Network` from the F9 menu
`install: no install code`	The boot's attestation got no code	Read the `attest.install` audit line in `journalctl -u fleetd` on spore, then set `desired_action` again
The node polls its config URL and gets 404	The node lost its config before Talos wrote it to STATE, as to a power loss in that second, or STATE was wiped on an installed node. Its code is spent either way. Within a day of the fetch, fleetd recognises the code, journals the ask and fires `FleetInstallConfigRefetched`	Set `desired_action` to `provision`, which renders a new code
The console shows `Talos is already installed to disk but booted from another media`	META is on the disk	Set `desired_action` to `wipe`
The node fetched its config and is cordoned	Its `attest_verdict` is `quarantined: <reason>`, so fleetd cordons it	Read the reason, and clear the verdict once it is explained. The next hourly check that passes restores the node's LDevID; then uncordon the node
`attest: FAILED` with `ek-hash-mismatch`	The approved binding or a pin names another TPM, such as after a board swap	Read the refused EK hash in the `attest.verdict` audit line in `journalctl -u fleetd`, and compare it with your evidence and with `approved_ek_hash` and `ek_hash`. A replaced TPM needs a binding change in the nodes file
`attest: FAILED` with `pending-approval`	A `planned` device whose entry approves no EK	Approve its EKs
Talos stops at `failed to read pcr signing public key`	Talos booted by kexec	Power-cycle the node
The journal says the run waits for `install_disk_serial`	The node's entry has no `disk_serial`, so fleetd starts no run that wipes	Add the serial that `talosctl --nodes <addr> get disks` shows for the install disk, as Hardware inventory says, and apply `terraform/netbox/folly`
Another symptom of a run	See Provision a node	Follow its table

Before you start#

Run it#

If something goes wrong#

Related#