infra_wiki

fleetd

The service on spore that checks machine identity and controls folly node installation.

live

fleetd controls how a machine becomes a folly Kubernetes node. It runs on spore and uses NetBox for device state and owner requests.

It checks the machine's identity, authorizes installation, and checks the running node each hour. Tinkerbell runs the installation actions.

Start here#

To learn aboutRead
The system and its trust boundariesFolly design
Provision, wipe and retirement requestsLifecycle
Identity checks and configuration deliveryAttestation and installation
Endpoints, alerts and recovery limitsOperate fleetd

How it works#

  1. A machine boots HookOS, the network installation environment.
  2. Its Trusted Platform Module (TPM) proves possession of the approved endorsement keys (EKs).
  3. fleetd checks the boot evidence and the device's declaration in NetBox.
  4. fleetd issues device certificates and authorizes an eligible installation.
  5. Tinkerbell wipes the declared disk and installs the class's operating system.
  6. The node fetches its configuration once: a Talos class its machine config, a nixos-image or alpine-image class its seed, the JSON that names the profile it switches into, its address and the fleet's endpoints. fleetd then marks the device active. NixOS node image says how a NixOS node's first boot reads its seed. A cloud-init-nocloud class's install fetches its seed, a tar of the NoCloud files, in HookOS and writes it onto the disk, so the device is active before the node's first boot.
  7. The running node sends a signed boot check each hour.

The device certificates are the Local Device Identity (LDevID) and Local Attestation Key (LAK) certificates. The LAK signs later boot checks.

Owner actions#

The owner sets desired_action in NetBox to request provision, attest or wipe. The decommissioning status requests retirement.

Important limits#

  • Provision, wipe and retirement erase data. Each requires the declared disk serial.
  • A successful installation Workflow means the reboot starts. It does not prove that Talos boots or Kubernetes becomes ready.
  • A spore reboot loses pending configurations. Recovery requires a new provision.
  • Retirement cannot revoke an issued kubelet client certificate.
  • TPM 1.2 nodes and class worker-g2 cannot use this installation path.

Reference#

  • Source and HTTP contract: apps/fleetd/
  • Service: nix/services/fleetd.nix
  • Class declarations: clusters/folly/fleet/classes.json
  • Kea reservations, one per NetBox device with a MAC: /var/lib/kea/fleet/reservations.json on spore (Kea)
  • Alerts: clusters/folly/monitoring/fleetd-rules.yaml

Edit on GitHub

Linked from 21
  • fleetd attestation and installationThe checks that authorize a node and deliver its configuration.
  • fleetd lifecycleHow fleetd schedules installation, disk actions and retirement.
  • Operate fleetdAccess, alerts and recovery limits for fleetd.
  • sporeA Raspberry Pi 5 on NVMe that serves x86 netboot, forge's HTTP boot image, the Talos boot image and Tinkerbell, lab DNS and time, and the SPIRE server.
  • HomeThe owner's homelab at two sites, folly and offsite, managed as code: its apps, platform, hosts and runbooks.
  • How changes shipThe path each kind of change takes from a pull request to the running system, and the GitOps rule that no one changes live state by hand.
  • IdentitySPIRE on spore gives each folly node with a TPM 2.0 an identity rooted in its TPM, by LDevID or EK hash, gives its pods SPIFFE IDs, and lets a pod reach GCP with no key file.
  • Talos LinuxHow the Talos roots install and configure each Kubernetes node, which nodes boot a signed image, and which nodes seal their disks to the TPM.
  • KeaThe DHCP server on spore for folly's Kubernetes network, which leases the netboot pool and names the signed iPXE, through the gateway's relay.
  • NixOS node imageThe signed disk image a folly node of the nixos class boots, how CI builds and signs it, and how its first boot reads the fleet seed.
  • Talos bootThe boot server on spore that serves folly's signed iPXE, HookOS and SecureBoot ISO on the bootstrap VLAN, beside fleetd's enrollment name.
  • TinkerbellThe provisioning stack on spore that power-cycles, wipes and reinstalls folly's Talos nodes, driven by fleetd from NetBox, with its own Kubernetes API.
  • Node lifecycleHow a folly node goes from a NetBox device to an attested Talos node with a SPIRE identity, and back out, with the system that owns each step.
  • PKIThe FML certificate chain, the Kubernetes cluster CAs and token signers, the cluster OIDC issuers, workload identity federation and the hardware DevID CAs.
  • Hardware inventoryThe nodes file that declares each folly Kubernetes node, the code that reads it, the OpenTofu root that applies it into NetBox, and the check that compares NetBox with the machines.
  • Get cluster admin accessGet kubectl access to folly and offsite with the admin certificate from talosctl, and withdraw access.
  • Install Talos on a nodeBoot a running NixOS node into the Talos installer with kexec, check it in maintenance mode, and check its volumes after the install.
  • Provision a nodeDeclare a folly machine in the nodes file, plug it into its node port, and let it attest and install as a Talos worker or a joining control plane.
  • Reprovision a nodeReinstall, attest again or wipe an installed folly node by setting its NetBox device's desired_action.
  • Retire a nodeDrain a folly worker, back up its volumes, and set its NetBox device to decommissioning, so that fleetd wipes it, then remove its entry from the nodes file.
  • Rotate the cluster bootstrap tokenReplace folly's cluster-wide Kubernetes bootstrap token in the Talos secrets bundle, apply it to the nodes that carry it, and delete the old token.

↑↓ move ↵ open esc close