infra_wiki

Netboot

The network boot service on spore, with TFTP and HTTP for folly's boot files and a signed RAM-boot image that forge falls back to.

Netboot is the network boot service on spore. It serves boot files over TFTP and HTTP, and publishes a signed RAM-boot image for forge, a headless Raspberry Pi 5. Talos boot serves the signed iPXE, HookOS and Talos ISO on the bootstrap VLAN.

Parts#

PartJobWhere it runs
dnsmasqServes TFTP from /var/lib/tftpboot, with no DHCP or DNS, from nix/services/talos-boot.nix. It holds only what Talos boot and Tinkerbell link there.every anchor
nginx spore-pxeServes /var/lib/tftpboot and the signed boot files on HTTP port 80, to the Anchor network and the future network only. TFTP answers on the bootstrap VLAN only. Neither is open to the Kubernetes network.spore
spore-native-boot-rackpi5 (the publisher)Signs and publishes the boot filesspore
spore-native-boot-artifact-check (the artifact check)Fetches each boot file each minute, and exports spore_native_boot_artifact_availablespore

Signed HTTP boot#

forge's EEPROM boots from NVMe first and falls back to rackpi5, a stateless NixOS image that spore serves over HTTP. The EEPROM settings are outside git. BOOT_ORDER digits run right to left: NVMe, HTTP, network, SD, restart. Change the forge EEPROM changes them.

BOOT_ORDER=0xf1276
HTTP_HOST=<SPORE_IP from clusters/folly/config/anchor-topology.json>
HTTP_PATH=rackpi5-ram

The publisher signs boot.img with /var/lib/pi-boot-sign/private.pem, outside the Nix store. It keeps each squashfs under the SHA-256 digest pinned in the signed command line. A rollback on spore publishes that generation's files again.

Rules#

  • The fallback is forge's only recovery path. Change forge's EEPROM only as Change the forge EEPROM says. Without HTTP_HOST, HTTP_PATH and the HTTP digit 7 in BOOT_ORDER, the fallback fails.
  • The BOOT_ORDER comment in nix/hardware/pi5/nvme-hat.nix is capsule's value, not forge's.
  • A stock EEPROM update erases the enrolled public key, and HTTP boot then fails.
  • If you change dnsmasq's single-port TFTP, the digest URL or the nginx unit order, change the spore-reliability check in nix/lib/checks.nix in the same PR, or mise run nix:check fails.

Alerts#

AlertMeaning
SporeTftpDowndnsmasq is not active
SporePxeHttpDownnginx is down
SporeRackpi5PublisherDownThe publisher is not active
SporeNativeBootArtifactUnavailableA boot file does not return HTTP 200

A failed publisher freezes the fallback at the last published image until spore's 7-day GC removes its squashfs. A fresh spore has no fallback.

Where it lives#

  • nix/services/pxe-netboot.nix: nginx and the firewall
  • nix/services/spore-native-boot.nix: the publisher and the artifact check
  • nix/hosts/rackpi5.nix: the fallback image
  • nix/lib/registry.nix: crossHostModules gives spore the piBootImg of rackpi5
  • clusters/folly/config/anchor-topology.json: SPORE_IP
  • clusters/folly/monitoring/spore.yaml: the alerts

Edit on GitHub

Linked from 7
  • forgeA Raspberry Pi 5 on NVMe that is the lab's native arm64 build host.
  • rackpi5The image-only NixOS config that spore signs and serves, and that forge boots over HTTP when its NVMe does not boot.
  • sporeA Raspberry Pi 5 on NVMe that serves x86 netboot, forge's HTTP boot image, the Talos boot image and Tinkerbell, lab DNS and time, and the SPIRE server.
  • NixOSThe flake that builds each Linux host and OS image, the fleet baseline each host shares, and the daily upgrade from main.
  • Build host and cacheHow GitHub Actions builds NixOS closures and images, the Nix binary caches that hosts pull from, and the arm64 build role of forge.
  • Talos bootThe boot server on spore that serves folly's signed iPXE, HookOS and SecureBoot ISO on the bootstrap VLAN, beside fleetd's enrollment name.
  • Change the forge EEPROMChange the boot settings in the EEPROM of forge, and keep its HTTP boot fallback to the rackpi5 image.

↑↓ move ↵ open esc close