infra_wiki

Deploy a NixOS host

Deploy the NixOS configuration of a host from your machine or from GitHub Actions, or restore the previous generation.

Use this runbook to deploy a host before its daily auto-upgrade, or to restore the previous generation.

GitHub Actions builds and Cachix serves. A target pulls the store paths it needs from Cachix and builds the rest, as Build host and cache describes.

Before you start#

  • Run mise run devshell.
  • You need SSH access to the host as jawn.
  • Read the host sheet.
  • To restore a host that does not boot, you need its console.
  • To deploy from GitHub Actions, sign in to gh.

<host> is the host name in nix/hosts/default.nix.

Host<target>
Raspberry Pi<host>.<tailnet>

<tailnet> is the tailnet key in terraform/network/tailscale/fleet.tf.json.

Deploy a change#

caution

Auto-upgrade rebuilds each host from main once a day and removes a change deployed from a branch. The Pi 4 hosts have no auto-upgrade.

note

CI pushes to Cachix only from main, so the target builds the store paths that a branch changes.

  1. Deploy the configuration.

    nixos-rebuild boot --sudo --flake .#<host> --build-host <target> --target-host <target>

    If the change must be active now, use switch in place of boot. If the host is a Raspberry Pi, add --no-reexec.

    Result: The command prints Done. The new configuration is and a store path.

  2. If you used boot, reboot the host.

  3. Make sure that no systemd unit failed.

    ssh <target> systemctl --failed --no-pager

    Result: The command prints 0 loaded units listed.

Deploy from GitHub Actions#

The nixos-deploy workflow builds a host on an arm64 runner and deploys a Pi 4 host over Tailscale. Its tag:ci identity reaches only tag:pi4 devices, so it cannot deploy the Pi Zero hosts that it lists. No Pi Zero runs its NixOS config. To deploy any other Raspberry Pi, use Deploy a change.

  1. Run the workflow.

    gh workflow run nixos-deploy.yaml -f host=<host> -f mode=boot

    The workflow deploys only main. If the change must be active now, use mode=switch.

    Result: The command prints Created workflow_dispatch event for nixos-deploy.yaml at main.

  2. Wait for the run to complete. When gh asks for a run, select the nixos-deploy run.

    gh run watch --exit-status

    Result: The command prints Run nixos-deploy (<run-id>) completed with 'success'.

  3. Do steps 2 and 3 of Deploy a change.

Restore the previous generation#

  1. If the host boots, roll back.

    ssh <target> sudo nixos-rebuild switch --rollback

    Result: The command prints Done. The new configuration is /nix/var/nix/profiles/system.

  2. If the host does not boot, restart it from the console. Select the previous generation in the boot menu.

  3. If the change is on main, merge a revert before the next auto-upgrade.

If something goes wrong#

SymptomCauseAction
A deployed change is gone.Auto-upgrade rebuilt the host from main.Merge the change.
A systemd unit failed.The change broke the unit.Run ssh <target> journalctl -u <unit> -n 80 --no-pager.
nixos-rebuild prints did you forget to use --ask-sudo-password?.A remote command failed.Read the lines above that message.
The rollback finds no earlier generation.nix.gc in nix/system/nixos.nix deleted it.Deploy the last good commit.

Edit on GitHub

Linked from 21
  • cloudpi4A wired Raspberry Pi 4 at folly that is a test DNS resolver and an iperf3 server.
  • HostsEvery host in the homelab, with its role, site, hardware, OS and status, and how to reach it.
  • How changes shipThe path each kind of change takes from a pull request to the running system, and the GitOps rule that no one changes live state by hand.
  • Remote accessThe Tailscale tailnet that reaches both sites' private addresses, and the grants that decide who reaches what.
  • NixOSThe flake that builds each Linux host and OS image, the fleet baseline each host shares, and the daily upgrade from main.
  • PKIThe FML certificate chain, the Kubernetes cluster CAs and token signers, the cluster OIDC issuers, workload identity federation and the hardware DevID CAs.
  • GlossaryThe lab-specific terms that the wiki pages use, what each means, and the page that defines it.
  • Add a TPM node to SPIREAdmit a folly node with a TPM 2.0 to spore's SPIRE server by its LDevID, or by its endorsement key hash as the fallback, and make sure its agent serves SVIDs.
  • Change the forge EEPROMChange the boot settings in the EEPROM of forge, and keep its HTTP boot fallback to the rackpi5 image.
  • Get cluster admin accessGet kubectl access to folly and offsite with the admin certificate from talosctl, and withdraw access.
  • RunbooksStep-by-step procedures for the hosts, clusters, network, secrets and apps, grouped by area.
  • Install a Windows desktopInstall the owner's Windows configuration on a desktop with one PowerShell script, update it, add or replace the NixOS WSL distro, and test a change to the Windows dotfiles.
  • Manage SOPS secretsRestore the operator age key, edit or add a SOPS file of encrypted secrets, give a NixOS host its own SOPS file, add a harmonia cache key, and rotate the operator key.
  • Migrate Garage to three nodesMove folly's backup store from the one-node Garage on spore to a replication factor 3 cluster on spore, capsule and forge.
  • Operate the Smiirl counterCheck the Smiirl counter, calibrate its drums, repair it when it does not show the value set on its page, and return it to Smiirl's cloud.
  • Provision a nodeDeclare a folly machine in the nodes file, plug it into its node port, and let it attest and install as a Talos worker or a joining control plane.
  • Repair a kioskCheck the Weather Hub kiosks on homepi4 and weatherpi4, restart them after an error page or a deploy, and start one that shows a blank screen.
  • Retire a nodeDrain a folly worker, back up its volumes, and set its NetBox device to decommissioning, so that fleetd wipes it, then remove its entry from the nodes file.
  • Make a YubiKey a DevID CAGenerate a DevID CA key on the YubiKey of an anchor, have Atlantis certify it under the FML intermediate, and write the certificate into the card.
  • Switch a NetBox token to its own userMint a v2 token for the terraform and fleetd NetBox users, move Terraform and fleetd onto them, and revoke the admin tokens.
  • Test a changeRun the local check for each kind of change before you open a pull request, with the CI workflow that runs the same check.

↑↓ move ↵ open esc close