infra_wiki

Kubernetes

The folly and offsite Kubernetes clusters, how Flux applies clusters/ from git, and the sandbox runtimes and storage the clusters offer.

The lab runs two Kubernetes clusters on Talos Linux nodes: folly at home and offsite at the remote site. clusters/<site>/talos/ declares each cluster's nodes. Flux, the GitOps controller in each cluster, applies clusters/ from main. Workloads lists what runs where.

Parts#

PartJobWhere it runs
Control planeRuns the API server, etcd and podsoptiplex on folly, retrofit on offsite
WorkersRun podsriptide and shale on folly, oldschool on offsite
Bootstrap rootThe OpenTofu root module that installs CoreDNS, flux-operator and the FluxInstance, the object that flux-operator installs Flux fromAtlantis
FluxApplies clusters/ from the infra GitRepositoryThe flux-system namespace
Shared controllersCloudNativePG, External Secrets, Kyverno and othersBoth clusters
Storagelocal-path volumes on each node's data volumeBoth clusters
GPU pluginOffers riptide's GPU to podsfolly

How state reaches a cluster#

A Flux Kustomization is a Flux object that applies one directory.

  1. instance.sync in the bootstrap root's flux-values.yaml points Flux at clusters/<site>/flux-system on main.
  2. That directory holds the Flux Kustomizations of the cluster, and includes clusters/base/flux-system/ for the shared controllers.
  3. Each Flux Kustomization applies its directory in dependsOn order. It decrypts SOPS files only with spec.decryption, and substitutes ${VAR} only from postBuild.substituteFrom.

Sandbox runtimes#

A pod that names no RuntimeClass runs on runc. Every node also has these:

RuntimeClassIsolation
gvisorgVisor, a user-space kernel (runsc)
kata-clhA Cloud Hypervisor microVM

Rowbutt sandboxes run on kata-clh. The packages/charts/app/ chart runs pods on gvisor unless its sandbox value is false.

On offsite, hotrod is the sandbox node. clusters/offsite/talos/talos.tf gives it the lolwtf.ca/sandbox label and a NoSchedule taint of the same key, so only a pod that selects the label and tolerates the taint lands there: the Rowbutt sandboxes, the GitHub Actions runners and the kthx engine's in-cluster build Jobs.

A DaemonSet that should run on every node tolerates the taint in its chart values. On folly, a node's labels and taints come from its class's workload in clusters/folly/fleet/classes.json, through the workloads map in clusters/folly/talos/talos.tf, which both the Talos root and fleetd's render read. Every folly class is general today, which sets none, so folly has no sandbox node. Talos hands a node's taints to the kubelet, which applies them when it registers the Node, and NodeRestriction refuses the kubelet's later change, so a taint declared for a registered node is applied once by hand with kubectl taint.

Signed Talos images#

.github/workflows/talos-images.yml builds folly's Secure Boot images with the siderolabs imager, and its netboot images with Nix. It uses the talos_version, extensions and any extra_kernel_args in clusters/folly/talos/talos.tf, and signs with the keys in PKI.

OutputWhereUse
Installerghcr.io/jonpulsifer/talos-installer-folly:<version>Installs and upgrades a node. Its UKI's talos.config is CONFIG_URL in the workflow: a node's first boot from disk, which has no STATE, fetches its config by the install code in META 0x0b, and every later boot loads the config from STATE. sd-boot's "Reset to maintenance mode" entry wipes STATE but not META, so that boot polls the URL with a burned code instead of entering maintenance mode. It carries no OAuth arguments.
ISOmetal-amd64-secureboot.iso on the release talos-folly/<version>Enrolls the Secure Boot keys at the machine, and rescues a node
iPXEipxe-amd64-secureboot.efi on the same releaseBoots HookOS from its menu. Talos boot has the details.
HookOS UKIhookos-amd64-secureboot-uki.efi on the same releaseRuns the Tinkerbell agent, which wipes a disk before a reinstall

A push to main rebuilds them when the version, the extensions, the kernel arguments, the public keys, clusters/folly/talos/netboot.json or the pipeline change. Nothing builds while netboot.json pins the all-zero agent digest, which the containers workflow's delivery replaces. The nodes still run the Image Factory installer that terraform/modules/talos-cluster/main.tf names. Build the signed Talos images is the procedure.

Rules#

  • Deploy a new containerd handler to every node from main before a workload names its RuntimeClass. The kubelet rejects a pod whose handler is missing.
  • Put a new CRD in its own Flux Kustomization, and add that to the dependsOn of each consumer. Flux dry-runs every object first, so one unknown kind stops the Flux Kustomization. If a HelmRelease installs the CRD, health-check that HelmRelease, as clusters/folly/flux-system/nodes.yaml does.
  • Keep helm.sh/resource-policy: keep in the prometheus-operator-crds HelmRelease values. Without it, an uninstall deletes the Prometheus Operator CRDs and every object of those kinds.
  • A change to only flux-values.yaml in clusters/<site>/bootstrap/ gets no autoplan. Plan and apply it as OpenTofu and Atlantis says.

Where it lives#

  • clusters/<site>/flux-system/ and clusters/base/flux-system/: the Flux Kustomizations
  • clusters/<site>/bootstrap/ and terraform/modules/flux-bootstrap/: the bootstrap root, with its state in gs://homelab-ng/clusters/<site>/bootstrap
  • clusters/base/platform/, clusters/*/storage/ and clusters/folly/nodes/: the shared controllers, storage and GPU plugin
  • clusters/folly/node-features/: the NodeFeatureRule that labels the GPU node
  • clusters/base/cluster-runtimeclass.yaml: the RuntimeClasses. The kata-containers and gvisor extensions in clusters/<site>/talos/talos.tf provide the handlers.
  • clusters/<site>/talos/talos.tf: each node's labels and taints, through terraform/modules/talos-cluster
  • clusters/<site>/config/: the settings, secrets and topology that Flux substitutes. See Topology.

Edit on GitHub

Linked from 27
  • Flame Boss exporterA Go exporter on folly that reads barbecue cooks from Flame Boss's cloud into Prometheus, for a Grafana dashboard and cook alerts in Discord.
  • Weather HubA web dashboard on offsite that shows the owner's WeatherFlow Tempest weather stations, on the kiosk displays and on phones.
  • kthxThe lab's hosting product on the offsite cluster, which the owner and agents use to serve a directory at <name>.kthx.dev or to build a repository and deploy it to a cluster or a cloud.
  • RowbuttA chat bot, code name mate, that answers the owner in Discord and Slack threads with a coding agent whose tools run in a sandbox on the offsite cluster.
  • How Rowbutt worksHow mate runs Rowbutt's agent loop, leases a sandbox on the offsite cluster for its tools and keeps its sessions, and the credentials and network policy of each part.
  • Smiirl counterA Go service on the folly Kubernetes cluster that replaces Smiirl's cloud, so the owner sets what the Smiirl counter shows from a web page.
  • Tidbyt appsSix Pixlet apps, rackstat, tempest, wishin, callerid, burnsafe and flameboss, that the Tronbyt server on folly renders for the lab's Tidbyt pixel displays.
  • hotrodAn HP ProDesk 600 G6 mini PC that is the offsite cluster's sandbox node, the worker that runs the untrusted work, on Talos Linux.
  • oldschoolAn HP EliteDesk 800 G3 mini PC that is the offsite cluster's worker node, on Talos Linux.
  • optiplexA Dell OptiPlex 3050 micro that is the folly cluster's only control-plane node, on Talos Linux.
  • retrofitAn HP EliteDesk 800 G2 mini PC that is the offsite cluster's only control-plane node, on Talos Linux.
  • riptideAn HP EliteDesk 800 G5 mini PC that is a folly worker node with an Intel GPU, on Talos Linux.
  • shaleAn HP EliteDesk 800 G2 mini PC that is a folly worker node, on Talos Linux.
  • Build and releaseHow first-party code builds, how its images and charts publish and deploy, how Renovate updates dependencies, and which CI checks run on a pull request.
  • PlatformThe shared systems that the apps and hosts depend on, and what applies each layer from git.
  • Talos LinuxHow the Talos roots install and configure each Kubernetes node, which nodes boot a signed image, and which nodes seal their disks to the TPM.
  • ObservabilityThe metrics, logs, traces, dashboards and alerts on each Kubernetes cluster, the hosts and Windows desktops they watch, and the daily k6 check.
  • PKIThe FML certificate chain, the Kubernetes cluster CAs and token signers, the cluster OIDC issuers, workload identity federation and the hardware DevID CAs.
  • WorkloadsWhat runs on the folly and offsite Kubernetes clusters, the files that list it, and the workloads that have their own page.
  • GlossaryThe lab-specific terms that the wiki pages use, what each means, and the page that defines it.
  • TopologyThe two JSON files that hold the lab's network facts, the format rule they follow, and the Flux, Nix and OpenTofu code that reads them.
  • Apply a Kubernetes changeChange the manifests under clusters/, share a resource between the two clusters, validate the change, and make sure Flux applies it after the merge.
  • Build the signed Talos imagesRun the Talos image pipeline for folly, and check the signed installer, ISO, iPXE and HookOS that it publishes.
  • Get cluster admin accessGet kubectl access to folly and offsite with the admin certificate from talosctl, and withdraw access.
  • Install Cilium and Flux on TalosWrite the kubeconfig context of a new Talos cluster, install Cilium with its pod IP pool and BGP peering, and bootstrap Flux.
  • Operate PostgresFind, connect to, inspect and restart the CloudNativePG Postgres databases with the kubectl cnpg plugin, and check which ones have a backup.
  • Verify a Talos clusterCheck a cluster that runs Talos after a rebuild or an upgrade, from the nodes and Cilium to federation, backups, OpenBao, the sandbox runtimes, the GPU and Falco.

↑↓ move ↵ open esc close