GitOps

• How a change actually ships. Desired state lives in git; an operator applies it after a PR check or a merge to main. Nobody applies by hand.

• Apply paths at a glance

  • Layer	Trigger	Applies via	Never
    Terraform (terraform/, clusters/*/bootstrap/)	PR opened/updated → autoplan; atlantis apply comment	Atlantis — successful apply automerges the PR	tofu/terraform apply against remote state, locally
    Kubernetes (clusters/**)	Merge to main	Flux reconciliation	kubectl apply to author state
    NixOS (nix/**)	Merge to main (daily auto-upgrade) or manual dispatch	nixos-rebuild switch/boot	out-of-band edits outside git

• Terraform → Atlantis

  • Every root under terraform/ and each clusters/<site>/bootstrap/ is a standalone Terraform root, applied through Atlantis (clusters/offsite/apps/atlantis, itself a Flux HelmRelease) on the PR — never locally against remote state.

  • The binary is OpenTofu (tofu), not Terraform: Atlantis's chart value defaultTFDistribution: opentofu sets ATLANTIS_DEFAULT_TF_DISTRIBUTION=opentofu, and CI (terraform.yml) installs via opentofu/setup-opentofu and runs tofu init -backend=false, tofu validate, and tofu test per affected root, plus a separate tofu fmt -check -recursive job. mise.toml installs both tofu and terraform, but mise run tf:* (tf:init, tf:validate, tf:fmt, tf:docs, tf:plan) is the local command source of truth.

  • Flow: open a PR touching a root → Atlantis autoplans it (ATLANTIS_AUTOPLAN_MODULES: "true", file list **/*.tf*,terraform/**/*.conf) and runs a policy_check (conftest-style policy set, owners jonpulsifer/rowbutt) → review the plan comment → comment atlantis apply → ATLANTIS_AUTOMERGE: "true" merges the PR once the apply succeeds.

  • trivy.yml additionally scans every changed .tf directory (and clusters/**) for CRITICAL/HIGH IaC findings on the PR.

  • Step-by-step: Runbooks/Terraform Change. Root/module layout: Architecture/Terraform.

• Kubernetes → Flux, and where ArgoCD fits

  • Merge to main → Flux reconciles clusters/**. Each cluster's root sync comes from its FluxInstance — clusters/<site>/bootstrap/bootstrap.tf installs flux-operator/flux-instance, and flux-values.yaml's instance.sync points it at the infra GitRepository (pullSecret: flux-github-app-credentials, ref: refs/heads/main, path: clusters/<site>/flux-system). There's no hand-applied root Kustomization.

  • Flux owns platform namespaces and installs Spindrift's control plane, target RBAC, policy engine, shared authentication, and edge workload. Spindrift then reconciles only its delegated App namespace and resources inside pre-provisioned vessels. This controller boundary is desired state too; Architecture/Spindrift names both sides.

  • CoreDNS is part of the Terraform bootstrap boundary alongside Flux itself. The shared bootstrap module creates cluster DNS before installing Flux; Flux owns the resources that reconcile after bootstrap.

  • kustomize.yml runs the configured render seam over both clusters' app overlays, their arc overlays, and folly monitoring on touching pushes/PRs — a render check, not an apply.

  • topology-contract.yml runs conftest against .github/policy/cluster-topology.rego on both clusters' cluster-topology.json before Flux ever substitutes those values into a manifest.

  • ArgoCD ships the same way as everything else: it's a Flux HelmRelease (clusters/folly/apps/argo). Its RBAC grants an atlantis API-key account full access to applications/applicationsets, and the Atlantis HelmRelease carries ARGOCD_SERVER/ARGOCD_AUTH_TOKEN — wiring for Atlantis to manage ArgoCD Application resources through terraform/argo, an argocd-provider Terraform root (state prefix terraform/argo). As of today that root declares only the provider — its generated README.md reads "No resources" — so ArgoCD is installed but owns no applications. argocd-diff-preview.yaml already posts a diff-preview PR comment for changes under clusters/**/argo/** or terraform/argo/**, ready for when that path is used.

  • Never kubectl apply to author state; kubectl, flux get, flux reconcile are for inspection or forcing a sync.

  • Step-by-step: Runbooks/Kubernetes GitOps Change. Cluster internals: Architecture/Kubernetes.

• NixOS → nixos-rebuild + daily auto-upgrade

  • Apply command: nixos-rebuild switch|boot --flake .#<host> --target-host <host> --sudo. boot installs the generation and activates on the next reboot; switch activates immediately.

  • Every NixOS host — including the k8s nodes (optiplex, riptide, shale, oldschool, retrofit, declared in nix/hosts/, like every other host) — carries system.autoUpgrade enabled by default (nix/system/nixos.nix: flake github:jonpulsifer/infra, daily at 03:37 plus up to a 1h random delay). Hosts self-pull and rebuild from main daily with no workflow run involved.

  • The microSD-rooted Pi 4 and Pi Zero hosts disable system.autoUpgrade; their generations are built elsewhere and pushed with nixos-rebuild --target-host. They update through nixos-deploy.yaml, a manual workflow_dispatch that builds on an aarch64 GitHub runner, joins the tailnet with a short-lived Tailscale OIDC key (tag:ci), and runs nixos-rebuild over SSH to <host>.pirate-musical.ts.net.

  • Implication: a host deployed straight from a branch stays live only until the next auto-upgrade cycle pulls main and reverts it. Merge promptly, or treat a branch deploy as a test.

  • nix-ci.yaml gates all of this: nix flake check, a build of optiplex's closure, a native-ARM build of spore, and the container/wsl/forge image outputs — scoped to changed paths through the same routing mechanism as Terraform (below).

  • Step-by-step: Runbooks/Deploy a NixOS Host. Host inventory: Fleet. Layer background: Architecture/NixOS.

• How CI decides what to validate

  • .github/scripts/validation-impact.sh is the shared routing seam behind both terraform.yml and nix-ci.yaml: given changed paths on stdin, its targets command prints one stable target per line — terraform:<root> (walking up from changed Terraform files, with explicit routes for clients.yaml and lab-topology.json) or nix:flake-check (Nix/flake inputs, apps/ddnsd, dotfiles, both cluster topologies, lab topology, and the Nix workflow).

  • A Terraform root is any directory (under terraform/ or clusters/*/bootstrap) whose .tf file has a backend " block — that's what separates an independently-applied root from a reusable module under terraform/modules/. A change to validation-impact.sh or terraform.yml itself fans out to every root.

  • terraform.yml's changed-directories job feeds the routed roots into a matrix so validate/fmt only run against what changed; nix-ci.yaml's changed-paths job checks for the literal nix:flake-check line to gate its check/nixos/spore/images jobs. A routing-tests job runs validation-impact_test.sh to unit-test the routing script itself.

  • containers.yml and kustomize.yml route independently, by path prefix rather than through validation-impact.sh: containers.yml's detect-containers.sh matches apps/**/packages/**/images/** against .github/containers.json's build/ignore classification — an image with a Dockerfile that isn't classified fails CI. kustomize.yml, trivy.yml, and topology-contract.yml key off their own workflow-level paths: filters.

• Renovate

  • .github/renovate.json5 opens PRs for Terraform providers, Helm charts, container images, GitHub Actions, Nix packages (including the in-repo dotfiles input), and npm packages, grouped and labeled by kind.

  • Automerge (GitHub-native, platformAutomerge: true) is broad but tiered: Terraform provider bumps use update-lockfile and automerge once CI passes — lockfile-only, so no atlantis apply is needed; container/Helm patch+digest and Flux minor/patch bumps automerge; GitHub Actions automerge (majors sit out a 3-day age gate first); non-major npm updates automerge after a 7-day cooldown. mise is not among them: it comes from nixpkgs like any other package, so it moves when the nixpkgs input does and Renovate has nothing to bump.

  • Anything touching clusters/** still only takes effect once its PR merges and Flux reconciles; anything touching a Terraform root with real resource changes still needs a reviewed atlantis apply.

  • An image with deploy targets in .github/containers.json doesn't wait for a Renovate pass: containers.yml's digest step resolves the digest that run just pushed, rewrites the pinned digest in each mapped manifest, and opens an automerging PR — so a merge to main ships the build, and Flux rolls it out on the infra GitRepository's next sync. Renovate still owns the same lines as the backstop.

  • The digest PR needs an App identity to merge. A pull request opened with GITHUB_TOKEN does not get its checks run — GitHub reports every workflow on the branch as action_required and waits for someone to press "Approve and run" — so a required status context is never created and gh pr merge --auto has nothing to wait on. The digest then sits unmerged while every step of the job reports green, and because a CD PR is only opened when none is already open, the next build force-pushes onto the same stuck branch and reuses it, so an image is built, signed, scanned, pushed, and never deployed. containers.yml mints an App installation token when vars.CD_APP_ID is set, which is a different actor and runs like anyone else's PR; with the variable unset it falls back to GITHUB_TOKEN and the old behaviour. The App needs contents: write and pull requests: write on this repo, its id in the CD_APP_ID variable, and its private key in the CD_APP_PRIVATE_KEY secret.

• Docs publish

  • wiki.yml builds docs/ — this Logseq graph — with the Bun SSG in apps/wiki (bun run build → dist/) on every push and PR touching docs/**, apps/wiki/**, or the workflow itself. On main it additionally runs bun x wrangler pages deploy dist --project-name=infra-wiki to Cloudflare Pages (project and DNS Terraform-managed in terraform/network/cloudflare/). A docs page goes live at wiki.lolwtf.ca the moment its PR merges to main — there's no separate publish step.

  • Validate before opening a docs PR: Runbooks/Validate Infra Changes.

Edit this page on GitHub