infra_wiki

OpenTofu and Atlantis

The OpenTofu roots that declare the network, cloud accounts, identity and cluster bootstrap, and the Atlantis server that plans and applies them from pull requests.

OpenTofu is the open-source fork of Terraform. The lab uses it to declare the network, the cloud accounts, the PKI and the bootstrap of each cluster. Atlantis, a server on the offsite cluster, plans each changed root, a directory with its own state, on a pull request (PR). It applies the root on a comment. The binary is tofu.

Parts#

PartJobWhere it lives
RootHas a backend "gcs" blockUnder terraform/, clusters/<site>/bootstrap/, and clusters/<site>/talos/ for a cluster on Talos Linux
ModuleShared code that roots callterraform/modules/, and a modules/ directory in some roots
StateOne object for each root in the homelab-ng bucketThe prefix in the root's backend block, which often differs from the root's path
AtlantisPlans, applies and merges PRsclusters/offsite/apps/atlantis/
CIRuns tofu validate and tofu test in each directory with a changed .tf or .tftest.hcl file, and tofu fmt -check.github/workflows/terraform.yml

Atlantis#

Atlantis plans each root with a changed file that matches ATLANTIS_AUTOPLAN_FILE_LIST, and each root that calls a changed module. Before a plan, the hook plan-hook.sh checks the GitHub identity that opened the PR or asked for the plan. only-me.rego, a Rego policy, lists the GitHub identities that pass.

A comment of atlantis apply applies the plans, and Atlantis then merges the PR. Before an apply or an atlantis import, the hook apply-hook.sh checks the commenter against appliers.rego, which admits the owner and clanky-bot[bot]. Both hooks run conftest from the policy mount, so a conftest.toml in a PR cannot change the result. Atlantis ignores the atlantis.yaml files in clusters/<site>/bootstrap/.

only-me.rego and appliers.rego also list clanky-bot[bot], the GitHub App of Rowbutt, so Rowbutt can plan and apply any PR. A plan runs the PR's code with Atlantis's credentials, so every identity in only-me.rego is trusted. The Atlantis ServiceAccount is cluster-admin on both clusters.

Rules#

  • Apply before you merge. A merge without an apply changes nothing, and the next plan shows the change as pending.
  • If a PR changes only a file that a root reads with file(), such as clients.yaml, a topology file or flux-values.yaml, comment atlantis plan -d <root>, then atlantis apply, before you merge. Autoplan sees only files that match ATLANTIS_AUTOPLAN_FILE_LIST, so the change is otherwise never applied.
  • Add every GitHub identity that opens PRs to atlantis_users in only-me.rego. atlantis/plan is a required check on every PR, so a PR from an unlisted GitHub identity is blocked.
  • Read network facts through terraform/modules/cluster-topology, which reads the topology files.
  • Keep terraform/pki on OpenTofu. Its opentofu/tls provider is only on the OpenTofu registry.
  • A plan of a clusters/<site>/talos/ root touches no node. An apply that changes the installer image reboots the nodes, the control plane first, and one that changes kubernetes_version upgrades the Kubernetes components, so read the plan before you comment atlantis apply.

Where it lives#

  • clusters/offsite/apps/atlantis/helm-release.yaml: the autoplan list and repo config
  • clusters/offsite/apps/atlantis/policies/only-me.rego: atlantis_users, who can plan
  • clusters/offsite/apps/atlantis/policies/appliers.rego: atlantis_appliers, who can apply and import
  • clusters/base/atlantis-bootstrap-rbac.yaml: the cluster-admin binding
  • terraform/modules/talos-cluster/: the Talos Linux machine configuration of one cluster, which a clusters/<site>/talos/ root calls. Its README has the inputs.
  • mise.toml: the tf:* tasks. No CI job checks the README tables that tf:docs writes.

Edit on GitHub

Linked from 12
  • Build and releaseHow first-party code builds, how its images and charts publish and deploy, how Renovate updates dependencies, and which CI checks run on a pull request.
  • Cloud accountsThe Google Cloud projects, Cloudflare zones and Google Workspace domain that OpenTofu declares, and what each one is for.
  • How changes shipThe path each kind of change takes from a pull request to the running system, and the GitOps rule that no one changes live state by hand.
  • PlatformThe shared systems that the apps and hosts depend on, and what applies each layer from git.
  • KubernetesThe folly and offsite Kubernetes clusters, how Flux applies clusters/ from git, and the sandbox runtimes and storage the clusters offer.
  • Talos LinuxHow the Talos roots install and configure each Kubernetes node, which nodes boot a signed image, and which nodes seal their disks to the TPM.
  • WorkloadsWhat runs on the folly and offsite Kubernetes clusters, the files that list it, and the workloads that have their own page.
  • GlossaryThe lab-specific terms that the wiki pages use, what each means, and the page that defines it.
  • TopologyThe two JSON files that hold the lab's network facts, the format rule they follow, and the Flux, Nix and OpenTofu code that reads them.
  • Apply an OpenTofu changeChange an OpenTofu root, check it on your machine, and apply it through Atlantis on the pull request.
  • Rebuild a cluster on TalosReplace a cluster's NixOS nodes with Talos Linux, bootstrap a fresh etcd, install Cilium and Flux, and restore the data that has a backup.
  • Release a stale state lockMake sure an OpenTofu state lock is stale, then release it and plan the root again.

↑↓ move ↵ open esc close