Kubernetes

• Layer 2 of Architecture: two Kubernetes clusters under clusters/, reconciled by FluxCD on every merge to main. The Terraform bootstrap roots own CoreDNS and Flux itself; Flux owns post-bootstrap cluster state. ArgoCD is installed as a Flux HelmRelease and owns no applications; terraform/argo/ wires the provider and declares no resources. Reconciliation mechanics and the apply model live on Architecture/GitOps.

• Clusters

  • clusters/folly/ — independent, fully capable on-site cluster. Nodes: optiplex (control-plane), riptide, shale (workers).

  • clusters/offsite/ — independent, fully capable remote-site cluster. Nodes: retrofit (control-plane), oldschool (worker).

  • clusters/base/ — resources shared by both clusters, referenced by relative path from each cluster's manifests.

  • Hardware, serials, and per-host quirks for all five nodes: Fleet.

  • The workloads each cluster reconciles: Architecture/Cluster Applications.

• The base/ sharing pattern

  • A cluster's flux-system/kustomization.yaml lists ../../base/flux-system as a resource alongside its own per-domain Kustomizations. That pulls in the shared platform layer — arc-system, external-secrets-operator, onepassword-connect, sandbox (agent-sandbox), valkey-operator — from clusters/base/flux-system/.

  • Individual per-cluster manifests also reference specific base/ subtrees directly wherever an app or component is identical across sites: clusters/folly/apps/kustomization.yaml lists ../../base/apps/iperf3 and ../../base/apps/reloader next to its cluster-only app directories; clusters/folly/storage/kustomization.yaml lists ../../base/storage next to nfs-provisioner; clusters/folly/networking/cloudflare/kustomization.yaml lists ../../../base/networking/cloudflare next to its own tunnel credentials.

  • Where a cluster has no local override at all, its Flux Kustomization custom resource points spec.path straight at the base directory — clusters/offsite/flux-system/storage.yaml's path is ./clusters/base/storage, since offsite carries no cluster-specific storage overlay.

  • clusters/base/kustomization.yaml itself holds just cluster-runtimeclass.yaml and cluster-settings.yaml — the shared settings ConfigMap. Per-domain Flux Kustomizations substitute it alongside their site's topology resources and secrets.

• Bootstrap

  • clusters/<site>/bootstrap/ is a standalone OpenTofu root (state gs://homelab-ng/clusters/<site>/bootstrap) that calls the shared terraform/modules/flux-bootstrap module. The module deploys CoreDNS before installing the flux-operator and flux-instance Helm releases (oci://ghcr.io/controlplaneio-fluxcd/charts) into the flux-system namespace. It also cuts an ECDSA deploy key, registers it read-only against the infra GitHub repository, and writes the key into the flux-github-app-credentials Secret that Flux uses to pull.

  • node-labels.tf in the same root labels each node: node-role.kubernetes.io/control-plane or /worker, plus bgp-enabled: "true" on every node in both clusters.

  • flux-values.yaml supplies the Helm values for flux-instance. Its instance.sync block is what makes the FluxInstance own the root git sync: GitRepository name infra, ref refs/heads/main, path clusters/<site>/flux-system, pullSecret: flux-github-app-credentials. The operator generates the root GitRepository/Kustomization from that block — there is no hand-applied root manifest.

  • This root applies through Atlantis like any other Terraform module — see Architecture/GitOps.

• Nodes

  • clusters/folly/nodes/ adds the Intel device-plugin operator, the GPU device plugin, and node-feature-discovery (intel-uhd-630.yaml) for the folly nodes' integrated GPUs. offsite has no nodes/ directory — no device-plugin layer runs there.

• Storage

  • clusters/base/storage/ provides local-path-provisioner, shared by both clusters.

  • clusters/folly/storage/ layers an NFS provisioner (nfs-provisioner/, backed by spore) and a static PV (spore-pv.yaml) on top. offsite uses base/storage unmodified.

• Monitoring

  • clusters/base/monitoring/ provides metrics-server and the whole log path, shared by both clusters: vector ships kubernetes_logs and journald into victoria-logs, which accepts the Loki push protocol at /insert/loki/api/v1/push. Grafana reads it through the victoriametrics-logs-datasource plugin.

  • clusters/folly/monitoring/ layers kube-prometheus-stack on top, plus Grafana dashboards and hand-written ServiceMonitor/EndpointSlice pairs that scrape node-exporter and CoreDNS metrics off hosts Prometheus can't discover via the Kubernetes API (capsule, spore, cloudpi4, radiopi0, forge). clusters/offsite/monitoring/ runs its own kube-prometheus-stack and Tempo, keeping metric/log/trace traffic on the local network — only Grafana is exposed (grafana-offsite.lolwtf.ca); Prometheus and Alertmanager stay ClusterIP-only. mise run k8s:render-apps renders both clusters' monitoring trees, and mise run k8s:check-rules lints every PrometheusRule that render produces and runs the alert unit tests written beside them in clusters/base/monitoring/.

  • clusters/base/monitoring-crds/ installs the Prometheus Operator CRDs as their own HelmRelease, ahead of monitoring rather than inside it — base/monitoring holds a raw ServiceMonitor, and Flux dry-runs every object in a Kustomization before applying any of them, so a cluster without the CRD already present refuses the whole monitoring Kustomization, including the chart that would supply the CRD. offsite's monitoring Kustomization dependsOn: monitoring-crds for that reason. folly has the same latent ordering problem but is not wired to monitoring-crds yet: its CRDs predate the split and carry no Helm ownership metadata for prometheus-operator-crds to adopt — see Runbooks/Adopt Folly Monitoring CRDs.

• Networking

  • Cilium (CNI + BGP load balancing) and the Gateway API live under each cluster's networking/, built from shared Helm releases in clusters/base/networking/{cert-manager,cloudflare,external-dns,tailscale} plus per-cluster secrets and config. Full detail, including the cross-site firewall gating, is on Architecture/Networking.

• Network facts: the cluster-topology SSOT

  • clusters/<site>/config/cluster-topology.json is the Flux cluster-topology ConfigMap (namespace: flux-system) — applied as-is, because JSON is valid YAML, with no generator step. It's a plain resource in clusters/<site>/config/kustomization.yaml.

  • data is a flat string → string map: CLUSTER_NAME, API_SERVER_IP, API_SERVER_HOSTNAME, API_SERVER_PORT, ROUTER_IP, K8S_NODE_CIDR, CILIUM_POD_CIDR, SERVICE_CIDR, CLUSTER_DNS, CILIUM_NATIVE_ROUTING_CIDR, LB_RANGE, BGP_GATEWAY_ASN, BGP_CILIUM_ASN. It stays flat because Flux's postBuild.substituteFrom only accepts string values, so a list (CLUSTER_DNS) is comma-separated and numbers (API_SERVER_PORT, both ASNs) are stringified.

  • Every other per-domain Flux Kustomization (apps, networking, storage, monitoring, arc-runners) dependsOn: config and substitutes ${VAR} from this ConfigMap — plus cluster-settings and the cluster-secrets Secret — via postBuild.substituteFrom. Folly storage and monitoring also substitute clusters/folly/config/lab-topology.json, the flat-string ConfigMap that owns Lab/future CIDRs and host addresses.

  • .github/workflows/topology-contract.yml runs conftest against both clusters' cluster-topology.json on every touching PR, enforcing the schema in .github/policy/cluster-topology.rego: every required key present and non-empty, IPs/CIDRs/ports/ASNs well-formed, API_SERVER_IP and ROUTER_IP inside K8S_NODE_CIDR, LB_RANGE disjoint from K8S_NODE_CIDR, and no CIDR overlap between the two clusters' files.

  • Consumers beyond Flux: nix/services/k8s/networks.nix reads cluster topology with builtins.fromJSON; nix/lib/lab.nix projects lab topology. OpenTofu roots instantiate terraform/modules/cluster-topology, selecting the site and ConfigMap name. Full picture on Architecture/Networking.

• Secrets

  • SOPS-encrypted in-repo (clusters/**/*.sops.yaml), decrypted per-Kustomization via decryption.provider: sops — see Architecture/Secrets and PKI.

Edit this page on GitHub