0003 Cluster topology single source of truth

accepted date 2026 (backfilled 2026-07-08) deciders jawn #adr

  • Context

    • Cluster IPs, CIDRs, BGP ASNs, and API endpoints were duplicated across Flux manifests, Nix modules, and Terraform roots. Renumbering anything meant a scavenger hunt, and the copies drifted.

  • Decision

    • Per-cluster cluster-topology ConfigMaps at clusters/<site>/config/cluster-topology.json are the single source of truth for network facts. Each JSON file is the Flux ConfigMap (JSON is valid YAML, applied as-is) — no generator.

    • Consumers read it, never copy it:

      • Flux substitutes ${VAR} via postBuild.substituteFrom

      • Nix reads it with builtins.fromJSON (nix/services/k8s/networks.nix)

      • Terraform roots read it with jsondecode(file(...)).data via a topology.tf each

  • Consequences

    • Renumbering is one file per cluster; every layer picks it up on its normal apply path.

    • data must stay flat string→string (a Flux substituteFrom requirement), so lists and numbers are encoded as strings (CLUSTER_DNS comma-separated, API_SERVER_PORT parsed to int in Nix).

    • Hardcoding a network fact anywhere else is now a review-blocking bug.

    • Not yet migrated: the FRR *.conf BGP files and terraform/network/tailscale/policy.hujson still hold literals.

  • Links

Linked references 8

ADR/0005 Cilium BGP load balancing

Architecture/Networking, ADR/0003 Cluster topology single source of truth, terraform/network/unifi/folly/README.md

ADR

ADR/0003 Cluster topology single source of truth — network facts live in per-cluster JSON ConfigMaps

Architecture/Kubernetes

config/ — cluster secrets and the cluster-topology ConfigMap (ADR/0003 Cluster topology single source of truth)

Architecture/Terraform

Roots that need network facts read the SSOT: jsondecode(file(".../cluster-topology.json")).data via a topology.tf per root (ADR/0003 Cluster topology single source of truth).

Home

Network facts come from the cluster-topology single source of truth — see ADR/0003 Cluster topology single source of truth.

Runbooks/Terraform Change

Network facts should come from the cluster topology single source of truth where a root already has a topology.tf; see ADR/0003 Cluster topology single source of truth.

Architecture/Networking

All cluster network facts — node/API IPs, pod CIDRs, BGP ASNs and peer addresses, DNS — live in per-cluster cluster-topology ConfigMaps at clusters/<site>/config/cluster-topology.json. Flux substitutes them, Nix parses them, Terraform jsondecodes them. Do not hardcode network facts anywhere else. Details in ADR/0003 Cluster topology single source of truth.

Architecture

Single source of truth for network facts. Cluster IPs, CIDRs, ASNs, and API endpoints live in per-cluster cluster-topology ConfigMaps consumed by Flux, Nix, and Terraform alike — ADR/0003 Cluster topology single source of truth.

Edit this page on GitHub