PKI
The FML certificate chain, the Kubernetes cluster CAs and token signers, the cluster OIDC issuers, workload identity federation and the hardware DevID CAs.
The FML (Folly Mountain Laboratories) PKI (public key infrastructure) is the private certificate chain of the lab. It signs a ServiceAccount token signer for each cluster. The signers make each cluster an OIDC issuer, which publishes the keys that other systems use to check its ServiceAccount tokens. With the issuers, workloads get GCP credentials and reach the other cluster with no stored key.
Parts#
| Part | Job | Where it runs |
|---|---|---|
| API server | Signs ServiceAccount tokens as the issuer https://oidc.lolwtf.ca/<cluster> | Each control plane |
| OIDC documents | Publish the public keys of each signer | Cloudflare Pages, from terraform/pki/oidc/<cluster>/ |
fml-pool | Exchanges a ServiceAccount token for a GCP credential | GCP |
Chain#
| Certificate | Signed by | Private key |
|---|---|---|
| FML Root CA | Itself | Offline. 1Password holds the certificate only. |
| FML Intermediate CA | FML Root CA | 1Password |
Kubernetes CA of <cluster> | Itself | The Talos secrets bundle, the 1Password item talos-<cluster>-secrets |
<cluster> ServiceAccount token signer | FML K8s <cluster> CA | The Talos secrets bundle, imported from the signer key |
FML DevID folly CA | FML Intermediate CA | terraform/pki state; no service signs with it, and the bundle keeps the LDevIDs it issued trusted |
folly-devid-ca-<host> | FML Intermediate CA | The YubiKey on the anchor <host> |
| A node's LDevID and LAK | A DevID CA | The node's TPM |
terraform/pki issues the signers and the DevID CAs, and its state holds every private key it generates or reads. The root and intermediate do not expire. Signers last one year, and the DevID CAs five years. The intermediate allows one CA below it, so a CA that a signer's issuer signs fails validation.
Both clusters run the self-signed Kubernetes CA from their secrets bundle, so nodes and pods trust that CA and no cluster CA file is committed under terraform/pki/certs/. apps/fml-pki checks the chain with mise run pki:verify and writes the OIDC documents. The other apps/fml-* directories hold an offline root ceremony. No real ceremony transcript is committed.
Workload identity#
fml-pool has one provider for each cluster issuer. Its provider spire takes SPIRE JWT-SVIDs, as Identity says.
| Workload | Cluster | Gets |
|---|---|---|
| OpenBao | folly | The KMS key that unseals it |
ESO store gcp-secret-manager | Both | Secret Manager in bluenose |
| Prowler | offsite | Read access to the GCP organization |
| kthx | offsite | See Ownership and security |
| mate database backup | offsite | Write access to the homelab-ng-mate bucket. See How Rowbutt works. |
backups:backup-push | Both | Access to the homelab-ng-backups-<site> bucket of its site. See Backups. |
Each API server also accepts tokens from the other cluster, for the ServiceAccounts atlantis/atlantis, mate/mate-sandbox-admin, mate/mate-sandbox-reader and spindrift/spindrift only. RBAC in clusters/ binds them as federated:<subject>.
Talos Secure Boot keys#
folly's signed Talos images use the lab's own Secure Boot keys, outside the FML chain. The 1Password item talos-secureboot in the homelab vault holds every key. clusters/folly/talos/secureboot/ holds the public halves. Kubernetes says what the build makes with them.
| Key | Signs | Public half in git |
|---|---|---|
UKI signing key, FML Secure Boot | Each UKI, systemd-boot and the netboot iPXE. The firmware boots them because db holds its certificate. | uki-signing-cert.pem, uki-signing-cert.der |
HookOS signing key, FML HookOS Secure Boot | The HookOS UKI. PCR 7 records which db certificate verified a boot, so a HookOS boot never measures Talos's PCR 7. | hookos-signing-cert.pem, hookos-signing-cert.der |
| PCR signing key | The expected PCR 11 value of each UKI. A TPM opens a sealed disk only under a policy this key signed. | None |
PK, KEK and db | The firmware's key lists. A machine in Setup Mode enrolls them. db holds both lab certificates and the Microsoft UEFI certificates. The UKI signing key signs all three. | PK.auth, KEK.auth, db.auth |
Key constraints#
The UKI and HookOS signing keys are RSA-2048. HP's EliteDesk 800 G2 and G5 firmware refused images signed with RSA-4096 keys whose certificates were in db. The item's PK_auth, KEK_auth and db_auth files match the committed .auth files. The PCR signing key is fleet-wide. Every disk that a TPM seals to the signed policy is bound to it, so a new key leaves each sealed disk unable to open after the next upgrade. Rotate it only with a plan that reinstalls each such node.
DevID CAs#
The DevID CAs sign each attested machine's LDevID and LAK certificates. Each one is a CA under the FML intermediate, which allows no further CA below it. Two of them are hardware CAs, one on a YubiKey in each of two anchors, so losing a key loses one CA, and its machines attest again under the other.
| CA | Key | Anchor |
|---|---|---|
FML DevID folly CA | ECDSA P-256, a software key in the terraform/pki state | Every anchor, from its sops file |
folly-devid-ca-spore | RSA-2048, generated in PIV slot 9c of a YubiKey 5 | spore |
folly-devid-ca-capsule | RSA-2048, generated in PIV slot 9c of a YubiKey 4c | capsule |
A key never leaves its card. Its PIN policy is once and its touch policy is never, so a service signs unattended after one login. The keys are RSA because YubiKey 5 firmware before 5.7 leaks ECDSA keys (Yubico's EUCLEAK advisory), and the 4c's firmware is past the ROCA fix in 4.3.5.
Issuance and deployment#
scripts/devid-ca-yubikey.shsets each card's PIN, PUK and management key, stores them in the host's SOPS file and the 1Password itemdevid-ca-<host>, and commits the key's CSR.terraform/pki/devid.tfissues the software CA.scripts/pki/devid-ca.shwrites it tofolly-devid-ca-software.pemand its chain tofolly-devid-ca-software-chain.peminterraform/pki/certs/.terraform/pki/devid-ca.tfgives each committed CSR a 5-year CA certificate,folly-devid-ca-<host>.pemin the same directory.folly-devid-ca.pemin the same directory bundles every issuing DevID CA, for a verifier that takes one trust file. Afolly-devid-ca-<name>-chain.pembeside a CA is its chain up to the FML root, and stays out of the bundle. spore's SPIRE server lists each CA file inhomelab.spire.devid.caChains, as Identity says.mise run pki:verifyfails when the bundle misses a CA, or a CA or chain file does not chain to the FML intermediate.nix/services/devid-ca-yubikey.nixruns pcscd and the ykcs11 PKCS#11 module on an anchor that setshomelab.devidCa.yubikey.enable. Members of thedevid-cagroup, fleetd on spore, may use the card and read its PIN.packages/pkcs11signeris the Gocrypto.Signerthat signs on the card. It finds the key whose public key matches the CA certificate. fleetd signs every LDevID with it on spore's YubiKey.
| Variable | Value |
|---|---|
ATTEST_CA_PKCS11_MODULE | The ykcs11 module, homelab.devidCa.yubikey.pkcs11Module |
ATTEST_CA_PKCS11_TOKEN | The token label, YubiKey PIV #<serial>, homelab.devidCa.yubikey.tokenLabel |
ATTEST_CA_PKCS11_PIN_FILE | The PIN, homelab.devidCa.yubikey.pinFile |
ATTEST_CA_CERT | A PEM file whose first certificate is the CA's |
Until scripts/pki/devid-ca.sh or the YubiKey runbook commits one, terraform/pki/certs/ holds no DevID CA. Make a YubiKey a DevID CA makes one.
Rules#
- Never trust the FML Root as a client CA of a cluster, or its API server would accept every client certificate under the FML Root.
- Read a DevID CSR before
atlantis apply. The intermediate makes a CA of every CSR thatterraform/pki/devid-ca.tffinds. - After Atlantis applies a new signer, follow the rotation steps in
terraform/pki/README.md. - After Atlantis applies or replaces the DevID CA, run
scripts/pki/devid-ca.shand commit what it writes. A new CA leaves every LDevID untrusted until its node attests again. - For a key change, follow the overlap and escrow steps in
terraform/pki/README.md.prevent_destroyon the escrow item blocks a replacement. - On a Talos cluster, a talosconfig's client certificate lasts one year, and the Talos API cannot revoke it. Issue the owner's and Rowbutt's again before they expire, as Issue a talosconfig says. Only
talosctl rotate-cawithdraws a lost one, and--kubernetes=falselimits the rotation to the Talos API CA, which is all a talosconfig needs. - On folly, rotate the Kubernetes CA only with a plan for the workers fleetd installed,
fleet_workersinclusters/folly/talos/talos.tf. Talos deletes a kubelet certificate that no accepted CA signed, and their kubelets then bootstrap with the token the Talos root gives them, which no Secret holds, so they stay off the cluster until they install again.
Rejoin fleetd's workers after a CA rotation#
A rotation of folly's Kubernetes CA leaves each worker in fleet_workers without a kubelet certificate and without a token that joins. Each one installs again through fleetd, which gives it a token of its own.
Commit the new CA to
clusters/folly/config/kubernetes-ca.pem, and deploy spore frommain, as Deploy a NixOS host describes. fleetd reaches folly's API with that CA.For one worker at a time, follow Reprovision a node. The reinstall erases the node's local-path volumes, so restore them as that runbook says.
Result: the node is
Readywith a new kubelet certificate.
Where it lives#
terraform/pki/: the certificates, the OIDC documents and the rotation procedure inREADME.mdclusters/<site>/talos/patches.tf: the issuer and the rule for tokens from the other clusterterraform/gcp/projects/homelab-ng/workload-identity.tf:fml-poolclusters/folly/talos/secureboot/: the public halves of the Talos Secure Boot keysterraform/pki/devid-ca.tf,scripts/devid-ca-yubikey.shandnix/services/devid-ca-yubikey.nix: the hardware DevID CAs
Related#
Linked from 22
- optiplexA Dell OptiPlex 3050 micro that is the folly cluster's only control-plane node, on Talos Linux.
- retrofitAn HP EliteDesk 800 G2 mini PC that is the offsite cluster's only control-plane node, on Talos Linux.
- BackupsHow Garage, Velero, CloudNativePG and etcd snapshots back up the lab's databases, volumes and etcd, and how each site copies them to GCS.
- Cloud accountsThe Google Cloud projects, Cloudflare zones and Google Workspace domain that OpenTofu declares, and what each one is for.
- IdentitySPIRE on spore gives each folly node with a TPM 2.0 an identity rooted in its TPM, by LDevID or EK hash, gives its pods SPIFFE IDs, and lets a pod reach GCP with no key file.
- KubernetesThe folly and offsite Kubernetes clusters, how Flux applies clusters/ from git, and the sandbox runtimes and storage the clusters offer.
- Talos LinuxHow the Talos roots install and configure each Kubernetes node, which nodes boot a signed image, and which nodes seal their disks to the TPM.
- NixOS node imageThe signed disk image a folly node of the nixos class boots, how CI builds and signs it, and how its first boot reads the fleet seed.
- Talos bootThe boot server on spore that serves folly's signed iPXE, HookOS and SecureBoot ISO on the bootstrap VLAN, beside fleetd's enrollment name.
- OpenTofu and AtlantisThe OpenTofu roots that declare the network, cloud accounts, identity and cluster bootstrap, and the Atlantis server that plans and applies them from pull requests.
- SecretsWhere secrets are stored, what decrypts them, and how they reach hosts and workloads.
- Folly node lifecycle and trust designThe technical design, tradeoffs, evidence and threat boundaries for node provisioning, trusted boot and workload identity at folly.
- GlossaryThe lab-specific terms that the wiki pages use, what each means, and the page that defines it.
- Build the signed Talos imagesRun the Talos image pipeline for folly, and check the signed installer, ISO, iPXE and HookOS that it publishes.
- Get cluster admin accessGet kubectl access to folly and offsite with the admin certificate from talosctl, and withdraw access.
- Initialize OpenBaoInitialize a new OpenBao instance on folly, store its recovery key and root token, and make sure it unseals itself.
- Install kthxInstall the kthx engine on the offsite cluster, from the OpenTofu roots to the first passkey.
- Issue a talosconfigMint the owner's os:admin talosconfig for a Talos cluster and Rowbutt's os:reader one, and store each in 1Password.
- Make a Talos secrets bundleGenerate a cluster's Talos secrets bundle with its imported service-account key, store it in 1Password, and pin its new Kubernetes CA in the cutover pull request.
- Manage SOPS secretsRestore the operator age key, edit or add a SOPS file of encrypted secrets, give a NixOS host its own SOPS file, add a harmonia cache key, and rotate the operator key.
- Prepare a machineSet a new folly machine's firmware and AMT address on its node port and enroll the lab's Secure Boot keys over PXE, before NetBox provisions it.
- Make a YubiKey a DevID CAGenerate a DevID CA key on the YubiKey of an anchor, have Atlantis certify it under the FML intermediate, and write the certificate into the card.