infra_wiki

Get cluster admin access

Get kubectl access to folly and offsite with the admin certificate from talosctl, and withdraw access.

Use this runbook to get kubectl access to the folly and offsite clusters, or to withdraw it. talosctl kubeconfig writes a kubeconfig whose one user is the cluster's admin client certificate, which is in system:masters. It comes from the admin talosconfig that Issue a talosconfig stores in 1Password.

Before you start#

  • You need kubectl, talosctl, op and the item talos-<site>-admin.

<site> is folly or offsite.

Get a kubeconfig#

  1. Add the cluster's admin context to ~/.talos/config.

    talosctl config remove <site> --noconfirm   # an old context; merge renames a duplicate
op document get talos-<site>-admin --vault homelab --out-file /dev/shm/talos-<site>
talosctl config merge /dev/shm/talos-<site> && rm /dev/shm/talos-<site>

  2. Write the kubeconfig for the cluster.

    talosctl --context <site> kubeconfig --merge=false ~/.kube/<site>.yaml

    Result: The file ~/.kube/<site>.yaml exists.

  3. Make sure that the certificate works.

    export KUBECONFIG=~/.kube/<site>.yaml
kubectl get nodes

    Result: Each node shows Ready.

Run the command again to renew an expired certificate. If it fails, check the control plane.

talosctl --context <site> health

Withdraw access#

Use this procedure if you lose a workstation or a talosconfig.

warning

A lost workstation holds the admin certificates and the talosconfig. Nothing can revoke them. Only a CA rotation withdraws them, as PKI describes.

caution

On folly, a Kubernetes CA rotation strands each worker fleetd installed: Talos deletes a kubelet certificate that no accepted CA signed, and the kubelet then bootstraps with the token the Talos root gives it, which no Secret holds. A lost talosconfig alone needs only the Talos API CA, talosctl rotate-ca --kubernetes=false. After a rotation of the Kubernetes CA, do Rejoin fleetd's workers after a CA rotation.

  1. Run talosctl rotate-ca on each cluster, as PKI describes.
  2. Issue a new talosconfig, as Issue a talosconfig describes.

note

nix/system/user.nix gives the jawn user the keys of https://github.com/jonpulsifer.keys (the keys flake input). It gives the rowbutt user, the host user for Rowbutt, the keys of https://github.com/rowbutt.keys (rowbuttkeys). Both users are in wheel, and nix/profiles/fleet.nix gives wheel passwordless sudo. The Talos nodes run no SSH server; these keys reach the NixOS hosts only.

  1. Remove the lost SSH key from the GitHub account that holds it.

  2. Update the keys and rowbuttkeys flake inputs.

    nix flake update keys rowbuttkeys

    Result: • Updated input for each input whose keys changed.

  3. Merge the change through a pull request.

note

The NixOS hosts remove the key at their next auto-upgrade from main. A host that sets system.autoUpgrade.enable = false keeps the key until you deploy it.

  1. Deploy each host that sets system.autoUpgrade.enable = false, as Deploy a NixOS host describes.

If something goes wrong#

SymptomCauseAction
talosctl kubeconfig fails.The Talos API failed, or ~/.talos/config has no context <site>.Run talosctl --context <site> version, or do step 1 of Get a kubeconfig.
kubectl prints You must be logged in to the server.The admin certificate expired.Run step 2 of Get a kubeconfig again.

Edit on GitHub

Linked from 26
  • KubernetesThe folly and offsite Kubernetes clusters, how Flux applies clusters/ from git, and the sandbox runtimes and storage the clusters offer.
  • PKIThe FML certificate chain, the Kubernetes cluster CAs and token signers, the cluster OIDC issuers, workload identity federation and the hardware DevID CAs.
  • Add a TPM node to SPIREAdmit a folly node with a TPM 2.0 to spore's SPIRE server by its LDevID, or by its endorsement key hash as the fallback, and make sure its agent serves SVIDs.
  • Apply a Kubernetes changeChange the manifests under clusters/, share a resource between the two clusters, validate the change, and make sure Flux applies it after the merge.
  • Evaluate the Rowbutt modelsRead what Rowbutt's recorded turns say about each model, and replay its recorded tool decisions against a candidate model before changing MATE_MODEL or the fallback.
  • RunbooksStep-by-step procedures for the hosts, clusters, network, secrets and apps, grouped by area.
  • Install kthxInstall the kthx engine on the offsite cluster, from the OpenTofu roots to the first passkey.
  • Install Talos on a nodeBoot a running NixOS node into the Talos installer with kexec, check it in maintenance mode, and check its volumes after the install.
  • Issue a talosconfigMint the owner's os:admin talosconfig for a Talos cluster and Rowbutt's os:reader one, and store each in 1Password.
  • Migrate a local-path volumeMove a PersistentVolumeClaim from a hostPath volume to a local volume under the same claim name, so that Velero can back it up.
  • Operate PostgresFind, connect to, inspect and restart the CloudNativePG Postgres databases with the kubectl cnpg plugin, and check which ones have a backup.
  • Operate the Flame Boss exporterCheck a cook in the Flame Boss exporter, act on its alerts, change the alert thresholds, replace the Flame Boss token, read the first controller payloads and get a past cook's log.
  • Operate the offsite PBXCheck the offsite PBX, which hands every call on the voip.ms sub-account 168847_elevenlabs to the ElevenLabs troll agent, and find why a call failed.
  • Operate the Rowbutt model fallbackCheck which model answers Rowbutt, keep turns off ChatGPT for a while or until a change, and drill the fallback.
  • Operate the Smiirl counterCheck the Smiirl counter, calibrate its drums, repair it when it does not show the value set on its page, and return it to Smiirl's cloud.
  • Recover a re-created databaseBring a Postgres database back from its barman backups when its CloudNativePG Cluster is created again under the same name, such as after a Kubernetes cluster rebuild.
  • Repair the Rowbutt GitHub credentialCheck, resync, rotate and replace the GitHub App credential that Rowbutt's sandboxes push with.
  • Restore a databaseRecover a Postgres database from its barman backups in Garage into a scratch CloudNativePG cluster, copy the data back, and run the restore drill.
  • Restore a volumeRestore the files of a volume from a Velero backup in Garage, or from the encrypted GCS copy, when the volume is lost or damaged.
  • Restore etcd on TalosRestore the etcd of a Talos control plane from a snapshot, when the etcd data is lost or corrupt.
  • Retire a nodeDrain a folly worker, back up its volumes, and set its NetBox device to decommissioning, so that fleetd wipes it, then remove its entry from the nodes file.
  • Rotate the cluster bootstrap tokenReplace folly's cluster-wide Kubernetes bootstrap token in the Talos secrets bundle, apply it to the nodes that carry it, and delete the old token.
  • Rotate the Rowbutt Slack tokenReplace the Slack App-Level Token that Rowbutt's Socket Mode connection uses, and revoke the tokens of clients that share its events.
  • Screen callers on the office phoneHow the folly PBX screens unknown callers on line 4 and hands them to the ElevenLabs troll agent, how to add or remove a contact who skips the screen, what the star codes do, how to render the prompts, and how to test the screen.
  • Sign Rowbutt in to ChatGPTSign mate in to the owner's ChatGPT subscription from chat, check the sign-in, and sign it out.
  • Verify a Talos clusterCheck a cluster that runs Talos after a rebuild or an upgrade, from the nodes and Cilium to federation, backups, OpenBao, the sandbox runtimes, the GPU and Falco.

↑↓ move ↵ open esc close