kthx
The lab's hosting product on the offsite cluster, which the owner and agents use to serve a directory at <name>.kthx.dev or to build a repository and deploy it to a cluster or a cloud.
kthx is the lab's hosting product. The owner and agents use it to put apps on the internet or on the lab network. Its quick-site server and built-app engine run on the offsite Kubernetes cluster, and each built app runs on its Target. It has two kinds of app.
| Kind | You give it | You get |
|---|---|---|
| Quick site | A directory, or a page the builder writes | https://<name>.kthx.dev, with a database, a websocket, a visitor ID and a file store |
| Built app | A GitHub repository or an uploaded archive | A signed build, deployed to a cluster, Cloud Run, Firebase Hosting, Vercel or Cloudflare Pages |
Ownership and security says which system owns each part and what it can reach.
Use it#
| Surface | Address | Who can reach it |
|---|---|---|
| Quick sites | https://<name>.kthx.dev | Anyone |
| Landing page, site directory, SDK, agent reference and CLI | https://kthx.dev | Anyone |
| Console | https://kthx.lolwtf.ca | The lab, the offsite LAN and the tailnet owner. The console needs a passkey. |
| Quick-site builder | https://kthx.<tailnet> | Every tailnet member |
| MCP endpoint for built apps | https://kthx-engine.lolwtf.dev/mcp | The internet, with an agent token |
How it works#
nginx in the kthx-sites namespace serves quick sites' files from the bluenose-kthx-sites bucket, and one Bun process in the kthx namespace serves their API. The kthx engine in the spindrift namespace builds and deploys built apps. The engine is a web process for the console, API and MCP, a reconciler that builds and deploys, and their CloudNativePG database.
Quick sites and the engine share the Cloudflare tunnel named kthx-engine, called the kthx Apps tunnel, and the kthx-apps Gateway on offsite. They also share the archive reader in packages/archive/ and the bluenose GCP project, and each has its own database and bucket.
Flux deploys quick sites from packages/charts/kthx/ in git. It deploys the engine, and the engine its Apps, from the OCI charts that .github/workflows/spindrift-charts.yml publishes. Build and release has the rule for a chart change.
Operate#
| Alert | Meaning | Runbook |
|---|---|---|
KthxEngineDeployFailed | A deploy ended FAILED. The console's Deploys page shows the reason. | |
KthxEngineDeployLost | A reconciler stopped during a deploy. Read its pod's logs. |
Quick sites lists its alerts. Install kthx installs the engine, and Connect an agent to kthx gives an agent the MCP endpoint. Operate Postgres covers both databases.
Reference#
- Quick sites:
apps/kthx/,packages/kthx/,packages/charts/kthx/andclusters/offsite/apps/kthx/. Imageghcr.io/jonpulsifer/kthx. - Built apps:
apps/kthx-engine/,packages/charts/kthx-engine/,packages/charts/kthx-app/,clusters/offsite/apps/kthx-engine/andclusters/base/platform/kthx-engine-target/. Imageghcr.io/jonpulsifer/kthx-engine. - Cloud:
terraform/network/cloudflare/kthx.dev.tf,terraform/network/cloudflare/kthx-engine.tfandterraform/gcp/projects/bluenose/
Linked from 15
- Built appsA kthx built app is a repository or archive that kthx builds into a signed artifact and deploys to Targets on the clusters, GCP, Vercel or Cloudflare.
- Ownership and securityWhich system owns each part of kthx, the identities and federation each part uses, and how built images are signed and admitted.
- Quick sitesA kthx quick site is a directory served at https://<name>.kthx.dev from a bucket, with its own database, websocket, visitor ID, file store, model endpoint and MCP server.
- RowbuttA chat bot, code name mate, that answers the owner in Discord and Slack threads with a coding agent whose tools run in a sandbox on the offsite cluster.
- hotrodAn HP ProDesk 600 G6 mini PC that is the offsite cluster's sandbox node, the worker that runs the untrusted work, on Talos Linux.
- oldschoolAn HP EliteDesk 800 G3 mini PC that is the offsite cluster's worker node, on Talos Linux.
- Cloud accountsThe Google Cloud projects, Cloudflare zones and Google Workspace domain that OpenTofu declares, and what each one is for.
- How changes shipThe path each kind of change takes from a pull request to the running system, and the GitOps rule that no one changes live state by hand.
- Ingress and DNSThe Gateways, DNS records, certificates, tunnels and lab resolvers that give cluster services a name and a path from the LAN, tailnet or internet.
- Remote accessThe Tailscale tailnet that reaches both sites' private addresses, and the grants that decide who reaches what.
- GlossaryThe lab-specific terms that the wiki pages use, what each means, and the page that defines it.
- Connect an agent to kthxMint an agent token in the kthx console and connect an MCP client, Rowbutt or the owner's workstation to the kthx engine, which serves each console command as a tool.
- Install kthxInstall the kthx engine on the offsite cluster, from the OpenTofu roots to the first passkey.
- Operate PostgresFind, connect to, inspect and restart the CloudNativePG Postgres databases with the kubectl cnpg plugin, and check which ones have a backup.
- Test a changeRun the local check for each kind of change before you open a pull request, with the CI workflow that runs the same check.