Restore a site
Copy a kthx quick site's files back from the noncurrent object versions that the sites bucket keeps.
Use this runbook when a deploy or kthx rm lost files of a quick site. The bluenose-kthx-sites bucket keeps three noncurrent versions of each object for 30 days, so a file can come back for that long. A site's database is not in the bucket.
Before you start#
- Sign in to
gcloudas an account that may write thebluenose-kthx-sitesbucket. - Know the site's name and the time of the deploy to undo.
Restore the files#
List every version of the site's objects.
gcloud storage ls --all-versions --long 'gs://bluenose-kthx-sites/<name>/**'Result: each line is a version, with its time and a
#<generation>suffix.For each file to restore, pick the newest version from before the deploy.
Copy that version over the live object.
gcloud storage cp 'gs://bluenose-kthx-sites/<name>/<path>#<generation>' 'gs://bluenose-kthx-sites/<name>/<path>'If
kthx rmdeleted the site, restore its marker<name>/.kthxlast. The copy keeps the owner in its metadata.Make sure the site serves the restored files.
curl -s "https://<name>.kthx.dev/?cachebust=$(date +%s)"Result: the restored page, within about 10 seconds of the copy.
note
After kthx rm, the kthx server drops the site's database within 10 minutes. Restore a database brings back the kthx-db cluster, with every site's later writes lost.
If something goes wrong#
| Symptom | Cause | Action |
|---|---|---|
ls shows no older version | The version is past 30 days, or was the fourth noncurrent one | Deploy the site again from its source. |
The site answers its API with 404 | The marker is not restored | Restore <name>/.kthx. |
cp answers 403 | The account may not write the bucket | Sign in as an account with object write access. |
Related#
Linked from 2
- Quick sitesA kthx quick site is a directory served at https://<name>.kthx.dev from a bucket, with its own database, websocket, visitor ID, file store, model endpoint and MCP server.
- RunbooksStep-by-step procedures for the hosts, clusters, network, secrets and apps, grouped by area.