24b7ef4c86
All checks were successful
PR build (required check) / changes (pull_request) Successful in 6s
secrets-guard / encrypted (pull_request) Successful in 5s
PR build (required check) / build-backend (pull_request) Has been skipped
shell-lint / shellcheck (pull_request) Successful in 7s
PR build (required check) / build-frontend (pull_request) Has been skipped
PR build (required check) / validate-observability (pull_request) Has been skipped
PR build (required check) / gate (pull_request) Successful in 1s
`caddy validate` run as root creates the log files the config names, owned by root. The caddy user then cannot open them and the reload fails with 'permission denied' on the log writer — which is exactly what happened on the live cutover, and what the root-owned centralis.log in that directory is a fossil of. Caddy rejected the config atomically and kept serving prod on the old one, so nothing broke; but the step reads as a no-op until it isn't.
451 lines
19 KiB
Markdown
451 lines
19 KiB
Markdown
# Runbook — moving beta to vps2 and dev to vps1
|
||
|
||
**Status: not executed.** Everything this runbook describes is landed in the
|
||
repo and live nowhere. The estate still runs the old shape until someone works
|
||
through the steps below.
|
||
|
||
## What changes
|
||
|
||
| | before | after |
|
||
|---|---|---|
|
||
| `75.119.132.91` (**vps1**) | beta + Forgejo + Grafana/Loki | Forgejo + Grafana/Loki + **dev** |
|
||
| `169.58.46.181` (**vps2**) | prod + Centralis + Postfix + backups | **prod + beta** + Centralis + Postfix + backups |
|
||
| desktop | LAN dev server + CI runner | AI models + flex Swarm worker, **no environment** |
|
||
| databases | one Postgres per environment | one instance on vps2, `thermograph` + `thermograph_beta` |
|
||
|
||
Mesh addresses do **not** move: vps1 stays `10.10.0.2`, vps2 stays `10.10.0.1`.
|
||
Every "beta = 75.119.132.91" reference anywhere is wrong after this — that
|
||
address is vps1, and sending a beta-intended command there is the most likely
|
||
way to do damage during this cutover.
|
||
|
||
## Order, and why it is this order
|
||
|
||
Beta moves first and completely, while dev stays where it is. Then dev moves.
|
||
The two halves are independent, so a problem in one never forces a rollback of
|
||
the other — and beta is the half that carries a public hostname and a TLS
|
||
certificate, so it gets done while you have the most attention.
|
||
|
||
Prod is touched twice, and both touches are additive: a new database + role on
|
||
its Postgres instance, and a new site block in its Caddyfile. Prod's stack file,
|
||
its services, its volumes and its env file are not modified by this work at all.
|
||
|
||
**Expected downtime:** beta, roughly the length of a dump/restore plus a stack
|
||
bring-up (~10–20 min). Dev, as long as you like. **Prod: none** — unless step 3
|
||
or 8 is done wrong, which is what the verification lines are for.
|
||
|
||
---
|
||
|
||
## 0. Before you start
|
||
|
||
- [ ] **Lower the DNS TTL** on `beta.thermograph.org` to 300s, at least an hour
|
||
before step 7. Do this first; it is the only step with a lead time.
|
||
- [ ] **Create the new Forgejo Actions secrets** (Settings → Secrets). The
|
||
workflows on this branch reference them and will fail without them:
|
||
|
||
VPS1_SSH_HOST=75.119.132.91 VPS1_SSH_USER=agent
|
||
VPS1_SSH_KEY=<the CI deploy key> VPS1_SSH_PORT=22
|
||
VPS2_SSH_HOST=169.58.46.181 VPS2_SSH_USER=agent
|
||
VPS2_SSH_KEY=<the CI deploy key> VPS2_SSH_PORT=22
|
||
|
||
Keep the old `SSH_*` / `PROD_SSH_*` secrets until step 11 — they are the
|
||
rollback path, and deleting them early strands you.
|
||
|
||
**Create these BEFORE merging.** `infra-sync.yml` fires on any push to
|
||
`dev` or `main` touching `infra/**`, and this change touches a great deal
|
||
of it. Without the new secrets those jobs SSH to an empty host and fail;
|
||
`sync-beta` additionally targets `/opt/thermograph-beta`, which does not
|
||
exist until step 1. Nothing is damaged either way — the jobs only fetch
|
||
and render — but you will get a red run and a misleading alert.
|
||
|
||
The app `Deploy` workflow does **not** fire on this merge: it is
|
||
path-filtered to `backend/**` and `frontend/**`, and this change touches
|
||
neither.
|
||
- [ ] **Take a fresh prod backup and confirm it landed**, rather than trusting
|
||
the schedule:
|
||
|
||
```
|
||
# from a machine with repo access
|
||
# Actions -> "Ops cron (backup + IndexNow)" -> Run workflow
|
||
ssh agent@169.58.46.181 'ls -lh ~/thermograph-backups | tail -3'
|
||
```
|
||
- [ ] **Confirm you can decrypt the vault** (`sops -d infra/deploy/secrets/beta.yaml`
|
||
from `infra/`). Steps 3 and 5 need the rendered beta password.
|
||
- [ ] Have the PR merged to `dev` and promoted to `main`. Hosts pull `main`, so
|
||
nothing below works from an unmerged branch.
|
||
|
||
---
|
||
|
||
## 1. Give beta a checkout on vps2
|
||
|
||
```
|
||
ssh agent@169.58.46.181
|
||
sudo git clone --branch main http://10.10.0.2:3080/emi/thermograph.git /opt/thermograph-beta
|
||
sudo chown -R agent:agent /opt/thermograph-beta
|
||
```
|
||
|
||
Prod's checkout at `/opt/thermograph` is untouched. Two checkouts side by side
|
||
is the whole point: a `git reset --hard` for beta must never be able to move
|
||
prod's tree.
|
||
|
||
**Verify:** `git -C /opt/thermograph-beta log --oneline -1` matches
|
||
`git -C /opt/thermograph log --oneline -1`.
|
||
|
||
---
|
||
|
||
## 2. Render beta's env file on vps2
|
||
|
||
```
|
||
ssh agent@169.58.46.181
|
||
cd /opt/thermograph-beta/infra
|
||
. deploy/render-secrets.sh
|
||
render_thermograph_secrets /opt/thermograph-beta/infra beta /etc/thermograph-beta.env
|
||
```
|
||
|
||
The age key is already at `/etc/thermograph/age.key` on this box (prod uses it);
|
||
beta needs no second copy.
|
||
|
||
**Verify** — beta's file must name beta's role and database, and prod's must be
|
||
untouched:
|
||
|
||
```
|
||
sudo grep -c . /etc/thermograph-beta.env # non-zero
|
||
sudo grep -o 'thermograph_beta' /etc/thermograph-beta.env | head -1
|
||
sudo grep -o '@db:5432/thermograph$' /etc/thermograph.env # prod: still 'thermograph'
|
||
```
|
||
|
||
---
|
||
|
||
## 3. Create beta's role and database on the shared instance
|
||
|
||
This is the first step that touches prod's Postgres. It only ever adds; it drops
|
||
nothing.
|
||
|
||
```
|
||
ssh agent@169.58.46.181
|
||
sudo bash /opt/thermograph-beta/infra/deploy/db/provision-env-db.sh beta
|
||
```
|
||
|
||
**Verify the isolation is real** — this is the whole justification for one
|
||
instance serving two environments, so actually run it:
|
||
|
||
```
|
||
DBC=$(docker ps -q --filter label=com.docker.swarm.service.name=thermograph_db | head -1)
|
||
# beta's role must NOT be able to reach prod's database:
|
||
docker exec "$DBC" psql -U thermograph_beta -d thermograph -c 'select 1' # must FAIL
|
||
# beta's read-only role exists and really is read-only:
|
||
docker exec "$DBC" psql -U thermograph_beta_ro -d thermograph_beta -c 'select 1' # must SUCCEED
|
||
docker exec "$DBC" psql -U thermograph_beta_ro -d thermograph_beta \
|
||
-c 'create table _x(i int)' # must FAIL
|
||
# prod's own role is UNTOUCHED — still a superuser:
|
||
docker exec "$DBC" psql -U thermograph -d postgres -tAc \
|
||
"select rolsuper from pg_roles where rolname='thermograph'" # must be t
|
||
# and prod is still healthy:
|
||
curl -fsS -o /dev/null -w '%{http_code}\n' https://thermograph.org/api/health
|
||
```
|
||
|
||
The script **refuses to run for prod** — prod's `thermograph` role is the
|
||
instance's bootstrap superuser, not a guest, and provisioning it would demote
|
||
it. That guard is why the superuser check above should never fail; run it
|
||
anyway.
|
||
|
||
If the first command SUCCEEDS, stop. `CONNECT` was not revoked and beta would be
|
||
able to open a session against the production database; re-run the provisioning
|
||
script and re-check before going further.
|
||
|
||
This is not hypothetical — it happened on the live cutover. The first version of
|
||
the script revoked `PUBLIC` connect on the *guest's* database only, and a fresh
|
||
Postgres database carries `=Tc` (TEMP **and** CONNECT) for `PUBLIC`, so
|
||
`thermograph_beta` could still reach `thermograph`. The script now hardens the
|
||
owner's database too, granting the owner's read-only role an explicit `CONNECT`
|
||
first so the revoke cannot strip `ops/dbq.sh` of its access. The check above is
|
||
what caught it.
|
||
|
||
**Rollback:** `DROP DATABASE thermograph_beta; DROP ROLE thermograph_beta;` —
|
||
prod is unaffected either way.
|
||
|
||
---
|
||
|
||
## 4. Move beta's data (optional)
|
||
|
||
Beta's existing database still lives on vps1. Decide honestly whether you want
|
||
it: beta's value is a rehearsal of prod, and a fresh schema with a handful of
|
||
test accounts is often *better* than carrying old beta state across. If you do
|
||
want it:
|
||
|
||
```
|
||
# on vps1 — dump the old beta database
|
||
ssh agent@75.119.132.91 \
|
||
'docker exec thermograph-db-1 pg_dump -U thermograph -d thermograph --format=custom' \
|
||
> /tmp/beta-premove.dump
|
||
|
||
# on vps2 — restore into beta's database as beta's role
|
||
scp /tmp/beta-premove.dump agent@169.58.46.181:/tmp/
|
||
ssh agent@169.58.46.181 '
|
||
DBC=$(docker ps -q --filter label=com.docker.swarm.service.name=thermograph_db | head -1)
|
||
docker cp /tmp/beta-premove.dump "$DBC":/tmp/
|
||
docker exec "$DBC" pg_restore -U thermograph -d thermograph_beta --no-owner --role=thermograph_beta /tmp/beta-premove.dump
|
||
'
|
||
```
|
||
|
||
`--no-owner --role=thermograph_beta` matters: the dump's objects are owned by the
|
||
old `thermograph` role, and restoring them verbatim would leave beta's tables
|
||
owned by a role beta does not connect as.
|
||
|
||
**Do not** restore a *prod* dump into beta. Beta's daemon runs the notification
|
||
timers, and beta's subscriber table would then be full of real people. Beta's
|
||
vault sets `THERMOGRAPH_MAIL_BACKEND=console` so nothing would actually send,
|
||
but that is one `sops edit` away from not being true.
|
||
|
||
---
|
||
|
||
## 5. Bring beta's stack up on vps2
|
||
|
||
```
|
||
ssh agent@169.58.46.181
|
||
cd /opt/thermograph-beta
|
||
THERMOGRAPH_ENV=beta SERVICE=all \
|
||
BACKEND_IMAGE_TAG=<current beta backend tag> \
|
||
FRONTEND_IMAGE_TAG=<current beta frontend tag> \
|
||
infra/deploy/deploy.sh
|
||
```
|
||
|
||
Get the current tags from the old beta host first
|
||
(`cat /opt/thermograph/infra/deploy/.image-tags.env` on vps1), so beta comes up
|
||
on exactly the code it was already running.
|
||
|
||
**Verify** — beta answers on its own loopback ports, and prod's are unmoved:
|
||
|
||
```
|
||
curl -fsS -o /dev/null -w 'beta %{http_code}\n' http://127.0.0.1:8237/healthz
|
||
curl -fsS -o /dev/null -w 'prod %{http_code}\n' http://127.0.0.1:8137/healthz
|
||
docker stack services thermograph-beta # 5 services, all 1/1
|
||
docker stack services thermograph # prod: unchanged, still 1/1
|
||
```
|
||
|
||
**Verify the DNS-collision fix actually held** — this is the failure mode that
|
||
would be subtle and awful in production:
|
||
|
||
```
|
||
# prod's frontend must resolve prod's web, not beta's
|
||
docker exec $(docker ps -q -f label=com.docker.swarm.service.name=thermograph_frontend | head -1) \
|
||
getent hosts web
|
||
docker exec $(docker ps -q -f label=com.docker.swarm.service.name=thermograph_frontend | head -1) \
|
||
getent hosts beta-web # should NOT resolve on prod's network
|
||
```
|
||
|
||
**Rollback:** `docker stack rm thermograph-beta` and remove the LB container
|
||
(`docker rm -f thermograph-beta-lb`). Beta on vps1 is still running and still
|
||
serving; nothing has moved yet.
|
||
|
||
---
|
||
|
||
## 6. Add beta's site block to vps2's Caddy
|
||
|
||
`infra/deploy/Caddyfile.vps2` is the reference copy. Append its
|
||
`beta.thermograph.org` block to the live `/etc/caddy/Caddyfile` on vps2.
|
||
|
||
```
|
||
ssh agent@169.58.46.181
|
||
sudo caddy validate --config /etc/caddy/Caddyfile # BEFORE reloading
|
||
# `validate` runs as root and CREATES any log file the config names, owned by
|
||
# root — which the caddy user then cannot open, and the reload fails. Fix the
|
||
# ownership before reloading, or the first reload rejects the whole config:
|
||
sudo chown caddy:caddy /var/log/caddy/beta.log
|
||
sudo systemctl reload caddy
|
||
```
|
||
|
||
Validate before reload, always: a malformed Caddyfile takes `thermograph.org`
|
||
down with it, and prod is on this box now.
|
||
|
||
The `chown` is not optional and it bit this cutover. `caddy validate` as root
|
||
left `/var/log/caddy/beta.log` as `root:root`, so the reload failed with
|
||
`permission denied` on the log writer. (The root-owned `centralis.log` sitting
|
||
in that directory is the same mistake, made earlier.) Note what did NOT happen:
|
||
Caddy rejects a bad config atomically and keeps serving the old one, so
|
||
`thermograph.org` stayed up throughout — verify it anyway.
|
||
|
||
Caddy cannot issue the certificate until DNS moves (step 7), so expect the beta
|
||
hostname to fail TLS until then. That is fine and expected.
|
||
|
||
---
|
||
|
||
## 7. Move DNS
|
||
|
||
Point `beta.thermograph.org`'s A record at **169.58.46.181**.
|
||
|
||
Then wait for propagation and the certificate:
|
||
|
||
```
|
||
dig +short beta.thermograph.org # 169.58.46.181
|
||
curl -fsS -o /dev/null -w '%{http_code}\n' https://beta.thermograph.org/api/health
|
||
ssh agent@169.58.46.181 'sudo journalctl -u caddy -n 30 --no-pager | grep -i certificate'
|
||
```
|
||
|
||
**Rollback:** point the A record back at 75.119.132.91. Beta on vps1 is still
|
||
up (you have not stopped it yet — that is step 9), so this is a clean revert
|
||
for as long as you leave it running.
|
||
|
||
---
|
||
|
||
## 8. Stand dev up on vps1
|
||
|
||
Independent of everything above; do it whenever.
|
||
|
||
```
|
||
ssh agent@75.119.132.91
|
||
sudo bash /opt/thermograph-dev/infra/deploy/provision-dev.sh # clones if absent
|
||
```
|
||
|
||
The script writes `/etc/thermograph/secrets-env` = `dev` and removes any
|
||
`deploy-mode` marker.
|
||
|
||
**That marker flip is load-bearing, and it is why this step comes after step 9.**
|
||
dev's env file is `/etc/thermograph.env` — the very path beta uses while beta is
|
||
still on vps1. `infra-sync`'s `sync-dev` job refuses to render unless the marker
|
||
already says `dev`, so until you run this script it syncs the checkout and skips
|
||
the render with a loud message. Flip the marker before beta has vacated and the
|
||
next dev push overwrites beta's live env file with dev's credentials; beta's
|
||
running containers survive it and then come up wrong on their next deploy.
|
||
|
||
(The checkout itself is safe to create at any time — only the render is gated.)
|
||
|
||
Then deploy:
|
||
|
||
```
|
||
cd /opt/thermograph-dev
|
||
SERVICE=all BACKEND_IMAGE_TAG=<dev tag> FRONTEND_IMAGE_TAG=<dev tag> \
|
||
infra/deploy/deploy-dev.sh
|
||
```
|
||
|
||
**Verify it is mesh-only.** This is the security-relevant check of the whole
|
||
cutover — vps1 is a public box and dev runs unreviewed branches:
|
||
|
||
```
|
||
ss -ltnp | grep 8137 # MUST show 10.10.0.2:8137, never 0.0.0.0:8137
|
||
curl -fsS -o /dev/null -w '%{http_code}\n' http://10.10.0.2:8137/healthz # from the mesh
|
||
curl --max-time 5 http://75.119.132.91:8137/healthz # MUST fail/refuse
|
||
```
|
||
|
||
Also confirm dev did **not** get the fleet's shared production credentials —
|
||
`dev.yaml` is rendered alone, and vps1 is exactly the box that must not hold
|
||
them:
|
||
|
||
```
|
||
sudo grep -c 'THERMOGRAPH_S3_SECRET_KEY\|REGISTRY_TOKEN\|VAPID_PRIVATE' /etc/thermograph.env # expect 0
|
||
```
|
||
|
||
---
|
||
|
||
## 9. Retire beta from vps1
|
||
|
||
Only after beta has been serving from vps2 through step 7 for long enough that
|
||
you believe it.
|
||
|
||
```
|
||
ssh agent@75.119.132.91
|
||
cd /opt/thermograph/infra
|
||
docker compose down # stops the old beta app stack
|
||
# remove beta's site block from /etc/caddy/Caddyfile (see Caddyfile.vps1)
|
||
sudo caddy validate --config /etc/caddy/Caddyfile && sudo systemctl reload caddy
|
||
```
|
||
|
||
**Keep the old `thermograph_pgdata` volume on vps1** until you are certain beta
|
||
on vps2 is healthy and backed up. It is the only copy of pre-move beta data.
|
||
Delete it deliberately, later, not as part of this runbook.
|
||
|
||
Forgejo, Grafana, Loki and the portfolio site all keep running on this box
|
||
untouched — do not stop anything else here.
|
||
|
||
---
|
||
|
||
## 10. Repoint the Alloy agents
|
||
|
||
Both nodes need the new variables; the label model changed (see
|
||
`observability/alloy/config.alloy`).
|
||
|
||
```
|
||
# vps2 — two environments, so two log volumes and the beta overlay
|
||
ssh agent@169.58.46.181
|
||
cd /opt/thermograph/observability/alloy
|
||
ALLOY_NODE=vps2 ALLOY_ENV=prod \
|
||
APPLOGS_VOLUME=thermograph_applogs BETA_APPLOGS_VOLUME=thermograph-beta_applogs \
|
||
LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push \
|
||
docker compose -f docker-compose.agent.yml -f docker-compose.agent.beta.yml up -d
|
||
|
||
# vps1
|
||
ssh agent@75.119.132.91
|
||
cd /opt/thermograph/observability/alloy
|
||
ALLOY_NODE=vps1 ALLOY_ENV=dev APPLOGS_VOLUME=thermograph-dev_applogs \
|
||
LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push \
|
||
docker compose -f docker-compose.agent.yml up -d
|
||
```
|
||
|
||
**Verify each environment is labelled as itself** — the point of the change is
|
||
that beta's logs on vps2 are not filed as prod:
|
||
|
||
```
|
||
# in Grafana Explore, or via the Loki API:
|
||
# {host="beta"} |= "" -> should show beta traffic, from node="vps2"
|
||
# {host="prod"} |= "" -> prod only
|
||
# {host="dev"} |= "" -> from node="vps1"
|
||
# count by (host) (count_over_time({job="docker"}[5m])) -> three values, not one
|
||
```
|
||
|
||
If `{host="beta"}` is empty while beta is clearly serving, the container-name
|
||
relabel did not match — check `docker ps --format '{{.Names}}'` on vps2 against
|
||
the `/thermograph-beta_.*` regex.
|
||
|
||
---
|
||
|
||
## 11. Clean up
|
||
|
||
- [ ] Delete the old `SSH_*` and `PROD_SSH_*` Forgejo secrets (only now — they
|
||
were the rollback path).
|
||
- [ ] Run the ops cron by hand and confirm **both** databases are dumped:
|
||
`ssh agent@169.58.46.181 'ls ~/thermograph-backups | tail -4'` should show
|
||
both a `thermograph-*.dump` and a `thermograph_beta-*.dump`.
|
||
- [ ] Confirm the Forgejo backup job still works — it now uses `VPS1_SSH_*`,
|
||
and it follows Forgejo (vps1), not beta.
|
||
- [ ] Restore the DNS TTL on `beta.thermograph.org`.
|
||
- [ ] Desktop: stop the old LAN dev stack and free the box for the AI models.
|
||
Leave it joined to the Swarm as a worker; nothing schedules onto it today
|
||
(every app service is pinned `node.role == manager`), which is what makes
|
||
it safe to also run models there.
|
||
- [ ] Update Centralis — see below.
|
||
|
||
---
|
||
|
||
## Centralis (outside this repo)
|
||
|
||
Centralis' own text still describes the old estate, and it is not in this
|
||
repository. These need a separate change in the Centralis repo:
|
||
|
||
- The `onboarding` tool's estate paragraph ("Four machines on a WireGuard
|
||
mesh... beta (beta.thermograph.org, Forgejo, Grafana + Loki), the operator's
|
||
desktop (LAN dev, the main CI runner)").
|
||
- The MCP server's own protocol-level `instructions` string, which carries a
|
||
second, separately-worded copy of the same paragraph.
|
||
- The `thermograph-orientation` skill's estate table, and `thermograph-ops`'
|
||
release-flow diagram ("LAN dev box"), its "ssh -L works for beta and is
|
||
impossible for prod" claim (beta is a Swarm overlay now, so it behaves like
|
||
prod), and its prod-only Swarm service-name list (beta has
|
||
`beta-`-prefixed ones now).
|
||
- Tool descriptions that hardcode the old shape: `fleet_status` ("Swarm services
|
||
on prod, compose containers on beta/dev"), `estate_status` and `rollback_to`
|
||
("Centralis cannot SSH to the dev desktop"), `logs_query`'s per-environment
|
||
service lists (Forgejo/Grafana/Loki are listed under `beta`; they belong under
|
||
`dev`/vps1 now), and `secrets_render`'s assumption that a host's own
|
||
`secrets-env` marker identifies one environment — vps2 has two.
|
||
|
||
---
|
||
|
||
## If it goes wrong
|
||
|
||
| symptom | most likely cause | action |
|
||
|---|---|---|
|
||
| prod 502s after step 5 | beta's stack collided with prod's service DNS | `docker stack rm thermograph-beta`; prod recovers on its own. Check beta's services are `beta-`-prefixed |
|
||
| beta 502s, prod fine | beta's LB or stack not up | `docker ps \| grep beta-lb`; `docker stack ps thermograph-beta --no-trunc` |
|
||
| beta can't reach the database | role/database missing, or the wrong network | re-run `provision-env-db.sh beta`; confirm beta's tasks are on `thermograph_internal` |
|
||
| beta serves prod's data | beta's env file has prod's URL | check `/etc/thermograph-beta.env` names `thermograph_beta` twice; re-render (step 2). **Stop beta until fixed** |
|
||
| prod's nightly backup fails | `thermograph_beta` doesn't exist yet | either finish step 3 or revert the ops-cron change; the job fails loudly by design rather than skipping silently |
|
||
| dev reachable from the internet | `DEV_BIND_ADDR` not applied | `ss -ltnp \| grep 8137`; redeploy via `deploy-dev.sh`, which sets it from `env-topology.sh` |
|