thermograph/infra/deploy/RUNBOOK-vps1-vps2-cutover.md

# Runbook — moving beta to vps2 and dev to vps1

**Status: not executed.** Everything this runbook describes is landed in the
repo and live nowhere. The estate still runs the old shape until someone works
through the steps below.

## What changes

| | before | after |
|---|---|---|
| `75.119.132.91` (**vps1**) | beta + Forgejo + Grafana/Loki | Forgejo + Grafana/Loki + **dev** |
| `169.58.46.181` (**vps2**) | prod + Centralis + Postfix + backups | **prod + beta** + Centralis + Postfix + backups |
| desktop | LAN dev server + CI runner | AI models + flex Swarm worker, **no environment** |
| databases | one Postgres per environment | one instance on vps2, `thermograph` + `thermograph_beta` |

Mesh addresses do **not** move: vps1 stays `10.10.0.2`, vps2 stays `10.10.0.1`.
Every "beta = 75.119.132.91" reference anywhere is wrong after this — that
address is vps1, and sending a beta-intended command there is the most likely
way to do damage during this cutover.

## Order, and why it is this order

Beta moves first and completely, while dev stays where it is. Then dev moves.
The two halves are independent, so a problem in one never forces a rollback of
the other — and beta is the half that carries a public hostname and a TLS
certificate, so it gets done while you have the most attention.

Prod is touched twice, and both touches are additive: a new database + role on
its Postgres instance, and a new site block in its Caddyfile. Prod's stack file,
its services, its volumes and its env file are not modified by this work at all.

**Expected downtime:** beta, roughly the length of a dump/restore plus a stack
bring-up (~10–20 min). Dev, as long as you like. **Prod: none** — unless step 3
or 8 is done wrong, which is what the verification lines are for.

---

## 0. Before you start

- [ ] **Lower the DNS TTL** on `beta.thermograph.org` to 300s, at least an hour
      before step 7. Do this first; it is the only step with a lead time.
- [ ] **Create the new Forgejo Actions secrets** (Settings → Secrets). The
      workflows on this branch reference them and will fail without them:

      VPS1_SSH_HOST=75.119.132.91   VPS1_SSH_USER=agent
      VPS1_SSH_KEY=<the CI deploy key>  VPS1_SSH_PORT=22
      VPS2_SSH_HOST=169.58.46.181   VPS2_SSH_USER=agent
      VPS2_SSH_KEY=<the CI deploy key>  VPS2_SSH_PORT=22

      Keep the old `SSH_*` / `PROD_SSH_*` secrets until step 11 — they are the
      rollback path, and deleting them early strands you.

      **Create these BEFORE merging.** `infra-sync.yml` fires on any push to
      `dev` or `main` touching `infra/**`, and this change touches a great deal
      of it. Without the new secrets those jobs SSH to an empty host and fail;
      `sync-beta` additionally targets `/opt/thermograph-beta`, which does not
      exist until step 1. Nothing is damaged either way — the jobs only fetch
      and render — but you will get a red run and a misleading alert.

      The app `Deploy` workflow does **not** fire on this merge: it is
      path-filtered to `backend/**` and `frontend/**`, and this change touches
      neither.
- [ ] **Take a fresh prod backup and confirm it landed**, rather than trusting
      the schedule:

      ```
      # from a machine with repo access
      # Actions -> "Ops cron (backup + IndexNow)" -> Run workflow
      ssh agent@169.58.46.181 'ls -lh ~/thermograph-backups | tail -3'
      ```
- [ ] **Confirm you can decrypt the vault** (`sops -d infra/deploy/secrets/beta.yaml`
      from `infra/`). Steps 3 and 5 need the rendered beta password.
- [ ] Have the PR merged to `dev` and promoted to `main`. Hosts pull `main`, so
      nothing below works from an unmerged branch.

---

## 1. Give beta a checkout on vps2

```
ssh agent@169.58.46.181
sudo git clone --branch main http://10.10.0.2:3080/emi/thermograph.git /opt/thermograph-beta
sudo chown -R agent:agent /opt/thermograph-beta
```

Prod's checkout at `/opt/thermograph` is untouched. Two checkouts side by side
is the whole point: a `git reset --hard` for beta must never be able to move
prod's tree.

**Verify:** `git -C /opt/thermograph-beta log --oneline -1` matches
`git -C /opt/thermograph log --oneline -1`.

---

## 2. Render beta's env file on vps2

```
ssh agent@169.58.46.181
cd /opt/thermograph-beta/infra
. deploy/render-secrets.sh
render_thermograph_secrets /opt/thermograph-beta/infra beta /etc/thermograph-beta.env
```

The age key is already at `/etc/thermograph/age.key` on this box (prod uses it);
beta needs no second copy.

**Verify** — beta's file must name beta's role and database, and prod's must be
untouched:

```
sudo grep -c . /etc/thermograph-beta.env                       # non-zero
sudo grep -o 'thermograph_beta' /etc/thermograph-beta.env | head -1
sudo grep -o '@db:5432/thermograph$' /etc/thermograph.env      # prod: still 'thermograph'
```

---

## 3. Create beta's role and database on the shared instance

This is the first step that touches prod's Postgres. It only ever adds; it drops
nothing.

```
ssh agent@169.58.46.181
sudo bash /opt/thermograph-beta/infra/deploy/db/provision-env-db.sh beta
```

**Verify the isolation is real** — this is the whole justification for one
instance serving two environments, so actually run it:

```
DBC=$(docker ps -q --filter label=com.docker.swarm.service.name=thermograph_db | head -1)
# beta's role must NOT be able to reach prod's database:
docker exec "$DBC" psql -U thermograph_beta -d thermograph -c 'select 1'   # must FAIL
# beta's read-only role exists and really is read-only:
docker exec "$DBC" psql -U thermograph_beta_ro -d thermograph_beta -c 'select 1'          # must SUCCEED
docker exec "$DBC" psql -U thermograph_beta_ro -d thermograph_beta \
  -c 'create table _x(i int)'                                                             # must FAIL
# prod's own role is UNTOUCHED — still a superuser:
docker exec "$DBC" psql -U thermograph -d postgres -tAc \
  "select rolsuper from pg_roles where rolname='thermograph'"                             # must be t
# and prod is still healthy:
curl -fsS -o /dev/null -w '%{http_code}\n' https://thermograph.org/api/health
```

The script **refuses to run for prod** — prod's `thermograph` role is the
instance's bootstrap superuser, not a guest, and provisioning it would demote
it. That guard is why the superuser check above should never fail; run it
anyway.

If the first command SUCCEEDS, stop. `CONNECT` was not revoked and beta would be
able to read production data; re-run the provisioning script and re-check before
going further.

**Rollback:** `DROP DATABASE thermograph_beta; DROP ROLE thermograph_beta;` —
prod is unaffected either way.

---

## 4. Move beta's data (optional)

Beta's existing database still lives on vps1. Decide honestly whether you want
it: beta's value is a rehearsal of prod, and a fresh schema with a handful of
test accounts is often *better* than carrying old beta state across. If you do
want it:

```
# on vps1 — dump the old beta database
ssh agent@75.119.132.91 \
  'docker exec thermograph-db-1 pg_dump -U thermograph -d thermograph --format=custom' \
  > /tmp/beta-premove.dump

# on vps2 — restore into beta's database as beta's role
scp /tmp/beta-premove.dump agent@169.58.46.181:/tmp/
ssh agent@169.58.46.181 '
  DBC=$(docker ps -q --filter label=com.docker.swarm.service.name=thermograph_db | head -1)
  docker cp /tmp/beta-premove.dump "$DBC":/tmp/
  docker exec "$DBC" pg_restore -U thermograph -d thermograph_beta --no-owner --role=thermograph_beta /tmp/beta-premove.dump
'
```

`--no-owner --role=thermograph_beta` matters: the dump's objects are owned by the
old `thermograph` role, and restoring them verbatim would leave beta's tables
owned by a role beta does not connect as.

**Do not** restore a *prod* dump into beta. Beta's daemon runs the notification
timers, and beta's subscriber table would then be full of real people. Beta's
vault sets `THERMOGRAPH_MAIL_BACKEND=console` so nothing would actually send,
but that is one `sops edit` away from not being true.

---

## 5. Bring beta's stack up on vps2

```
ssh agent@169.58.46.181
cd /opt/thermograph-beta
THERMOGRAPH_ENV=beta SERVICE=all \
  BACKEND_IMAGE_TAG=<current beta backend tag> \
  FRONTEND_IMAGE_TAG=<current beta frontend tag> \
  infra/deploy/deploy.sh
```

Get the current tags from the old beta host first
(`cat /opt/thermograph/infra/deploy/.image-tags.env` on vps1), so beta comes up
on exactly the code it was already running.

**Verify** — beta answers on its own loopback ports, and prod's are unmoved:

```
curl -fsS -o /dev/null -w 'beta %{http_code}\n' http://127.0.0.1:8237/healthz
curl -fsS -o /dev/null -w 'prod %{http_code}\n' http://127.0.0.1:8137/healthz
docker stack services thermograph-beta        # 5 services, all 1/1
docker stack services thermograph             # prod: unchanged, still 1/1
```

**Verify the DNS-collision fix actually held** — this is the failure mode that
would be subtle and awful in production:

```
# prod's frontend must resolve prod's web, not beta's
docker exec $(docker ps -q -f label=com.docker.swarm.service.name=thermograph_frontend | head -1) \
  getent hosts web
docker exec $(docker ps -q -f label=com.docker.swarm.service.name=thermograph_frontend | head -1) \
  getent hosts beta-web    # should NOT resolve on prod's network
```

**Rollback:** `docker stack rm thermograph-beta` and remove the LB container
(`docker rm -f thermograph-beta-lb`). Beta on vps1 is still running and still
serving; nothing has moved yet.

---

## 6. Add beta's site block to vps2's Caddy

`infra/deploy/Caddyfile.vps2` is the reference copy. Append its
`beta.thermograph.org` block to the live `/etc/caddy/Caddyfile` on vps2.

```
ssh agent@169.58.46.181
sudo caddy validate --config /etc/caddy/Caddyfile     # BEFORE reloading
sudo systemctl reload caddy
```

Validate before reload, always: a malformed Caddyfile takes `thermograph.org`
down with it, and prod is on this box now.

Caddy cannot issue the certificate until DNS moves (step 7), so expect the beta
hostname to fail TLS until then. That is fine and expected.

---

## 7. Move DNS

Point `beta.thermograph.org`'s A record at **169.58.46.181**.

Then wait for propagation and the certificate:

```
dig +short beta.thermograph.org                  # 169.58.46.181
curl -fsS -o /dev/null -w '%{http_code}\n' https://beta.thermograph.org/api/health
ssh agent@169.58.46.181 'sudo journalctl -u caddy -n 30 --no-pager | grep -i certificate'
```

**Rollback:** point the A record back at 75.119.132.91. Beta on vps1 is still
up (you have not stopped it yet — that is step 9), so this is a clean revert
for as long as you leave it running.

---

## 8. Stand dev up on vps1

Independent of everything above; do it whenever.

```
ssh agent@75.119.132.91
sudo bash /opt/thermograph-dev/infra/deploy/provision-dev.sh   # clones if absent
```

The script writes `/etc/thermograph/secrets-env` = `dev` and removes any
`deploy-mode` marker. Then deploy:

```
cd /opt/thermograph-dev
SERVICE=all BACKEND_IMAGE_TAG=<dev tag> FRONTEND_IMAGE_TAG=<dev tag> \
  infra/deploy/deploy-dev.sh
```

**Verify it is mesh-only.** This is the security-relevant check of the whole
cutover — vps1 is a public box and dev runs unreviewed branches:

```
ss -ltnp | grep 8137            # MUST show 10.10.0.2:8137, never 0.0.0.0:8137
curl -fsS -o /dev/null -w '%{http_code}\n' http://10.10.0.2:8137/healthz   # from the mesh
curl --max-time 5 http://75.119.132.91:8137/healthz                        # MUST fail/refuse
```

Also confirm dev did **not** get the fleet's shared production credentials —
`dev.yaml` is rendered alone, and vps1 is exactly the box that must not hold
them:

```
sudo grep -c 'THERMOGRAPH_S3_SECRET_KEY\|REGISTRY_TOKEN\|VAPID_PRIVATE' /etc/thermograph.env   # expect 0
```

---

## 9. Retire beta from vps1

Only after beta has been serving from vps2 through step 7 for long enough that
you believe it.

```
ssh agent@75.119.132.91
cd /opt/thermograph/infra
docker compose down                    # stops the old beta app stack
# remove beta's site block from /etc/caddy/Caddyfile (see Caddyfile.vps1)
sudo caddy validate --config /etc/caddy/Caddyfile && sudo systemctl reload caddy
```

**Keep the old `thermograph_pgdata` volume on vps1** until you are certain beta
on vps2 is healthy and backed up. It is the only copy of pre-move beta data.
Delete it deliberately, later, not as part of this runbook.

Forgejo, Grafana, Loki and the portfolio site all keep running on this box
untouched — do not stop anything else here.

---

## 10. Repoint the Alloy agents

Both nodes need the new variables; the label model changed (see
`observability/alloy/config.alloy`).

```
# vps2 — two environments, so two log volumes and the beta overlay
ssh agent@169.58.46.181
cd /opt/thermograph/observability/alloy
ALLOY_NODE=vps2 ALLOY_ENV=prod \
APPLOGS_VOLUME=thermograph_applogs BETA_APPLOGS_VOLUME=thermograph-beta_applogs \
LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push \
  docker compose -f docker-compose.agent.yml -f docker-compose.agent.beta.yml up -d

# vps1
ssh agent@75.119.132.91
cd /opt/thermograph/observability/alloy
ALLOY_NODE=vps1 ALLOY_ENV=dev APPLOGS_VOLUME=thermograph-dev_applogs \
LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push \
  docker compose -f docker-compose.agent.yml up -d
```

**Verify each environment is labelled as itself** — the point of the change is
that beta's logs on vps2 are not filed as prod:

```
# in Grafana Explore, or via the Loki API:
#   {host="beta"} |= ""      -> should show beta traffic, from node="vps2"
#   {host="prod"} |= ""      -> prod only
#   {host="dev"}  |= ""      -> from node="vps1"
#   count by (host) (count_over_time({job="docker"}[5m]))   -> three values, not one
```

If `{host="beta"}` is empty while beta is clearly serving, the container-name
relabel did not match — check `docker ps --format '{{.Names}}'` on vps2 against
the `/thermograph-beta_.*` regex.

---

## 11. Clean up

- [ ] Delete the old `SSH_*` and `PROD_SSH_*` Forgejo secrets (only now — they
      were the rollback path).
- [ ] Run the ops cron by hand and confirm **both** databases are dumped:
      `ssh agent@169.58.46.181 'ls ~/thermograph-backups | tail -4'` should show
      both a `thermograph-*.dump` and a `thermograph_beta-*.dump`.
- [ ] Confirm the Forgejo backup job still works — it now uses `VPS1_SSH_*`,
      and it follows Forgejo (vps1), not beta.
- [ ] Restore the DNS TTL on `beta.thermograph.org`.
- [ ] Desktop: stop the old LAN dev stack and free the box for the AI models.
      Leave it joined to the Swarm as a worker; nothing schedules onto it today
      (every app service is pinned `node.role == manager`), which is what makes
      it safe to also run models there.
- [ ] Update Centralis — see below.

---

## Centralis (outside this repo)

Centralis' own text still describes the old estate, and it is not in this
repository. These need a separate change in the Centralis repo:

- The `onboarding` tool's estate paragraph ("Four machines on a WireGuard
  mesh... beta (beta.thermograph.org, Forgejo, Grafana + Loki), the operator's
  desktop (LAN dev, the main CI runner)").
- The MCP server's own protocol-level `instructions` string, which carries a
  second, separately-worded copy of the same paragraph.
- The `thermograph-orientation` skill's estate table, and `thermograph-ops`'
  release-flow diagram ("LAN dev box"), its "ssh -L works for beta and is
  impossible for prod" claim (beta is a Swarm overlay now, so it behaves like
  prod), and its prod-only Swarm service-name list (beta has
  `beta-`-prefixed ones now).
- Tool descriptions that hardcode the old shape: `fleet_status` ("Swarm services
  on prod, compose containers on beta/dev"), `estate_status` and `rollback_to`
  ("Centralis cannot SSH to the dev desktop"), `logs_query`'s per-environment
  service lists (Forgejo/Grafana/Loki are listed under `beta`; they belong under
  `dev`/vps1 now), and `secrets_render`'s assumption that a host's own
  `secrets-env` marker identifies one environment — vps2 has two.

---

## If it goes wrong

| symptom | most likely cause | action |
|---|---|---|
| prod 502s after step 5 | beta's stack collided with prod's service DNS | `docker stack rm thermograph-beta`; prod recovers on its own. Check beta's services are `beta-`-prefixed |
| beta 502s, prod fine | beta's LB or stack not up | `docker ps \| grep beta-lb`; `docker stack ps thermograph-beta --no-trunc` |
| beta can't reach the database | role/database missing, or the wrong network | re-run `provision-env-db.sh beta`; confirm beta's tasks are on `thermograph_internal` |
| beta serves prod's data | beta's env file has prod's URL | check `/etc/thermograph-beta.env` names `thermograph_beta` twice; re-render (step 2). **Stop beta until fixed** |
| prod's nightly backup fails | `thermograph_beta` doesn't exist yet | either finish step 3 or revert the ops-cron change; the job fails loudly by design rather than skipping silently |
| dev reachable from the internet | `DEV_BIND_ADDR` not applied | `ss -ltnp \| grep 8137`; redeploy via `deploy-dev.sh`, which sets it from `env-topology.sh` |