infra/deploy/RUNBOOK-vps1-vps2-cutover.md

@@ -149,8 +149,16 @@ it. That guard is why the superuser check above should never fail; run it
 anyway.
 
 If the first command SUCCEEDS, stop. `CONNECT` was not revoked and beta would be
-able to read production data; re-run the provisioning script and re-check before
+able to open a session against the production database; re-run the provisioning
-going further.
+script and re-check before going further.
+
+This is not hypothetical — it happened on the live cutover. The first version of
+the script revoked `PUBLIC` connect on the *guest's* database only, and a fresh
+Postgres database carries `=Tc` (TEMP **and** CONNECT) for `PUBLIC`, so
+`thermograph_beta` could still reach `thermograph`. The script now hardens the
+owner's database too, granting the owner's read-only role an explicit `CONNECT`
+first so the revoke cannot strip `ops/dbq.sh` of its access. The check above is
+what caught it.
 
 **Rollback:** `DROP DATABASE thermograph_beta; DROP ROLE thermograph_beta;` —
 prod is unaffected either way.
@@ -239,12 +247,23 @@ serving; nothing has moved yet.
 ```
 ssh agent@169.58.46.181
 sudo caddy validate --config /etc/caddy/Caddyfile     # BEFORE reloading
+# `validate` runs as root and CREATES any log file the config names, owned by
+# root — which the caddy user then cannot open, and the reload fails. Fix the
+# ownership before reloading, or the first reload rejects the whole config:
+sudo chown caddy:caddy /var/log/caddy/beta.log
 sudo systemctl reload caddy
 ```
 
 Validate before reload, always: a malformed Caddyfile takes `thermograph.org`
 down with it, and prod is on this box now.
 
+The `chown` is not optional and it bit this cutover. `caddy validate` as root
+left `/var/log/caddy/beta.log` as `root:root`, so the reload failed with
+`permission denied` on the log writer. (The root-owned `centralis.log` sitting
+in that directory is the same mistake, made earlier.) Note what did NOT happen:
+Caddy rejects a bad config atomically and keeps serving the old one, so
+`thermograph.org` stayed up throughout — verify it anyway.
+
 Caddy cannot issue the certificate until DNS moves (step 7), so expect the beta
 hostname to fail TLS until then. That is fine and expected.
 

infra/deploy/db/provision-env-db.sh

@@ -156,6 +156,46 @@ ALTER DEFAULT PRIVILEGES FOR ROLE :"role" IN SCHEMA public
   GRANT SELECT ON SEQUENCES TO :"rorole";
 SQL
 
+# --- harden the OWNER environment's database too -------------------------------
+#
+# Revoking CONNECT from PUBLIC on the guest's own database is only half the job,
+# and the missing half is the half that matters. A fresh Postgres database
+# carries `=Tc` for PUBLIC — TEMP *and* CONNECT — so the moment a second role
+# exists on the instance, that role can connect to every database that still has
+# the default. Provisioning beta and stopping here left `thermograph_beta` able
+# to open a session against prod's database; caught live by the runbook's own
+# isolation check, which is why that check exists.
+#
+# Order matters: grant the owner's read-only role an EXPLICIT connect first, so
+# the revoke below cannot strip the ad-hoc query path (ops/dbq.sh) of its access.
+# The owner's app role is the database owner and a superuser, so it is unaffected
+# either way.
+OWNER_DB=$(
+  # shellcheck disable=SC1090 # sourced above; re-resolving prod in a subshell
+  thermograph_topology prod >/dev/null 2>&1
+  printf '%s' "$TG_DB_NAME"
+)
+OWNER_RO=$(
+  thermograph_topology prod >/dev/null 2>&1
+  printf '%s_ro' "$TG_DB_USER"
+)
+# Restore this run's environment — thermograph_topology overwrote the TG_* vars.
+thermograph_topology "$ENV_NAME"
+
+if [ "$OWNER_DB" != "$TG_DB_NAME" ]; then
+  echo "==> Hardening the owner database '${OWNER_DB}' against PUBLIC connects"
+  docker exec -i "$DB_CID" \
+    psql -v ON_ERROR_STOP=1 -U thermograph -d postgres \
+         -v ownerdb="$OWNER_DB" -v ownerro="$OWNER_RO" <<'SQL'
+-- Only if that read-only role actually exists (it predates this script).
+SELECT format('GRANT CONNECT ON DATABASE %I TO %I', :'ownerdb', :'ownerro')
+WHERE EXISTS (SELECT 1 FROM pg_roles WHERE rolname = :'ownerro')
+\gexec
+
+REVOKE CONNECT ON DATABASE :"ownerdb" FROM PUBLIC;
+SQL
+fi
+
 echo "==> OK: ${TG_DB_USER} owns ${TG_DB_NAME} (timescaledb enabled, PUBLIC revoked)"
 echo "    Verify isolation:"
 echo "      docker exec $DB_CID psql -U ${TG_DB_USER} -d thermograph -c 'select 1'   # must FAIL"