From 1aeea6bdc5059493bac9e05839c2e8bac919ee77 Mon Sep 17 00:00:00 2001 From: Emi Griffith Date: Sun, 26 Jul 2026 00:12:25 -0700 Subject: [PATCH 1/2] provision-env-db: revoke PUBLIC connect on the OWNER database too MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Revoking PUBLIC's CONNECT on the guest's own database is the half that does not matter. A fresh Postgres database carries '=Tc' for PUBLIC — TEMP and CONNECT — so as soon as a second role exists on the instance it can open a session against every database still holding the default. Provisioning beta and stopping there left thermograph_beta able to connect to prod's database. The runbook's isolation check caught it on the live cutover; this makes the script produce the state that check expects instead of a false sense of it. Grants the owner's read-only role an explicit CONNECT before the revoke, so the ad-hoc query path (ops/dbq.sh, which connects as thermograph_ro) cannot lose access to prod. The owner's app role owns the database and is a superuser, so it is unaffected either way. --- infra/deploy/RUNBOOK-vps1-vps2-cutover.md | 12 +++++-- infra/deploy/db/provision-env-db.sh | 40 +++++++++++++++++++++++ 2 files changed, 50 insertions(+), 2 deletions(-) diff --git a/infra/deploy/RUNBOOK-vps1-vps2-cutover.md b/infra/deploy/RUNBOOK-vps1-vps2-cutover.md index b1874da..3bbfcd6 100644 --- a/infra/deploy/RUNBOOK-vps1-vps2-cutover.md +++ b/infra/deploy/RUNBOOK-vps1-vps2-cutover.md @@ -149,8 +149,16 @@ it. That guard is why the superuser check above should never fail; run it anyway. If the first command SUCCEEDS, stop. `CONNECT` was not revoked and beta would be -able to read production data; re-run the provisioning script and re-check before -going further. +able to open a session against the production database; re-run the provisioning +script and re-check before going further. + +This is not hypothetical — it happened on the live cutover. The first version of +the script revoked `PUBLIC` connect on the *guest's* database only, and a fresh +Postgres database carries `=Tc` (TEMP **and** CONNECT) for `PUBLIC`, so +`thermograph_beta` could still reach `thermograph`. The script now hardens the +owner's database too, granting the owner's read-only role an explicit `CONNECT` +first so the revoke cannot strip `ops/dbq.sh` of its access. The check above is +what caught it. **Rollback:** `DROP DATABASE thermograph_beta; DROP ROLE thermograph_beta;` — prod is unaffected either way. diff --git a/infra/deploy/db/provision-env-db.sh b/infra/deploy/db/provision-env-db.sh index 334612c..3695027 100755 --- a/infra/deploy/db/provision-env-db.sh +++ b/infra/deploy/db/provision-env-db.sh @@ -156,6 +156,46 @@ ALTER DEFAULT PRIVILEGES FOR ROLE :"role" IN SCHEMA public GRANT SELECT ON SEQUENCES TO :"rorole"; SQL +# --- harden the OWNER environment's database too ------------------------------- +# +# Revoking CONNECT from PUBLIC on the guest's own database is only half the job, +# and the missing half is the half that matters. A fresh Postgres database +# carries `=Tc` for PUBLIC — TEMP *and* CONNECT — so the moment a second role +# exists on the instance, that role can connect to every database that still has +# the default. Provisioning beta and stopping here left `thermograph_beta` able +# to open a session against prod's database; caught live by the runbook's own +# isolation check, which is why that check exists. +# +# Order matters: grant the owner's read-only role an EXPLICIT connect first, so +# the revoke below cannot strip the ad-hoc query path (ops/dbq.sh) of its access. +# The owner's app role is the database owner and a superuser, so it is unaffected +# either way. +OWNER_DB=$( + # shellcheck disable=SC1090 # sourced above; re-resolving prod in a subshell + thermograph_topology prod >/dev/null 2>&1 + printf '%s' "$TG_DB_NAME" +) +OWNER_RO=$( + thermograph_topology prod >/dev/null 2>&1 + printf '%s_ro' "$TG_DB_USER" +) +# Restore this run's environment — thermograph_topology overwrote the TG_* vars. +thermograph_topology "$ENV_NAME" + +if [ "$OWNER_DB" != "$TG_DB_NAME" ]; then + echo "==> Hardening the owner database '${OWNER_DB}' against PUBLIC connects" + docker exec -i "$DB_CID" \ + psql -v ON_ERROR_STOP=1 -U thermograph -d postgres \ + -v ownerdb="$OWNER_DB" -v ownerro="$OWNER_RO" <<'SQL' +-- Only if that read-only role actually exists (it predates this script). +SELECT format('GRANT CONNECT ON DATABASE %I TO %I', :'ownerdb', :'ownerro') +WHERE EXISTS (SELECT 1 FROM pg_roles WHERE rolname = :'ownerro') +\gexec + +REVOKE CONNECT ON DATABASE :"ownerdb" FROM PUBLIC; +SQL +fi + echo "==> OK: ${TG_DB_USER} owns ${TG_DB_NAME} (timescaledb enabled, PUBLIC revoked)" echo " Verify isolation:" echo " docker exec $DB_CID psql -U ${TG_DB_USER} -d thermograph -c 'select 1' # must FAIL" -- 2.45.2 From 24b7ef4c8669865c62b6dcef02f34ba990614516 Mon Sep 17 00:00:00 2001 From: Emi Griffith Date: Sun, 26 Jul 2026 00:16:36 -0700 Subject: [PATCH 2/2] runbook: chown the new Caddy log before reloading MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `caddy validate` run as root creates the log files the config names, owned by root. The caddy user then cannot open them and the reload fails with 'permission denied' on the log writer — which is exactly what happened on the live cutover, and what the root-owned centralis.log in that directory is a fossil of. Caddy rejected the config atomically and kept serving prod on the old one, so nothing broke; but the step reads as a no-op until it isn't. --- infra/deploy/RUNBOOK-vps1-vps2-cutover.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/infra/deploy/RUNBOOK-vps1-vps2-cutover.md b/infra/deploy/RUNBOOK-vps1-vps2-cutover.md index 3bbfcd6..fdb379a 100644 --- a/infra/deploy/RUNBOOK-vps1-vps2-cutover.md +++ b/infra/deploy/RUNBOOK-vps1-vps2-cutover.md @@ -247,12 +247,23 @@ serving; nothing has moved yet. ``` ssh agent@169.58.46.181 sudo caddy validate --config /etc/caddy/Caddyfile # BEFORE reloading +# `validate` runs as root and CREATES any log file the config names, owned by +# root — which the caddy user then cannot open, and the reload fails. Fix the +# ownership before reloading, or the first reload rejects the whole config: +sudo chown caddy:caddy /var/log/caddy/beta.log sudo systemctl reload caddy ``` Validate before reload, always: a malformed Caddyfile takes `thermograph.org` down with it, and prod is on this box now. +The `chown` is not optional and it bit this cutover. `caddy validate` as root +left `/var/log/caddy/beta.log` as `root:root`, so the reload failed with +`permission denied` on the log writer. (The root-owned `centralis.log` sitting +in that directory is the same mistake, made earlier.) Note what did NOT happen: +Caddy rejects a bad config atomically and keeps serving the old one, so +`thermograph.org` stayed up throughout — verify it anyway. + Caddy cannot issue the certificate until DNS moves (step 7), so expect the beta hostname to fail TLS until then. That is fine and expected. -- 2.45.2