diff --git a/infra/deploy/RUNBOOK-vps1-vps2-cutover.md b/infra/deploy/RUNBOOK-vps1-vps2-cutover.md index b1874da..fdb379a 100644 --- a/infra/deploy/RUNBOOK-vps1-vps2-cutover.md +++ b/infra/deploy/RUNBOOK-vps1-vps2-cutover.md @@ -149,8 +149,16 @@ it. That guard is why the superuser check above should never fail; run it anyway. If the first command SUCCEEDS, stop. `CONNECT` was not revoked and beta would be -able to read production data; re-run the provisioning script and re-check before -going further. +able to open a session against the production database; re-run the provisioning +script and re-check before going further. + +This is not hypothetical — it happened on the live cutover. The first version of +the script revoked `PUBLIC` connect on the *guest's* database only, and a fresh +Postgres database carries `=Tc` (TEMP **and** CONNECT) for `PUBLIC`, so +`thermograph_beta` could still reach `thermograph`. The script now hardens the +owner's database too, granting the owner's read-only role an explicit `CONNECT` +first so the revoke cannot strip `ops/dbq.sh` of its access. The check above is +what caught it. **Rollback:** `DROP DATABASE thermograph_beta; DROP ROLE thermograph_beta;` — prod is unaffected either way. @@ -239,12 +247,23 @@ serving; nothing has moved yet. ``` ssh agent@169.58.46.181 sudo caddy validate --config /etc/caddy/Caddyfile # BEFORE reloading +# `validate` runs as root and CREATES any log file the config names, owned by +# root — which the caddy user then cannot open, and the reload fails. Fix the +# ownership before reloading, or the first reload rejects the whole config: +sudo chown caddy:caddy /var/log/caddy/beta.log sudo systemctl reload caddy ``` Validate before reload, always: a malformed Caddyfile takes `thermograph.org` down with it, and prod is on this box now. +The `chown` is not optional and it bit this cutover. `caddy validate` as root +left `/var/log/caddy/beta.log` as `root:root`, so the reload failed with +`permission denied` on the log writer. (The root-owned `centralis.log` sitting +in that directory is the same mistake, made earlier.) Note what did NOT happen: +Caddy rejects a bad config atomically and keeps serving the old one, so +`thermograph.org` stayed up throughout — verify it anyway. + Caddy cannot issue the certificate until DNS moves (step 7), so expect the beta hostname to fail TLS until then. That is fine and expected. diff --git a/infra/deploy/db/provision-env-db.sh b/infra/deploy/db/provision-env-db.sh index 334612c..3695027 100755 --- a/infra/deploy/db/provision-env-db.sh +++ b/infra/deploy/db/provision-env-db.sh @@ -156,6 +156,46 @@ ALTER DEFAULT PRIVILEGES FOR ROLE :"role" IN SCHEMA public GRANT SELECT ON SEQUENCES TO :"rorole"; SQL +# --- harden the OWNER environment's database too ------------------------------- +# +# Revoking CONNECT from PUBLIC on the guest's own database is only half the job, +# and the missing half is the half that matters. A fresh Postgres database +# carries `=Tc` for PUBLIC — TEMP *and* CONNECT — so the moment a second role +# exists on the instance, that role can connect to every database that still has +# the default. Provisioning beta and stopping here left `thermograph_beta` able +# to open a session against prod's database; caught live by the runbook's own +# isolation check, which is why that check exists. +# +# Order matters: grant the owner's read-only role an EXPLICIT connect first, so +# the revoke below cannot strip the ad-hoc query path (ops/dbq.sh) of its access. +# The owner's app role is the database owner and a superuser, so it is unaffected +# either way. +OWNER_DB=$( + # shellcheck disable=SC1090 # sourced above; re-resolving prod in a subshell + thermograph_topology prod >/dev/null 2>&1 + printf '%s' "$TG_DB_NAME" +) +OWNER_RO=$( + thermograph_topology prod >/dev/null 2>&1 + printf '%s_ro' "$TG_DB_USER" +) +# Restore this run's environment — thermograph_topology overwrote the TG_* vars. +thermograph_topology "$ENV_NAME" + +if [ "$OWNER_DB" != "$TG_DB_NAME" ]; then + echo "==> Hardening the owner database '${OWNER_DB}' against PUBLIC connects" + docker exec -i "$DB_CID" \ + psql -v ON_ERROR_STOP=1 -U thermograph -d postgres \ + -v ownerdb="$OWNER_DB" -v ownerro="$OWNER_RO" <<'SQL' +-- Only if that read-only role actually exists (it predates this script). +SELECT format('GRANT CONNECT ON DATABASE %I TO %I', :'ownerdb', :'ownerro') +WHERE EXISTS (SELECT 1 FROM pg_roles WHERE rolname = :'ownerro') +\gexec + +REVOKE CONNECT ON DATABASE :"ownerdb" FROM PUBLIC; +SQL +fi + echo "==> OK: ${TG_DB_USER} owns ${TG_DB_NAME} (timescaledb enabled, PUBLIC revoked)" echo " Verify isolation:" echo " docker exec $DB_CID psql -U ${TG_DB_USER} -d thermograph -c 'select 1' # must FAIL" diff --git a/infra/deploy/deploy.sh b/infra/deploy/deploy.sh index 98fd9de..9efeaf7 100755 --- a/infra/deploy/deploy.sh +++ b/infra/deploy/deploy.sh @@ -417,7 +417,21 @@ docker images --format '{{.Repository}}:{{.Tag}}' \ # Post-deploy warm/IndexNow only make sense once the backend is (re)deployed -- # they exec inside the backend container. Skip them on a frontend-only roll. -if [ "$SERVICE" = backend ] || [ "$SERVICE" = all ]; then +# +# TG_POST_DEPLOY gates them per ENVIRONMENT (env-topology.sh): prod does both, +# beta and dev do neither. Both are wrong outside prod. The warm spends the +# shared upstream archive quota to fill a cache nobody serves from, and the +# IndexNow ping announces URLs to Bing/DuckDuckGo/Yandex — from dev that means +# submitting a mesh address no crawler can reach. +# +# This gate existed in deploy/stack/deploy-stack.sh (beta, prod) but was missed +# here, on the compose path, which is the ONE environment where the default is +# wrong: dev. Standing dev up on vps1 duly warmed 1000 cities and tried to +# submit 14,007 URLs to IndexNow before this was fixed. +# +# Defaults to 1 when unset so a checkout predating env-topology.sh keeps prod's +# historical behaviour. +if { [ "$SERVICE" = backend ] || [ "$SERVICE" = all ]; } && [ "${TG_POST_DEPLOY:-1}" = 1 ]; then warm_city_archives ping_indexnow fi