provision-env-db: revoke PUBLIC connect on the owner database too #108

Merged

admin_emi merged 2 commits from fix/harden-owner-db-connect into dev

2026-07-26 07:17:21 +00:00

admin_emi commented

2026-07-26 07:12:42 +00:00

Owner

Caught live during the vps1/vps2 cutover, by the runbook's own isolation check.

provision-env-db.sh beta revoked CONNECT from PUBLIC on beta's database and stopped there. But a fresh Postgres database carries =Tc for PUBLIC — TEMP and CONNECT — so prod's database still had the default, and the newly created thermograph_beta role could open a session against it:

beta role -> prod database                  SUCCEEDED     # must FAIL

Revoking on the guest's own database is the half that doesn't matter; the half that does is the database you are trying to protect.

The fix hardens the owner environment's database as part of provisioning a guest, and orders it carefully: the owner's read-only role (thermograph_ro, which ops/dbq.sh connects as) gets an explicit CONNECT grant before the revoke, so hardening cannot strip the ad-hoc query path of its access. The owner's app role owns the database and is a superuser, so it is unaffected either way.

Already applied by hand on vps2 — this makes the script reproduce that state rather than a false sense of it, which matters on any rebuild or re-run. Post-fix, live:

MUST FAIL:     beta role -> prod db · beta_ro -> prod db · beta_ro writes      all FAILED
MUST SUCCEED:  prod app role · thermograph_ro -> prod db · beta -> beta db     all SUCCEEDED
prod site -> 200, all 8 prod services 1/1

The runbook now records the incident next to the check that found it, so the check reads as load-bearing rather than ceremonial.

Caught live during the vps1/vps2 cutover, by the runbook's own isolation check. `provision-env-db.sh beta` revoked `CONNECT` from `PUBLIC` on **beta's** database and stopped there. But a fresh Postgres database carries `=Tc` for `PUBLIC` — `TEMP` *and* `CONNECT` — so prod's database still had the default, and the newly created `thermograph_beta` role could open a session against it: ``` beta role -> prod database SUCCEEDED # must FAIL ``` Revoking on the guest's own database is the half that doesn't matter; the half that does is the database you are trying to protect. The fix hardens the owner environment's database as part of provisioning a guest, and orders it carefully: the owner's read-only role (`thermograph_ro`, which `ops/dbq.sh` connects as) gets an **explicit** `CONNECT` grant *before* the revoke, so hardening cannot strip the ad-hoc query path of its access. The owner's app role owns the database and is a superuser, so it is unaffected either way. Already applied by hand on vps2 — this makes the script reproduce that state rather than a false sense of it, which matters on any rebuild or re-run. Post-fix, live: ``` MUST FAIL: beta role -> prod db · beta_ro -> prod db · beta_ro writes all FAILED MUST SUCCEED: prod app role · thermograph_ro -> prod db · beta -> beta db all SUCCEEDED prod site -> 200, all 8 prod services 1/1 ``` The runbook now records the incident next to the check that found it, so the check reads as load-bearing rather than ceremonial.

admin_emi added 1 commit 2026-07-26 07:12:43 +00:00

provision-env-db: revoke PUBLIC connect on the OWNER database too

PR build (required check) / changes (pull_request) Successful in 6s

Details

secrets-guard / encrypted (pull_request) Successful in 5s

Details

PR build (required check) / build-backend (pull_request) Has been skipped

Details

PR build (required check) / build-frontend (pull_request) Has been skipped

Details

shell-lint / shellcheck (pull_request) Successful in 6s

Details

PR build (required check) / validate-observability (pull_request) Has been skipped

Details

PR build (required check) / gate (pull_request) Successful in 1s

Details

1aeea6bdc5

Revoking PUBLIC's CONNECT on the guest's own database is the half that does not
matter. A fresh Postgres database carries '=Tc' for PUBLIC — TEMP and CONNECT —
so as soon as a second role exists on the instance it can open a session against
every database still holding the default.

Provisioning beta and stopping there left thermograph_beta able to connect to
prod's database. The runbook's isolation check caught it on the live cutover;
this makes the script produce the state that check expects instead of a false
sense of it.

Grants the owner's read-only role an explicit CONNECT before the revoke, so the
ad-hoc query path (ops/dbq.sh, which connects as thermograph_ro) cannot lose
access to prod. The owner's app role owns the database and is a superuser, so it
is unaffected either way.

admin_emi added 1 commit 2026-07-26 07:16:40 +00:00

runbook: chown the new Caddy log before reloading

PR build (required check) / changes (pull_request) Successful in 6s

Details

secrets-guard / encrypted (pull_request) Successful in 5s

Details

PR build (required check) / build-backend (pull_request) Has been skipped

Details

shell-lint / shellcheck (pull_request) Successful in 7s

Details

PR build (required check) / build-frontend (pull_request) Has been skipped

Details

PR build (required check) / validate-observability (pull_request) Has been skipped

Details

PR build (required check) / gate (pull_request) Successful in 1s

Details

24b7ef4c86

`caddy validate` run as root creates the log files the config names, owned by
root. The caddy user then cannot open them and the reload fails with
'permission denied' on the log writer — which is exactly what happened on the
live cutover, and what the root-owned centralis.log in that directory is a
fossil of.

Caddy rejected the config atomically and kept serving prod on the old one, so
nothing broke; but the step reads as a no-op until it isn't.