1aeea6bdc5
All checks were successful
PR build (required check) / changes (pull_request) Successful in 6s
secrets-guard / encrypted (pull_request) Successful in 5s
PR build (required check) / build-backend (pull_request) Has been skipped
PR build (required check) / build-frontend (pull_request) Has been skipped
shell-lint / shellcheck (pull_request) Successful in 6s
PR build (required check) / validate-observability (pull_request) Has been skipped
PR build (required check) / gate (pull_request) Successful in 1s
Revoking PUBLIC's CONNECT on the guest's own database is the half that does not matter. A fresh Postgres database carries '=Tc' for PUBLIC — TEMP and CONNECT — so as soon as a second role exists on the instance it can open a session against every database still holding the default. Provisioning beta and stopping there left thermograph_beta able to connect to prod's database. The runbook's isolation check caught it on the live cutover; this makes the script produce the state that check expects instead of a false sense of it. Grants the owner's read-only role an explicit CONNECT before the revoke, so the ad-hoc query path (ops/dbq.sh, which connects as thermograph_ro) cannot lose access to prod. The owner's app role owns the database and is a superuser, so it is unaffected either way.
201 lines
9.5 KiB
Bash
Executable file
201 lines
9.5 KiB
Bash
Executable file
#!/usr/bin/env bash
|
|
# Give an environment its own role + database on the SHARED Postgres instance.
|
|
#
|
|
# sudo bash infra/deploy/db/provision-env-db.sh beta # run on vps2
|
|
#
|
|
# Since beta moved onto vps2 there is ONE TimescaleDB instance serving two
|
|
# environments. That is a capacity decision — one Postgres to tune, back up and
|
|
# keep on one extension build — and it is emphatically NOT a decision to let the
|
|
# two environments see each other's data. This script is what makes the second
|
|
# half true:
|
|
#
|
|
# * a role per environment (thermograph_beta), with its own password taken
|
|
# from that environment's own vault render,
|
|
# * a database per environment (thermograph_beta) OWNED by that role,
|
|
# * CONNECT revoked from PUBLIC on it, so the split is enforced by Postgres
|
|
# rather than by everyone remembering to use the right URL,
|
|
# * and NO superuser, NO CREATEDB, NO CREATEROLE on that role.
|
|
#
|
|
# The prod role keeps its own database and cannot be reached with beta's
|
|
# credentials; beta's role cannot connect to prod's database at all.
|
|
#
|
|
# Idempotent by construction — safe to re-run after a password rotation (it
|
|
# re-applies the password) or on a fresh instance. It never drops anything.
|
|
#
|
|
# WHY THIS IS NOT WIRED INTO deploy.sh: creating roles and databases is a
|
|
# privileged, once-per-environment act, and a deploy that can mint database
|
|
# roles is a deploy that can mint them wrongly at 3am. Run it by hand from the
|
|
# runbook when an environment is first stood up, and after a password rotation.
|
|
set -euo pipefail
|
|
|
|
ENV_NAME="${1:?usage: provision-env-db.sh <env> (beta|prod)}"
|
|
|
|
SELF_DIR=$(cd "$(dirname "$0")" && pwd)
|
|
# shellcheck source=infra/deploy/env-topology.sh
|
|
. "$SELF_DIR/../env-topology.sh"
|
|
thermograph_topology "$ENV_NAME"
|
|
|
|
if [ "$TG_DEPLOY_MODE" != stack ]; then
|
|
echo "!! $ENV_NAME does not use the shared instance (dev keeps its own db container)" >&2
|
|
exit 2
|
|
fi
|
|
|
|
# REFUSE to run for the environment that OWNS the instance.
|
|
#
|
|
# This script provisions a GUEST environment onto someone else's Postgres. Run
|
|
# for prod, it would target prod's `thermograph` role — which is the instance's
|
|
# bootstrap superuser, not a guest — and the `ALTER ROLE ... NOSUPERUSER` below
|
|
# would strip superuser from the role the whole instance is administered with.
|
|
# Prod's roles predate this script and are created by the container's own
|
|
# initdb; there is nothing here for prod to need.
|
|
if [ "$TG_DB_SERVICE" = "${TG_STACK_NAME}_db" ]; then
|
|
echo "!! '$ENV_NAME' OWNS this Postgres instance (${TG_DB_SERVICE} is its own stack's db)." >&2
|
|
echo "!! This script provisions a guest environment onto a shared instance." >&2
|
|
echo "!! Running it here would demote ${TG_DB_USER} from superuser. Refusing." >&2
|
|
exit 2
|
|
fi
|
|
|
|
# The database SERVER is prod's, wherever we are provisioning FOR.
|
|
DB_CID=$(docker ps -q --filter "label=com.docker.swarm.service.name=${TG_DB_SERVICE}" | head -1)
|
|
if [ -z "$DB_CID" ]; then
|
|
echo "!! no running task for ${TG_DB_SERVICE} on this host" >&2
|
|
echo "!! run this on vps2, with prod's stack up." >&2
|
|
exit 1
|
|
fi
|
|
|
|
# The new role's password is whatever that environment's vault says it is, so
|
|
# the database and the app can never disagree about it. Read from the rendered
|
|
# env file (a deploy of that environment writes it) rather than from sops here:
|
|
# this script should not need the age key.
|
|
if [ ! -r "$TG_ENV_FILE" ]; then
|
|
PW=$(sudo grep -m1 '^POSTGRES_PASSWORD=' "$TG_ENV_FILE" 2>/dev/null | cut -d= -f2- || true)
|
|
else
|
|
PW=$(grep -m1 '^POSTGRES_PASSWORD=' "$TG_ENV_FILE" | cut -d= -f2- || true)
|
|
fi
|
|
if [ -z "${PW:-}" ]; then
|
|
echo "!! no POSTGRES_PASSWORD in $TG_ENV_FILE" >&2
|
|
echo "!! deploy $ENV_NAME once first so the vault renders it, then re-run." >&2
|
|
exit 1
|
|
fi
|
|
|
|
echo "==> Provisioning role/database '${TG_DB_USER}'/'${TG_DB_NAME}' on ${TG_DB_SERVICE}"
|
|
|
|
# The bootstrap superuser of the instance is prod's POSTGRES_USER (`thermograph`),
|
|
# and psql runs inside the container over its local socket — the database is
|
|
# never exposed on a TCP port anyone outside the overlay can reach.
|
|
#
|
|
# The password reaches psql as an environment variable rather than a -v
|
|
# argument, so it is not in psql's argv inside the container. It IS briefly in
|
|
# the `docker exec` argv on the host; that is visible only to root on vps2, who
|
|
# can read the vault render anyway. It is never interpolated into SQL text:
|
|
# :'pw' is psql's quote-and-escape form, which is also what makes a password
|
|
# containing a quote safe.
|
|
#
|
|
# Note both statements below are generated and run via \gexec rather than a
|
|
# DO block. psql does NOT substitute :variables inside dollar-quoted strings,
|
|
# so a DO $$ ... :'role' ... $$ body would be sent to the server literally.
|
|
docker exec -i -e ENV_DB_PASSWORD="$PW" "$DB_CID" \
|
|
psql -v ON_ERROR_STOP=1 -U thermograph -d postgres \
|
|
-v role="$TG_DB_USER" -v dbname="$TG_DB_NAME" <<'SQL'
|
|
\set pw `echo "$ENV_DB_PASSWORD"`
|
|
|
|
-- Role: create if absent.
|
|
SELECT format('CREATE ROLE %I LOGIN', :'role')
|
|
WHERE NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = :'role')
|
|
\gexec
|
|
|
|
-- Always (re)apply the password and the negative privileges, so a vault
|
|
-- rotation is just "rotate, redeploy, re-run this" — and so a role that was
|
|
-- created by hand with more rights than it should have gets corrected.
|
|
ALTER ROLE :"role" WITH LOGIN PASSWORD :'pw' NOSUPERUSER NOCREATEDB NOCREATEROLE;
|
|
|
|
-- CREATE DATABASE cannot run inside a transaction block, hence \gexec here too.
|
|
SELECT format('CREATE DATABASE %I OWNER %I', :'dbname', :'role')
|
|
WHERE NOT EXISTS (SELECT 1 FROM pg_database WHERE datname = :'dbname')
|
|
\gexec
|
|
SQL
|
|
|
|
# Extension + privileges have to run INSIDE the new database, hence a second
|
|
# connection. CREATE EXTENSION needs superuser, which is why it is done here
|
|
# rather than left to the app's own migration to attempt as the env role.
|
|
docker exec -i "$DB_CID" \
|
|
psql -v ON_ERROR_STOP=1 -U thermograph -d "$TG_DB_NAME" \
|
|
-v role="$TG_DB_USER" -v dbname="$TG_DB_NAME" -v rorole="${TG_DB_USER}_ro" <<'SQL'
|
|
CREATE EXTENSION IF NOT EXISTS timescaledb;
|
|
|
|
-- Nobody but this environment's roles connects to this database. Without this,
|
|
-- PUBLIC retains CONNECT and any role on the instance could read it.
|
|
REVOKE CONNECT ON DATABASE :"dbname" FROM PUBLIC;
|
|
GRANT CONNECT ON DATABASE :"dbname" TO :"role";
|
|
|
|
-- The role owns the database but not necessarily the public schema, which is
|
|
-- owned by the bootstrap superuser on a fresh database; Alembic needs to create
|
|
-- tables in it.
|
|
ALTER SCHEMA public OWNER TO :"role";
|
|
|
|
-- The read-only role that ad-hoc queries use (ops/dbq.sh). Every environment
|
|
-- gets one, named <role>_ro, so a human or an agent poking at data cannot
|
|
-- write. Without it, beta's queries would have had to connect as the OWNER —
|
|
-- read-write on its own database — losing a guarantee prod and dev already had
|
|
-- purely because beta was newer.
|
|
SELECT format('CREATE ROLE %I LOGIN', :'rorole')
|
|
WHERE NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = :'rorole')
|
|
\gexec
|
|
ALTER ROLE :"rorole" WITH LOGIN NOSUPERUSER NOCREATEDB NOCREATEROLE;
|
|
|
|
GRANT CONNECT ON DATABASE :"dbname" TO :"rorole";
|
|
GRANT USAGE ON SCHEMA public TO :"rorole";
|
|
GRANT SELECT ON ALL TABLES IN SCHEMA public TO :"rorole";
|
|
GRANT SELECT ON ALL SEQUENCES IN SCHEMA public TO :"rorole";
|
|
-- Future tables too: without this, every new migration would create a table the
|
|
-- read-only role cannot see, and the omission would only surface as a confusing
|
|
-- "permission denied" months later.
|
|
ALTER DEFAULT PRIVILEGES FOR ROLE :"role" IN SCHEMA public
|
|
GRANT SELECT ON TABLES TO :"rorole";
|
|
ALTER DEFAULT PRIVILEGES FOR ROLE :"role" IN SCHEMA public
|
|
GRANT SELECT ON SEQUENCES TO :"rorole";
|
|
SQL
|
|
|
|
# --- harden the OWNER environment's database too -------------------------------
|
|
#
|
|
# Revoking CONNECT from PUBLIC on the guest's own database is only half the job,
|
|
# and the missing half is the half that matters. A fresh Postgres database
|
|
# carries `=Tc` for PUBLIC — TEMP *and* CONNECT — so the moment a second role
|
|
# exists on the instance, that role can connect to every database that still has
|
|
# the default. Provisioning beta and stopping here left `thermograph_beta` able
|
|
# to open a session against prod's database; caught live by the runbook's own
|
|
# isolation check, which is why that check exists.
|
|
#
|
|
# Order matters: grant the owner's read-only role an EXPLICIT connect first, so
|
|
# the revoke below cannot strip the ad-hoc query path (ops/dbq.sh) of its access.
|
|
# The owner's app role is the database owner and a superuser, so it is unaffected
|
|
# either way.
|
|
OWNER_DB=$(
|
|
# shellcheck disable=SC1090 # sourced above; re-resolving prod in a subshell
|
|
thermograph_topology prod >/dev/null 2>&1
|
|
printf '%s' "$TG_DB_NAME"
|
|
)
|
|
OWNER_RO=$(
|
|
thermograph_topology prod >/dev/null 2>&1
|
|
printf '%s_ro' "$TG_DB_USER"
|
|
)
|
|
# Restore this run's environment — thermograph_topology overwrote the TG_* vars.
|
|
thermograph_topology "$ENV_NAME"
|
|
|
|
if [ "$OWNER_DB" != "$TG_DB_NAME" ]; then
|
|
echo "==> Hardening the owner database '${OWNER_DB}' against PUBLIC connects"
|
|
docker exec -i "$DB_CID" \
|
|
psql -v ON_ERROR_STOP=1 -U thermograph -d postgres \
|
|
-v ownerdb="$OWNER_DB" -v ownerro="$OWNER_RO" <<'SQL'
|
|
-- Only if that read-only role actually exists (it predates this script).
|
|
SELECT format('GRANT CONNECT ON DATABASE %I TO %I', :'ownerdb', :'ownerro')
|
|
WHERE EXISTS (SELECT 1 FROM pg_roles WHERE rolname = :'ownerro')
|
|
\gexec
|
|
|
|
REVOKE CONNECT ON DATABASE :"ownerdb" FROM PUBLIC;
|
|
SQL
|
|
fi
|
|
|
|
echo "==> OK: ${TG_DB_USER} owns ${TG_DB_NAME} (timescaledb enabled, PUBLIC revoked)"
|
|
echo " Verify isolation:"
|
|
echo " docker exec $DB_CID psql -U ${TG_DB_USER} -d thermograph -c 'select 1' # must FAIL"
|