thermograph/.claude/hooks/prod-guard.sh
Emi Griffith 4e97d8e5dc
All checks were successful
PR build (required check) / changes (pull_request) Successful in 6s
secrets-guard / encrypted (pull_request) Successful in 5s
PR build (required check) / build-backend (pull_request) Has been skipped
shell-lint / shellcheck (pull_request) Successful in 7s
PR build (required check) / build-frontend (pull_request) Has been skipped
PR build (required check) / validate-observability (pull_request) Successful in 18s
PR build (required check) / gate (pull_request) Successful in 2s
infra: split the estate into vps1/vps2, moving beta next to prod and dev onto vps1
Environments stop being machines. Until now each box WAS an environment --
"beta" named both a deploy target and a host, "the desktop" named both the
operator's computer and the dev server -- so every path could assume one
environment per host and hardcode /opt/thermograph, /etc/thermograph.env and
ports 8137/8080. That assumption ends here:

  vps1  75.119.132.91  Forgejo, Grafana/Loki, the portfolio site, and DEV
                       (own Postgres, mesh-only on 10.10.0.2:8137)
  vps2  169.58.46.181  PROD and BETA as two Swarm stacks sharing one
                       TimescaleDB instance, plus Centralis, Postfix, backups
  desktop              AI model hosting + flex Swarm capacity, no environment

Nothing live has moved; the ordered cutover is in
infra/deploy/RUNBOOK-vps1-vps2-cutover.md.

deploy/env-topology.sh is the single source of truth: env -> host, checkout,
branch, deploy mode, stack name, env file, LB ports, DB role/database, service
prefix. THERMOGRAPH_ENV is the input, and on vps2 it is the only thing
distinguishing a beta deploy from a prod one -- deploy.sh refuses to run when it
disagrees with the checkout it was invoked from. The host-wide secrets-env and
deploy-mode markers survive only as a fallback for a single-environment box.

Beta gets its own stack file rather than an overlay (a merge cannot REMOVE the
db service, and beta having no database of its own is the point). Its services
are prefixed beta-*: Swarm registers a service's short name as a DNS alias on
every network it joins, so two stacks both calling a service `web` on the shared
data network would let prod's frontend resolve a beta task. Prod's stack, env
and LB config are untouched.

One Postgres, two databases with two roles: deploy/db/provision-env-db.sh
creates thermograph_beta (NOSUPERUSER, owns only its own database, CONNECT
revoked from PUBLIC) plus a <role>_ro for ad-hoc queries, and refuses to run for
the environment that owns the instance -- doing so would demote prod's bootstrap
superuser.

Fixes that co-residency would otherwise have broken silently:

- CI secrets are keyed by host (VPS1_SSH_*, VPS2_SSH_*). SSH_* meant "beta" and
  also "the Forgejo box" because those shared a machine; that conflation is what
  once had the prod backup dumping beta.
- The nightly backup dumps BOTH application databases to separate off-box
  prefixes, and fails loudly on a missing one instead of skipping it.
- Alloy stopped deriving the `host` label from the node -- on vps2 that would
  have filed every beta line as prod, feeding prod's alert rules with beta's
  traffic. It is now derived per source, with a new `node` label for the machine,
  and beta's log volume is mounted via a vps2-only overlay.
- ops/dbq.sh, ops/iceberg.sh and the secrets seed scripts derived their SSH
  target and env-file path from the topology instead of hardcoding beta to
  75.119.132.91 -- which is vps1's address now.
- Dev's overlay binds DEV_BIND_ADDR (loopback by default, the mesh address on
  vps1) instead of 0.0.0.0: correct for a home LAN box, a public exposure of
  unreviewed branches on a VPS.

Terraform's hosts variable is keyed by machine with a nested environments map,
and main.tf flattens (host, environment) pairs so two environments on one box
cannot share a checkout. Caddy's single reference config is split per host.

Docs, onboarding and the runbooks are updated throughout, including the security
rationales that co-residency makes false: separate SSH credentials no longer put
a host boundary between beta and prod, and the boundary that remains is the
database and the filesystem.
2026-07-25 15:01:29 -07:00

165 lines
7.7 KiB
Bash
Executable file

#!/usr/bin/env bash
# PreToolUse guard for commands that reach a live host.
#
# Policy: reads stay friction-free, mutations ask first.
#
# The estate's widest privilege is that `agent` has passwordless sudo on prod and
# beta, and the global settings allow `ssh prod:*` outright — so before this hook
# existed, any session in any directory could roll, delete or restart production
# with no prompt. CI cannot help here: nothing about `ssh prod 'docker service rm'`
# goes through a pull request.
#
# Classification is an ALLOWLIST of read-only commands, not a blocklist of
# dangerous ones. A blocklist is wrong by construction: the first destructive verb
# nobody thought of sails straight through. Anything this script does not
# positively recognise as a read becomes an "ask", which is the safe direction to
# be wrong in.
#
# Exit 0 with no output = "no opinion", and the normal permission flow proceeds.
# That is also what happens if this script errors or jq is missing, so a bug here
# fails toward the prompt rather than toward silent execution.
#
# Beta is guarded as strictly as prod -- more strictly than the names alone
# suggest, now: vps2 (169.58.46.181) runs prod AND beta as two Swarm stacks on
# the SAME manager, sharing the same TimescaleDB instance, so a "beta" command
# there has PROD's blast radius (shared Docker host, shared Postgres server --
# see infra/deploy/env-topology.sh). vps1 (75.119.132.91) is the other guarded
# host: Forgejo (git + CI + registry), Grafana/Loki, and the dev environment --
# a destructive command there can take out the git server, CI and the container
# registry at once. Mesh IPs did NOT move in the vps1/vps2 split (vps1 is still
# 10.10.0.2, vps2 is still 10.10.0.1); what moved is which environment runs where.
set -uo pipefail
payload=$(cat)
command -v jq >/dev/null 2>&1 || exit 0
tool=$(printf '%s' "$payload" | jq -r '.tool_name // ""')
# An ssh target naming a live host, matched as a whole token after any `user@`.
# Backslashes are doubled because awk -v processes escape sequences in the value
# before the regex ever sees it; a single \. arrives as a bare dot (and warns).
# vps2 (169.58.46.181) carries BOTH prod and beta; vps1 (75.119.132.91) carries
# Forgejo/Grafana/dev -- "vps1"/"vps2" are recognised alongside the environment
# names and raw IPs so a command naming the host either way still gets caught.
readonly HOST_TOKEN_RE='^(prod|beta|vps1|vps2|169\\.58\\.46\\.181|75\\.119\\.132\\.91|10\\.10\\.0\\.[12])$'
emit() { # $1=allow|ask|deny $2=reason
jq -n --arg d "$1" --arg r "$2" \
'{hookSpecificOutput:{hookEventName:"PreToolUse",permissionDecision:$d,permissionDecisionReason:$r}}'
exit 0
}
# --- is this shell command read-only? ----------------------------------------
# Returns 0 only when every segment leads with a recognised read-only command.
is_readonly() {
local cmd="$1" seg w1 w2 stripped
# Discard the harmless redirections that appear in ordinary read commands,
# then treat any surviving redirect as a write.
stripped=${cmd//2>&1/}
stripped=${stripped//&>\/dev\/null/}
stripped=${stripped//2>\/dev\/null/}
stripped=${stripped//>\/dev\/null/}
stripped=${stripped//> \/dev\/null/}
case "$stripped" in *'>'*) return 1 ;; esac
# Command substitution can hide anything; refuse to reason about it.
# shellcheck disable=SC2016 # single quotes are the point: these are literal
# `$(` and backtick characters being searched for inside the command text, not
# expansions to perform. Expanding them here would defeat the check entirely.
case "$cmd" in *'$('*|*'`'*) return 1 ;; esac
while IFS= read -r seg; do
seg="${seg#"${seg%%[![:space:]]*}"}"
[ -z "$seg" ] && continue
w1=$(printf '%s' "$seg" | awk '{print $1}')
w2=$(printf '%s' "$seg" | awk '{print $2}')
case "$w1" in
cat|ls|head|tail|wc|grep|egrep|fgrep|rg|find|stat|df|du|free|uptime|whoami) ;;
id|hostname|date|ps|env|printenv|pwd|which|uname|lsblk|ip|ss|netstat) ;;
pg_isready|echo|true|sort|uniq|cut|tr|awk|jq|nproc|readlink|realpath|test) ;;
sed) case "$seg" in *' -i'*) return 1 ;; esac ;;
curl) case "$seg" in *' -X'*|*' -d'*|*'--data'*|*' -T'*|*' -F'*|*'--upload'*) return 1 ;; esac ;;
journalctl) ;;
systemctl)
case "$w2" in status|show|is-active|is-enabled|is-failed|list-units|cat) ;; *) return 1 ;; esac ;;
docker)
case "$w2" in
ps|logs|inspect|stats|images|version|info|top|port|events|history|diff) ;;
service) case "$seg" in *'service ls'*|*'service ps'*|*'service inspect'*|*'service logs'*) ;; *) return 1 ;; esac ;;
stack) case "$seg" in *'stack ls'*|*'stack ps'*|*'stack services'*) ;; *) return 1 ;; esac ;;
compose) case "$seg" in *'compose ps'*|*'compose logs'*|*'compose config'*|*'compose top'*|*'compose images'*) ;; *) return 1 ;; esac ;;
node) case "$seg" in *'node ls'*|*'node inspect'*) ;; *) return 1 ;; esac ;;
volume) case "$seg" in *'volume ls'*|*'volume inspect'*) ;; *) return 1 ;; esac ;;
*) return 1 ;;
esac ;;
git)
case "$w2" in status|log|diff|show|branch|remote|rev-parse|describe|blame|ls-files|config) ;; *) return 1 ;; esac ;;
*) return 1 ;;
esac
# NOTE: the trailing newline matters. Without it `read` returns non-zero on the
# final (only) line, the loop body never runs, and every command classifies as
# read-only — a silent false negative that allows anything.
done < <(printf '%s\n' "$cmd" | sed 's/&&/\n/g; s/||/\n/g; s/;/\n/g; s/|/\n/g')
return 0
}
# --- find an ssh target and return what would run on it -----------------------
# Prints "<host>\t<remote command>" when a live host is addressed, nothing
# otherwise. The remote command is empty for a bare login shell.
ssh_target_of() {
printf '%s' "$1" | awk -v re="$HOST_TOKEN_RE" '
{
for (i = 1; i <= NF; i++) {
t = $i; sub(/^[^@]*@/, "", t)
if (t ~ re) {
out = ""
for (j = i + 1; j <= NF; j++) out = out (out == "" ? "" : " ") $j
printf "%s\t%s\n", t, out
exit
}
}
}'
}
case "$tool" in
Bash)
cmd=$(printf '%s' "$payload" | jq -r '.tool_input.command // ""')
case "$cmd" in *ssh*) ;; *) exit 0 ;; esac
found=$(ssh_target_of "$cmd")
[ -n "$found" ] || exit 0 # no live host addressed
host=${found%%$'\t'*}
remote=${found#*$'\t'}
remote=$(printf '%s' "$remote" | sed -E "s/^'(.*)'$/\1/; s/^\"(.*)\"$/\1/")
[ -z "$remote" ] && exit 0 # bare login shell
if is_readonly "$remote"; then
emit allow "read-only command on ${host}"
fi
emit ask "This mutates ${host}, a live host where the agent user has passwordless sudo. Command: ${remote}"
;;
mcp__centralis__run_on_host)
env=$(printf '%s' "$payload" | jq -r '.tool_input.env // ""')
cmd=$(printf '%s' "$payload" | jq -r '.tool_input.command // ""')
case "$env" in prod|beta) ;; *) exit 0 ;; esac
if is_readonly "$cmd"; then
emit allow "read-only command on ${env}"
fi
emit ask "run_on_host would mutate ${env}. Command: ${cmd}"
;;
mcp__centralis__sql_query)
env=$(printf '%s' "$payload" | jq -r '.tool_input.env // ""')
write=$(printf '%s' "$payload" | jq -r '.tool_input.write // false')
[ "$write" = "true" ] || exit 0
case "$env" in prod|beta) ;; *) exit 0 ;; esac
sql=$(printf '%s' "$payload" | jq -r '.tool_input.sql // ""' | tr '\n' ' ' | cut -c1-300)
emit ask "Write query against the ${env} database. sql_query escalates to the app role and has no confirmation of its own. SQL: ${sql}"
;;
mcp__centralis__rollback_to|mcp__centralis__promote|mcp__centralis__secrets_rotate)
emit ask "${tool#mcp__centralis__} changes production state directly."
;;
*) exit 0 ;;
esac