thermograph/infra/deploy/env-topology.sh
Emi Griffith 4e97d8e5dc
All checks were successful
PR build (required check) / changes (pull_request) Successful in 6s
secrets-guard / encrypted (pull_request) Successful in 5s
PR build (required check) / build-backend (pull_request) Has been skipped
shell-lint / shellcheck (pull_request) Successful in 7s
PR build (required check) / build-frontend (pull_request) Has been skipped
PR build (required check) / validate-observability (pull_request) Successful in 18s
PR build (required check) / gate (pull_request) Successful in 2s
infra: split the estate into vps1/vps2, moving beta next to prod and dev onto vps1
Environments stop being machines. Until now each box WAS an environment --
"beta" named both a deploy target and a host, "the desktop" named both the
operator's computer and the dev server -- so every path could assume one
environment per host and hardcode /opt/thermograph, /etc/thermograph.env and
ports 8137/8080. That assumption ends here:

  vps1  75.119.132.91  Forgejo, Grafana/Loki, the portfolio site, and DEV
                       (own Postgres, mesh-only on 10.10.0.2:8137)
  vps2  169.58.46.181  PROD and BETA as two Swarm stacks sharing one
                       TimescaleDB instance, plus Centralis, Postfix, backups
  desktop              AI model hosting + flex Swarm capacity, no environment

Nothing live has moved; the ordered cutover is in
infra/deploy/RUNBOOK-vps1-vps2-cutover.md.

deploy/env-topology.sh is the single source of truth: env -> host, checkout,
branch, deploy mode, stack name, env file, LB ports, DB role/database, service
prefix. THERMOGRAPH_ENV is the input, and on vps2 it is the only thing
distinguishing a beta deploy from a prod one -- deploy.sh refuses to run when it
disagrees with the checkout it was invoked from. The host-wide secrets-env and
deploy-mode markers survive only as a fallback for a single-environment box.

Beta gets its own stack file rather than an overlay (a merge cannot REMOVE the
db service, and beta having no database of its own is the point). Its services
are prefixed beta-*: Swarm registers a service's short name as a DNS alias on
every network it joins, so two stacks both calling a service `web` on the shared
data network would let prod's frontend resolve a beta task. Prod's stack, env
and LB config are untouched.

One Postgres, two databases with two roles: deploy/db/provision-env-db.sh
creates thermograph_beta (NOSUPERUSER, owns only its own database, CONNECT
revoked from PUBLIC) plus a <role>_ro for ad-hoc queries, and refuses to run for
the environment that owns the instance -- doing so would demote prod's bootstrap
superuser.

Fixes that co-residency would otherwise have broken silently:

- CI secrets are keyed by host (VPS1_SSH_*, VPS2_SSH_*). SSH_* meant "beta" and
  also "the Forgejo box" because those shared a machine; that conflation is what
  once had the prod backup dumping beta.
- The nightly backup dumps BOTH application databases to separate off-box
  prefixes, and fails loudly on a missing one instead of skipping it.
- Alloy stopped deriving the `host` label from the node -- on vps2 that would
  have filed every beta line as prod, feeding prod's alert rules with beta's
  traffic. It is now derived per source, with a new `node` label for the machine,
  and beta's log volume is mounted via a vps2-only overlay.
- ops/dbq.sh, ops/iceberg.sh and the secrets seed scripts derived their SSH
  target and env-file path from the topology instead of hardcoding beta to
  75.119.132.91 -- which is vps1's address now.
- Dev's overlay binds DEV_BIND_ADDR (loopback by default, the mesh address on
  vps1) instead of 0.0.0.0: correct for a home LAN box, a public exposure of
  unreviewed branches on a VPS.

Terraform's hosts variable is keyed by machine with a nested environments map,
and main.tf flattens (host, environment) pairs so two environments on one box
cannot share a checkout. Caddy's single reference config is split per host.

Docs, onboarding and the runbooks are updated throughout, including the security
rationales that co-residency makes false: separate SSH credentials no longer put
a host boundary between beta and prod, and the boundary that remains is the
database and the filesystem.
2026-07-25 15:01:29 -07:00

237 lines
12 KiB
Bash

#!/usr/bin/env bash
# WHERE EACH ENVIRONMENT LIVES — the single source of truth, sourced by every
# deploy path. Not executable on its own; `. env-topology.sh` then call
# `thermograph_topology <env>`.
#
# Why this file exists at all. Until the vps1/vps2 split, "which environment is
# this?" was answerable from the machine you were standing on: one host ran one
# environment, so a host-wide marker (/etc/thermograph/secrets-env) and a
# host-wide deploy mode (/etc/thermograph/deploy-mode) were enough, and every
# path could hardcode /opt/thermograph, /etc/thermograph.env, ports 8137/8080.
# vps2 now runs BOTH beta and prod. Every one of those assumptions becomes a
# collision: two checkouts, two rendered env files, two stacks, two loopback
# port pairs, two image-tag files, two deploy locks — on one box. So the
# environment is now an INPUT (THERMOGRAPH_ENV, passed by the deploy caller),
# and everything else is derived here rather than re-guessed per script.
#
# The host markers survive as the fallback for a by-hand run with no explicit
# env (`deploy.sh` on prod still means prod), but they are no longer the thing
# that decides where files go.
#
# THE ESTATE
#
# vps1 75.119.132.91 10.10.0.2 "operational programs"
# Forgejo (git + CI + registry), Grafana/Loki/Alloy, emigriffith.dev,
# and the DEV environment with its own Postgres. Nothing here is
# reachable by an external user except git/dashboard/the portfolio site;
# dev itself is mesh-only, deliberately (see DEV_BIND_ADDR below).
#
# vps2 169.58.46.181 10.10.0.1 "the deployed environment"
# Everything an external user can touch: prod (thermograph.org) AND beta
# (beta.thermograph.org), Centralis, Postfix, the backups, and the ONE
# TimescaleDB instance that serves both environments on separate
# databases with separate roles.
#
# desktop 10.10.0.3 operator's box
# AI model hosting (voice-to-text, the LLM behind upcoming features) and
# flex capacity as a Swarm worker. It hosts NO Thermograph environment —
# that is the whole point of the vps1/vps2 split. A laptop-style
# `make dev-up` still works locally; it is just not the dev server.
#
# WHY BETA SITS NEXT TO PROD RATHER THAN NEXT TO DEV
#
# Beta is a pre-production rehearsal, so what it needs to rehearse is prod's
# environment: the same orchestrator (Swarm, not compose), the same Postgres
# major and extension build, the same Caddy in front, the same mail path, the
# same mesh position. Co-locating beta with dev bought resemblance to the thing
# it is NOT trying to predict. Co-locating it with prod means a beta green light
# is evidence about prod. The cost — a bad beta deploy shares a machine with
# prod — is bounded by the per-environment isolation this file defines: separate
# stacks, separate volumes, separate DB roles, separate CPU limits.
# thermograph_topology <env>
#
# Exports the TG_* variables describing that environment. Every value is a
# derived fact about the estate, not a preference: change one here and the
# deploy scripts, the stack files and the runbooks all follow.
thermograph_topology() {
local env_name="${1:?thermograph_topology needs an environment: dev|beta|prod}"
# Reset first: this function is sourced into long-lived shells (deploy.sh
# sources it before and after its self re-exec), and a stale TG_STACK_NAME
# from a previous call would silently target the wrong stack.
TG_ENV=""; TG_HOST=""; TG_APP_DIR=""; TG_BRANCH=""; TG_DEPLOY_MODE=""
TG_STACK_NAME=""; TG_STACK_FILE=""; TG_LB_CONFIG=""
TG_COMPOSE_PROJECT=""; TG_COMPOSE_FILE=""
TG_ENV_FILE=""; TG_STACK_ENV_FILE=""; TG_LB_NAME=""
TG_LB_HTTP_PORT=""; TG_LB_FE_PORT=""; TG_DB_NAME=""; TG_DB_USER=""
TG_TAGS_FILE=""; TG_LOCK_FILE=""; TG_BIND_ADDR=""; TG_SKIP_COMMON=0
TG_SVC_PREFIX=""; TG_DATA_NETWORK=""; TG_DB_SERVICE=""; TG_POST_DEPLOY=0
TG_SSH_HOST=""; TG_SSH_TARGET=""
case "$env_name" in
prod)
TG_ENV=prod
TG_HOST=vps2
# Unchanged from before the split — prod keeps every path it already has,
# so nothing about the live prod host moves during the cutover.
TG_SSH_HOST=169.58.46.181
TG_SSH_TARGET=agent@169.58.46.181
TG_APP_DIR=/opt/thermograph
TG_BRANCH=main
TG_DEPLOY_MODE=stack
TG_STACK_NAME=thermograph
TG_STACK_FILE=deploy/stack/thermograph-stack.yml
TG_ENV_FILE=/etc/thermograph.env
TG_STACK_ENV_FILE=/etc/thermograph/stack.env
TG_LB_NAME=thermograph-lb
TG_LB_CONFIG=deploy/stack/lb/Caddyfile
TG_LB_HTTP_PORT=8137
TG_LB_FE_PORT=8080
TG_DB_NAME=thermograph
TG_DB_USER=thermograph
# Prod's services keep their bare names (web, worker, frontend, ...): the
# cutover must not touch prod's stack file, its env or its LB config.
TG_SVC_PREFIX=""
# Where the database lives. Prod's own overlay, which is `attachable` and
# which beta joins as an external network.
TG_DATA_NETWORK=thermograph_internal
TG_DB_SERVICE=thermograph_db
# Post-deploy city warm + IndexNow ping. Prod only — see the beta note.
TG_POST_DEPLOY=1
;;
beta)
TG_ENV=beta
TG_HOST=vps2
# Same box, same public IP as prod — beta and prod are two stacks on one
# Swarm manager now, not two separate hosts. A beta SSH target is
# therefore prod's blast radius too (see .claude/hooks/prod-guard.sh).
TG_SSH_HOST=169.58.46.181
TG_SSH_TARGET=agent@169.58.46.181
# A SECOND checkout on the same box. Separate from prod's so the two can
# sit on different commits of this repo, and so `git reset --hard` in one
# deploy can never yank the tree out from under the other's running
# deploy. Everything below is likewise a distinct name/port/path from
# prod's, on purpose: co-location is only safe if nothing is shared by
# accident. The one deliberate exception is the database SERVER (see
# TG_DB_* — separate role and database on a shared instance).
TG_APP_DIR=/opt/thermograph-beta
TG_BRANCH=main
TG_DEPLOY_MODE=stack
TG_STACK_NAME=thermograph-beta
TG_STACK_FILE=deploy/stack/thermograph-beta-stack.yml
TG_ENV_FILE=/etc/thermograph-beta.env
TG_STACK_ENV_FILE=/etc/thermograph/beta-stack.env
TG_LB_NAME=thermograph-beta-lb
TG_LB_CONFIG=deploy/stack/lb/Caddyfile.beta
# NOT 8137/8080: prod's loopback LB already owns those on this host.
TG_LB_HTTP_PORT=8237
TG_LB_FE_PORT=8180
TG_DB_NAME=thermograph_beta
# Its own role, not prod's `thermograph` superuser-of-its-own-database.
# One instance is a capacity decision; it is not a decision to let a beta
# deploy running an unmerged branch read or write the production database.
TG_DB_USER=thermograph_beta
# Beta's Swarm services are named beta-web, beta-worker, ... Swarm
# registers a service's SHORT name as a DNS alias on every network it
# joins, so two stacks that both call a service `web` on one shared
# network make `web` ambiguous — prod's frontend could resolve a beta
# task. Prefixing beta's names removes the collision without editing a
# single line of prod's stack.
TG_SVC_PREFIX="beta-"
# Beta has no db service of its own: it joins prod's overlay (declared
# `external` in its stack file) purely to reach `db`, and keeps its own
# `internal` overlay for beta-to-beta traffic.
TG_DATA_NETWORK=thermograph_internal
TG_DB_SERVICE=thermograph_db
# No warm, no IndexNow on beta. IndexNow announces URLs to Bing/DDG/
# Yandex — from beta that means asking search engines to index
# beta.thermograph.org, which is the opposite of what beta is for. The
# city warm is worse than useless here: it spends the shared upstream
# archive quota to fill a cache that only a rehearsal environment reads.
# (Both ran on beta under compose; that was a side effect of beta and
# prod sharing one script, not a decision.)
TG_POST_DEPLOY=0
;;
dev)
TG_ENV=dev
TG_HOST=vps1
# vps1 — Forgejo/Grafana/dev, mesh-only for dev itself (see TG_BIND_ADDR).
TG_SSH_HOST=75.119.132.91
TG_SSH_TARGET=agent@75.119.132.91
TG_APP_DIR=/opt/thermograph-dev
# The only environment that tracks `dev`; beta and prod both run this
# repo's `main` (infra is not environment-staged — app code is, via image
# tags). See the branch model in infra/README.md.
TG_BRANCH=dev
# Compose, not Swarm: dev's value is a fast, legible, single-box stack
# you can `docker compose logs` at. It is the one environment not
# pretending to be prod.
TG_DEPLOY_MODE=compose
TG_COMPOSE_PROJECT=thermograph-dev
TG_COMPOSE_FILE="docker-compose.yml:docker-compose.dev.yml"
TG_ENV_FILE=/etc/thermograph.env
# Dev keeps its OWN Postgres container (a `db` service in its compose
# project). It is not on the shared instance and must not be: the shared
# instance lives on vps2 and holds real user data.
TG_DB_NAME=thermograph
TG_DB_USER=thermograph
# Mesh-only. vps1 is a public VPS, so the dev overlay must not publish on
# 0.0.0.0 the way it did on the operator's LAN box — dev runs whatever
# branch is in flight, including unreviewed ones. Reachable at
# http://10.10.0.2:8137 from anything on the WireGuard mesh, and from
# nowhere else. There is deliberately no dev DNS record and no Caddy site.
TG_BIND_ADDR=10.10.0.2
# Dev NEVER layers common.yaml — that file is the internet-facing fleet's
# shared production credential set (both S3 keypairs, the VAPID signing
# key, REGISTRY_TOKEN, the metrics token), and dev runs whatever branch is
# in flight on the same box as Forgejo and its CI runner. This lives here
# rather than only in deploy-dev.sh so the protection is a property of the
# ENVIRONMENT, not of which entry point happened to be used. See the long
# note in render-secrets.sh.
TG_SKIP_COMMON=1
TG_DB_SERVICE=db
# Dev warms nothing and pings nothing: it must never spend the upstream
# archive quota, and it must never tell a search engine it exists.
TG_POST_DEPLOY=0
;;
*)
echo "!! unknown environment '$env_name' (expected dev|beta|prod)" >&2
return 2
;;
esac
# Derived, never hand-set: keeping these next to the definitions above is what
# stops a second environment on one host from sharing a lock or a tag file.
TG_LOCK_FILE="$TG_APP_DIR/infra/deploy/.deploy.lock"
if [ "$TG_DEPLOY_MODE" = stack ]; then
TG_TAGS_FILE="$TG_APP_DIR/infra/deploy/.stack-image-tags.env"
else
TG_TAGS_FILE="$TG_APP_DIR/infra/deploy/.image-tags.env"
fi
export TG_ENV TG_HOST TG_APP_DIR TG_BRANCH TG_DEPLOY_MODE
export TG_STACK_NAME TG_STACK_FILE TG_LB_CONFIG TG_COMPOSE_PROJECT TG_COMPOSE_FILE
export TG_ENV_FILE TG_STACK_ENV_FILE TG_LB_NAME
export TG_LB_HTTP_PORT TG_LB_FE_PORT TG_DB_NAME TG_DB_USER
export TG_TAGS_FILE TG_LOCK_FILE TG_BIND_ADDR TG_SKIP_COMMON
export TG_SVC_PREFIX TG_DATA_NETWORK TG_DB_SERVICE TG_POST_DEPLOY
export TG_SSH_HOST TG_SSH_TARGET
}
# thermograph_env_name
#
# Which environment is this run for? Explicit input first (the deploy workflows
# pass THERMOGRAPH_ENV, and on vps2 it is the ONLY thing distinguishing a beta
# deploy from a prod one), then the host marker for a by-hand run on a
# single-environment box. Empty output means "cannot tell" and the caller must
# fail rather than guess — guessing wrong on vps2 means deploying beta's image
# tag onto prod.
thermograph_env_name() {
local marker="${THERMOGRAPH_SECRETS_ENV_FILE:-/etc/thermograph/secrets-env}"
if [ -n "${THERMOGRAPH_ENV:-}" ]; then
printf '%s\n' "$THERMOGRAPH_ENV"
return 0
fi
cat "$marker" 2>/dev/null || true
}