thermograph/docs/onboarding/08-infra-secrets.md

8. Infra and secrets

infra/ owns how and where the already-built images run. It never builds app source. Hosts' /opt/thermograph is a checkout of this whole monorepo.

Read infra/CLAUDE.md alongside this.

The machines

Two VPS boxes named by role, not by environment, plus the operator's desktop:

Host	Public	Mesh	Runs	Login
vps1	75.119.132.91 / git.thermograph.org, dashboard.thermograph.org	10.10.0.2	Forgejo, Grafana + Loki + Alloy, emigriffith.dev, and dev (own Postgres, compose, mesh-only)	agent, passwordless sudo
vps2	169.58.46.181 / thermograph.org, beta.thermograph.org	10.10.0.1	prod and beta, as two co-resident Docker Swarm stacks; Centralis; Postfix; backups	agent, passwordless sudo
desktop	—	10.10.0.3	AI-model hosting + flex Swarm capacity. No Thermograph environment; make dev-up is a laptop convenience there	it's your box

ssh -i ~/.ssh/thermograph_agent_ed25519 agent@169.58.46.181   # vps2 (prod + beta)
ssh -i ~/.ssh/thermograph_agent_ed25519 agent@75.119.132.91   # vps1 (Forgejo + Grafana + dev)

The mesh IPs did not move when this topology landed — vps1 is the box that used to be called "beta" (still mesh 10.10.0.2), vps2 the one that used to be called "prod" (still mesh 10.10.0.1). What moved is which environment runs on which box: beta moved from vps1 onto vps2 to sit next to prod, and dev moved onto vps1.

vps1 is guarded as strictly as vps2, not more loosely because it "used to be beta" or because dev now lives there. It hosts Forgejo, its CI runner and the registry, plus Grafana/Loki — a destructive command there takes out git, CI and the registry at once. Dev being mesh-only and a public VPS's tenant, rather than someone's desktop, is exactly why it gets the same guard as everything else on that box: it shares a kernel with the estate's source of truth for code and the monitoring stack, sudo there is passwordless, and it runs whatever branch is currently in flight.

Every root-effective command on both VPSes is logged by auditd (ausearch -k agentcmd). That's a feature — fixes are attributable.

Prefer Centralis for routine work (run_on_host, fleet_status, deployed_version) over raw SSH.

The two orchestrators

Which path an environment takes is decided by infra/deploy/env-topology.sh (thermograph_topology <env>, keyed on THERMOGRAPH_ENV=dev|beta|prod): it derives the checkout, branch, deploy mode, stack/compose name, env file, LB ports and DB role for that environment, and deploy.sh execs deploy/stack/deploy-stack.sh when the mode is stack, or rolls compose otherwise. The old host-wide /etc/thermograph/deploy-mode marker still exists as a fallback for a by-hand run on a box that predates this file, but it cannot describe vps2 on its own — vps2 runs two environments, so "which mode does this host use" is no longer a well-formed question there.

	compose (dev)	Swarm (prod, beta)
Host	vps1	vps2, both stacks co-resident
File	infra/docker-compose.yml	infra/deploy/stack/thermograph-stack.yml (prod), thermograph-beta-stack.yml (beta)
Services	db, backend, lake, daemon, frontend	prod: db, web, worker, lake, daemon, frontend, autoscaler, autoscaler-lake. beta: beta-web, beta-worker, beta-lake, beta-daemon, beta-frontend — prefixed, no db, no autoscalers
Rolling	up -d --no-deps <targets>	start-first, health-gated, auto-rollback
Tag file	deploy/.image-tags.env	deploy/.stack-image-tags.env

Note prod splits backend into web and worker (the THERMOGRAPH_ROLE split from 04); dev's compose stack runs one backend service in role all. Beta's Swarm services are prefixed (beta-web, not web) because Swarm registers a service's short name as a DNS alias on every network it joins — two stacks both naming a service web on the shared data network would make web ambiguous, and beta joins that network (declared external in its stack file) purely to reach prod's db. Beta keeps its own internal overlay for beta-to-beta traffic and has no db service of its own: one TimescaleDB instance serves both environments, on separate databases (thermograph / thermograph_beta) and separate roles (beta's is NOSUPERUSER/NOCREATEDB/NOCREATEROLE, CONNECT revoked from PUBLIC on prod's database).

STACK_TEST=1 rehearses the entire Swarm deploy under stack name thermograph-test on throwaway volumes and ports 18137/18080. Leftover thermograph-test_* containers in logs are rehearsal residue — ignore them.

The volume/project-name coupling

Compose creates thermograph_pgdata / _appdata / _applogs; the Swarm stack declares those exact names as external: true at the same mount paths. That coupling is why name: thermograph is pinned in the compose file — running compose from infra/ without it derives project infra, silently creating a whole new stack with empty volumes beside the running one. deploy-dev.sh exports COMPOSE_PROJECT_NAME=thermograph-dev (env wins over the file key) to keep dev separate on purpose. Keep both halves.

deploy/deploy.sh — read this before you touch a deploy

Single entry point for dev, beta and prod — the same script for all three; infra/deploy/env-topology.sh is what tells it which checkout, branch, stack and ports apply. Contract, run from that environment's own checkout (/opt/thermograph for prod, /opt/thermograph-beta for beta, both on vps2; /opt/thermograph-dev on vps1 via the deploy-dev.sh wrapper):

SERVICE=backend  BACKEND_IMAGE_TAG=sha-<12hex>  /opt/thermograph/infra/deploy/deploy.sh
SERVICE=frontend FRONTEND_IMAGE_TAG=sha-<12hex> /opt/thermograph/infra/deploy/deploy.sh
SERVICE=all BACKEND_IMAGE_TAG=sha-<a> FRONTEND_IMAGE_TAG=sha-<b> …/deploy.sh

What it does, in order, and why each step is the way it is:

1. flock on deploy/.deploy.lock, with a self re-exec so the lock spans the whole run. Backend and frontend deploys can fire for the same push seconds apart and both SSH into one checkout; concurrent runs race the git reset, the compose project and the tag file. -w 600 bounds the wait.
2. Render secrets from the SOPS vault into that environment's own env file (/etc/thermograph.env for prod, /etc/thermograph-beta.env for beta, likewise /etc/thermograph.env on dev's separate host), then source it, so a by-hand run interpolates the same as CI's does. Guarded on the helper's existence so the very deploy that introduces render-secrets.sh is safe.
3. git reset --hard origin/$BRANCH on the checkout root (BRANCH defaults to main for beta/prod, dev for dev). ⚠️ Uncommitted edits in that checkout evaporate here — on vps2 that's /opt/thermograph for a prod deploy or /opt/thermograph-beta for a beta one; each reset only ever touches its own checkout, never the sibling's. The one exception is deploy/.image-tags.env, untracked on purpose.
4. Read .image-tags.env so a single-service roll re-renders compose with both services' real tags and never accidentally recreates or downgrades the sibling.
5. Registry login only if REGISTRY_TOKEN is set. The CI paths don't pass one — the host is already docker logined — and an unconditional login with an empty token would abort under set -e.
6. Pull, with a bounded ~5-minute retry. build-push.yml is a separate workflow triggered by the same push, and Forgejo's needs: only orders jobs within one file. A deploy racing ahead of its build and failing "not found" is confirmed-live behaviour; the retry covers a normal build and still fails loudly on a genuine problem.
7. Daemon capability probe. Infra tracks main while image tags are env-staged, so a host can legitimately be asked to roll a backend image older than the compose file — one built before the daemon binary existed. The script docker runs the image to test for /usr/local/bin/thermograph-daemon and drops daemon from this run if it's absent, rather than leaving a container crash-looping on a missing binary.
8. Roll. TARGETS are backend → (backend, lake, daemon), frontend → (frontend), all → everything. daemon and lake ride with backend and are never targets on their own — all three run the same image at the same tag, and rolling one without the others is exactly the version skew the /internal/* contract has no negotiation for. Single-service rolls use --no-deps; a full all uses --remove-orphans (a renamed-away service's old container would otherwise squat its port — confirmed live).
9. Write .image-tags.env after up, so a failed pull never records a tag that isn't running.
10. Health check (backend 8137, frontend 8080), then detached city warming.

Rollback

Rollback is redeploying the previous image tag. But do not treat image retention as a rollback guarantee: both deploy.sh and deploy-stack.sh end by deleting images outside the running pair. Beta now runs Swarm exactly like prod (it used to be compose, on the old topology, where this pruning reliably left no rollback target at all); on both Swarm stacks it usually fails only because Swarm's stopped task containers still hold references, which is why prod and beta both tend to keep a handful. Dev, the one remaining compose environment, is the one where this prune still reliably succeeds and leaves nothing to roll back to. Verify the target tag is actually present before promising a rollback — Centralis's rollback_to(dry_run: true) checks both host and registry.

docker system prune -a on a live box is forbidden for the same reason. docker image prune -f (dangling only) is the safe form.

Secrets — the SOPS + age vault

The single source of truth is infra/deploy/secrets/*.yaml, committed encrypted (values only — keys stay readable so diffs mean something) and rendered at deploy time by deploy/render-secrets.sh into each environment's own env file (/etc/thermograph.env for prod, /etc/thermograph-beta.env for beta, /etc/thermograph.env again on dev's separate host).

Cycling a key is: edit → commit → deploy. No SSH, no hand-edited root file, no per-host duplication.

File	Holds
common.yaml	the 16 values identical on prod and beta — VAPID keypair, metrics token, IndexNow key, REGISTRY_TOKEN, S3 endpoint/bucket and both S3 keypairs, plus shared non-secret config. Never layered under dev — see below
prod.yaml	prod's own — the three held-back credentials, sizing, THERMOGRAPH_BASE_URL, Discord + mail credentials that exist nowhere else
beta.yaml	beta's own — the three held-back credentials, sizing, base URL
dev.yaml	dev's own, self-contained, 12 values, no production credential
centralis.prod.yaml	Centralis's nine variables → /etc/centralis.env, its own renderer
example.yaml	format reference / CI fixture (fake values)
../../.sops.yaml	which age recipients files encrypt to (plaintext config)

The renderer concatenates common.yaml then <env>.yaml, so a host value wins (last occurrence of a duplicate key). <env> is passed explicitly as THERMOGRAPH_ENV by the deploy caller — required now that vps2 alone hosts two environments and a host-wide marker can't tell them apart; the marker (/etc/thermograph/secrets-env) survives only as the fallback for a by-hand run on a single-environment box.

Two design decisions worth understanding — re-derived for vps1/vps2

Three credentials are deliberately not in common.yaml — POSTGRES_PASSWORD, THERMOGRAPH_AUTH_SECRET, THERMOGRAPH_DATABASE_URL. They hold identical values on prod and beta today (beta was seeded from prod), so by the mechanical rule they'd belong in common.yaml. They're kept per-host anyway, and the reason used to be phrased as a host-isolation argument — "a foothold on beta is a foothold on prod's database, and beta is the more exposed box." That framing no longer holds: beta and prod are now two Swarm stacks co-resident on vps2, so a foothold on the host reaches both regardless of which file a credential lives in. What these three credentials still buy, correctly stated, is a database-level and file-level boundary, not a host-level one: beta's THERMOGRAPH_DATABASE_URL names its own NOSUPERUSER role (thermograph_beta) against its own database on the shared TimescaleDB instance, with CONNECT revoked from PUBLIC on prod's database, and each environment's POSTGRES_PASSWORD/THERMOGRAPH_AUTH_SECRET live in a separate file so rotating prod's stops implying "and beta's too." Keeping them per-host costs one extra line and buys the ability to diverge — that part is unchanged. What changed is what "diverge" is defending against: not a beta compromise reaching a separate prod host, but a beta compromise reaching prod's role and session-signing key on the host they now already share.

The credential boundary that actually maps to hosts now is vps1-vs-vps2, not beta-vs-prod — which is exactly why dev renders dev.yaml alone, never layering common.yaml (the fleet's shared production credentials): deploy-dev.sh exports THERMOGRAPH_SECRETS_SKIP_COMMON=1. Eleven of common.yaml's sixteen values are live production credentials, including a read-write object-storage keypair on the bucket that also holds prod's database backups, and the VAPID private key that signs push to real subscribers. The render is plaintext concatenation, so layering it would put all of those into every container in the dev stack — on vps1, the box that also runs Forgejo and its CI runner, whose docker-labelled jobs get the host socket, running whatever branch is currently dev (unreviewed by definition: that's the branch PRs land on before the gate promotes them to main). An override in dev.yaml wouldn't help: last-wins governs consumers, but the production value is still physically a line in the file. This is the load- bearing boundary in the vault today, not a per-environment credential split that co-residency has already dissolved at the host level.

Dev works fine without common.yaml's values: no S3 keys means the lake service answers 503 and history falls through to the Open-Meteo archive (an accelerator, never a dependency); VAPID and IndexNow both self-generate and persist to the appdata volume; with no metrics token /api/v2/metrics is direct-loopback-only, which is right for a mesh-only box. One common.yaml value is simply wrong for dev — THERMOGRAPH_COOKIE_SECURE=1 silently breaks login over dev's plain-HTTP mesh URL.

None of this makes dev unimportant or unguarded. It's a public VPS's tenant now, sharing a box with Forgejo and the monitoring stack — not somebody's desktop — which is precisely why it gets its own exclusion from common.yaml rather than being treated as beneath the vault's concern.

Working with the vault

sops edit infra/deploy/secrets/beta.yaml     # never a plain editor
infra/deploy/secrets/dry-run-render.sh       # preview a render

• Never hand-edit /etc/thermograph.env — it's a rendered artifact, overwritten on the next deploy.
• Never paste a plaintext secret onto a box.
• secrets-guard CI rejects a plaintext commit; the repo's secrets-guard.sh hook blocks a direct Write/Edit to those files locally.
• deploy/secrets/seed-from-live.sh reads production secrets and is not for an agent to run.
• The key-gaps skill audits which keys are missing or drifting across environments. It reads key names only, never values.

There's a cautionary tale in deploy/secrets/README.md worth reading: Centralis was hand-configured until a hand-edit wrote CENTRALIS_TOKENS as bare JSON into a file that gets sourced, bash stripped the quotes, and Centralis — correctly failing closed on malformed JSON — dropped to a single identity. Nothing logged an error. It looked exactly like the file had never been written. That's why centralis.prod.yaml exists: the only durable fix was a human stops writing the file.

Terraform

infra/terraform/ provisions and configures hosts, SSH-driven.

Never run terraform apply casually. No tfstate is persisted anywhere, so an apply would attempt full re-provisioning of live hosts. Terraform here is executable documentation until state is bootstrapped.

Ops query tooling

Read-only, uniform across environments:

infra/ops/dbq.sh dev  "select count(*) from climate_history"
infra/ops/dbq.sh prod -tA -c "select max(date) from climate_history"
infra/ops/iceberg.sh prod -c "select count(*) from era5_daily"

Both exec into the container. No database is exposed over TCP — each listens only on its private docker network, and prod's and beta's are both Swarm overlays the vps2 host itself cannot route to, so ssh -L is impossible for either. (Dev's, on vps1, is plain compose, so ssh -L would actually work there — but exec is used uniformly across environments anyway, so it doesn't matter which one happens to allow the shortcut.) Exec works identically everywhere with no ports, no tunnels, no infra changes. The prod and beta container names are Swarm task names that change on every redeploy, so they're resolved at call time via docker ps --filter name=, never hardcoded.

Queries connect as thermograph_ro — NOSUPERUSER, granted only pg_read_all_data. Read-only is enforced by Postgres, not by convention:

$ infra/ops/dbq.sh prod -c "create table t(i int)"
ERROR:  permission denied for schema public

Centralis's sql_query does the same thing with the same role. write: true escalates to the app's superuser and takes effect immediately with no confirmation — on prod, know what you are doing.

Backups

The nightly ops-cron.yml pg_dump into agent@vps2:~/thermograph-backups/ (prod's data specifically — see CI and release) is it. Known gaps, stated plainly: a single copy on the same box as the database, no offsite yet, and restores must handle TimescaleDB's continuous_agg circular-FK warning (--disable-triggers). The backups/ prefix in the object-storage bucket belongs to those jobs — don't write outside the lake's own prefix.

Next: Observability.