thermograph/infra/deploy/secrets

Secrets — the git-native vault (SOPS + age)

The single source of truth for Thermograph's secrets is the set of SOPS-encrypted YAML files in this directory. They're committed to the repo encrypted (values only — keys stay readable so diffs are meaningful) and rendered into /etc/thermograph.env at deploy time by deploy/render-secrets.sh, which the prod/beta deploy.sh invokes.

Cycling a key is: edit → commit → deploy. No SSHing in to hand-edit a root file, no per-host duplication.

Files

File	Holds	Encrypted?
common.yaml	The 16 values identical on prod and beta — VAPID keypair, metrics token, IndexNow key, REGISTRY_TOKEN, S3 endpoint/bucket and both S3 keypairs, plus shared non-secret config. Not layered under dev — see below	✅
prod.yaml	Prod's own — the three held-back credentials below, sizing (APP_CPUS, DB_CPUS, DB_MEMORY, WORKERS), THERMOGRAPH_BASE_URL, and the Discord + mail credentials that exist nowhere else	✅
beta.yaml	Beta's own — the three held-back credentials, THERMOGRAPH_BASE_URL, APP_CPUS/WORKERS, plus THERMOGRAPH_MAIL_BACKEND=console and THERMOGRAPH_DISCORD_BOT=0. Its POSTGRES_PASSWORD and THERMOGRAPH_DATABASE_URL name beta's own role and database (thermograph_beta) on the instance it shares with prod. DB_CPUS/DB_MEMORY were removed: beta no longer runs a database of its own, and leaving them would have implied it sized one — the shared instance is sized by prod.yaml's values (see deploy/stack/thermograph-stack.yml's db service)	✅
dev.yaml	Dev's own (vps1) — self-contained, 12 values, no production credential among them	✅
centralis.prod.yaml	Centralis's nine variables, rendered to /etc/centralis.env on prod. A different service, a different output file, a different renderer — see below	✅
example.yaml	Format reference / CI fixture (fake values)	✅
../../.sops.yaml	Which age recipient files are encrypted to (plaintext config)	—

Three credentials are deliberately NOT in common.yaml

POSTGRES_PASSWORD, THERMOGRAPH_AUTH_SECRET and THERMOGRAPH_DATABASE_URL hold different values on prod and beta today (beta has its own database role/password on the shared instance, and its own auth secret), so unlike the rest of common.yaml they were never candidates for sharing on values alone. They stay per-environment for a reason that matters more now than it used to:

These are the credentials that let one environment act as another. With matching values, a foothold in one environment's rendered env file is a foothold on the other's database and the other's session signing. That used to matter because beta and prod were separate boxes; it matters for a different reason now that they're co-resident on vps2 — keeping the credentials themselves separate is what stops co-location on one host from also becoming equivalence at the credential level. deploy/db/provision-env-db.sh enforces the same boundary from the database side: beta's role (thermograph_beta) is NOSUPERUSER/NOCREATEDB/NOCREATEROLE, owns only its own database, and CONNECT is revoked from PUBLIC on it — so even a leaked beta credential cannot reach prod's data, and the reverse.

What actually separates beta and prod now, stated plainly: they share a host by design (see deploy/env-topology.sh's header), so a host-level compromise of vps2 is shared between them — separate SSH credentials no longer buy a host boundary between beta and prod, the way they once did when beta and prod were different machines. What remains is isolation at the database level (separate roles, separate databases, CONNECT revoked from PUBLIC) and the file level (separate checkouts, separate rendered env files, separate stack env files, separate loopback ports). Keeping POSTGRES_PASSWORD/THERMOGRAPH_AUTH_SECRET/THERMOGRAPH_DATABASE_URL out of a shared file is part of that file-level isolation: nothing on vps2 would ever hand you prod's auth secret merely because you have beta's.

vps1 is the box that must never hold prod credentials. It runs Forgejo, Forgejo's CI runner, Grafana, and the dev environment — which runs whatever branch is currently in flight, reviewed or not. That is the actual reason dev renders dev.yaml alone and never layers common.yaml (see below): not squeamishness about a dev box, but that vps1 is uniquely positioned to leak whatever it's handed, and common.yaml is the fleet's shared production credential set.

Keeping the three held-back credentials per-environment costs nothing (one line each) and buys the ability for them to diverge further, or for a future environment to be added without inheriting anyone else's values by accident.

dev.yaml is the case that already differs completely: all three of its values are its own, generated on vps1 and never shared with prod or beta.

At deploy the renderer concatenates common.yaml then <env>.yaml, so a host value wins (env_file / source take the last occurrence of a duplicate key). <env> comes from an explicit THERMOGRAPH_ENV/argument on vps2 (which runs two environments) or /etc/thermograph/secrets-env as a fallback elsewhere — see deploy/env-topology.sh.

dev is vaulted, but renders dev.yaml alone

All three environments are managed here now. dev is the one that does not get common.yaml layered under it: deploy-dev.sh exports THERMOGRAPH_SECRETS_SKIP_COMMON=1, and the renderer skips the shared layer.

That is not squeamishness about a dev box — it is what common.yaml contains. Eleven of its sixteen values are live production credentials: REGISTRY_TOKEN, THERMOGRAPH_S3_ACCESS_KEY/_SECRET_KEY and THERMOGRAPH_LAKE_S3_ACCESS_KEY/_SECRET_KEY (one keypair, read-write, on the bucket that also holds prod's database backups — see ../../ops/README.md), THERMOGRAPH_VAPID_PRIVATE_KEY/_PUBLIC_KEY (they sign Web Push to real subscribers), THERMOGRAPH_INDEXNOW_KEY, THERMOGRAPH_METRICS_TOKEN. The render is a plaintext concatenation, so layering it would put every one of those in /etc/thermograph.env on the dev box and, through env_file:, into the environment of every container in the dev stack.

The dev box is vps1 — the same box that runs Forgejo and its CI. Its docker-labelled runner jobs get the host docker socket automounted, and the stack it runs is the dev branch — code that has not been through the gate that guards prod. Handing that the estate's read-write object-storage keys and the push signing key is a strictly larger blast radius than anything the vault buys back. An override in dev.yaml would not help: last-wins governs consumers, but the production value is still physically a line in the file.

One value in common.yaml is also simply wrong for dev: THERMOGRAPH_COOKIE_SECURE=1 is right for the TLS hosts and silently breaks login over dev's plain-HTTP mesh-only URL. dev.yaml sets 0.

So dev.yaml is self-contained. Where the value is shared-but-harmless (PORT, THERMOGRAPH_BASE, TIMESCALEDB_TAG) it carries its own copy — a duplicated constant is the cheap half of this trade.

What dev.yaml holds (12 keys):

Key	Why
POSTGRES_PASSWORD, THERMOGRAPH_DATABASE_URL	dev's own. Deliberately the weak, well-known thermograph-dev — the value dev's pgdata volume is already initialized with, and the DB is never published off the compose network. Kept per-host for the reason below, and not shared with prod
THERMOGRAPH_AUTH_SECRET	dev's own, freshly generated. Fixes a live crash loop: the daemon service refuses to start without it (it derives the /internal/* token from it), so dev's daemon had been restarting every 60s. Prod's value must never be here — it signs sessions and verification links
THERMOGRAPH_BASE_URL	http://10.10.0.2:8137 (vps1's mesh address). Unset, the app defaults to https://thermograph.org, so dev's IndexNow pings and Discord links claimed to be prod
THERMOGRAPH_COOKIE_SECURE=0	plain HTTP on the mesh-only URL
THERMOGRAPH_MAIL_BACKEND=console, THERMOGRAPH_DISCORD_BOT=0	explicit fail-safes. Both are already the code defaults; stating them means dev cannot start mailing or open a Discord gateway by inheriting a value
PORT, THERMOGRAPH_BASE, TIMESCALEDB_TAG, DB_MEMORY, WORKERS	non-secret config that would otherwise have come from common.yaml

Deliberately absent, and why nothing breaks:

• Both S3 keypairs. Without them the lake service answers 503 and history reads fall through to the Open-Meteo archive — an accelerator, never a dependency (docker-compose.yml, backend/data/climate.py).
• VAPID keypair. backend/notifications/push.py resolves env → file → generates once and persists to the appdata volume. Dev gets its own identity for free; prod's private key stays off the box.
• THERMOGRAPH_INDEXNOW_KEY. Same env → file → self-generate ladder (backend/indexnow.py). Prod's key is what authenticates URL submissions for thermograph.org.
• THERMOGRAPH_METRICS_TOKEN. With no token /api/v2/metrics is direct-loopback-only, which is exactly right on a mesh-only box; dev's Alloy agent ships logs, not metrics, so nothing scrapes it.
• REGISTRY_TOKEN. dev pulls with the persistent docker login credential already in the host's docker config, like the prod/beta CI paths. A vault copy would only duplicate a credential the box already has into a more readable file.
• All Discord and mail credentials. Dev must not be able to act as the real bot or send real mail.
• APP_CPUS, DB_CPUS. docker-compose.dev.yml !resets both — dev runs uncapped, so setting them would be a lie. (DB_MEMORY is live: the db service's mem_limit is not reset.)

If dev ever needs one of these for real, it gets its own — a dev Discord app, a scoped read-only bucket key — never a copy of prod's.

Centralis: its own file, its own renderer

/etc/centralis.env on prod is the control plane's env file — four real credentials (CENTRALIS_AUTH_TOKEN, CENTRALIS_TOKENS, CENTRALIS_FORGEJO_TOKEN, DISCORD_TOKEN) and five non-secret settings. It was hand-edited until 2026-07-24, when a hand-edit wrote CENTRALIS_TOKENS as bare JSON into a file that gets sourced, bash stripped the double quotes, {"emi":"…"} arrived as {emi:…}, and Centralis — correctly failing closed on malformed JSON — dropped to the single shared identity. Nothing logged an error. It looked exactly like the file had never been written.

centralis.prod.yaml + render_centralis_secrets() in ../render-secrets.sh end that, by the only durable route: a human stops writing the file.

Seed / verify / rotate:

# One-time seed from the live box (plaintext never leaves prod):
deploy/secrets/seed-centralis-on-host.sh prod agent@169.58.46.181

# Prove a render matches the live file before cutting over. Renders to a 0700
# scratch dir on the host, sources both, compares — key NAMES and a verdict only:
deploy/secrets/verify-centralis-render.sh prod agent@169.58.46.181

# Thereafter, rotation is edit -> commit -> merge to main -> redeploy Centralis:
sops deploy/secrets/centralis.prod.yaml

Merging to main is what puts the new ciphertext on prod (infra-sync.yml fast-forwards /opt/thermograph); the render happens when Centralis is redeployed. See Centralis's own deploy/README.md §2 and §5.

Why CENTRALIS_* is not just added to prod.yaml

It is the obvious move, and it hands the estate's control-plane bearer token to every application container.

/etc/thermograph.env is loaded into the app stack through env_file: and through . /host/thermograph.env in the stack services' entrypoints. Every key in it is in the environment of every container in the stack. CENTRALIS_TOKENS and CENTRALIS_AUTH_TOKEN authenticate an endpoint that carries sql_query with write:true and run_on_host over production, and CENTRALIS_FORGEJO_TOKEN is repo-scope on the monorepo. Putting them in prod.yaml would mean that a read-anything bug in the web backend, or a leaked container env dump, is a foothold on the control plane. The two files are separate for a reason that predates the vault, and the vault is not a reason to merge them.

The converse — pointing Centralis's deploy at /etc/thermograph.env — has the same problem from the other side, plus 32 unrelated variables in the shell that runs its compose, plus a lifecycle coupling: every app deploy re-renders that file, and Centralis and the app stack are deliberately deployed on different cadences.

Why this file is fully quoted and /etc/thermograph.env is not

Different consumers.

/etc/thermograph.env is read by docker-compose env_file: and by systemd EnvironmentFile=, which parse KEY=value literally. Raw sops --output-type dotenv is the right representation for it, and it must stay that way: Centralis's own secrets_inventory / secrets_render tools compare rendered values against that file textually, so quoting it would report all 32 keys as value-changed.

/etc/centralis.env is read by exactly one thing, and it is a shell:

sudo bash -c 'set -a && . /etc/centralis.env && set +a && docker compose up -d --build'

set -a; . file runs every line through bash's full expansion pipeline. Raw dotenv output is unsafe there and sops cannot make it safe — it emits values verbatim. So render_centralis_secrets decrypts to JSON (lossless; dotenv escapes a newline to a literal \n and cannot be told apart from a literal backslash-n) and writes POSIX single-quoted assignments, in which every byte is literal except ' itself.

Then it proves the file before letting it near /etc: it sources the rendered file in a clean shell (env -i), reads every variable back, and compares to the plaintext it started from. If any value does not survive, or if CENTRALIS_TOKENS does not parse as a JSON object of subject → token, the render aborts and the existing file is left untouched. The 2026-07-24 bug cannot be written by this path, and cannot be written through it either.

Scoped by environment even though Centralis is prod-only

The file is centralis.prod.yaml, not centralis.yaml, and the renderer takes the prod from /etc/thermograph/secrets-env like everything else here. An unscoped file would render prod's bearer tokens onto any host that ran the function. If Centralis ever gains a second home, that is a new file rather than a redesign — and the two boxes get different tokens, which is the point.

centralis.prod.yaml is not layered under common.yaml. Centralis shares no key with the app stack (verified: zero overlap between the two live env files), so layering would only push the VAPID private key, both S3 keypairs and REGISTRY_TOKEN into a file that wants none of them.

The age key (the one thing that matters)

• The private key is the single recovery root. Lose it and every secret here is unrecoverable; leak it and every secret is compromised.
• It is never in the repo. It lives at:
  • ~/.config/sops/age/keys.txt on the operator's machine (for editing), and
  • /etc/thermograph/age.key (0400) on each VPS that renders at deploy time.
• Back it up in the password manager. The public recipient is in .sops.yaml.

The two on-host copies are a deliberate quorum

/etc/thermograph/age.key on vps1 and vps2 is not provisioning residue and must not be tidied away. Together they are the recovery quorum for the entire estate: five SOPS vaults, and every off-box backup, since ops-cron.yml:142,197 pipe those dumps through age -r to this same recipient.

That is not theoretical. On 2026-07-30 the operator's copy was shredded from the desktop before it had been copied to its replacement machine, leaving zero operator-side copies. These two were the only surviving material, and the key was restored from vps2. The ordering rule below exists because of that hour.

• Never remove the last operator-side copy until its replacement has been verified against all five vaults (sops -d … | grep -c '=' → 16 / 12 / 8 / 16 / 9). Deleting first and verifying second is unrecoverable if the copy is bad.
• Both hosts must hold byte-identical copies. Divergence is as bad as absence: two different keys means one host renders secrets the other cannot read, and a restore silently picks the wrong one.
• Verify it, don't assume it — deploy/secrets/verify-age-quorum.sh asserts presence, permissions and hash equality across both hosts. It prints SHA-256 digests and modes only, never the key, so it is safe in a CI log and suitable as a cron gate. Deletion and divergence are both silent failures; nothing else in the estate would notice either until a restore.

Verified 2026-07-30: vps1 0440 root:deploy, vps2 0400 root:root, digests equal. Note the asymmetry — on vps1 the group deploy can read the key that decrypts common.yaml and prod.yaml, on the box that also runs Forgejo, its CI runner and dev's unreviewed branches. The dev render does not need that group bit: it runs as agent, which is not in group deploy and reaches the key through its passwordless sudo instead (/opt/thermograph-dev is agent:agent). Tightening vps1 to 0400 root:root is therefore expected to be safe and is the obvious follow-up; it is called out rather than done here because it changes a live host on the deploy path.

This weakens — in practice, not in intent — the rule in infra/CLAUDE.md that "vps1 must never hold prod credentials". Withholding common.yaml from dev's render does not withhold the key that decrypts it from the box, because every vault is encrypted to a single recipient. Fixing that properly means a second age identity for dev, not a policy line.

Prerequisites (once per machine)

Install sops + age (single static binaries). On a host, use deploy/provision-secrets.sh; on your laptop:

# see the pinned URLs in provision-secrets.sh, or:
go install github.com/getsops/sops/v3/cmd/sops@latest   # if you have Go

Everyday: rotate a key

sops deploy/secrets/common.yaml        # opens DECRYPTED in $EDITOR; re-encrypts on save
git commit -am "rotate <thing>" && git push forgejo
# beta and prod both auto-deploy on push (main -> beta, release -> prod) via
# the Deploy workflow. To force it by hand on vps2, say which environment:
ssh agent@169.58.46.181 'cd /opt/thermograph && git pull && THERMOGRAPH_ENV=prod deploy/deploy.sh'
ssh agent@169.58.46.181 'cd /opt/thermograph-beta && git pull && THERMOGRAPH_ENV=beta deploy/deploy.sh'

A host-specific value (e.g. POSTGRES_PASSWORD) is the same, editing prod.yaml / beta.yaml / dev.yaml instead. POSTGRES_PASSWORD is special: the database only reads it on a fresh volume, so also ALTER ROLE thermograph PASSWORD '…' inside the running Postgres to match — the env change alone won't re-key an initialized DB. (That applies to dev too. dev.yaml deliberately carries the value dev's existing pgdata volume was initialized with, so nothing rotates the moment dev is provisioned; rotating it later is the two-step above, plus the matching default in deploy-dev.sh.)

Dev is edited and deployed exactly like the others — sops deploy/secrets/dev.yaml, commit, and the next deploy-dev.sh run (merge to dev, or by hand) renders it.

Setting up a new dev machine

Dev is provisioned like any fleet host now (deploy/provision-dev.sh, run as root on vps1), not the sudo-free desktop bootstrap this predates. The secrets-specific steps below are still worth doing by hand rather than folding into that script: provision-secrets.sh is not the right tool here — it installs the age key root-owned at /etc/thermograph/age.key, which the CI-runner-driven render may not be able to read (see the age-key note above).

# 1. sops + age on PATH, and the age private key in the operator's keyring:
#      ~/.config/sops/age/keys.txt   (0600, restored from the password manager)
sops --version && age --version

# 2. Name the environment. This marker is what switches rendering ON — until it
#    exists the box renders nothing, which is the safe default.
sudo mkdir -p /etc/thermograph
printf 'dev\n' | sudo tee /etc/thermograph/secrets-env >/dev/null
sudo chmod 0644 /etc/thermograph/secrets-env

# 3. Pre-create the render target owned by the deploying user, so no deploy ever
#    needs sudo (the renderer writes in place when the file is writable — the same
#    path beta's non-root `deploy` user takes).
sudo install -m 0600 -o "$USER" -g "$USER" /dev/null /etc/thermograph.env

# 4. Dry-run the render before letting a deploy do it, and read the key names back.
#    Nothing from common.yaml may appear here.
THERMOGRAPH_SECRETS_SKIP_COMMON=1 \
SOPS_AGE_KEY_FILE=~/.config/sops/age/keys.txt \
  sops -d --input-type yaml --output-type dotenv deploy/secrets/dev.yaml | sed 's/=.*/=/'

# 5. Deploy. deploy-dev.sh refuses to run if render-secrets.sh in the checkout
#    cannot honour THERMOGRAPH_SECRETS_SKIP_COMMON, rather than leak.
APP_DIR=/opt/thermograph-dev bash deploy/deploy-dev.sh

To take a dev box back off the vault, delete /etc/thermograph/secrets-env: presence detection turns rendering off and deploy-dev.sh falls back to its built-in POSTGRES_PASSWORD default.

A different dev machine (a laptop running make dev-up locally, or a future second dev host) needs its own values, not a copy of vps1's: give it its own <env>.yaml and its own marker name, so the two can never be confused for one host — the same argument as the three held-back credentials above.

Add a new secret

1. sops deploy/secrets/common.yaml, add THERMOGRAPH_NEW_THING: value.
2. Add the app reader (os.environ.get("THERMOGRAPH_NEW_THING")).
3. Commit + deploy. It flows into /etc/thermograph.env automatically.

common.yaml reaches prod and beta only. If dev needs the new thing, decide which it is and act on it: harmless config → copy it into dev.yaml; a real credential → mint dev its own and put that in dev.yaml; neither → leave dev without it and make sure the code degrades rather than crashes. Do not reach for prod's copy.

Rotate the age identity itself (re-key)

age-keygen -o new.key                              # new identity
# add the new PUBLIC recipient alongside the old in .sops.yaml, then:
sops updatekeys deploy/secrets/*.yaml              # re-encrypt to BOTH keys
# distribute new.key to /etc/thermograph/age.key on every host + ~/.config/sops/age,
# deploy once so every host can decrypt with the new key, then remove the OLD
# recipient from .sops.yaml and:
sops updatekeys deploy/secrets/*.yaml              # drop the old key
git commit -am "rotate age identity" && push + deploy

First-time bootstrap (seed from the live boxes)

Historical — this ran once, before the vps1/vps2 split, when beta was its own box at 75.119.132.91. That address is vps1 now (dev + Forgejo + Grafana), not beta; re-running the beta line below as written would seed beta.yaml from the wrong host. Kept for the shape of the process, which is unchanged if a new environment is ever bootstrapped the same way — just point seed-from-live.sh at wherever that environment actually lives today.

Order matters — values must match the live env exactly so AUTH_SECRET / VAPID / POSTGRES_PASSWORD don't rotate unintentionally:

1. Install sops+age and drop /etc/thermograph/age.key (0400) + /etc/thermograph/secrets-env on the target hosts (provision-secrets.sh).
2. Seed each host's file from its live env with the helper (pulls over SSH, encrypts, and verifies the render round-trips — run it on your own machine, it reads live secrets), e.g. as originally run:

   deploy/secrets/seed-from-live.sh prod agent@169.58.46.181
   deploy/secrets/seed-from-live.sh beta agent@75.119.132.91   # beta's address AT THE TIME; now vps1's
   

   That writes exact per-host copies (prod.yaml/beta.yaml); factor shared values into common.yaml later. Commit.
3. Dry-run: on the box, render to a temp file and diff against the live /etc/thermograph.env — must be byte-identical before cutover.
4. Merge the deploy.sh wiring; run one deploy; confirm the app is healthy.

Safety

• CI (.forgejo/workflows/secrets-guard.yml) fails if any *.yaml here is committed unencrypted.
• Rendering is presence-detected: a host without the age key + marker keeps its existing /etc/thermograph.env, so nothing breaks before a host is migrated.
• Rendering happens on the target box, so CI/runners never see the age key or the plaintext — except on dev, where the box is the runner. That is not a hole this vault can close; it is the reason dev.yaml holds nothing that would matter if the box were owned. Keep it that way: the day dev needs a real credential, mint it a scoped one of its own.
• deploy-dev.sh fails closed: if the render-secrets.sh it finds cannot honour THERMOGRAPH_SECRETS_SKIP_COMMON, it aborts the deploy rather than render the shared production layer onto the dev box.