thermograph/infra/deploy/secrets

Secrets — the git-native vault (SOPS + age)

The single source of truth for Thermograph's secrets is the set of SOPS-encrypted YAML files in this directory. They're committed to the repo encrypted (values only — keys stay readable so diffs are meaningful) and rendered into /etc/thermograph.env at deploy time by deploy/render-secrets.sh, which the prod/beta deploy.sh invokes.

Cycling a key is: edit → commit → deploy. No SSHing in to hand-edit a root file, no per-host duplication.

Files

File	Holds	Encrypted?
common.yaml	The 16 values identical on prod and beta — VAPID keypair, metrics token, IndexNow key, REGISTRY_TOKEN, S3 endpoint/bucket and both S3 keypairs, plus shared non-secret config. Not layered under dev — see below	✅
prod.yaml	Prod's own — the three held-back credentials below, sizing (APP_CPUS, DB_CPUS, DB_MEMORY, WORKERS), THERMOGRAPH_BASE_URL, and the Discord + mail credentials that exist nowhere else	✅
beta.yaml	Beta's own — the three held-back credentials, sizing, THERMOGRAPH_BASE_URL	✅
dev.yaml	The LAN dev box's own — self-contained, 12 values, no production credential among them	✅
centralis.prod.yaml	Centralis's nine variables, rendered to /etc/centralis.env on prod. A different service, a different output file, a different renderer — see below	✅
example.yaml	Format reference / CI fixture (fake values)	✅
../../.sops.yaml	Which age recipient files are encrypted to (plaintext config)	—

Three credentials are deliberately NOT in common.yaml

POSTGRES_PASSWORD, THERMOGRAPH_AUTH_SECRET and THERMOGRAPH_DATABASE_URL hold identical values on prod and beta today, so by the mechanical rule they belong in common.yaml. They are kept per-host anyway.

These are the credentials that let one environment act as another: with them, a foothold on beta is a foothold on prod's database and prod's session signing. Beta is the more exposed box — it serves public Forgejo and Grafana. They match only because beta was seeded from prod, not because the two are meant to be one system.

Keeping them per-host costs nothing now (same values, one extra line each) and buys the ability to diverge: rotating prod's database password stops implying "and beta's too". Moving them into common.yaml would encode the equivalence as intentional and turn breaking it into a migration rather than an edit.

If you ever do want them to differ, change one file. That is the whole point.

dev.yaml is the case that already differs: all three of its copies are its own, generated on the desktop. The rule earned its keep the first time it was used.

At deploy the renderer concatenates common.yaml then <env>.yaml, so a host value wins (env_file / source take the last occurrence of a duplicate key). <env> comes from /etc/thermograph/secrets-env on the box (prod/beta/dev).

dev is vaulted, but renders dev.yaml alone

All three environments are managed here now. dev is the one that does not get common.yaml layered under it: deploy-dev.sh exports THERMOGRAPH_SECRETS_SKIP_COMMON=1, and the renderer skips the shared layer.

That is not squeamishness about a dev box — it is what common.yaml contains. Eleven of its sixteen values are live production credentials: REGISTRY_TOKEN, THERMOGRAPH_S3_ACCESS_KEY/_SECRET_KEY and THERMOGRAPH_LAKE_S3_ACCESS_KEY/_SECRET_KEY (one keypair, read-write, on the bucket that also holds prod's database backups — see ../../ops/README.md), THERMOGRAPH_VAPID_PRIVATE_KEY/_PUBLIC_KEY (they sign Web Push to real subscribers), THERMOGRAPH_INDEXNOW_KEY, THERMOGRAPH_METRICS_TOKEN. The render is a plaintext concatenation, so layering it would put every one of those in /etc/thermograph.env on the dev box and, through env_file:, into the environment of every container in the dev stack.

The dev box is the operator's desktop. It is also the Forgejo CI runner, whose docker-labelled jobs get the host docker socket automounted, and the stack it runs is the dev branch — code that has not been through the gate that guards prod. Handing that the estate's read-write object-storage keys and the push signing key is a strictly larger blast radius than anything the vault buys back. An override in dev.yaml would not help: last-wins governs consumers, but the production value is still physically a line in the file.

One value in common.yaml is also simply wrong for dev: THERMOGRAPH_COOKIE_SECURE=1 is right for the TLS hosts and silently breaks login over dev's plain-HTTP LAN URL. dev.yaml sets 0.

So dev.yaml is self-contained. Where the value is shared-but-harmless (PORT, THERMOGRAPH_BASE, TIMESCALEDB_TAG) it carries its own copy — a duplicated constant is the cheap half of this trade.

What dev.yaml holds (12 keys):

Key	Why
POSTGRES_PASSWORD, THERMOGRAPH_DATABASE_URL	dev's own. Deliberately the weak, well-known thermograph-dev — the value dev's pgdata volume is already initialized with, and the DB is never published off the compose network. Kept per-host for the reason below, and not shared with prod
THERMOGRAPH_AUTH_SECRET	dev's own, freshly generated. Fixes a live crash loop: the daemon service refuses to start without it (it derives the /internal/* token from it), so dev's daemon had been restarting every 60s. Prod's value must never be here — it signs sessions and verification links
THERMOGRAPH_BASE_URL	http://10.0.1.216:8137. Unset, the app defaults to https://thermograph.org, so dev's IndexNow pings and Discord links claimed to be prod
THERMOGRAPH_COOKIE_SECURE=0	plain HTTP on the LAN
THERMOGRAPH_MAIL_BACKEND=console, THERMOGRAPH_DISCORD_BOT=0	explicit fail-safes. Both are already the code defaults; stating them means dev cannot start mailing or open a Discord gateway by inheriting a value
PORT, THERMOGRAPH_BASE, TIMESCALEDB_TAG, DB_MEMORY, WORKERS	non-secret config that would otherwise have come from common.yaml

Deliberately absent, and why nothing breaks:

• Both S3 keypairs. Without them the lake service answers 503 and history reads fall through to the Open-Meteo archive — an accelerator, never a dependency (docker-compose.yml, backend/data/climate.py).
• VAPID keypair. backend/notifications/push.py resolves env → file → generates once and persists to the appdata volume. Dev gets its own identity for free; prod's private key stays off the box.
• THERMOGRAPH_INDEXNOW_KEY. Same env → file → self-generate ladder (backend/indexnow.py). Prod's key is what authenticates URL submissions for thermograph.org.
• THERMOGRAPH_METRICS_TOKEN. With no token /api/v2/metrics is direct-loopback-only, which is exactly right on a LAN box; dev's Alloy agent ships logs, not metrics, so nothing scrapes it.
• REGISTRY_TOKEN. dev pulls with the persistent docker login credential already in the host's docker config, like the prod/beta CI paths. A vault copy would only duplicate a credential the box already has into a more readable file.
• All Discord and mail credentials. Dev must not be able to act as the real bot or send real mail.
• APP_CPUS, DB_CPUS. docker-compose.dev.yml !resets both — dev runs uncapped, so setting them would be a lie. (DB_MEMORY is live: the db service's mem_limit is not reset.)

If dev ever needs one of these for real, it gets its own — a dev Discord app, a scoped read-only bucket key — never a copy of prod's.

Centralis: its own file, its own renderer

/etc/centralis.env on prod is the control plane's env file — four real credentials (CENTRALIS_AUTH_TOKEN, CENTRALIS_TOKENS, CENTRALIS_FORGEJO_TOKEN, DISCORD_TOKEN) and five non-secret settings. It was hand-edited until 2026-07-24, when a hand-edit wrote CENTRALIS_TOKENS as bare JSON into a file that gets sourced, bash stripped the double quotes, {"emi":"…"} arrived as {emi:…}, and Centralis — correctly failing closed on malformed JSON — dropped to the single shared identity. Nothing logged an error. It looked exactly like the file had never been written.

centralis.prod.yaml + render_centralis_secrets() in ../render-secrets.sh end that, by the only durable route: a human stops writing the file.

Seed / verify / rotate:

# One-time seed from the live box (plaintext never leaves prod):
deploy/secrets/seed-centralis-on-host.sh prod agent@169.58.46.181

# Prove a render matches the live file before cutting over. Renders to a 0700
# scratch dir on the host, sources both, compares — key NAMES and a verdict only:
deploy/secrets/verify-centralis-render.sh prod agent@169.58.46.181

# Thereafter, rotation is edit -> commit -> merge to main -> redeploy Centralis:
sops deploy/secrets/centralis.prod.yaml

Merging to main is what puts the new ciphertext on prod (infra-sync.yml fast-forwards /opt/thermograph); the render happens when Centralis is redeployed. See Centralis's own deploy/README.md §2 and §5.

Why CENTRALIS_* is not just added to prod.yaml

It is the obvious move, and it hands the estate's control-plane bearer token to every application container.

/etc/thermograph.env is loaded into the app stack through env_file: and through . /host/thermograph.env in the stack services' entrypoints. Every key in it is in the environment of every container in the stack. CENTRALIS_TOKENS and CENTRALIS_AUTH_TOKEN authenticate an endpoint that carries sql_query with write:true and run_on_host over production, and CENTRALIS_FORGEJO_TOKEN is repo-scope on the monorepo. Putting them in prod.yaml would mean that a read-anything bug in the web backend, or a leaked container env dump, is a foothold on the control plane. The two files are separate for a reason that predates the vault, and the vault is not a reason to merge them.

The converse — pointing Centralis's deploy at /etc/thermograph.env — has the same problem from the other side, plus 32 unrelated variables in the shell that runs its compose, plus a lifecycle coupling: every app deploy re-renders that file, and Centralis and the app stack are deliberately deployed on different cadences.

Why this file is fully quoted and /etc/thermograph.env is not

Different consumers.

/etc/thermograph.env is read by docker-compose env_file: and by systemd EnvironmentFile=, which parse KEY=value literally. Raw sops --output-type dotenv is the right representation for it, and it must stay that way: Centralis's own secrets_inventory / secrets_render tools compare rendered values against that file textually, so quoting it would report all 32 keys as value-changed.

/etc/centralis.env is read by exactly one thing, and it is a shell:

sudo bash -c 'set -a && . /etc/centralis.env && set +a && docker compose up -d --build'

set -a; . file runs every line through bash's full expansion pipeline. Raw dotenv output is unsafe there and sops cannot make it safe — it emits values verbatim. So render_centralis_secrets decrypts to JSON (lossless; dotenv escapes a newline to a literal \n and cannot be told apart from a literal backslash-n) and writes POSIX single-quoted assignments, in which every byte is literal except ' itself.

Then it proves the file before letting it near /etc: it sources the rendered file in a clean shell (env -i), reads every variable back, and compares to the plaintext it started from. If any value does not survive, or if CENTRALIS_TOKENS does not parse as a JSON object of subject → token, the render aborts and the existing file is left untouched. The 2026-07-24 bug cannot be written by this path, and cannot be written through it either.

Scoped by environment even though Centralis is prod-only

The file is centralis.prod.yaml, not centralis.yaml, and the renderer takes the prod from /etc/thermograph/secrets-env like everything else here. An unscoped file would render prod's bearer tokens onto any host that ran the function. If Centralis ever gains a second home, that is a new file rather than a redesign — and the two boxes get different tokens, which is the point.

centralis.prod.yaml is not layered under common.yaml. Centralis shares no key with the app stack (verified: zero overlap between the two live env files), so layering would only push the VAPID private key, both S3 keypairs and REGISTRY_TOKEN into a file that wants none of them.

The age key (the one thing that matters)

• The private key is the single recovery root. Lose it and every secret here is unrecoverable; leak it and every secret is compromised.
• It is never in the repo. It lives at:
  • ~/.config/sops/age/keys.txt on the operator's machine (for editing), and
  • /etc/thermograph/age.key (0400) on each VPS that renders at deploy time.
• Back it up in the password manager. The public recipient is in .sops.yaml.

On the dev desktop those two are the same file, on purpose. render-secrets.sh falls back to sudo cat for a root-owned key, the desktop has no passwordless sudo, and the Forgejo runner is a systemd --user service with no tty — a /etc copy would make every CI-triggered dev deploy hang on a prompt it cannot answer. deploy-dev.sh points THERMOGRAPH_AGE_KEY at the operator's keyring instead, so the box keeps one copy of the recovery root rather than two.

Prerequisites (once per machine)

Install sops + age (single static binaries). On a host, use deploy/provision-secrets.sh; on your laptop:

# see the pinned URLs in provision-secrets.sh, or:
go install github.com/getsops/sops/v3/cmd/sops@latest   # if you have Go

Everyday: rotate a key

sops deploy/secrets/common.yaml        # opens DECRYPTED in $EDITOR; re-encrypts on save
git commit -am "rotate <thing>" && git push forgejo
# beta auto-deploys on push to main. prod (no CI) — one command:
ssh agent@169.58.46.181 'cd /opt/thermograph && git pull && deploy/deploy.sh'

A host-specific value (e.g. POSTGRES_PASSWORD) is the same, editing prod.yaml / beta.yaml / dev.yaml instead. POSTGRES_PASSWORD is special: the database only reads it on a fresh volume, so also ALTER ROLE thermograph PASSWORD '…' inside the running Postgres to match — the env change alone won't re-key an initialized DB. (That applies to dev too. dev.yaml deliberately carries the value dev's existing pgdata volume was initialized with, so nothing rotates the moment dev is provisioned; rotating it later is the two-step above, plus the matching default in deploy-dev.sh.)

Dev is edited and deployed exactly like the others — sops deploy/secrets/dev.yaml, commit, and the next deploy-dev.sh run (merge to dev, or by hand) renders it.

Setting up a new dev machine

The desktop is hands-on: Centralis cannot SSH to it, so this is typed, not automated. provision-secrets.sh is not the right tool here — it installs the age key root-owned at /etc/thermograph/age.key, which the runner cannot read (above).

# 1. sops + age on PATH, and the age private key in the operator's keyring:
#      ~/.config/sops/age/keys.txt   (0600, restored from the password manager)
sops --version && age --version

# 2. Name the environment. This marker is what switches rendering ON — until it
#    exists the box renders nothing, which is the safe default.
sudo mkdir -p /etc/thermograph
printf 'dev\n' | sudo tee /etc/thermograph/secrets-env >/dev/null
sudo chmod 0644 /etc/thermograph/secrets-env

# 3. Pre-create the render target owned by the deploying user, so no deploy ever
#    needs sudo (the renderer writes in place when the file is writable — the same
#    path beta's non-root `deploy` user takes).
sudo install -m 0600 -o "$USER" -g "$USER" /dev/null /etc/thermograph.env

# 4. Dry-run the render before letting a deploy do it, and read the key names back.
#    Nothing from common.yaml may appear here.
THERMOGRAPH_SECRETS_SKIP_COMMON=1 \
SOPS_AGE_KEY_FILE=~/.config/sops/age/keys.txt \
  sops -d --input-type yaml --output-type dotenv deploy/secrets/dev.yaml | sed 's/=.*/=/'

# 5. Deploy. deploy-dev.sh refuses to run if render-secrets.sh in the checkout
#    cannot honour THERMOGRAPH_SECRETS_SKIP_COMMON, rather than leak.
APP_DIR=~/thermograph-dev bash deploy/deploy-dev.sh

To take a dev box back off the vault, delete /etc/thermograph/secrets-env: presence detection turns rendering off and deploy-dev.sh falls back to its built-in POSTGRES_PASSWORD default.

A different dev machine (a second desktop, a laptop) needs its own values, not a copy of this one's: give it its own <env>.yaml and its own marker name, so the two can never be confused for one host — the same argument as the three held-back credentials above.

Add a new secret

1. sops deploy/secrets/common.yaml, add THERMOGRAPH_NEW_THING: value.
2. Add the app reader (os.environ.get("THERMOGRAPH_NEW_THING")).
3. Commit + deploy. It flows into /etc/thermograph.env automatically.

common.yaml reaches prod and beta only. If dev needs the new thing, decide which it is and act on it: harmless config → copy it into dev.yaml; a real credential → mint dev its own and put that in dev.yaml; neither → leave dev without it and make sure the code degrades rather than crashes. Do not reach for prod's copy.

Rotate the age identity itself (re-key)

age-keygen -o new.key                              # new identity
# add the new PUBLIC recipient alongside the old in .sops.yaml, then:
sops updatekeys deploy/secrets/*.yaml              # re-encrypt to BOTH keys
# distribute new.key to /etc/thermograph/age.key on every host + ~/.config/sops/age,
# deploy once so every host can decrypt with the new key, then remove the OLD
# recipient from .sops.yaml and:
sops updatekeys deploy/secrets/*.yaml              # drop the old key
git commit -am "rotate age identity" && push + deploy

First-time bootstrap (seed from the live boxes)

Order matters — values must match the live env exactly so AUTH_SECRET / VAPID / POSTGRES_PASSWORD don't rotate unintentionally:

1. Install sops+age and drop /etc/thermograph/age.key (0400) + /etc/thermograph/secrets-env on prod & beta (provision-secrets.sh).
2. Seed each host's file from its live env with the helper (pulls over SSH, encrypts, and verifies the render round-trips — run it on your own machine, it reads live secrets):

   deploy/secrets/seed-from-live.sh prod agent@169.58.46.181
   deploy/secrets/seed-from-live.sh beta agent@75.119.132.91
   

   That writes exact per-host copies (prod.yaml/beta.yaml); factor shared values into common.yaml later. Commit.
3. Dry-run: on the box, render to a temp file and diff against the live /etc/thermograph.env — must be byte-identical before cutover.
4. Merge the deploy.sh wiring; run one deploy; confirm the app is healthy.

Safety

• CI (.forgejo/workflows/secrets-guard.yml) fails if any *.yaml here is committed unencrypted.
• Rendering is presence-detected: a host without the age key + marker keeps its existing /etc/thermograph.env, so nothing breaks before a host is migrated.
• Rendering happens on the target box, so CI/runners never see the age key or the plaintext — except on dev, where the box is the runner. That is not a hole this vault can close; it is the reason dev.yaml holds nothing that would matter if the box were owned. Keep it that way: the day dev needs a real credential, mint it a scoped one of its own.
• deploy-dev.sh fails closed: if the render-secrets.sh it finds cannot honour THERMOGRAPH_SECRETS_SKIP_COMMON, it aborts the deploy rather than render the shared production layer onto the dev box.