thermograph/infra/deploy/secrets/README.md

Secrets — the git-native vault (SOPS + age)

The single source of truth for Thermograph's secrets is the set of SOPS-encrypted YAML files in this directory. They're committed to the repo encrypted (values only — keys stay readable so diffs are meaningful) and rendered into /etc/thermograph.env at deploy time by deploy/render-secrets.sh, which the prod/beta deploy.sh invokes.

Cycling a key is: edit → commit → deploy. No SSHing in to hand-edit a root file, no per-host duplication.

Files

File	Holds	Encrypted?
common.yaml	The 16 values identical on every host — VAPID keypair, metrics token, IndexNow key, REGISTRY_TOKEN, S3 endpoint/bucket and both S3 keypairs, plus shared non-secret config	✅
prod.yaml	Prod's own — the three held-back credentials below, sizing (APP_CPUS, DB_CPUS, DB_MEMORY, WORKERS), THERMOGRAPH_BASE_URL, and the Discord + mail credentials that exist nowhere else	✅
beta.yaml	Beta's own — the three held-back credentials, sizing, THERMOGRAPH_BASE_URL	✅
example.yaml	Format reference / CI fixture (fake values)	✅
../../.sops.yaml	Which age recipient files are encrypted to (plaintext config)	—

Three credentials are deliberately NOT in common.yaml

POSTGRES_PASSWORD, THERMOGRAPH_AUTH_SECRET and THERMOGRAPH_DATABASE_URL hold identical values on prod and beta today, so by the mechanical rule they belong in common.yaml. They are kept per-host anyway.

These are the credentials that let one environment act as another: with them, a foothold on beta is a foothold on prod's database and prod's session signing. Beta is the more exposed box — it serves public Forgejo and Grafana. They match only because beta was seeded from prod, not because the two are meant to be one system.

Keeping them per-host costs nothing now (same values, one extra line each) and buys the ability to diverge: rotating prod's database password stops implying "and beta's too". Moving them into common.yaml would encode the equivalence as intentional and turn breaking it into a migration rather than an edit.

If you ever do want them to differ, change one file. That is the whole point.

At deploy the renderer concatenates common.yaml then <env>.yaml, so a host value wins (env_file / source take the last occurrence of a duplicate key). <env> comes from /etc/thermograph/secrets-env on the box (prod/beta/dev). dev has no real secrets and is intentionally not wired in — see deploy-dev.sh.

The age key (the one thing that matters)

• The private key is the single recovery root. Lose it and every secret here is unrecoverable; leak it and every secret is compromised.
• It is never in the repo. It lives at:
  • ~/.config/sops/age/keys.txt on the operator's machine (for editing), and
  • /etc/thermograph/age.key (0400) on each host that renders at deploy time.
• Back it up in the password manager. The public recipient is in .sops.yaml.

Prerequisites (once per machine)

Install sops + age (single static binaries). On a host, use deploy/provision-secrets.sh; on your laptop:

# see the pinned URLs in provision-secrets.sh, or:
go install github.com/getsops/sops/v3/cmd/sops@latest   # if you have Go

Everyday: rotate a key

sops deploy/secrets/common.yaml        # opens DECRYPTED in $EDITOR; re-encrypts on save
git commit -am "rotate <thing>" && git push forgejo
# beta auto-deploys on push to main. prod (no CI) — one command:
ssh agent@169.58.46.181 'cd /opt/thermograph && git pull && deploy/deploy.sh'

A host-specific value (e.g. POSTGRES_PASSWORD) is the same, editing prod.yaml / beta.yaml instead. POSTGRES_PASSWORD is special: the database only reads it on a fresh volume, so also ALTER ROLE thermograph PASSWORD '…' inside the running Postgres to match — the env change alone won't re-key an initialized DB.

Add a new secret

1. sops deploy/secrets/common.yaml, add THERMOGRAPH_NEW_THING: value.
2. Add the app reader (os.environ.get("THERMOGRAPH_NEW_THING")).
3. Commit + deploy. It flows into /etc/thermograph.env automatically.

Rotate the age identity itself (re-key)

age-keygen -o new.key                              # new identity
# add the new PUBLIC recipient alongside the old in .sops.yaml, then:
sops updatekeys deploy/secrets/*.yaml              # re-encrypt to BOTH keys
# distribute new.key to /etc/thermograph/age.key on every host + ~/.config/sops/age,
# deploy once so every host can decrypt with the new key, then remove the OLD
# recipient from .sops.yaml and:
sops updatekeys deploy/secrets/*.yaml              # drop the old key
git commit -am "rotate age identity" && push + deploy

First-time bootstrap (seed from the live boxes)

Order matters — values must match the live env exactly so AUTH_SECRET / VAPID / POSTGRES_PASSWORD don't rotate unintentionally:

1. Install sops+age and drop /etc/thermograph/age.key (0400) + /etc/thermograph/secrets-env on prod & beta (provision-secrets.sh).
2. Seed each host's file from its live env with the helper (pulls over SSH, encrypts, and verifies the render round-trips — run it on your own machine, it reads live secrets):

   deploy/secrets/seed-from-live.sh prod agent@169.58.46.181
   deploy/secrets/seed-from-live.sh beta agent@75.119.132.91
   

   That writes exact per-host copies (prod.yaml/beta.yaml); factor shared values into common.yaml later. Commit.
3. Dry-run: on the box, render to a temp file and diff against the live /etc/thermograph.env — must be byte-identical before cutover.
4. Merge the deploy.sh wiring; run one deploy; confirm the app is healthy.

Safety

• CI (.forgejo/workflows/secrets-guard.yml) fails if any *.yaml here is committed unencrypted.
• Rendering is presence-detected: a host without the age key + marker keeps its existing /etc/thermograph.env, so nothing breaks before a host is migrated.
• Rendering happens on the target box, so CI/runners never see the age key or the plaintext.