2026-07-22 03:21:46 +00:00
# Secrets — the git-native vault (SOPS + age)
The **single source of truth** for Thermograph's secrets is the set of
SOPS-encrypted YAML files in this directory. They're committed to the repo
**encrypted** (values only — keys stay readable so diffs are meaningful) and
**rendered into `/etc/thermograph.env` at deploy time** by
[`deploy/render-secrets.sh` ](../render-secrets.sh ), which the prod/beta
`deploy.sh` invokes.
Cycling a key is: **edit → commit → deploy** . No SSHing in to hand-edit a root file,
no per-host duplication.
## Files
| File | Holds | Encrypted? |
|------|-------|------------|
secrets: factor shared values into common.yaml; stop the renderer failing open
Two changes to how credentials are stored and distributed.
1. common.yaml now exists.
The renderer has always concatenated common.yaml then <env>.yaml, host winning,
and the README has always documented common.yaml with a checkmark. The file was
never created. So every value shared by prod and beta was duplicated in both
vaults, free to drift apart with nothing to detect it.
16 values move to common.yaml: VAPID keypair, metrics token, IndexNow key,
REGISTRY_TOKEN, the S3 endpoint/bucket and both S3 keypairs, plus shared config.
5 stay per-host because their values genuinely differ (APP_CPUS, DB_CPUS,
DB_MEMORY, WORKERS, THERMOGRAPH_BASE_URL). 8 exist only on prod — the Discord
and mail credentials, which beta does not have at all.
Three more are held back deliberately despite being identical today:
POSTGRES_PASSWORD, THERMOGRAPH_AUTH_SECRET, THERMOGRAPH_DATABASE_URL. These are
the credentials that let one environment act as another, and beta is the more
exposed box — it serves public Forgejo and Grafana. They match only because beta
was seeded from prod. Keeping them per-host costs one line each and preserves
the ability to diverge; putting them in common.yaml would encode the equivalence
as intentional and make breaking it a migration rather than an edit. Reasoning
recorded in the vault README, since a future reader will otherwise "fix" it.
Done without any plaintext leaving prod: ciphertext was shipped up, decrypted
against the host's age key, recombined, re-encrypted, and shipped back. The
consolidation refuses to write unless it has proved merged(common + env) equals
the original env vault exactly — same keys, same values — and re-verifies from
the written files afterwards. Both checks passed for prod and beta.
2. render_thermograph_secrets no longer fails open.
One `return 0` covered two different situations: "this host is not configured
for SOPS" (true of the LAN dev box, and correct to succeed) and "this host IS
configured but its vault is not where we looked". The second is a failure, and
returning 0 made it a silent no-op wearing a success code — a deploy would
report success having rendered nothing, and the host would keep serving whatever
/etc/thermograph.env already held, including after a rotation.
The likely cause is passing the wrong root: the function wants the directory
containing deploy/secrets, which on the hosts is /opt/thermograph/infra, not
/opt/thermograph. That mistake looked exactly like "not configured here", which
is why it went unnoticed. It now exits 1 and names the probable cause.
Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY
2026-07-25 00:40:26 +00:00
| `common.yaml` | The 16 values identical on every host — VAPID keypair, metrics token, IndexNow key, `REGISTRY_TOKEN` , S3 endpoint/bucket and both S3 keypairs, plus shared non-secret config | ✅ |
| `prod.yaml` | Prod's own — the three held-back credentials below, sizing (`APP_CPUS`, `DB_CPUS` , `DB_MEMORY` , `WORKERS` ), `THERMOGRAPH_BASE_URL` , and the Discord + mail credentials that exist nowhere else | ✅ |
| `beta.yaml` | Beta's own — the three held-back credentials, sizing, `THERMOGRAPH_BASE_URL` | ✅ |
2026-07-22 03:21:46 +00:00
| `example.yaml` | Format reference / CI fixture (fake values) | ✅ |
| `../../.sops.yaml` | Which age recipient files are encrypted to (plaintext config) | — |
secrets: factor shared values into common.yaml; stop the renderer failing open
Two changes to how credentials are stored and distributed.
1. common.yaml now exists.
The renderer has always concatenated common.yaml then <env>.yaml, host winning,
and the README has always documented common.yaml with a checkmark. The file was
never created. So every value shared by prod and beta was duplicated in both
vaults, free to drift apart with nothing to detect it.
16 values move to common.yaml: VAPID keypair, metrics token, IndexNow key,
REGISTRY_TOKEN, the S3 endpoint/bucket and both S3 keypairs, plus shared config.
5 stay per-host because their values genuinely differ (APP_CPUS, DB_CPUS,
DB_MEMORY, WORKERS, THERMOGRAPH_BASE_URL). 8 exist only on prod — the Discord
and mail credentials, which beta does not have at all.
Three more are held back deliberately despite being identical today:
POSTGRES_PASSWORD, THERMOGRAPH_AUTH_SECRET, THERMOGRAPH_DATABASE_URL. These are
the credentials that let one environment act as another, and beta is the more
exposed box — it serves public Forgejo and Grafana. They match only because beta
was seeded from prod. Keeping them per-host costs one line each and preserves
the ability to diverge; putting them in common.yaml would encode the equivalence
as intentional and make breaking it a migration rather than an edit. Reasoning
recorded in the vault README, since a future reader will otherwise "fix" it.
Done without any plaintext leaving prod: ciphertext was shipped up, decrypted
against the host's age key, recombined, re-encrypted, and shipped back. The
consolidation refuses to write unless it has proved merged(common + env) equals
the original env vault exactly — same keys, same values — and re-verifies from
the written files afterwards. Both checks passed for prod and beta.
2. render_thermograph_secrets no longer fails open.
One `return 0` covered two different situations: "this host is not configured
for SOPS" (true of the LAN dev box, and correct to succeed) and "this host IS
configured but its vault is not where we looked". The second is a failure, and
returning 0 made it a silent no-op wearing a success code — a deploy would
report success having rendered nothing, and the host would keep serving whatever
/etc/thermograph.env already held, including after a rotation.
The likely cause is passing the wrong root: the function wants the directory
containing deploy/secrets, which on the hosts is /opt/thermograph/infra, not
/opt/thermograph. That mistake looked exactly like "not configured here", which
is why it went unnoticed. It now exits 1 and names the probable cause.
Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY
2026-07-25 00:40:26 +00:00
### Three credentials are deliberately NOT in `common.yaml`
`POSTGRES_PASSWORD` , `THERMOGRAPH_AUTH_SECRET` and `THERMOGRAPH_DATABASE_URL`
hold **identical values on prod and beta today** , so by the mechanical rule they
belong in `common.yaml` . They are kept per-host anyway.
These are the credentials that let one environment act as another: with them, a
foothold on beta is a foothold on prod's database and prod's session signing.
Beta is the *more* exposed box — it serves public Forgejo and Grafana. They match
only because beta was seeded from prod, not because the two are meant to be one
system.
Keeping them per-host costs nothing now (same values, one extra line each) and
buys the ability to diverge: rotating prod's database password stops implying
"and beta's too". Moving them into `common.yaml` would encode the equivalence as
intentional and turn breaking it into a migration rather than an edit.
If you ever *do* want them to differ, change one file. That is the whole point.
2026-07-22 03:21:46 +00:00
At deploy the renderer concatenates `common.yaml` then `<env>.yaml` , so a **host
value wins** (env_file / `source` take the last occurrence of a duplicate key).
`<env>` comes from `/etc/thermograph/secrets-env` on the box (`prod`/`beta`/`dev`).
`dev` has no real secrets and is intentionally not wired in — see `deploy-dev.sh` .
## The age key (the one thing that matters)
- The **private key is the single recovery root.** Lose it and every secret here is
unrecoverable; leak it and every secret is compromised.
- It is **never in the repo.** It lives at:
- `~/.config/sops/age/keys.txt` on the operator's machine (for editing), and
- `/etc/thermograph/age.key` (`0400`) on each host that renders at deploy time.
- **Back it up** in the password manager. The public recipient is in `.sops.yaml` .
## Prerequisites (once per machine)
Install `sops` + `age` (single static binaries). On a host, use
[`deploy/provision-secrets.sh` ](../provision-secrets.sh ); on your laptop:
```sh
# see the pinned URLs in provision-secrets.sh, or:
go install github.com/getsops/sops/v3/cmd/sops@latest # if you have Go
```
## Everyday: rotate a key
```sh
sops deploy/secrets/common.yaml # opens DECRYPTED in $EDITOR; re-encrypts on save
git commit -am "rotate < thing > " & & git push forgejo
# beta auto-deploys on push to main. prod (no CI) — one command:
ssh agent@169.58.46.181 'cd /opt/thermograph & & git pull & & deploy/deploy.sh'
```
A host-specific value (e.g. `POSTGRES_PASSWORD` ) is the same, editing `prod.yaml` /
`beta.yaml` instead. ** `POSTGRES_PASSWORD` is special**: the database only reads it on
a *fresh* volume, so also `ALTER ROLE thermograph PASSWORD '…'` inside the running
Postgres to match — the env change alone won't re-key an initialized DB.
## Add a new secret
1. `sops deploy/secrets/common.yaml` , add `THERMOGRAPH_NEW_THING: value` .
2. Add the app reader (`os.environ.get("THERMOGRAPH_NEW_THING")`).
3. Commit + deploy. It flows into `/etc/thermograph.env` automatically.
## Rotate the age identity itself (re-key)
```sh
age-keygen -o new.key # new identity
# add the new PUBLIC recipient alongside the old in .sops.yaml, then:
sops updatekeys deploy/secrets/*.yaml # re-encrypt to BOTH keys
# distribute new.key to /etc/thermograph/age.key on every host + ~/.config/sops/age,
# deploy once so every host can decrypt with the new key, then remove the OLD
# recipient from .sops.yaml and:
sops updatekeys deploy/secrets/*.yaml # drop the old key
git commit -am "rotate age identity" & & push + deploy
```
## First-time bootstrap (seed from the live boxes)
Order matters — values must match the live env exactly so `AUTH_SECRET` / VAPID /
`POSTGRES_PASSWORD` don't rotate unintentionally:
1. Install `sops` +`age` and drop `/etc/thermograph/age.key` (0400) +
`/etc/thermograph/secrets-env` on prod & beta (`provision-secrets.sh`).
2. Seed each host's file from its live env with the helper (pulls over SSH, encrypts,
and verifies the render round-trips — run it on your own machine, it reads live
secrets):
```sh
deploy/secrets/seed-from-live.sh prod agent@169.58.46.181
deploy/secrets/seed-from-live.sh beta agent@75.119.132.91
```
That writes exact per-host copies (`prod.yaml`/`beta.yaml`); factor shared values
into `common.yaml` later. Commit.
3. **Dry-run:** on the box, render to a temp file and `diff` against the live
`/etc/thermograph.env` — must be byte-identical before cutover.
4. Merge the `deploy.sh` wiring; run one deploy; confirm the app is healthy.
## Safety
- CI (`.forgejo/workflows/secrets-guard.yml`) fails if any `*.yaml` here is committed
unencrypted.
- Rendering is **presence-detected** : a host without the age key + marker keeps its
existing `/etc/thermograph.env` , so nothing breaks before a host is migrated.
- Rendering happens **on the target box** , so CI/runners never see the age key or the
plaintext.
