thermograph/infra/deploy/secrets/README.md

# Secrets — the git-native vault (SOPS + age)

The **single source of truth** for Thermograph's secrets is the set of
SOPS-encrypted YAML files in this directory. They're committed to the repo
**encrypted** (values only — keys stay readable so diffs are meaningful) and
**rendered into `/etc/thermograph.env` at deploy time** by
[`deploy/render-secrets.sh`](../render-secrets.sh), which the prod/beta
`deploy.sh` invokes.

Cycling a key is: **edit → commit → deploy**. No SSHing in to hand-edit a root file,
no per-host duplication.

## Files

| File | Holds | Encrypted? |
|------|-------|------------|
| `common.yaml` | The 16 values identical on **prod and beta** — VAPID keypair, metrics token, IndexNow key, `REGISTRY_TOKEN`, S3 endpoint/bucket and both S3 keypairs, plus shared non-secret config. **Not layered under `dev`** — see below | ✅ |
| `prod.yaml` | Prod's own — the three held-back credentials below, sizing (`APP_CPUS`, `DB_CPUS`, `DB_MEMORY`, `WORKERS`), `THERMOGRAPH_BASE_URL`, and the Discord + mail credentials that exist nowhere else | ✅ |
| `beta.yaml` | Beta's own — the three held-back credentials, `THERMOGRAPH_BASE_URL`, `APP_CPUS`/`WORKERS`, plus `THERMOGRAPH_MAIL_BACKEND=console` and `THERMOGRAPH_DISCORD_BOT=0`. Its `POSTGRES_PASSWORD` and `THERMOGRAPH_DATABASE_URL` name beta's **own role and database** (`thermograph_beta`) on the instance it shares with prod. `DB_CPUS`/`DB_MEMORY` were **removed**: beta no longer runs a database of its own, and leaving them would have implied it sized one — the shared instance is sized by `prod.yaml`'s values (see `deploy/stack/thermograph-stack.yml`'s `db` service) | ✅ |
| `dev.yaml` | Dev's own (vps1) — **self-contained**, 12 values, no production credential among them | ✅ |
| `centralis.prod.yaml` | **Centralis's** nine variables, rendered to `/etc/centralis.env` on prod. A different service, a different output file, a different renderer — see below | ✅ |
| `example.yaml` | Format reference / CI fixture (fake values) | ✅ |
| `../../.sops.yaml` | Which age recipient files are encrypted to (plaintext config) | — |

### Three credentials are deliberately NOT in `common.yaml`

`POSTGRES_PASSWORD`, `THERMOGRAPH_AUTH_SECRET` and `THERMOGRAPH_DATABASE_URL`
hold **different values on prod and beta today** (beta has its own database
role/password on the shared instance, and its own auth secret), so unlike the
rest of `common.yaml` they were never candidates for sharing on values alone.
They stay per-environment for a reason that matters more now than it used to:

**These are the credentials that let one environment act as another.** With
matching values, a foothold in one environment's rendered env file is a
foothold on the other's database and the other's session signing. That used
to matter because beta and prod were separate boxes; it matters for a
different reason now that they're co-resident on vps2 — keeping the
credentials themselves separate is what stops co-location on one host from
also becoming equivalence at the credential level. `deploy/db/provision-env-db.sh`
enforces the same boundary from the database side: beta's role
(`thermograph_beta`) is `NOSUPERUSER`/`NOCREATEDB`/`NOCREATEROLE`, owns only
its own database, and `CONNECT` is revoked from `PUBLIC` on it — so even a
leaked beta credential cannot reach prod's data, and the reverse.

**What actually separates beta and prod now, stated plainly:** they share a
host by design (see `deploy/env-topology.sh`'s header), so a host-level
compromise of vps2 is shared between them — separate SSH credentials no
longer buy a host boundary between beta and prod, the way they once did when
beta and prod were different machines. What remains is isolation at the
**database** level (separate roles, separate databases, `CONNECT` revoked from
`PUBLIC`) and the **file** level (separate checkouts, separate rendered env
files, separate stack env files, separate loopback ports). Keeping
`POSTGRES_PASSWORD`/`THERMOGRAPH_AUTH_SECRET`/`THERMOGRAPH_DATABASE_URL` out of
a shared file is part of that file-level isolation: nothing on vps2 would ever
hand you prod's auth secret merely because you have beta's.

**vps1 is the box that must never hold prod credentials.** It runs Forgejo,
Forgejo's CI runner, Grafana, and the `dev` environment — which runs whatever
branch is currently in flight, reviewed or not. That is the actual reason
`dev` renders `dev.yaml` **alone** and never layers `common.yaml` (see below):
not squeamishness about a dev box, but that vps1 is uniquely positioned to
leak whatever it's handed, and `common.yaml` is the fleet's shared production
credential set.

Keeping the three held-back credentials per-environment costs nothing (one
line each) and buys the ability for them to diverge further, or for a future
environment to be added without inheriting anyone else's values by accident.

`dev.yaml` is the case that already differs completely: all three of its
values are its own, generated on vps1 and never shared with prod or beta.

At deploy the renderer concatenates `common.yaml` then `<env>.yaml`, so a **host
value wins** (env_file / `source` take the last occurrence of a duplicate key).
`<env>` comes from an explicit `THERMOGRAPH_ENV`/argument on vps2 (which runs
two environments) or `/etc/thermograph/secrets-env` as a fallback elsewhere —
see `deploy/env-topology.sh`.

### `dev` is vaulted, but renders `dev.yaml` **alone**

All three environments are managed here now. `dev` is the one that does **not**
get `common.yaml` layered under it: `deploy-dev.sh` exports
`THERMOGRAPH_SECRETS_SKIP_COMMON=1`, and the renderer skips the shared layer.

That is not squeamishness about a dev box — it is what `common.yaml` contains.
Eleven of its sixteen values are live production credentials: `REGISTRY_TOKEN`,
`THERMOGRAPH_S3_ACCESS_KEY`/`_SECRET_KEY` and
`THERMOGRAPH_LAKE_S3_ACCESS_KEY`/`_SECRET_KEY` (one keypair, **read-write**, on the
bucket that also holds prod's database backups — see `../../ops/README.md`),
`THERMOGRAPH_VAPID_PRIVATE_KEY`/`_PUBLIC_KEY` (they sign Web Push to real
subscribers), `THERMOGRAPH_INDEXNOW_KEY`, `THERMOGRAPH_METRICS_TOKEN`. The render
is a **plaintext concatenation**, so layering it would put every one of those in
`/etc/thermograph.env` on the dev box and, through `env_file:`, into the
environment of every container in the dev stack.

The dev box is **vps1** — the same box that runs Forgejo and its CI. Its
`docker`-labelled runner jobs get the host docker socket automounted, and the
stack it runs is the `dev` branch — code that has not been through the gate
that guards prod. Handing that the estate's read-write object-storage keys and
the push signing key is a strictly larger blast radius than anything the vault
buys back. An override in `dev.yaml` would not help: last-wins governs
*consumers*, but the production value is still physically a line in the file.

One value in `common.yaml` is also simply **wrong** for dev:
`THERMOGRAPH_COOKIE_SECURE=1` is right for the TLS hosts and silently breaks login
over dev's plain-HTTP mesh-only URL. `dev.yaml` sets `0`.

So `dev.yaml` is self-contained. Where the value is shared-but-harmless
(`PORT`, `THERMOGRAPH_BASE`, `TIMESCALEDB_TAG`) it carries its own copy — a
duplicated constant is the cheap half of this trade.

**What `dev.yaml` holds** (12 keys):

| Key | Why |
|-----|-----|
| `POSTGRES_PASSWORD`, `THERMOGRAPH_DATABASE_URL` | dev's own. Deliberately the weak, well-known `thermograph-dev` — the value dev's `pgdata` volume is already initialized with, and the DB is never published off the compose network. Kept per-host for the reason below, and *not* shared with prod |
| `THERMOGRAPH_AUTH_SECRET` | dev's **own**, freshly generated. Fixes a live crash loop: the `daemon` service refuses to start without it (it derives the `/internal/*` token from it), so dev's daemon had been restarting every 60s. Prod's value must never be here — it signs sessions and verification links |
| `THERMOGRAPH_BASE_URL` | `http://10.10.0.2:8137` (vps1's mesh address). Unset, the app defaults to `https://thermograph.org`, so dev's IndexNow pings and Discord links claimed to be prod |
| `THERMOGRAPH_COOKIE_SECURE=0` | plain HTTP on the mesh-only URL |
| `THERMOGRAPH_MAIL_BACKEND=console`, `THERMOGRAPH_DISCORD_BOT=0` | explicit fail-safes. Both are already the code defaults; stating them means dev cannot start mailing or open a Discord gateway by inheriting a value |
| `PORT`, `THERMOGRAPH_BASE`, `TIMESCALEDB_TAG`, `DB_MEMORY`, `WORKERS` | non-secret config that would otherwise have come from `common.yaml` |

**Deliberately absent, and why nothing breaks:**

- **Both S3 keypairs.** Without them the `lake` service answers 503 and history
  reads fall through to the Open-Meteo archive — an accelerator, never a
  dependency (`docker-compose.yml`, `backend/data/climate.py`).
- **VAPID keypair.** `backend/notifications/push.py` resolves env → file →
  generates once and persists to the `appdata` volume. Dev gets its own identity
  for free; prod's private key stays off the box.
- **`THERMOGRAPH_INDEXNOW_KEY`.** Same env → file → self-generate ladder
  (`backend/indexnow.py`). Prod's key is what authenticates URL submissions *for
  thermograph.org*.
- **`THERMOGRAPH_METRICS_TOKEN`.** With no token `/api/v2/metrics` is
  direct-loopback-only, which is exactly right on a mesh-only box; dev's Alloy
  agent ships logs, not metrics, so nothing scrapes it.
- **`REGISTRY_TOKEN`.** dev pulls with the persistent `docker login` credential
  already in the host's docker config, like the prod/beta CI paths. A vault copy
  would only duplicate a credential the box already has into a more readable file.
- **All Discord and mail credentials.** Dev must not be able to act as the real
  bot or send real mail.
- **`APP_CPUS`, `DB_CPUS`.** `docker-compose.dev.yml` `!reset`s both — dev runs
  uncapped, so setting them would be a lie. (`DB_MEMORY` *is* live: the `db`
  service's `mem_limit` is not reset.)

If dev ever needs one of these for real, it gets **its own** — a dev Discord app,
a scoped read-only bucket key — never a copy of prod's.

## Centralis: its own file, its own renderer

`/etc/centralis.env` on prod is the control plane's env file — four real
credentials (`CENTRALIS_AUTH_TOKEN`, `CENTRALIS_TOKENS`,
`CENTRALIS_FORGEJO_TOKEN`, `DISCORD_TOKEN`) and five non-secret settings. It was
**hand-edited** until 2026-07-24, when a hand-edit wrote `CENTRALIS_TOKENS` as
bare JSON into a file that gets `source`d, bash stripped the double quotes,
`{"emi":"…"}` arrived as `{emi:…}`, and Centralis — correctly failing closed on
malformed JSON — dropped to the single `shared` identity. Nothing logged an
error. It looked exactly like the file had never been written.

`centralis.prod.yaml` + `render_centralis_secrets()` in
[`../render-secrets.sh`](../render-secrets.sh) end that, by the only durable
route: **a human stops writing the file.**

Seed / verify / rotate:

```sh
# One-time seed from the live box (plaintext never leaves prod):
deploy/secrets/seed-centralis-on-host.sh prod agent@169.58.46.181

# Prove a render matches the live file before cutting over. Renders to a 0700
# scratch dir on the host, sources both, compares — key NAMES and a verdict only:
deploy/secrets/verify-centralis-render.sh prod agent@169.58.46.181

# Thereafter, rotation is edit -> commit -> merge to main -> redeploy Centralis:
sops deploy/secrets/centralis.prod.yaml
```

Merging to `main` is what puts the new ciphertext on prod (`infra-sync.yml`
fast-forwards `/opt/thermograph`); the render happens when Centralis is
redeployed. See Centralis's own `deploy/README.md` §2 and §5.

### Why `CENTRALIS_*` is not just added to `prod.yaml`

It is the obvious move, and it hands the estate's control-plane bearer token to
every application container.

`/etc/thermograph.env` is loaded into the app stack through `env_file:` and
through `. /host/thermograph.env` in the stack services' entrypoints. Every key
in it is in the environment of every container in the stack. `CENTRALIS_TOKENS`
and `CENTRALIS_AUTH_TOKEN` authenticate an endpoint that carries `sql_query`
with `write:true` and `run_on_host` over production, and `CENTRALIS_FORGEJO_TOKEN`
is repo-scope on the monorepo. Putting them in `prod.yaml` would mean that a
read-anything bug in the web backend, or a leaked container env dump, is a
foothold on the control plane. The two files are separate for a reason that
predates the vault, and the vault is not a reason to merge them.

The converse — pointing Centralis's deploy at `/etc/thermograph.env` — has the
same problem from the other side, plus 32 unrelated variables in the shell that
runs its compose, plus a lifecycle coupling: every app deploy re-renders that
file, and Centralis and the app stack are deliberately deployed on different
cadences.

### Why this file is fully quoted and `/etc/thermograph.env` is not

Different consumers.

`/etc/thermograph.env` is read by docker-compose `env_file:` and by systemd
`EnvironmentFile=`, which parse `KEY=value` literally. Raw `sops --output-type
dotenv` is the right representation for it, and it must stay that way: Centralis's
own `secrets_inventory` / `secrets_render` tools compare rendered values against
that file **textually**, so quoting it would report all 32 keys as value-changed.

`/etc/centralis.env` is read by exactly one thing, and it is a shell:

```sh
sudo bash -c 'set -a && . /etc/centralis.env && set +a && docker compose up -d --build'
```

`set -a; . file` runs every line through bash's full expansion pipeline. Raw
dotenv output is *unsafe* there and sops cannot make it safe — it emits values
verbatim. So `render_centralis_secrets` decrypts to **JSON** (lossless; dotenv
escapes a newline to a literal `\n` and cannot be told apart from a literal
backslash-n) and writes POSIX single-quoted assignments, in which every byte is
literal except `'` itself.

Then it **proves the file** before letting it near `/etc`: it sources the
rendered file in a clean shell (`env -i`), reads every variable back, and
compares to the plaintext it started from. If any value does not survive, or if
`CENTRALIS_TOKENS` does not parse as a JSON object of subject → token, the
render aborts and the existing file is left untouched. The 2026-07-24 bug cannot
be written by this path, and cannot be written *through* it either.

### Scoped by environment even though Centralis is prod-only

The file is `centralis.prod.yaml`, not `centralis.yaml`, and the renderer takes
the `prod` from `/etc/thermograph/secrets-env` like everything else here. An
unscoped file would render prod's bearer tokens onto any host that ran the
function. If Centralis ever gains a second home, that is a new file rather than
a redesign — and the two boxes get different tokens, which is the point.

`centralis.prod.yaml` is **not** layered under `common.yaml`. Centralis shares
no key with the app stack (verified: zero overlap between the two live env
files), so layering would only push the VAPID private key, both S3 keypairs and
`REGISTRY_TOKEN` into a file that wants none of them.

## The age key (the one thing that matters)

- The **private key is the single recovery root.** Lose it and every secret here is
  unrecoverable; leak it and every secret is compromised.
- It is **never in the repo.** It lives at:
  - `~/.config/sops/age/keys.txt` on the operator's machine (for editing), and
  - `/etc/thermograph/age.key` (`0400`) on each VPS that renders at deploy time.
- **Back it up** in the password manager. The public recipient is in `.sops.yaml`.

On **vps1** (dev) those two can end up being the same file. `render-secrets.sh`
falls back to `sudo cat` for a root-owned key, and `deploy-dev.sh` still assumes
the account driving a CI-triggered dev deploy has no passwordless sudo and no
tty to answer a `sudo` prompt (a `systemd --user` Forgejo-runner service) —
inherited from the old desktop setup, where that was true of the whole box.
Whether it's still true of vps1's CI-runner account specifically (as opposed
to the `agent` login, which does have passwordless sudo there per `ACCESS.md`)
wasn't re-verified for this pass; `deploy-dev.sh` points `THERMOGRAPH_AGE_KEY`
at the operator's keyring as a fallback either way, so the box keeps **one**
copy of the recovery root rather than two regardless of which account ends up
mattering.

## Prerequisites (once per machine)

Install `sops` + `age` (single static binaries). On a host, use
[`deploy/provision-secrets.sh`](../provision-secrets.sh); on your laptop:

```sh
# see the pinned URLs in provision-secrets.sh, or:
go install github.com/getsops/sops/v3/cmd/sops@latest   # if you have Go
```

## Everyday: rotate a key

```sh
sops deploy/secrets/common.yaml        # opens DECRYPTED in $EDITOR; re-encrypts on save
git commit -am "rotate <thing>" && git push forgejo
# beta and prod both auto-deploy on push (main -> beta, release -> prod) via
# the Deploy workflow. To force it by hand on vps2, say which environment:
ssh agent@169.58.46.181 'cd /opt/thermograph && git pull && THERMOGRAPH_ENV=prod deploy/deploy.sh'
ssh agent@169.58.46.181 'cd /opt/thermograph-beta && git pull && THERMOGRAPH_ENV=beta deploy/deploy.sh'
```

A host-specific value (e.g. `POSTGRES_PASSWORD`) is the same, editing `prod.yaml` /
`beta.yaml` / `dev.yaml` instead. **`POSTGRES_PASSWORD` is special**: the database
only reads it on a *fresh* volume, so also `ALTER ROLE thermograph PASSWORD '…'`
inside the running Postgres to match — the env change alone won't re-key an
initialized DB. (That applies to dev too. `dev.yaml` deliberately carries the value
dev's existing `pgdata` volume was initialized with, so nothing rotates the moment
dev is provisioned; rotating it later is the two-step above, plus the matching
default in `deploy-dev.sh`.)

Dev is edited and deployed exactly like the others — `sops deploy/secrets/dev.yaml`,
commit, and the next `deploy-dev.sh` run (merge to `dev`, or by hand) renders it.

## Setting up a new dev machine

Dev is provisioned like any fleet host now (`deploy/provision-dev.sh`, run as
root on vps1), not the sudo-free desktop bootstrap this predates. The
secrets-specific steps below are still worth doing by hand rather than
folding into that script: `provision-secrets.sh` is **not** the right tool
here — it installs the age key root-owned at `/etc/thermograph/age.key`,
which the CI-runner-driven render may not be able to read (see the age-key
note above).

```sh
# 1. sops + age on PATH, and the age private key in the operator's keyring:
#      ~/.config/sops/age/keys.txt   (0600, restored from the password manager)
sops --version && age --version

# 2. Name the environment. This marker is what switches rendering ON — until it
#    exists the box renders nothing, which is the safe default.
sudo mkdir -p /etc/thermograph
printf 'dev\n' | sudo tee /etc/thermograph/secrets-env >/dev/null
sudo chmod 0644 /etc/thermograph/secrets-env

# 3. Pre-create the render target owned by the deploying user, so no deploy ever
#    needs sudo (the renderer writes in place when the file is writable — the same
#    path beta's non-root `deploy` user takes).
sudo install -m 0600 -o "$USER" -g "$USER" /dev/null /etc/thermograph.env

# 4. Dry-run the render before letting a deploy do it, and read the key names back.
#    Nothing from common.yaml may appear here.
THERMOGRAPH_SECRETS_SKIP_COMMON=1 \
SOPS_AGE_KEY_FILE=~/.config/sops/age/keys.txt \
  sops -d --input-type yaml --output-type dotenv deploy/secrets/dev.yaml | sed 's/=.*/=/'

# 5. Deploy. deploy-dev.sh refuses to run if render-secrets.sh in the checkout
#    cannot honour THERMOGRAPH_SECRETS_SKIP_COMMON, rather than leak.
APP_DIR=/opt/thermograph-dev bash deploy/deploy-dev.sh
```

To take a dev box **back** off the vault, delete `/etc/thermograph/secrets-env`:
presence detection turns rendering off and `deploy-dev.sh` falls back to its
built-in `POSTGRES_PASSWORD` default.

A *different* dev machine (a laptop running `make dev-up` locally, or a future
second dev host) needs its own values, not a copy of vps1's: give it its own
`<env>.yaml` and its own marker name, so the two can never be confused for one
host — the same argument as the three held-back credentials above.

## Add a new secret

1. `sops deploy/secrets/common.yaml`, add `THERMOGRAPH_NEW_THING: value`.
2. Add the app reader (`os.environ.get("THERMOGRAPH_NEW_THING")`).
3. Commit + deploy. It flows into `/etc/thermograph.env` automatically.

`common.yaml` reaches prod and beta only. If dev needs the new thing, decide
which it is and act on it: harmless config → copy it into `dev.yaml`; a real
credential → mint dev **its own** and put that in `dev.yaml`; neither → leave dev
without it and make sure the code degrades rather than crashes. Do not reach for
prod's copy.

## Rotate the age identity itself (re-key)

```sh
age-keygen -o new.key                              # new identity
# add the new PUBLIC recipient alongside the old in .sops.yaml, then:
sops updatekeys deploy/secrets/*.yaml              # re-encrypt to BOTH keys
# distribute new.key to /etc/thermograph/age.key on every host + ~/.config/sops/age,
# deploy once so every host can decrypt with the new key, then remove the OLD
# recipient from .sops.yaml and:
sops updatekeys deploy/secrets/*.yaml              # drop the old key
git commit -am "rotate age identity" && push + deploy
```

## First-time bootstrap (seed from the live boxes)

**Historical** — this ran once, before the vps1/vps2 split, when beta was its
own box at `75.119.132.91`. That address is **vps1** now (dev + Forgejo +
Grafana), not beta; re-running the beta line below as written would seed
`beta.yaml` from the wrong host. Kept for the shape of the process, which is
unchanged if a new environment is ever bootstrapped the same way — just point
`seed-from-live.sh` at wherever that environment actually lives today.

Order matters — values must match the live env exactly so `AUTH_SECRET` / VAPID /
`POSTGRES_PASSWORD` don't rotate unintentionally:

1. Install `sops`+`age` and drop `/etc/thermograph/age.key` (0400) +
   `/etc/thermograph/secrets-env` on the target hosts (`provision-secrets.sh`).
2. Seed each host's file from its live env with the helper (pulls over SSH, encrypts,
   and verifies the render round-trips — run it on your own machine, it reads live
   secrets), e.g. as originally run:
   ```sh
   deploy/secrets/seed-from-live.sh prod agent@169.58.46.181
   deploy/secrets/seed-from-live.sh beta agent@75.119.132.91   # beta's address AT THE TIME; now vps1's
   ```
   That writes exact per-host copies (`prod.yaml`/`beta.yaml`); factor shared values
   into `common.yaml` later. Commit.
3. **Dry-run:** on the box, render to a temp file and `diff` against the live
   `/etc/thermograph.env` — must be byte-identical before cutover.
4. Merge the `deploy.sh` wiring; run one deploy; confirm the app is healthy.

## Safety

- CI (`.forgejo/workflows/secrets-guard.yml`) fails if any `*.yaml` here is committed
  unencrypted.
- Rendering is **presence-detected**: a host without the age key + marker keeps its
  existing `/etc/thermograph.env`, so nothing breaks before a host is migrated.
- Rendering happens **on the target box**, so CI/runners never see the age key or the
  plaintext — **except on dev**, where the box *is* the runner. That is not a hole
  this vault can close; it is the reason `dev.yaml` holds nothing that would matter
  if the box were owned. Keep it that way: the day dev needs a real credential,
  mint it a scoped one of its own.
- `deploy-dev.sh` **fails closed**: if the `render-secrets.sh` it finds cannot honour
  `THERMOGRAPH_SECRETS_SKIP_COMMON`, it aborts the deploy rather than render the
  shared production layer onto the dev box.