thermograph/infra/deploy/secrets/README.md
Emi Griffith 7fd364bb8a
All checks were successful
PR build (required check) / changes (pull_request) Successful in 6s
secrets-guard / encrypted (pull_request) Successful in 5s
shell-lint / shellcheck (pull_request) Successful in 7s
PR build (required check) / gate (pull_request) Successful in 1s
PR build (required check) / build-backend (pull_request) Has been skipped
PR build (required check) / build-frontend (pull_request) Has been skipped
PR build (required check) / validate-observability (pull_request) Has been skipped
secrets: make the two-host age key quorum a checked policy
The copies of /etc/thermograph/age.key on vps1 and vps2 have always read as
provisioning residue — a side effect of provision-secrets.sh rather than a
decision. They are in fact the estate's recovery quorum: that key is the single
recovery root for all five SOPS vaults AND for every off-box backup, since
ops-cron.yml:142,197 pipe those dumps through `age -r` to the same recipient.

On 2026-07-30 the operator's copy was shredded from the desktop before it had been
copied to its replacement machine, leaving zero operator-side copies. These two were
the only surviving material and the key was restored from vps2. Documenting the
ordering rule that would have prevented it, and adding the check that makes the
quorum verifiable rather than assumed.

verify-age-quorum.sh asserts presence, permissions and hash equality across both
hosts. Divergence is treated as seriously as absence: two hosts holding different
keys means one renders secrets the other cannot read, and a restore silently picks
the wrong one. Prints SHA-256 digests and modes only — a digest of a 32-byte random
key is not reversible, so it is safe in a CI log, and the key is never read or
copied anywhere. Exit status makes it usable as a cron gate.

Also replaces the "wasn't re-verified for this pass" hedge in the README with what
was actually measured: vps1 0440 root:deploy, vps2 0400 root:root, digests equal.
The dev render does not need vps1's group bit — it runs as `agent`, which is not in
group `deploy` and reaches the key through passwordless sudo, with
/opt/thermograph-dev owned agent:agent. Tightening vps1 to 0400 root:root is
therefore expected to be safe, and is called out as the follow-up rather than done
here because it changes a live host on the deploy path.

The check reports that group read as an advisory rather than a failure. It is the
current provisioned state, and a check that fails on day one is a check that is
ignored by day three.
2026-07-31 05:10:47 +00:00

437 lines
25 KiB
Markdown

# Secrets — the git-native vault (SOPS + age)
The **single source of truth** for Thermograph's secrets is the set of
SOPS-encrypted YAML files in this directory. They're committed to the repo
**encrypted** (values only — keys stay readable so diffs are meaningful) and
**rendered into `/etc/thermograph.env` at deploy time** by
[`deploy/render-secrets.sh`](../render-secrets.sh), which the prod/beta
`deploy.sh` invokes.
Cycling a key is: **edit → commit → deploy**. No SSHing in to hand-edit a root file,
no per-host duplication.
## Files
| File | Holds | Encrypted? |
|------|-------|------------|
| `common.yaml` | The 16 values identical on **prod and beta** — VAPID keypair, metrics token, IndexNow key, `REGISTRY_TOKEN`, S3 endpoint/bucket and both S3 keypairs, plus shared non-secret config. **Not layered under `dev`** — see below | ✅ |
| `prod.yaml` | Prod's own — the three held-back credentials below, sizing (`APP_CPUS`, `DB_CPUS`, `DB_MEMORY`, `WORKERS`), `THERMOGRAPH_BASE_URL`, and the Discord + mail credentials that exist nowhere else | ✅ |
| `beta.yaml` | Beta's own — the three held-back credentials, `THERMOGRAPH_BASE_URL`, `APP_CPUS`/`WORKERS`, plus `THERMOGRAPH_MAIL_BACKEND=console` and `THERMOGRAPH_DISCORD_BOT=0`. Its `POSTGRES_PASSWORD` and `THERMOGRAPH_DATABASE_URL` name beta's **own role and database** (`thermograph_beta`) on the instance it shares with prod. `DB_CPUS`/`DB_MEMORY` were **removed**: beta no longer runs a database of its own, and leaving them would have implied it sized one — the shared instance is sized by `prod.yaml`'s values (see `deploy/stack/thermograph-stack.yml`'s `db` service) | ✅ |
| `dev.yaml` | Dev's own (vps1) — **self-contained**, 12 values, no production credential among them | ✅ |
| `centralis.prod.yaml` | **Centralis's** nine variables, rendered to `/etc/centralis.env` on prod. A different service, a different output file, a different renderer — see below | ✅ |
| `example.yaml` | Format reference / CI fixture (fake values) | ✅ |
| `../../.sops.yaml` | Which age recipient files are encrypted to (plaintext config) | — |
### Three credentials are deliberately NOT in `common.yaml`
`POSTGRES_PASSWORD`, `THERMOGRAPH_AUTH_SECRET` and `THERMOGRAPH_DATABASE_URL`
hold **different values on prod and beta today** (beta has its own database
role/password on the shared instance, and its own auth secret), so unlike the
rest of `common.yaml` they were never candidates for sharing on values alone.
They stay per-environment for a reason that matters more now than it used to:
**These are the credentials that let one environment act as another.** With
matching values, a foothold in one environment's rendered env file is a
foothold on the other's database and the other's session signing. That used
to matter because beta and prod were separate boxes; it matters for a
different reason now that they're co-resident on vps2 — keeping the
credentials themselves separate is what stops co-location on one host from
also becoming equivalence at the credential level. `deploy/db/provision-env-db.sh`
enforces the same boundary from the database side: beta's role
(`thermograph_beta`) is `NOSUPERUSER`/`NOCREATEDB`/`NOCREATEROLE`, owns only
its own database, and `CONNECT` is revoked from `PUBLIC` on it — so even a
leaked beta credential cannot reach prod's data, and the reverse.
**What actually separates beta and prod now, stated plainly:** they share a
host by design (see `deploy/env-topology.sh`'s header), so a host-level
compromise of vps2 is shared between them — separate SSH credentials no
longer buy a host boundary between beta and prod, the way they once did when
beta and prod were different machines. What remains is isolation at the
**database** level (separate roles, separate databases, `CONNECT` revoked from
`PUBLIC`) and the **file** level (separate checkouts, separate rendered env
files, separate stack env files, separate loopback ports). Keeping
`POSTGRES_PASSWORD`/`THERMOGRAPH_AUTH_SECRET`/`THERMOGRAPH_DATABASE_URL` out of
a shared file is part of that file-level isolation: nothing on vps2 would ever
hand you prod's auth secret merely because you have beta's.
**vps1 is the box that must never hold prod credentials.** It runs Forgejo,
Forgejo's CI runner, Grafana, and the `dev` environment — which runs whatever
branch is currently in flight, reviewed or not. That is the actual reason
`dev` renders `dev.yaml` **alone** and never layers `common.yaml` (see below):
not squeamishness about a dev box, but that vps1 is uniquely positioned to
leak whatever it's handed, and `common.yaml` is the fleet's shared production
credential set.
Keeping the three held-back credentials per-environment costs nothing (one
line each) and buys the ability for them to diverge further, or for a future
environment to be added without inheriting anyone else's values by accident.
`dev.yaml` is the case that already differs completely: all three of its
values are its own, generated on vps1 and never shared with prod or beta.
At deploy the renderer concatenates `common.yaml` then `<env>.yaml`, so a **host
value wins** (env_file / `source` take the last occurrence of a duplicate key).
`<env>` comes from an explicit `THERMOGRAPH_ENV`/argument on vps2 (which runs
two environments) or `/etc/thermograph/secrets-env` as a fallback elsewhere —
see `deploy/env-topology.sh`.
### `dev` is vaulted, but renders `dev.yaml` **alone**
All three environments are managed here now. `dev` is the one that does **not**
get `common.yaml` layered under it: `deploy-dev.sh` exports
`THERMOGRAPH_SECRETS_SKIP_COMMON=1`, and the renderer skips the shared layer.
That is not squeamishness about a dev box — it is what `common.yaml` contains.
Eleven of its sixteen values are live production credentials: `REGISTRY_TOKEN`,
`THERMOGRAPH_S3_ACCESS_KEY`/`_SECRET_KEY` and
`THERMOGRAPH_LAKE_S3_ACCESS_KEY`/`_SECRET_KEY` (one keypair, **read-write**, on the
bucket that also holds prod's database backups — see `../../ops/README.md`),
`THERMOGRAPH_VAPID_PRIVATE_KEY`/`_PUBLIC_KEY` (they sign Web Push to real
subscribers), `THERMOGRAPH_INDEXNOW_KEY`, `THERMOGRAPH_METRICS_TOKEN`. The render
is a **plaintext concatenation**, so layering it would put every one of those in
`/etc/thermograph.env` on the dev box and, through `env_file:`, into the
environment of every container in the dev stack.
The dev box is **vps1** — the same box that runs Forgejo and its CI. Its
`docker`-labelled runner jobs get the host docker socket automounted, and the
stack it runs is the `dev` branch — code that has not been through the gate
that guards prod. Handing that the estate's read-write object-storage keys and
the push signing key is a strictly larger blast radius than anything the vault
buys back. An override in `dev.yaml` would not help: last-wins governs
*consumers*, but the production value is still physically a line in the file.
One value in `common.yaml` is also simply **wrong** for dev:
`THERMOGRAPH_COOKIE_SECURE=1` is right for the TLS hosts and silently breaks login
over dev's plain-HTTP mesh-only URL. `dev.yaml` sets `0`.
So `dev.yaml` is self-contained. Where the value is shared-but-harmless
(`PORT`, `THERMOGRAPH_BASE`, `TIMESCALEDB_TAG`) it carries its own copy — a
duplicated constant is the cheap half of this trade.
**What `dev.yaml` holds** (12 keys):
| Key | Why |
|-----|-----|
| `POSTGRES_PASSWORD`, `THERMOGRAPH_DATABASE_URL` | dev's own. Deliberately the weak, well-known `thermograph-dev` — the value dev's `pgdata` volume is already initialized with, and the DB is never published off the compose network. Kept per-host for the reason below, and *not* shared with prod |
| `THERMOGRAPH_AUTH_SECRET` | dev's **own**, freshly generated. Fixes a live crash loop: the `daemon` service refuses to start without it (it derives the `/internal/*` token from it), so dev's daemon had been restarting every 60s. Prod's value must never be here — it signs sessions and verification links |
| `THERMOGRAPH_BASE_URL` | `http://10.10.0.2:8137` (vps1's mesh address). Unset, the app defaults to `https://thermograph.org`, so dev's IndexNow pings and Discord links claimed to be prod |
| `THERMOGRAPH_COOKIE_SECURE=0` | plain HTTP on the mesh-only URL |
| `THERMOGRAPH_MAIL_BACKEND=console`, `THERMOGRAPH_DISCORD_BOT=0` | explicit fail-safes. Both are already the code defaults; stating them means dev cannot start mailing or open a Discord gateway by inheriting a value |
| `PORT`, `THERMOGRAPH_BASE`, `TIMESCALEDB_TAG`, `DB_MEMORY`, `WORKERS` | non-secret config that would otherwise have come from `common.yaml` |
**Deliberately absent, and why nothing breaks:**
- **Both S3 keypairs.** Without them the `lake` service answers 503 and history
reads fall through to the Open-Meteo archive — an accelerator, never a
dependency (`docker-compose.yml`, `backend/data/climate.py`).
- **VAPID keypair.** `backend/notifications/push.py` resolves env → file →
generates once and persists to the `appdata` volume. Dev gets its own identity
for free; prod's private key stays off the box.
- **`THERMOGRAPH_INDEXNOW_KEY`.** Same env → file → self-generate ladder
(`backend/indexnow.py`). Prod's key is what authenticates URL submissions *for
thermograph.org*.
- **`THERMOGRAPH_METRICS_TOKEN`.** With no token `/api/v2/metrics` is
direct-loopback-only, which is exactly right on a mesh-only box; dev's Alloy
agent ships logs, not metrics, so nothing scrapes it.
- **`REGISTRY_TOKEN`.** dev pulls with the persistent `docker login` credential
already in the host's docker config, like the prod/beta CI paths. A vault copy
would only duplicate a credential the box already has into a more readable file.
- **All Discord and mail credentials.** Dev must not be able to act as the real
bot or send real mail.
- **`APP_CPUS`, `DB_CPUS`.** `docker-compose.dev.yml` `!reset`s both — dev runs
uncapped, so setting them would be a lie. (`DB_MEMORY` *is* live: the `db`
service's `mem_limit` is not reset.)
If dev ever needs one of these for real, it gets **its own** — a dev Discord app,
a scoped read-only bucket key — never a copy of prod's.
## Centralis: its own file, its own renderer
`/etc/centralis.env` on prod is the control plane's env file — four real
credentials (`CENTRALIS_AUTH_TOKEN`, `CENTRALIS_TOKENS`,
`CENTRALIS_FORGEJO_TOKEN`, `DISCORD_TOKEN`) and five non-secret settings. It was
**hand-edited** until 2026-07-24, when a hand-edit wrote `CENTRALIS_TOKENS` as
bare JSON into a file that gets `source`d, bash stripped the double quotes,
`{"emi":"…"}` arrived as `{emi:…}`, and Centralis — correctly failing closed on
malformed JSON — dropped to the single `shared` identity. Nothing logged an
error. It looked exactly like the file had never been written.
`centralis.prod.yaml` + `render_centralis_secrets()` in
[`../render-secrets.sh`](../render-secrets.sh) end that, by the only durable
route: **a human stops writing the file.**
Seed / verify / rotate:
```sh
# One-time seed from the live box (plaintext never leaves prod):
deploy/secrets/seed-centralis-on-host.sh prod agent@169.58.46.181
# Prove a render matches the live file before cutting over. Renders to a 0700
# scratch dir on the host, sources both, compares — key NAMES and a verdict only:
deploy/secrets/verify-centralis-render.sh prod agent@169.58.46.181
# Thereafter, rotation is edit -> commit -> merge to main -> redeploy Centralis:
sops deploy/secrets/centralis.prod.yaml
```
Merging to `main` is what puts the new ciphertext on prod (`infra-sync.yml`
fast-forwards `/opt/thermograph`); the render happens when Centralis is
redeployed. See Centralis's own `deploy/README.md` §2 and §5.
### Why `CENTRALIS_*` is not just added to `prod.yaml`
It is the obvious move, and it hands the estate's control-plane bearer token to
every application container.
`/etc/thermograph.env` is loaded into the app stack through `env_file:` and
through `. /host/thermograph.env` in the stack services' entrypoints. Every key
in it is in the environment of every container in the stack. `CENTRALIS_TOKENS`
and `CENTRALIS_AUTH_TOKEN` authenticate an endpoint that carries `sql_query`
with `write:true` and `run_on_host` over production, and `CENTRALIS_FORGEJO_TOKEN`
is repo-scope on the monorepo. Putting them in `prod.yaml` would mean that a
read-anything bug in the web backend, or a leaked container env dump, is a
foothold on the control plane. The two files are separate for a reason that
predates the vault, and the vault is not a reason to merge them.
The converse — pointing Centralis's deploy at `/etc/thermograph.env` — has the
same problem from the other side, plus 32 unrelated variables in the shell that
runs its compose, plus a lifecycle coupling: every app deploy re-renders that
file, and Centralis and the app stack are deliberately deployed on different
cadences.
### Why this file is fully quoted and `/etc/thermograph.env` is not
Different consumers.
`/etc/thermograph.env` is read by docker-compose `env_file:` and by systemd
`EnvironmentFile=`, which parse `KEY=value` literally. Raw `sops --output-type
dotenv` is the right representation for it, and it must stay that way: Centralis's
own `secrets_inventory` / `secrets_render` tools compare rendered values against
that file **textually**, so quoting it would report all 32 keys as value-changed.
`/etc/centralis.env` is read by exactly one thing, and it is a shell:
```sh
sudo bash -c 'set -a && . /etc/centralis.env && set +a && docker compose up -d --build'
```
`set -a; . file` runs every line through bash's full expansion pipeline. Raw
dotenv output is *unsafe* there and sops cannot make it safe — it emits values
verbatim. So `render_centralis_secrets` decrypts to **JSON** (lossless; dotenv
escapes a newline to a literal `\n` and cannot be told apart from a literal
backslash-n) and writes POSIX single-quoted assignments, in which every byte is
literal except `'` itself.
Then it **proves the file** before letting it near `/etc`: it sources the
rendered file in a clean shell (`env -i`), reads every variable back, and
compares to the plaintext it started from. If any value does not survive, or if
`CENTRALIS_TOKENS` does not parse as a JSON object of subject → token, the
render aborts and the existing file is left untouched. The 2026-07-24 bug cannot
be written by this path, and cannot be written *through* it either.
### Scoped by environment even though Centralis is prod-only
The file is `centralis.prod.yaml`, not `centralis.yaml`, and the renderer takes
the `prod` from `/etc/thermograph/secrets-env` like everything else here. An
unscoped file would render prod's bearer tokens onto any host that ran the
function. If Centralis ever gains a second home, that is a new file rather than
a redesign — and the two boxes get different tokens, which is the point.
`centralis.prod.yaml` is **not** layered under `common.yaml`. Centralis shares
no key with the app stack (verified: zero overlap between the two live env
files), so layering would only push the VAPID private key, both S3 keypairs and
`REGISTRY_TOKEN` into a file that wants none of them.
## The age key (the one thing that matters)
- The **private key is the single recovery root.** Lose it and every secret here is
unrecoverable; leak it and every secret is compromised.
- It is **never in the repo.** It lives at:
- `~/.config/sops/age/keys.txt` on the operator's machine (for editing), and
- `/etc/thermograph/age.key` (`0400`) on each VPS that renders at deploy time.
- **Back it up** in the password manager. The public recipient is in `.sops.yaml`.
### The two on-host copies are a deliberate quorum
`/etc/thermograph/age.key` on **vps1** and **vps2** is not provisioning residue and
must not be tidied away. Together they are the recovery quorum for the entire
estate: five SOPS vaults, and every off-box backup, since `ops-cron.yml:142,197`
pipe those dumps through `age -r` to this same recipient.
That is not theoretical. On **2026-07-30** the operator's copy was shredded from the
desktop *before* it had been copied to its replacement machine, leaving zero
operator-side copies. These two were the only surviving material, and the key was
restored from vps2. The ordering rule below exists because of that hour.
- **Never remove the last operator-side copy** until its replacement has been
verified against all five vaults (`sops -d … | grep -c '='` → 16 / 12 / 8 / 16 / 9).
Deleting first and verifying second is unrecoverable if the copy is bad.
- **Both hosts must hold byte-identical copies.** Divergence is as bad as absence:
two different keys means one host renders secrets the other cannot read, and a
restore silently picks the wrong one.
- **Verify it, don't assume it** — `deploy/secrets/verify-age-quorum.sh` asserts
presence, permissions and hash equality across both hosts. It prints SHA-256
digests and modes only, never the key, so it is safe in a CI log and suitable as
a cron gate. Deletion and divergence are both silent failures; nothing else in the
estate would notice either until a restore.
Verified 2026-07-30: vps1 `0440 root:deploy`, vps2 `0400 root:root`, digests equal.
Note the asymmetry — on vps1 the group `deploy` can read the key that decrypts
`common.yaml` and `prod.yaml`, on the box that also runs Forgejo, its CI runner and
dev's unreviewed branches. The dev render does not need that group bit: it runs as
`agent`, which is not in group `deploy` and reaches the key through its passwordless
`sudo` instead (`/opt/thermograph-dev` is `agent:agent`). Tightening vps1 to `0400
root:root` is therefore expected to be safe and is the obvious follow-up; it is
called out rather than done here because it changes a live host on the deploy path.
This weakens — in practice, not in intent — the rule in `infra/CLAUDE.md` that "vps1
must never hold prod credentials". Withholding `common.yaml` from dev's *render* does
not withhold the key that decrypts it from the *box*, because every vault is
encrypted to a single recipient. Fixing that properly means a second age identity for
dev, not a policy line.
## Prerequisites (once per machine)
Install `sops` + `age` (single static binaries). On a host, use
[`deploy/provision-secrets.sh`](../provision-secrets.sh); on your laptop:
```sh
# see the pinned URLs in provision-secrets.sh, or:
go install github.com/getsops/sops/v3/cmd/sops@latest # if you have Go
```
## Everyday: rotate a key
```sh
sops deploy/secrets/common.yaml # opens DECRYPTED in $EDITOR; re-encrypts on save
git commit -am "rotate <thing>" && git push forgejo
# beta and prod both auto-deploy on push (main -> beta, release -> prod) via
# the Deploy workflow. To force it by hand on vps2, say which environment:
ssh agent@169.58.46.181 'cd /opt/thermograph && git pull && THERMOGRAPH_ENV=prod deploy/deploy.sh'
ssh agent@169.58.46.181 'cd /opt/thermograph-beta && git pull && THERMOGRAPH_ENV=beta deploy/deploy.sh'
```
A host-specific value (e.g. `POSTGRES_PASSWORD`) is the same, editing `prod.yaml` /
`beta.yaml` / `dev.yaml` instead. **`POSTGRES_PASSWORD` is special**: the database
only reads it on a *fresh* volume, so also `ALTER ROLE thermograph PASSWORD '…'`
inside the running Postgres to match — the env change alone won't re-key an
initialized DB. (That applies to dev too. `dev.yaml` deliberately carries the value
dev's existing `pgdata` volume was initialized with, so nothing rotates the moment
dev is provisioned; rotating it later is the two-step above, plus the matching
default in `deploy-dev.sh`.)
Dev is edited and deployed exactly like the others — `sops deploy/secrets/dev.yaml`,
commit, and the next `deploy-dev.sh` run (merge to `dev`, or by hand) renders it.
## Setting up a new dev machine
Dev is provisioned like any fleet host now (`deploy/provision-dev.sh`, run as
root on vps1), not the sudo-free desktop bootstrap this predates. The
secrets-specific steps below are still worth doing by hand rather than
folding into that script: `provision-secrets.sh` is **not** the right tool
here — it installs the age key root-owned at `/etc/thermograph/age.key`,
which the CI-runner-driven render may not be able to read (see the age-key
note above).
```sh
# 1. sops + age on PATH, and the age private key in the operator's keyring:
# ~/.config/sops/age/keys.txt (0600, restored from the password manager)
sops --version && age --version
# 2. Name the environment. This marker is what switches rendering ON — until it
# exists the box renders nothing, which is the safe default.
sudo mkdir -p /etc/thermograph
printf 'dev\n' | sudo tee /etc/thermograph/secrets-env >/dev/null
sudo chmod 0644 /etc/thermograph/secrets-env
# 3. Pre-create the render target owned by the deploying user, so no deploy ever
# needs sudo (the renderer writes in place when the file is writable — the same
# path beta's non-root `deploy` user takes).
sudo install -m 0600 -o "$USER" -g "$USER" /dev/null /etc/thermograph.env
# 4. Dry-run the render before letting a deploy do it, and read the key names back.
# Nothing from common.yaml may appear here.
THERMOGRAPH_SECRETS_SKIP_COMMON=1 \
SOPS_AGE_KEY_FILE=~/.config/sops/age/keys.txt \
sops -d --input-type yaml --output-type dotenv deploy/secrets/dev.yaml | sed 's/=.*/=/'
# 5. Deploy. deploy-dev.sh refuses to run if render-secrets.sh in the checkout
# cannot honour THERMOGRAPH_SECRETS_SKIP_COMMON, rather than leak.
APP_DIR=/opt/thermograph-dev bash deploy/deploy-dev.sh
```
To take a dev box **back** off the vault, delete `/etc/thermograph/secrets-env`:
presence detection turns rendering off and `deploy-dev.sh` falls back to its
built-in `POSTGRES_PASSWORD` default.
A *different* dev machine (a laptop running `make dev-up` locally, or a future
second dev host) needs its own values, not a copy of vps1's: give it its own
`<env>.yaml` and its own marker name, so the two can never be confused for one
host — the same argument as the three held-back credentials above.
## Add a new secret
1. `sops deploy/secrets/common.yaml`, add `THERMOGRAPH_NEW_THING: value`.
2. Add the app reader (`os.environ.get("THERMOGRAPH_NEW_THING")`).
3. Commit + deploy. It flows into `/etc/thermograph.env` automatically.
`common.yaml` reaches prod and beta only. If dev needs the new thing, decide
which it is and act on it: harmless config → copy it into `dev.yaml`; a real
credential → mint dev **its own** and put that in `dev.yaml`; neither → leave dev
without it and make sure the code degrades rather than crashes. Do not reach for
prod's copy.
## Rotate the age identity itself (re-key)
```sh
age-keygen -o new.key # new identity
# add the new PUBLIC recipient alongside the old in .sops.yaml, then:
sops updatekeys deploy/secrets/*.yaml # re-encrypt to BOTH keys
# distribute new.key to /etc/thermograph/age.key on every host + ~/.config/sops/age,
# deploy once so every host can decrypt with the new key, then remove the OLD
# recipient from .sops.yaml and:
sops updatekeys deploy/secrets/*.yaml # drop the old key
git commit -am "rotate age identity" && push + deploy
```
## First-time bootstrap (seed from the live boxes)
**Historical** — this ran once, before the vps1/vps2 split, when beta was its
own box at `75.119.132.91`. That address is **vps1** now (dev + Forgejo +
Grafana), not beta; re-running the beta line below as written would seed
`beta.yaml` from the wrong host. Kept for the shape of the process, which is
unchanged if a new environment is ever bootstrapped the same way — just point
`seed-from-live.sh` at wherever that environment actually lives today.
Order matters — values must match the live env exactly so `AUTH_SECRET` / VAPID /
`POSTGRES_PASSWORD` don't rotate unintentionally:
1. Install `sops`+`age` and drop `/etc/thermograph/age.key` (0400) +
`/etc/thermograph/secrets-env` on the target hosts (`provision-secrets.sh`).
2. Seed each host's file from its live env with the helper (pulls over SSH, encrypts,
and verifies the render round-trips — run it on your own machine, it reads live
secrets), e.g. as originally run:
```sh
deploy/secrets/seed-from-live.sh prod agent@169.58.46.181
deploy/secrets/seed-from-live.sh beta agent@75.119.132.91 # beta's address AT THE TIME; now vps1's
```
That writes exact per-host copies (`prod.yaml`/`beta.yaml`); factor shared values
into `common.yaml` later. Commit.
3. **Dry-run:** on the box, render to a temp file and `diff` against the live
`/etc/thermograph.env` — must be byte-identical before cutover.
4. Merge the `deploy.sh` wiring; run one deploy; confirm the app is healthy.
## Safety
- CI (`.forgejo/workflows/secrets-guard.yml`) fails if any `*.yaml` here is committed
unencrypted.
- Rendering is **presence-detected**: a host without the age key + marker keeps its
existing `/etc/thermograph.env`, so nothing breaks before a host is migrated.
- Rendering happens **on the target box**, so CI/runners never see the age key or the
plaintext — **except on dev**, where the box *is* the runner. That is not a hole
this vault can close; it is the reason `dev.yaml` holds nothing that would matter
if the box were owned. Keep it that way: the day dev needs a real credential,
mint it a scoped one of its own.
- `deploy-dev.sh` **fails closed**: if the `render-secrets.sh` it finds cannot honour
`THERMOGRAPH_SECRETS_SKIP_COMMON`, it aborts the deploy rather than render the
shared production layer onto the dev box.