Jinemi/thermograph
4
0
Fork 0
Code Issues 2 Pull requests 9 Projects Releases Packages Wiki Activity Actions

secrets: vault dev and Centralis; render dev without common.yaml
Some checks failed
PR build (required check) / changes (pull_request) Successful in 7s
Details
secrets-guard / encrypted (pull_request) Successful in 5s
Details
PR build (required check) / build-backend (pull_request) Has been skipped
Details
PR build (required check) / build-frontend (pull_request) Has been skipped
Details
shell-lint / shellcheck (pull_request) Failing after 9s
Details
PR build (required check) / validate-observability (pull_request) Has been skipped
Details
PR build (required check) / gate (pull_request) Successful in 2s
Details

Browse source 
Brings the last two credential stores under SOPS, so all three environments and
the control plane are managed the same way.

dev.yaml — dev was not "intentionally not wired in", it was broken

The vault README claimed dev had no real secrets. Both halves were false.

Dev needed secrets it did not have: thermograph-dev-daemon-1 has been in a crash
loop, restarting every ~60s with "neither THERMOGRAPH_INTERNAL_TOKEN nor
THERMOGRAPH_AUTH_SECRET is set; refusing to start" — so dev has had no Discord
gateway and no job timers. With THERMOGRAPH_BASE_URL unset the app also falls
back to https://thermograph.org, so dev's IndexNow pings and verification links
claimed to be prod. dev.yaml gives it its own generated AUTH_SECRET and a real
base URL, and 12 values total.

And dev already held production credentials: backend-deploy-dev.yml injects
S3_ACCESS_KEY/S3_SECRET_KEY into every dev deploy, and the only provisioned
keypair for that bucket is read-write — on the bucket holding prod's backups.
Dev does not need them; without bucket creds the lake service 503s and history
falls through to the archive. Those workflow lines still need removing.

dev renders dev.yaml ALONE, via THERMOGRAPH_SECRETS_SKIP_COMMON=1. Eleven of
common.yaml's sixteen values are live production credentials, the render is a
plaintext concatenation consumed through env_file:, and dev is the operator's
desktop AND the Forgejo runner executing unreviewed dev-branch code with the
docker socket mounted. An override in dev.yaml would not help — last-wins
governs consumers, but prod's value is still physically a line in the file.
Verified: current renderer leaks 28 lines onto dev including both S3 keypairs
and the VAPID private key; patched gives exactly 12 keys and no prod credential.
Beta's render is byte-identical either way.

centralis.prod.yaml — its own file, its own renderer

/etc/centralis.env was hand-edited, which is how a JSON registry got written
unquoted into a shell-sourced file tonight and silently collapsed three
identities to one. The renderer now owns the file.

It is service- AND environment-scoped, not folded into prod.yaml, because
/etc/thermograph.env is loaded into every container in the app stack. Folding
these in would put CENTRALIS_AUTH_TOKEN and CENTRALIS_TOKENS — which
authenticate an endpoint carrying run_on_host and sql_query(write) — into the
web backend's environment, turning a read-anything bug in the app into a
foothold on the control plane.

Quoting is guaranteed three ways rather than assumed. Critically, `sops -d
--output-type dotenv` emits values verbatim, so reusing the existing render path
would have reproduced tonight's bug exactly. The new function decrypts to JSON
and emits POSIX single-quoted assignments; it then sources its own output in a
clean shell and compares every value before touching /etc; and it refuses to
write unless CENTRALIS_TOKENS parses as a non-empty subject->token object after
sourcing. Tested against 16 hostile values including embedded quotes, newlines
and ;rm -rf /.

Value-identity proven, not asserted: rendered vs live compared as *effective*
values on prod (a textual diff would report a false difference, and would report
a false match if the vault had captured quote characters as part of the value),
plus both env files producing a byte-identical `docker compose config` digest.
9 keys, both token subjects preserved. Nothing deployed.

Note for later: /etc/thermograph.env is also shell-sourced, and survives on raw
dotenv only because none of its 32 current values contains a shell-special
character. It is one quoted secret away from the same bug.

Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY
This commit is contained in:
Emi Griffith 2026-07-24 18:10:51 -07:00
parent a00b2c016b
commit 43c9e2052a
8 changed files with 969 additions and 31 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

60
infra/DEPLOY-DEV.md
View file

@ -47,6 +47,7 @@ old in-workflow `ci-cd.yml` (retired along with the rest of `.github/`).
| `.forgejo/workflows/deploy-dev.yml` | build + deploy on pushes to `dev` (direct, or a PR merge) |
| `.forgejo/workflows/build.yml` | shared build gate: deps, backend tests, JS syntax check, boot/health check |
| `deploy/deploy-dev.sh` | pull `dev` into `~/thermograph-dev`, `docker compose up` the uncapped LAN stack, health check |
| `deploy/secrets/dev.yaml` | dev's SOPS vault — its own secrets and config, rendered to `/etc/thermograph.env` at deploy time (see "Config and secrets") |
| `docker-compose.dev.yml` | dev overlay: no CPU limits, backend published on `0.0.0.0:8137` (frontend unpublished) |
| `deploy/provision-dev-lan.sh` | one-time sudo-free bootstrap of the checkout + runner |
| `deploy/forgejo/register-lan-runner.sh` | registers the Forgejo Actions runner on this machine |
@ -119,10 +120,54 @@ Bootstrap (or rebuild) it by hand:
bash deploy/provision-dev-lan.sh
```
Config: port/base path are baked into the unit from `deploy/deploy-dev.sh`
(`PORT`, `THERMOGRAPH_BASE`). Change them there and redeploy. The dedicated
checkout at `~/thermograph-dev` is separate from your working tree, so a deploy
never touches uncommitted edits in `~/Code/Thermograph`.
The dedicated checkout at `~/thermograph-dev` is separate from your working tree,
so a deploy never touches uncommitted edits in `~/Code/Thermograph`.
## Config and secrets
Dev's configuration lives in the same SOPS vault as prod's and beta's —
`deploy/secrets/dev.yaml`, encrypted to the same age recipient, rendered to
`/etc/thermograph.env` by the same `render-secrets.sh` at deploy time. Changing a
dev value is `sops deploy/secrets/dev.yaml` → commit → deploy, exactly as for the
VPS boxes. **`deploy/secrets/README.md` is the reference**; what follows is only
what is specific to this machine.
Dev renders `dev.yaml` **alone**. It does *not* layer `common.yaml`, which is the
fleet's shared production credential set — registry token, both S3 keypairs (one
read-write pair, on the bucket that also holds prod's backups), the VAPID private
key that signs push to real subscribers, the IndexNow key, the metrics token. This
box is the CI runner and runs unreviewed `dev`-branch code; a plaintext render of
that set into `/etc/thermograph.env` would put all of it in every dev container's
environment. `deploy-dev.sh` exports `THERMOGRAPH_SECRETS_SKIP_COMMON=1` to prevent
it, and **aborts the deploy** if the renderer in the checkout cannot honour that.
So dev holds its own `THERMOGRAPH_AUTH_SECRET` (its own, not prod's — the `daemon`
service refuses to start without one), its own `POSTGRES_PASSWORD`, a LAN
`THERMOGRAPH_BASE_URL`, `THERMOGRAPH_COOKIE_SECURE=0` (plain HTTP — the shared
value is `1` and would silently break login here), and the non-secret config it
would otherwise have inherited. It holds **no** S3 keys, no VAPID keypair, no
IndexNow key, no Discord or mail credentials: the lake falls through to the
Open-Meteo archive without bucket creds, and the app generates and persists its own
VAPID and IndexNow keys into the `appdata` volume.
Two dev-only mechanics, both because this box has **no passwordless sudo** and the
runner is a `systemd --user` service with no tty:
- the age key is read from `~/.config/sops/age/keys.txt`, not
`/etc/thermograph/age.key` — a root-owned `0400` key would send the renderer down
its `sudo cat` path, where it would hang on a prompt no CI job can answer;
- `/etc/thermograph.env` is **pre-created** owned by the deploying user, so the
renderer takes its in-place write path and no deploy needs `sudo` at all.
Setting this up on a fresh dev machine is four typed commands — see **"Setting up a
new dev machine"** in `deploy/secrets/README.md`. Until the
`/etc/thermograph/secrets-env` marker exists the box renders nothing and falls back
to `deploy-dev.sh`'s built-in `POSTGRES_PASSWORD` default, which is the safe state
and the one to return to (delete the marker) if the vault ever gets in the way.
`REGISTRY_TOKEN` is deliberately not in the vault: dev pulls with the persistent
`docker login` credential already in this host's docker config, like the prod/beta
CI paths do.
## Before monitoring a PR to land
@ -168,5 +213,8 @@ container/app logs under `host="dev"`, so LAN dev shows up alongside prod and
beta in the same dashboards. This replaced the old `scripts/dashboard.py`.
For a quick terminal check without Grafana, the app still serves raw counters at
the gated `GET /api/v2/metrics` route (loopback-only; set
`THERMOGRAPH_METRICS_TOKEN` to read it over an SSH tunnel).
the gated `GET /api/v2/metrics` route. Dev sets no `THERMOGRAPH_METRICS_TOKEN`, so
the route is direct-loopback-only — `curl localhost:8137/api/v2/metrics` from this
machine, which is where you already are. (A token exists on the VPS boxes to allow
reads through an SSH tunnel; dev has no reason to carry a copy, and the estate's
copy must not land here — see `deploy/secrets/README.md`.)

91
infra/deploy/deploy-dev.sh
View file

@ -50,25 +50,82 @@ export COMPOSE_FILE="docker-compose.yml:docker-compose.dev.yml"
# var wins over the compose-file `name:` key, preserving that split.
export COMPOSE_PROJECT_NAME="${COMPOSE_PROJECT_NAME:-thermograph-dev}"
# deploy.sh needs POSTGRES_PASSWORD to interpolate the compose file (both to init
# the db container and to build backend's THERMOGRAPH_DATABASE_URL) -- normally
# supplied by /etc/thermograph.env via render-secrets.sh's SOPS render, but dev has
# no entry in deploy/secrets/ (only common/beta/prod.yaml exist), so
# render_thermograph_secrets degrades to a no-op here (see render-secrets.sh) and
# /etc/thermograph.env likely doesn't exist on a fresh dev box either. Postgres is
# loopback-only inside the compose network on dev (never published, no real
# credentials on this box), so a fixed default is fine -- same value the monorepo's
# deploy-dev.sh used historically. Override via the environment if you ever want a
# different one; this only sets it when nothing else already has.
# --- Secrets -----------------------------------------------------------------
# dev renders from the SOPS vault the same way prod and beta do -- same files, same
# tool, same edit/commit/deploy loop -- with two dev-only adjustments. Both are
# expressed as environment that render-secrets.sh already parameterises, so nothing
# about the rendering logic is forked. See deploy/secrets/README.md.
# (1) dev renders deploy/secrets/dev.yaml ALONE; it does NOT layer common.yaml.
#
# common.yaml is the internet-facing fleet's SHARED PRODUCTION credential set: the
# registry token, both S3 keypairs (the bucket keypair is read-write, and that
# bucket also holds prod's database backups), the VAPID private key that signs push
# to real subscribers, the IndexNow key, the metrics token. Layering it here would
# write all of them in plaintext to /etc/thermograph.env on the box that is also the
# Forgejo CI runner, and hand them to every container in the dev stack through
# `env_file:` -- that is, to whatever unreviewed branch dev happens to be running.
# It would also silently apply THERMOGRAPH_COOKIE_SECURE=1, which is correct for the
# TLS hosts and breaks login on dev's plain-HTTP LAN URL.
#
# dev.yaml is self-contained instead: it carries dev's own copy of everything dev
# genuinely needs, and nothing else.
export THERMOGRAPH_SECRETS_SKIP_COMMON=1
# (2) The age key is read from wherever it is READABLE, not necessarily
# /etc/thermograph/age.key. render-secrets.sh falls back to `sudo cat` for a
# root-owned 0400 key, and this box has no passwordless sudo -- under the Forgejo
# runner (a systemd --user service, no tty) that sudo cannot prompt, so the
# fleet-standard placement would make every dev deploy fail at the render. Prefer
# the standard path when it is readable; otherwise the operator's own keyring, which
# is where dev's key already lives (this box is the machine `sops` is run on), so
# provisioning dev needs no second copy of the estate's recovery root.
if [ -z "${THERMOGRAPH_AGE_KEY:-}" ] && [ ! -r /etc/thermograph/age.key ] \
&& [ -r "$HOME/.config/sops/age/keys.txt" ]; then
export THERMOGRAPH_AGE_KEY="$HOME/.config/sops/age/keys.txt"
fi
# Fail closed rather than leak. If this box is provisioned to render (an env marker
# AND a key) but the render-secrets.sh in this checkout predates
# THERMOGRAPH_SECRETS_SKIP_COMMON, the render would quietly concatenate common.yaml
# into dev's /etc/thermograph.env -- exactly the leak (1) exists to prevent, with a
# success exit code. Checking a sibling script for the flag is blunt, but the two
# files are versioned together in this same checkout, and the alternative failure
# mode is unreviewed code running with the estate's object-storage keys. Remove this
# once the flag is unconditionally present in render-secrets.sh.
_render_sh="$(dirname "$0")/render-secrets.sh"
_secrets_marker="${THERMOGRAPH_SECRETS_ENV_FILE:-/etc/thermograph/secrets-env}"
_age_key="${THERMOGRAPH_AGE_KEY:-/etc/thermograph/age.key}"
if [ -s "$_secrets_marker" ] && [ -f "$_age_key" ] && [ -f "$_render_sh" ] \
&& ! grep -q 'THERMOGRAPH_SECRETS_SKIP_COMMON' "$_render_sh"; then
echo "!! This box is provisioned to render SOPS secrets (env marker + age key)," >&2
echo "!! but $_render_sh does not honour" >&2
echo "!! THERMOGRAPH_SECRETS_SKIP_COMMON, so it would render common.yaml -- the" >&2
echo "!! fleet's shared PRODUCTION credentials -- into /etc/thermograph.env on the" >&2
echo "!! CI-runner box. Refusing to deploy." >&2
echo "!! Fix: update deploy/render-secrets.sh, or remove $_secrets_marker to fall" >&2
echo "!! back to the un-vaulted dev path." >&2
exit 1
fi
# deploy.sh needs POSTGRES_PASSWORD to interpolate the compose file (both to init the
# db container and to build backend's THERMOGRAPH_DATABASE_URL). On a provisioned dev
# box it arrives from dev.yaml via the render above -- deploy.sh sources
# /etc/thermograph.env AFTER calling the renderer, so the vault value wins over this
# line. This default is the fallback for the un-vaulted paths that remain: a fresh
# box before provisioning, and a bare `make dev-up`. It is the same value dev.yaml
# holds and the value dev's existing pgdata volume was initialized with -- keep the
# two in step, and note that changing it needs an ALTER ROLE against the running
# database as well (see deploy/secrets/README.md), not just an edit.
export POSTGRES_PASSWORD="${POSTGRES_PASSWORD:-thermograph-dev}"
# REGISTRY_TOKEN (for `docker login` in deploy.sh) is a REAL credential -- read
# access to git.thermograph.org's registry -- and deliberately has NO default here.
# Until dev gets its own deploy/secrets/dev.yaml entry (see deploy/secrets/README.md)
# it must come from the calling environment: exported by hand for a manual run, or
# injected by whatever eventually SSHes in to trigger a dev deploy (analogous to
# how deploy.yml supplies it for prod/beta today). deploy.sh's own `docker login`
# step fails loudly if it's unset or wrong -- nothing silently no-ops.
# REGISTRY_TOKEN (for `docker login` in deploy.sh) is deliberately NOT in dev.yaml
# and has no default here. dev pulls with the persistent `docker login` credential
# already in this host's docker config -- the same arrangement the prod/beta CI
# deploy paths use, and the reason deploy.sh's login step is conditional. Putting a
# registry token in dev's vault would only add a second copy of a credential the box
# already has, in a file more processes can read. Export it by hand for a one-off run
# on a box that has never logged in; a genuine auth problem fails loudly at `pull`.
echo "==> LAN-dev deploy: APP_DIR=$APP_DIR BRANCH=$BRANCH COMPOSE_FILE=$COMPOSE_FILE"
exec "$(dirname "$0")/deploy.sh" "$@"

293
infra/deploy/render-secrets.sh
View file

@ -84,7 +84,21 @@ render_thermograph_secrets() {
# Decrypt failures return 1 explicitly, so a partial env is never written and
# correctness doesn't depend on the caller's shell options. common first, host
# second, so a host value overrides a shared one (last-wins).
if [ -f "$repo/deploy/secrets/common.yaml" ]; then
#
# THERMOGRAPH_SECRETS_SKIP_COMMON=1 renders the host file ALONE. deploy-dev.sh
# sets it, and the reason is what common.yaml contains: eleven of its sixteen
# values are live production credentials — both S3 keypairs (one is read-write
# on the bucket holding prod's database backups), the VAPID private key that
# signs Web Push to real subscribers, REGISTRY_TOKEN, the IndexNow and metrics
# tokens. This render is a plaintext concatenation consumed via `env_file:`, so
# layering common on dev would put every one of those into the environment of
# every container on a desktop that is also the Forgejo CI runner, executing
# unreviewed dev-branch code with the docker socket mounted.
#
# An override in dev.yaml would not fix it: last-wins governs *consumers*, but
# the production value is still physically a line in the file.
if [ -f "$repo/deploy/secrets/common.yaml" ] \
&& [ "${THERMOGRAPH_SECRETS_SKIP_COMMON:-0}" != 1 ]; then
env "${key_env[@]}" sops -d --input-type yaml --output-type dotenv \
"$repo/deploy/secrets/common.yaml" >> "$tmp" || { rm -f "$tmp"; return 1; }
fi
@ -116,3 +130,280 @@ render_thermograph_secrets() {
rm -f "$tmp"
return "$rc"
}
# ============================================================================
# Centralis — render /etc/centralis.env from deploy/secrets/centralis.<env>.yaml
# ============================================================================
#
# Centralis (the estate's control plane, prod-only) keeps its own env file,
# deliberately separate from the app stack's /etc/thermograph.env. This function
# brings that file under the same vault, so its four credentials
# (CENTRALIS_AUTH_TOKEN, CENTRALIS_TOKENS, CENTRALIS_FORGEJO_TOKEN,
# DISCORD_TOKEN) stop being hand-edited on the box.
#
# ---------------------------------------------------------------------------
# WHY THIS IS A SEPARATE FUNCTION, NOT AN ARGUMENT TO THE ONE ABOVE
# ---------------------------------------------------------------------------
# The two files have DIFFERENT OUTPUT CONTRACTS, and that is the whole reason
# this migration is being done.
#
# /etc/thermograph.env is consumed by docker-compose `env_file:`, by the systemd
# unit's EnvironmentFile=, and by the stack containers' `. /host/thermograph.env`.
# It is written as raw `KEY=value` — sops' own dotenv output, unquoted.
#
# /etc/centralis.env is consumed by EXACTLY ONE thing, and it is a shell:
#
# sudo bash -c 'set -a && . /etc/centralis.env && set +a && docker compose up'
#
# `set -a; . file` runs every line through bash's full word-expansion pipeline:
# quote removal, parameter expansion, command substitution, field splitting.
# Raw dotenv output is therefore NOT a safe representation for this file, and
# sops cannot make it one — `sops -d --output-type dotenv` emits values verbatim.
# On a value of {"emi":"..."} that yields
#
# CENTRALIS_TOKENS={"emi":"..."} -> bash strips the quotes -> {emi:...}
#
# which is exactly the hand-edit that took the token registry out on 2026-07-24:
# Centralis rejected the malformed JSON, failed closed as designed, and served a
# single `shared` identity — indistinguishable from the file never having been
# written. A value containing a space would truncate; one containing a backtick
# or $( ) would EXECUTE. So this function does not reuse the dotenv path at all.
# It decrypts to JSON (lossless, unambiguous — dotenv escapes a newline to a
# literal \n and cannot be told apart from a literal backslash-n) and emits
# POSIX single-quoted assignments, then PROVES the result by sourcing it in a
# clean shell and comparing every value back against the plaintext before the
# file is allowed anywhere near /etc.
#
# Making render_thermograph_secrets take an output format would put a flag on
# the app stack's deploy path whose wrong value is a production outage, to save
# a few lines. It also could not be changed to emit quoted values for BOTH
# files: /etc/thermograph.env's consumers include Centralis's own secrets tools,
# which compare rendered values against the live file textually, so quoting it
# would report all 32 keys as value-changed. Two consumers, two formats.
#
# ---------------------------------------------------------------------------
# WHY common.yaml IS NOT READ HERE
# ---------------------------------------------------------------------------
# Not an oversight. Centralis shares no key with the app stack (verified: zero
# overlap between the two live env files). Concatenating common.yaml would put
# the VAPID private key, both S3 keypairs and REGISTRY_TOKEN into
# /etc/centralis.env and into the shell that runs Centralis's compose — 16
# credentials handed to a service that wants none of them, for no benefit.
#
# ---------------------------------------------------------------------------
# WHY THIS ONE FAILS LOUDLY WITH NO "not configured here" PATH
# ---------------------------------------------------------------------------
# render_thermograph_secrets is called unconditionally by every deploy on every
# box, so it needs the (a) branch for hosts that have not been migrated. This
# function is called only by Centralis's own deploy, on the one host that runs
# Centralis. A caller asking for it has already asserted it should happen, so
# every missing prerequisite is an error. There is no silent no-op path, because
# a silent no-op here means deploying against a stale hand-edited file — the
# state this exists to end.
render_centralis_secrets() {
local repo="${1:-.}" # dir CONTAINING deploy/secrets
local key="${THERMOGRAPH_AGE_KEY:-/etc/thermograph/age.key}"
local marker="${THERMOGRAPH_SECRETS_ENV_FILE:-/etc/thermograph/secrets-env}"
# Overridable so the pre-cutover dry run can render somewhere harmless and diff
# it against the live file. Nothing in the deploy path sets it.
local dest="${CENTRALIS_RENDER_DEST:-/etc/centralis.env}"
local env_name vault
env_name=$(cat "$marker" 2>/dev/null || true)
if [ -z "$env_name" ]; then
echo "!! no environment name at ${marker} — cannot tell which Centralis vault to render" >&2
return 1
fi
if [ ! -f "$key" ]; then
echo "!! no age key at ${key}; this host cannot decrypt the vault" >&2
return 1
fi
# Scoped by environment even though Centralis runs only on prod today. The
# alternative (one unscoped centralis.yaml) would render prod's bearer tokens
# onto any host that ran this, which is the wrong failure to leave lying around
# for the day Centralis gains a second home. Adding one is then a new file,
# not a redesign.
vault="$repo/deploy/secrets/centralis.${env_name}.yaml"
if [ ! -f "$vault" ]; then
echo "!! this host is configured for SOPS (env='${env_name}', age key present)" >&2
echo "!! but no Centralis vault was found at ${vault}" >&2
echo "!! pass the directory containing deploy/secrets — on the hosts that is" >&2
echo "!! /opt/thermograph/infra — or add centralis.${env_name}.yaml to the vault." >&2
return 1
fi
command -v sops >/dev/null 2>&1 || {
echo "!! sops not installed but this host is configured for SOPS secrets" >&2; return 1; }
command -v python3 >/dev/null 2>&1 || {
echo "!! python3 not installed; needed to quote values safely for a sourced file" >&2; return 1; }
echo "==> Rendering ${dest} from ${vault}"
# Same age-key handling as render_thermograph_secrets: read the 0400 root key
# directly when we can, else lift it through sudo into SOPS_AGE_KEY, so the key
# never has to be made readable by the deploy user.
local key_env=()
if [ -r "$key" ]; then
key_env=("SOPS_AGE_KEY_FILE=$key")
else
local keymat; keymat=$(sudo cat "$key" 2>/dev/null | grep '^AGE-SECRET-KEY-' || true)
[ -n "$keymat" ] || { echo "!! cannot read age key at $key (need sudo)" >&2; return 1; }
key_env=("SOPS_AGE_KEY=$keymat")
fi
# ONE temp directory, 0700, holding plaintext — so cleanup is a single rm on
# every path. Deliberately NOT a `trap ... RETURN`: this file is SOURCED, and a
# RETURN trap set inside a sourced function persists into the caller's shell and
# re-fires when its next `source` completes (see the note in the function above).
local work rc=0
work=$(mktemp -d) || { echo "!! mktemp -d failed" >&2; return 1; }
chmod 700 "$work"
# The work happens in a subshell so that no failure path can skip the cleanup
# below, and so umask/cwd changes cannot leak into deploy.sh.
#
# Every step carries its own `|| exit 1` rather than relying on `set -e`. That
# is not belt-and-braces: `set -e` is DISABLED inside a compound command that
# is the left operand of `||`, which this subshell is (`( ... ) || rc=1`, needed
# so a failure here cannot abort a `set -e` caller mid-deploy). A `set -e` in
# here reads as protection and provides none — caught by a test that fed the
# renderer a nested mapping and watched it sail past the rejection.
(
umask 077
# JSON, not dotenv. Lossless and unambiguous — see the header.
env "${key_env[@]}" sops -d --input-type yaml --output-type json "$vault" \
> "$work/plain.json" || exit 1
python3 - "$work/plain.json" "$work/out.env" "$work/want.json" <<'EMIT' || exit 1
import json, re, sys
src, out_env, want_json = sys.argv[1], sys.argv[2], sys.argv[3]
NAME = re.compile(r"[A-Za-z_][A-Za-z0-9_]*\Z")
def shell_quote(v):
"""POSIX single-quoting. Inside '...' every byte is literal except ' itself,
which is closed, backslash-escaped and reopened. There is no escape sequence
to get wrong and no character class to keep up to date: it is total, and
total is the only property worth having for a file bash will expand."""
return "'" + v.replace("'", "'\\''") + "'"
def scalar(k, v):
if isinstance(v, str):
return v
if isinstance(v, bool): # before int: bool is an int in Python
return "true" if v else "false"
if isinstance(v, (int, float)):
return json.dumps(v)
if v is None:
return ""
sys.stderr.write(
"!! %s decrypts to a %s, but an env file holds only scalars\n" % (k, type(v).__name__))
sys.exit(1)
data = json.load(open(src, encoding="utf-8"))
if not isinstance(data, dict):
sys.stderr.write("!! the vault did not decrypt to a mapping\n")
sys.exit(1)
want, lines = {}, []
for k, v in data.items():
if not NAME.match(k):
sys.stderr.write("!! %r is not a usable shell variable name\n" % (k,))
sys.exit(1)
want[k] = scalar(k, v)
lines.append("%s=%s\n" % (k, shell_quote(want[k])))
with open(out_env, "w", encoding="utf-8") as fh:
fh.write("# Rendered by infra/deploy/render-secrets.sh from the SOPS vault.\n")
fh.write("# DO NOT EDIT BY HAND: the next render overwrites this file, and a\n")
fh.write("# hand-edit is how the token registry was lost on 2026-07-24.\n")
fh.write("# Rotate by editing infra/deploy/secrets/centralis.<env>.yaml instead.\n")
fh.writelines(lines)
with open(want_json, "w", encoding="utf-8") as fh:
json.dump(want, fh)
sys.stderr.write(" %d keys quoted\n" % len(want))
EMIT
# PROVE IT. Source the rendered file the way the deploy does — in a clean
# environment, `set -a`, no inherited variables to mask a dropped key — and
# read every value back out. This is the test that the file cannot pass while
# carrying the 2026-07-24 bug, and it runs on every render, not once.
env -i PATH="$PATH" bash -c \
'set -a; . "$1"; set +a; exec python3 -c "
import json, os, sys
sys.stdout.write(json.dumps(dict(os.environ)))"' _ "$work/out.env" \
> "$work/got.json" || exit 1
python3 - "$work/want.json" "$work/got.json" <<'VERIFY' || exit 1
import json, sys
want = json.load(open(sys.argv[1], encoding="utf-8"))
got = json.load(open(sys.argv[2], encoding="utf-8"))
bad = []
for k, v in want.items():
if k not in got:
bad.append("%s: not set at all after sourcing" % k)
elif got[k] != v:
# NEVER print either value. The difference is the finding; the content is
# not, and this runs inside a deploy log.
bad.append("%s: survives sourcing as a DIFFERENT value (len %d -> %d)"
% (k, len(v), len(got[k])))
if bad:
sys.stderr.write("!! the rendered file does not round-trip through bash:\n")
for b in bad:
sys.stderr.write("!! %s\n" % b)
sys.exit(1)
# CENTRALIS_TOKENS is the reason for all of the above: JSON, full of double
# quotes, in a file bash expands. Centralis fails CLOSED on malformed JSON — it
# drops to the single `shared` identity, which looks exactly like the registry
# having never been configured. Refuse to write a file that would do that,
# rather than discovering it from a dashboard that is missing two people.
raw = got.get("CENTRALIS_TOKENS", "")
if raw:
try:
reg = json.loads(raw)
except ValueError as exc:
sys.stderr.write("!! CENTRALIS_TOKENS is not valid JSON after sourcing: %s\n" % exc)
sys.exit(1)
if not isinstance(reg, dict) or not reg:
sys.stderr.write("!! CENTRALIS_TOKENS must be a non-empty JSON object\n")
sys.exit(1)
if not all(isinstance(s, str) and isinstance(t, str) and t for s, t in reg.items()):
sys.stderr.write("!! CENTRALIS_TOKENS must map subject -> non-empty token string\n")
sys.exit(1)
# Subjects are names on audit lines, not credentials. Printing them is the
# whole point: it is how an operator sees at a glance that nobody was lost.
sys.stderr.write(" CENTRALIS_TOKENS parses: %d subject(s) — %s\n"
% (len(reg), ", ".join(sorted(reg))))
sys.stderr.write(" %d keys survive `set -a; . <file>` byte-for-byte\n" % len(want))
VERIFY
) || rc=1
# Only now, with a file that has been read back through bash and matched value
# for value, does anything touch $dest.
if [ "$rc" -eq 0 ]; then
# 0600, not 0640: this file is four bearer credentials for the estate's
# control plane and has no group that needs it. Prefer an in-place write so
# the existing owner/mode survive; fall back the same way the app renderer
# does, chowning on the sudo path so a non-root deploy user can still source
# what it just rendered.
if [ -f "$dest" ] && [ -w "$dest" ]; then
cat "$work/out.env" > "$dest" || rc=1
elif install -m 0600 "$work/out.env" "$dest" 2>/dev/null; then :
elif sudo install -m 0600 -o "$(id -un)" -g "$(id -gn)" "$work/out.env" "$dest" 2>/dev/null; then :
else
echo "!! cannot write ${dest} (need file write access or passwordless sudo)" >&2
rc=1
fi
else
echo "!! render aborted; ${dest} left untouched" >&2
fi
rm -rf "$work"
return "$rc"
}

254
infra/deploy/secrets/README.md
View file

@ -14,9 +14,11 @@ no per-host duplication.
| File | Holds | Encrypted? |
|------|-------|------------|
| `common.yaml` | The 16 values identical on every host — VAPID keypair, metrics token, IndexNow key, `REGISTRY_TOKEN`, S3 endpoint/bucket and both S3 keypairs, plus shared non-secret config | ✅ |
| `common.yaml` | The 16 values identical on **prod and beta** — VAPID keypair, metrics token, IndexNow key, `REGISTRY_TOKEN`, S3 endpoint/bucket and both S3 keypairs, plus shared non-secret config. **Not layered under `dev`** — see below | ✅ |
| `prod.yaml` | Prod's own — the three held-back credentials below, sizing (`APP_CPUS`, `DB_CPUS`, `DB_MEMORY`, `WORKERS`), `THERMOGRAPH_BASE_URL`, and the Discord + mail credentials that exist nowhere else | ✅ |
| `beta.yaml` | Beta's own — the three held-back credentials, sizing, `THERMOGRAPH_BASE_URL` | ✅ |
| `dev.yaml` | The LAN dev box's own — **self-contained**, 12 values, no production credential among them | ✅ |
| `centralis.prod.yaml` | **Centralis's** nine variables, rendered to `/etc/centralis.env` on prod. A different service, a different output file, a different renderer — see below | ✅ |
| `example.yaml` | Format reference / CI fixture (fake values) | ✅ |
| `../../.sops.yaml` | Which age recipient files are encrypted to (plaintext config) | — |
@ -39,10 +41,179 @@ intentional and turn breaking it into a migration rather than an edit.
If you ever *do* want them to differ, change one file. That is the whole point.
`dev.yaml` is the case that already differs: all three of its copies are its own,
generated on the desktop. The rule earned its keep the first time it was used.
At deploy the renderer concatenates `common.yaml` then `<env>.yaml`, so a **host
value wins** (env_file / `source` take the last occurrence of a duplicate key).
`<env>` comes from `/etc/thermograph/secrets-env` on the box (`prod`/`beta`/`dev`).
`dev` has no real secrets and is intentionally not wired in — see `deploy-dev.sh`.
### `dev` is vaulted, but renders `dev.yaml` **alone**
All three environments are managed here now. `dev` is the one that does **not**
get `common.yaml` layered under it: `deploy-dev.sh` exports
`THERMOGRAPH_SECRETS_SKIP_COMMON=1`, and the renderer skips the shared layer.
That is not squeamishness about a dev box — it is what `common.yaml` contains.
Eleven of its sixteen values are live production credentials: `REGISTRY_TOKEN`,
`THERMOGRAPH_S3_ACCESS_KEY`/`_SECRET_KEY` and
`THERMOGRAPH_LAKE_S3_ACCESS_KEY`/`_SECRET_KEY` (one keypair, **read-write**, on the
bucket that also holds prod's database backups — see `../../ops/README.md`),
`THERMOGRAPH_VAPID_PRIVATE_KEY`/`_PUBLIC_KEY` (they sign Web Push to real
subscribers), `THERMOGRAPH_INDEXNOW_KEY`, `THERMOGRAPH_METRICS_TOKEN`. The render
is a **plaintext concatenation**, so layering it would put every one of those in
`/etc/thermograph.env` on the dev box and, through `env_file:`, into the
environment of every container in the dev stack.
The dev box is the operator's desktop. It is also the Forgejo CI runner, whose
`docker`-labelled jobs get the host docker socket automounted, and the stack it
runs is the `dev` branch — code that has not been through the gate that guards
prod. Handing that the estate's read-write object-storage keys and the push
signing key is a strictly larger blast radius than anything the vault buys back.
An override in `dev.yaml` would not help: last-wins governs *consumers*, but the
production value is still physically a line in the file.
One value in `common.yaml` is also simply **wrong** for dev:
`THERMOGRAPH_COOKIE_SECURE=1` is right for the TLS hosts and silently breaks login
over dev's plain-HTTP LAN URL. `dev.yaml` sets `0`.
So `dev.yaml` is self-contained. Where the value is shared-but-harmless
(`PORT`, `THERMOGRAPH_BASE`, `TIMESCALEDB_TAG`) it carries its own copy — a
duplicated constant is the cheap half of this trade.
**What `dev.yaml` holds** (12 keys):
| Key | Why |
|-----|-----|
| `POSTGRES_PASSWORD`, `THERMOGRAPH_DATABASE_URL` | dev's own. Deliberately the weak, well-known `thermograph-dev` — the value dev's `pgdata` volume is already initialized with, and the DB is never published off the compose network. Kept per-host for the reason below, and *not* shared with prod |
| `THERMOGRAPH_AUTH_SECRET` | dev's **own**, freshly generated. Fixes a live crash loop: the `daemon` service refuses to start without it (it derives the `/internal/*` token from it), so dev's daemon had been restarting every 60s. Prod's value must never be here — it signs sessions and verification links |
| `THERMOGRAPH_BASE_URL` | `http://10.0.1.216:8137`. Unset, the app defaults to `https://thermograph.org`, so dev's IndexNow pings and Discord links claimed to be prod |
| `THERMOGRAPH_COOKIE_SECURE=0` | plain HTTP on the LAN |
| `THERMOGRAPH_MAIL_BACKEND=console`, `THERMOGRAPH_DISCORD_BOT=0` | explicit fail-safes. Both are already the code defaults; stating them means dev cannot start mailing or open a Discord gateway by inheriting a value |
| `PORT`, `THERMOGRAPH_BASE`, `TIMESCALEDB_TAG`, `DB_MEMORY`, `WORKERS` | non-secret config that would otherwise have come from `common.yaml` |
**Deliberately absent, and why nothing breaks:**
- **Both S3 keypairs.** Without them the `lake` service answers 503 and history
reads fall through to the Open-Meteo archive — an accelerator, never a
dependency (`docker-compose.yml`, `backend/data/climate.py`).
- **VAPID keypair.** `backend/notifications/push.py` resolves env → file →
generates once and persists to the `appdata` volume. Dev gets its own identity
for free; prod's private key stays off the box.
- **`THERMOGRAPH_INDEXNOW_KEY`.** Same env → file → self-generate ladder
(`backend/indexnow.py`). Prod's key is what authenticates URL submissions *for
thermograph.org*.
- **`THERMOGRAPH_METRICS_TOKEN`.** With no token `/api/v2/metrics` is
direct-loopback-only, which is exactly right on a LAN box; dev's Alloy agent
ships logs, not metrics, so nothing scrapes it.
- **`REGISTRY_TOKEN`.** dev pulls with the persistent `docker login` credential
already in the host's docker config, like the prod/beta CI paths. A vault copy
would only duplicate a credential the box already has into a more readable file.
- **All Discord and mail credentials.** Dev must not be able to act as the real
bot or send real mail.
- **`APP_CPUS`, `DB_CPUS`.** `docker-compose.dev.yml` `!reset`s both — dev runs
uncapped, so setting them would be a lie. (`DB_MEMORY` *is* live: the `db`
service's `mem_limit` is not reset.)
If dev ever needs one of these for real, it gets **its own** — a dev Discord app,
a scoped read-only bucket key — never a copy of prod's.
## Centralis: its own file, its own renderer
`/etc/centralis.env` on prod is the control plane's env file — four real
credentials (`CENTRALIS_AUTH_TOKEN`, `CENTRALIS_TOKENS`,
`CENTRALIS_FORGEJO_TOKEN`, `DISCORD_TOKEN`) and five non-secret settings. It was
**hand-edited** until 2026-07-24, when a hand-edit wrote `CENTRALIS_TOKENS` as
bare JSON into a file that gets `source`d, bash stripped the double quotes,
`{"emi":"…"}` arrived as `{emi:…}`, and Centralis — correctly failing closed on
malformed JSON — dropped to the single `shared` identity. Nothing logged an
error. It looked exactly like the file had never been written.
`centralis.prod.yaml` + `render_centralis_secrets()` in
[`../render-secrets.sh`](../render-secrets.sh) end that, by the only durable
route: **a human stops writing the file.**
Seed / verify / rotate:
```sh
# One-time seed from the live box (plaintext never leaves prod):
deploy/secrets/seed-centralis-on-host.sh prod agent@169.58.46.181
# Prove a render matches the live file before cutting over. Renders to a 0700
# scratch dir on the host, sources both, compares — key NAMES and a verdict only:
deploy/secrets/verify-centralis-render.sh prod agent@169.58.46.181
# Thereafter, rotation is edit -> commit -> merge to main -> redeploy Centralis:
sops deploy/secrets/centralis.prod.yaml
```
Merging to `main` is what puts the new ciphertext on prod (`infra-sync.yml`
fast-forwards `/opt/thermograph`); the render happens when Centralis is
redeployed. See Centralis's own `deploy/README.md` §2 and §5.
### Why `CENTRALIS_*` is not just added to `prod.yaml`
It is the obvious move, and it hands the estate's control-plane bearer token to
every application container.
`/etc/thermograph.env` is loaded into the app stack through `env_file:` and
through `. /host/thermograph.env` in the stack services' entrypoints. Every key
in it is in the environment of every container in the stack. `CENTRALIS_TOKENS`
and `CENTRALIS_AUTH_TOKEN` authenticate an endpoint that carries `sql_query`
with `write:true` and `run_on_host` over production, and `CENTRALIS_FORGEJO_TOKEN`
is repo-scope on the monorepo. Putting them in `prod.yaml` would mean that a
read-anything bug in the web backend, or a leaked container env dump, is a
foothold on the control plane. The two files are separate for a reason that
predates the vault, and the vault is not a reason to merge them.
The converse — pointing Centralis's deploy at `/etc/thermograph.env` — has the
same problem from the other side, plus 32 unrelated variables in the shell that
runs its compose, plus a lifecycle coupling: every app deploy re-renders that
file, and Centralis and the app stack are deliberately deployed on different
cadences.
### Why this file is fully quoted and `/etc/thermograph.env` is not
Different consumers.
`/etc/thermograph.env` is read by docker-compose `env_file:` and by systemd
`EnvironmentFile=`, which parse `KEY=value` literally. Raw `sops --output-type
dotenv` is the right representation for it, and it must stay that way: Centralis's
own `secrets_inventory` / `secrets_render` tools compare rendered values against
that file **textually**, so quoting it would report all 32 keys as value-changed.
`/etc/centralis.env` is read by exactly one thing, and it is a shell:
```sh
sudo bash -c 'set -a && . /etc/centralis.env && set +a && docker compose up -d --build'
```
`set -a; . file` runs every line through bash's full expansion pipeline. Raw
dotenv output is *unsafe* there and sops cannot make it safe — it emits values
verbatim. So `render_centralis_secrets` decrypts to **JSON** (lossless; dotenv
escapes a newline to a literal `\n` and cannot be told apart from a literal
backslash-n) and writes POSIX single-quoted assignments, in which every byte is
literal except `'` itself.
Then it **proves the file** before letting it near `/etc`: it sources the
rendered file in a clean shell (`env -i`), reads every variable back, and
compares to the plaintext it started from. If any value does not survive, or if
`CENTRALIS_TOKENS` does not parse as a JSON object of subject → token, the
render aborts and the existing file is left untouched. The 2026-07-24 bug cannot
be written by this path, and cannot be written *through* it either.
### Scoped by environment even though Centralis is prod-only
The file is `centralis.prod.yaml`, not `centralis.yaml`, and the renderer takes
the `prod` from `/etc/thermograph/secrets-env` like everything else here. An
unscoped file would render prod's bearer tokens onto any host that ran the
function. If Centralis ever gains a second home, that is a new file rather than
a redesign — and the two boxes get different tokens, which is the point.
`centralis.prod.yaml` is **not** layered under `common.yaml`. Centralis shares
no key with the app stack (verified: zero overlap between the two live env
files), so layering would only push the VAPID private key, both S3 keypairs and
`REGISTRY_TOKEN` into a file that wants none of them.
## The age key (the one thing that matters)
@ -50,9 +221,16 @@ value wins** (env_file / `source` take the last occurrence of a duplicate key).
unrecoverable; leak it and every secret is compromised.
- It is **never in the repo.** It lives at:
- `~/.config/sops/age/keys.txt` on the operator's machine (for editing), and
- `/etc/thermograph/age.key` (`0400`) on each host that renders at deploy time.
- `/etc/thermograph/age.key` (`0400`) on each VPS that renders at deploy time.
- **Back it up** in the password manager. The public recipient is in `.sops.yaml`.
On the **dev desktop** those two are the same file, on purpose. `render-secrets.sh`
falls back to `sudo cat` for a root-owned key, the desktop has no passwordless sudo,
and the Forgejo runner is a `systemd --user` service with no tty — a `/etc` copy
would make every CI-triggered dev deploy hang on a prompt it cannot answer.
`deploy-dev.sh` points `THERMOGRAPH_AGE_KEY` at the operator's keyring instead, so
the box keeps **one** copy of the recovery root rather than two.
## Prerequisites (once per machine)
Install `sops` + `age` (single static binaries). On a host, use
@ -73,9 +251,59 @@ ssh agent@169.58.46.181 'cd /opt/thermograph && git pull && deploy/deploy.sh'
```
A host-specific value (e.g. `POSTGRES_PASSWORD`) is the same, editing `prod.yaml` /
`beta.yaml` instead. **`POSTGRES_PASSWORD` is special**: the database only reads it on
a *fresh* volume, so also `ALTER ROLE thermograph PASSWORD '…'` inside the running
Postgres to match — the env change alone won't re-key an initialized DB.
`beta.yaml` / `dev.yaml` instead. **`POSTGRES_PASSWORD` is special**: the database
only reads it on a *fresh* volume, so also `ALTER ROLE thermograph PASSWORD '…'`
inside the running Postgres to match — the env change alone won't re-key an
initialized DB. (That applies to dev too. `dev.yaml` deliberately carries the value
dev's existing `pgdata` volume was initialized with, so nothing rotates the moment
dev is provisioned; rotating it later is the two-step above, plus the matching
default in `deploy-dev.sh`.)
Dev is edited and deployed exactly like the others — `sops deploy/secrets/dev.yaml`,
commit, and the next `deploy-dev.sh` run (merge to `dev`, or by hand) renders it.
## Setting up a new dev machine
The desktop is hands-on: Centralis cannot SSH to it, so this is typed, not
automated. `provision-secrets.sh` is **not** the right tool here — it installs the
age key root-owned at `/etc/thermograph/age.key`, which the runner cannot read
(above).
```sh
# 1. sops + age on PATH, and the age private key in the operator's keyring:
# ~/.config/sops/age/keys.txt (0600, restored from the password manager)
sops --version && age --version
# 2. Name the environment. This marker is what switches rendering ON — until it
# exists the box renders nothing, which is the safe default.
sudo mkdir -p /etc/thermograph
printf 'dev\n' | sudo tee /etc/thermograph/secrets-env >/dev/null
sudo chmod 0644 /etc/thermograph/secrets-env
# 3. Pre-create the render target owned by the deploying user, so no deploy ever
# needs sudo (the renderer writes in place when the file is writable — the same
# path beta's non-root `deploy` user takes).
sudo install -m 0600 -o "$USER" -g "$USER" /dev/null /etc/thermograph.env
# 4. Dry-run the render before letting a deploy do it, and read the key names back.
# Nothing from common.yaml may appear here.
THERMOGRAPH_SECRETS_SKIP_COMMON=1 \
SOPS_AGE_KEY_FILE=~/.config/sops/age/keys.txt \
sops -d --input-type yaml --output-type dotenv deploy/secrets/dev.yaml | sed 's/=.*/=/'
# 5. Deploy. deploy-dev.sh refuses to run if render-secrets.sh in the checkout
# cannot honour THERMOGRAPH_SECRETS_SKIP_COMMON, rather than leak.
APP_DIR=~/thermograph-dev bash deploy/deploy-dev.sh
```
To take a dev box **back** off the vault, delete `/etc/thermograph/secrets-env`:
presence detection turns rendering off and `deploy-dev.sh` falls back to its
built-in `POSTGRES_PASSWORD` default.
A *different* dev machine (a second desktop, a laptop) needs its own values, not a
copy of this one's: give it its own `<env>.yaml` and its own marker name, so the two
can never be confused for one host — the same argument as the three held-back
credentials above.
## Add a new secret
@ -83,6 +311,12 @@ Postgres to match — the env change alone won't re-key an initialized DB.
2. Add the app reader (`os.environ.get("THERMOGRAPH_NEW_THING")`).
3. Commit + deploy. It flows into `/etc/thermograph.env` automatically.
`common.yaml` reaches prod and beta only. If dev needs the new thing, decide
which it is and act on it: harmless config → copy it into `dev.yaml`; a real
credential → mint dev **its own** and put that in `dev.yaml`; neither → leave dev
without it and make sure the code degrades rather than crashes. Do not reach for
prod's copy.
## Rotate the age identity itself (re-key)
```sh
@ -123,4 +357,10 @@ Order matters — values must match the live env exactly so `AUTH_SECRET` / VAPI
- Rendering is **presence-detected**: a host without the age key + marker keeps its
existing `/etc/thermograph.env`, so nothing breaks before a host is migrated.
- Rendering happens **on the target box**, so CI/runners never see the age key or the
plaintext.
plaintext — **except on dev**, where the box *is* the runner. That is not a hole
this vault can close; it is the reason `dev.yaml` holds nothing that would matter
if the box were owned. Keep it that way: the day dev needs a real credential,
mint it a scoped one of its own.
- `deploy-dev.sh` **fails closed**: if the `render-secrets.sh` it finds cannot honour
`THERMOGRAPH_SECRETS_SKIP_COMMON`, it aborts the deploy rather than render the
shared production layer onto the dev box.

24
infra/deploy/secrets/centralis.prod.yaml Normal file
View file

@ -0,0 +1,24 @@
CENTRALIS_AUTH_TOKEN: ENC[AES256_GCM,data:MoOrGKQoe0mFu0gk4koa0kX+I6SgcKSlDAh+motBvBy22kiWucumgrl9avCdepPP,iv:0Vn/8bld2+EZhcwEPepCnmn4ktXp63LJUrZ36uajZp8=,tag:cjM/AVlG21Mk36QB7TjfSw==,type:str]
CENTRALIS_FORGEJO_TOKEN: ENC[AES256_GCM,data:p1KKmZZHhB3IO9QdWe2CYuN+rho+lZf3aeADT2sxZrD5SYeA05O1VA==,iv:93QFRj4tNU44wo6eCn3oCGmLP2tm571S4IhVCXpSQk8=,tag:/yH0O5Y25RfLBmKOWWJkZQ==,type:str]
CENTRALIS_FORGEJO_URL: ENC[AES256_GCM,data:z3ZTFg/2F9ZeilolVXW2XlyjyQy9,iv:hX0lj94vXFG2lZCG3gykaE1oEjTsciREey7onyQgQsc=,tag:415VJUTErR9SG82mkS2jOg==,type:str]
CENTRALIS_LOKI_URL: ENC[AES256_GCM,data:cFzoMoUYhxEVIF/K8JArTB49GRAo,iv:BTgmKBOsTdnWcvYOww4Zve5dJd+ZnFnBbWv3y5V2Vi4=,tag:kZ6LKLIVXk4UOZYbAr483w==,type:str]
DISCORD_GUILD_ID: ENC[AES256_GCM,data:qUFkitxFHOZdA7h2xtNycUpdLA==,iv:RGfbTEdjWY4Wu9i8WQ0rAnDAwqLZZbTj/bsWavYEnjk=,tag:L2b2t+tLhGXpqmGcamBpNw==,type:str]
CENTRALIS_TRANSCRIPT_CHANNEL: ENC[AES256_GCM,data:poxdL903opQV5waTCnO4wM3eIg==,iv:m5VWJpWXb49qak0wYoS1/W7S22+BAl+AFzjTY2Zu0a8=,tag:8iILm7mM0S/9/aAG1DywkQ==,type:str]
CENTRALIS_VOICE_SUMMARY_CHANNEL: ENC[AES256_GCM,data:Ztvy9w8wpmEcROyRDJOosBJlWw==,iv:iugYisN38csVNIZamZ84mzXaPTABxU+I2yZ+CzL/ddQ=,tag:wNGJcoYN2obnoKY7uchYiw==,type:str]
DISCORD_TOKEN: ENC[AES256_GCM,data:hdrjzSqwKmUdmdLl8dOMUPzLxfZf3H4Z6D9b+PIn/nzapQAPECihrxHZswayinYV3nbcZKj2Qh3Aw2jzaP0OpYM6HFdZmosA,iv:nE4utRtwNCDEWOKjTaobpXJHWkXY6LQE1EvHGH+tUH0=,tag:+3UC7N8B1WCCPzBzmB4yog==,type:str]
CENTRALIS_TOKENS: ENC[AES256_GCM,data:nlDM6Y4rUuAiITo1iN/khDFiwCH8UlVA42SaB1d1a4CG7uVMZnl369MVTMWEXhe4GFOtXlSYgHt7XUMkVmCGQzfBwXed6VZcMPMD9XVkbdYz8+FtrOsf0KVQWvNOHQ6MnvOGMb/CVFBeo0A=,iv:OAw7hXOgAlUQWViCb/fkfqXaNEcyu7947ttbaqHYceM=,tag:sUoVL7Fac9c4dil/+bz7fw==,type:str]
sops:
age:
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBGcitwT3BoSE9LWE5zejlh
WlZPYnJUWTlycC9nZnJaK2d3dmFLQWhsS1M4Ckx0c0hPaXVnaUZra3dFdFVJbm93
N29XNXZtVUN6d3FjNFB5UDNseXNwMDgKLS0tIGZ4ckI1ejEyc2w5alJNSXF3YklW
cWZrUWdGd3BEdTYvZU5lV1BDcEE4Z0UK2Q/UGFJwBxLeSib1vJhfGAsR+RMLq144
JEUSbfXvUHvuMyY94DhbXmh2WgqxWYQuvF9UrDLeDw14tNPIGoIpFg==
-----END AGE ENCRYPTED FILE-----
recipient: age1xx4dzs0dxlwvkv9sjuqzsphl7lfrxannkfken374yu2qvvcte9sqzktqt2
lastmodified: "2026-07-25T01:02:49Z"
mac: ENC[AES256_GCM,data:HsxO525FU/AfZw6Y5nTrfPIefl1InTV4M7e03hGsxZRHS2Fwsdqiyl/swI2oUFt3tfEyn0RwNB9cEeG/yo4UPPY80E5wnBvv0kaV0gfibBNwBRuIbPJZR/vXm6k8+60shcCOrtvpby4zUmtuk84kEK4mHWpVzh4xs5eYY37la6I=,iv:yR1Zq5TusAdzt7JnarXqRU3X896JMXJzrJw7p6tTzWs=,tag:lK9ph8OJyF/W1E4CLX8AGw==,type:str]
unencrypted_suffix: _unencrypted
version: 3.13.2

27
infra/deploy/secrets/dev.yaml Normal file
View file

@ -0,0 +1,27 @@
PORT: ENC[AES256_GCM,data:0LjaWw==,iv:3e8Q71QFlMAhQStgoiVh/c2ELI3+7UkWseAFv8BnElU=,tag:/wYfKmqErJpsLBjg1gs4Tw==,type:str]
THERMOGRAPH_BASE: ENC[AES256_GCM,data:Ww==,iv:e2yNIAAw4dtlahfbA+PCM/jyODKFwWFNfVpVJ1JoREc=,tag:QzoGwDgIwdA9OFAMIPej7Q==,type:str]
TIMESCALEDB_TAG: ENC[AES256_GCM,data:7Oj7OoxoaG51whg=,iv:PPbfdn4DHMxTPA5FUqv+lEY29wQsVX+g0aVuWvraYrU=,tag:NAyJZlRXU1iKo4jwBctUtw==,type:str]
DB_MEMORY: ENC[AES256_GCM,data:/3U=,iv:jFI/9NQaz8oIqBWAKydWUEEI7bI3SmLnsoBlVgfKw7I=,tag:oBoFANuTgZV0jXC9qR5IHQ==,type:str]
WORKERS: ENC[AES256_GCM,data:VQ==,iv:+d84ByEU22XVQi43NcRn8cHmbW9XudJMj0guidsHxBQ=,tag:XM//7y0VQCwini4CGL7pSg==,type:str]
THERMOGRAPH_BASE_URL: ENC[AES256_GCM,data:FK2kOaZG8ESwDi3VMOlo0XvBKvdADQ==,iv:c6GvCrAnAf+ZFh53xg3ZOlf934Ls+PyR/+CXRi933lE=,tag:cYLXvxP5hk7/6gmz/CCP0A==,type:str]
THERMOGRAPH_COOKIE_SECURE: ENC[AES256_GCM,data:zg==,iv:sVr33CvZoZmn0LcHlP+FVA8esjGy0aKe1A2iPrrrT0I=,tag:3YneNF7m05fCRGfMgYzByA==,type:str]
THERMOGRAPH_MAIL_BACKEND: ENC[AES256_GCM,data:pRBtOu6cTw==,iv:88QgyY6rwwp/6Y32r6Us1M+b7YpSLPfKIsCL2j/YuIY=,tag:FrchZZrMcCSeOFRWbePM4Q==,type:str]
THERMOGRAPH_DISCORD_BOT: ENC[AES256_GCM,data:NQ==,iv:Zl888EtEoQQd8FdqGb4/N7gGMT4yEymQfWd2fZZCNVA=,tag:Xspr8/taeQwh1qHhfFbLfw==,type:str]
POSTGRES_PASSWORD: ENC[AES256_GCM,data:l076hUutOAKw7+X6wboX,iv:9WVz3SVUikWREVjB6GsJ+EXWO3PvX+g+OaZNGN9f+jU=,tag:OkVuEqEbTSikpCNEySPX5A==,type:str]
THERMOGRAPH_DATABASE_URL: ENC[AES256_GCM,data:kXCYmZXewa8JSY5iJd0cHDtsKrxyKEl5tSVwWeQEd8u7XU+/dn4BExT4RDB3vUWVFVoHXYIXedMnDO8h0QfDCxSI13s=,iv:s1bDjdH4E4T+kF126saOrKlx3AnKEkRWF68KEPcCUe0=,tag:nCzD9U/7g28bJmUXkiYh3Q==,type:str]
THERMOGRAPH_AUTH_SECRET: ENC[AES256_GCM,data:EjwKGKh1od5vgvm8LLv1l6TXO7nshUwh5ihXDpETl+3ldtF8L4H3CWszsw==,iv:4mqwEXWpGpLYbIUotHj9Qf7FSD0ATwkxwhfuL7pG+Zc=,tag:0iLU0ee86H0CgmgPCCtatQ==,type:str]
sops:
age:
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBzV0lrcUpjeU81TmJDRVpa
elppeU1mMUlpc3lKSTVtU3MxUi83bzN2dVVRCmxZbDJFUkJQVjhCSmljNFpPaWVu
dkNqTkJTd1ZEaHZsZitpTWlJdElCK28KLS0tIEdSTXN6S1Y2eE12U3o2RGxBNXNN
bDBlYkNxMDVTVEFLYVIyVkhIRnNwOGMKv/TMt307XqvzBLCOCA5C4kXFV9iJeVBn
7gyzb5MRbHbDoNAY/5ckU3341uWUXUQ+IrPvVt2snAdXIlWghm+HwA==
-----END AGE ENCRYPTED FILE-----
recipient: age1xx4dzs0dxlwvkv9sjuqzsphl7lfrxannkfken374yu2qvvcte9sqzktqt2
lastmodified: "2026-07-25T00:59:09Z"
mac: ENC[AES256_GCM,data:Ga1lMooQrBhU+YasAbN3THZvjrct21KRbuxOoXf+72K5mrgPmqZyFDdB71B8dOUzmz4cqTPw3+jQx2s2ZrZCEiR3qmb897I8hWbTiFWnURSC7rjytxIe6kLhDBjCtso8ThKUMqNjAglqjkytHXljP7Cg4zCuGLEgczoakEJVB7s=,iv:NFHljXdG619XJMjy+1OuUknu+QaG5nH68fvo/zAf/TE=,tag:MyFf/edujKzypOwloVUGAA==,type:str]
unencrypted_suffix: _unencrypted
version: 3.13.2

99
infra/deploy/secrets/seed-centralis-on-host.sh Executable file
View file

@ -0,0 +1,99 @@
#!/usr/bin/env bash
# Seed deploy/secrets/centralis.<env>.yaml by encrypting prod's live
# /etc/centralis.env ENTIRELY ON THE BOX. The plaintext never leaves the box;
# only already-encrypted ciphertext is written back here. Encryption uses the
# age PUBLIC key (safe to hardcode), so the machine you run this from needs
# nothing but SSH access — no sops, no age, no private key.
#
# deploy/secrets/seed-centralis-on-host.sh prod agent@169.58.46.181
#
# Then commit the resulting deploy/secrets/centralis.<env>.yaml.
#
# ---------------------------------------------------------------------------
# WHY THIS IS NOT seed-encrypt-on-host.sh WITH A DIFFERENT PATH
# ---------------------------------------------------------------------------
# `seed-encrypt-on-host.sh` captures the RAW TEXT to the right of the `=`. That
# is correct for /etc/thermograph.env, which is machine-rendered and whose 32
# values happen to be bare tokens, but it is WRONG for a hand-maintained,
# shell-SOURCED file. /etc/centralis.env is consumed as
#
# set -a && . /etc/centralis.env && set +a && docker compose up
#
# so the value the program actually receives is the value AFTER bash's quote
# removal. `CENTRALIS_TOKENS` is stored there single-quoted, because it is JSON
# and contains double quotes. Capturing the raw text would put the surrounding
# single quotes INSIDE the vault; the renderer would then quote them again, and
# Centralis would receive `'{"emi":...}'` — literal quotes and all — and fail
# its JSON parse. Fails closed, so it would look like the token registry had
# been deleted. That is the same class of bug as the one this whole migration
# exists to remove, arriving from the other direction.
#
# So this script asks BASH what the values are, by sourcing the file in a clean
# environment (`env -i`) and reading the resulting variables. What goes into the
# vault is exactly what Centralis receives today, by construction.
set -euo pipefail
cd "$(dirname "$(readlink -f "$0")")/../.."
ENV_NAME="${1:?usage: seed-centralis-on-host.sh <env> <ssh-target> [ssh-key]}"
SSH_TARGET="${2:?ssh target, e.g. agent@169.58.46.181}"
SSH_KEY="${3:-$HOME/.ssh/thermograph_agent_ed25519}"
OUT="deploy/secrets/centralis.${ENV_NAME}.yaml"
# Public age recipient — NOT a secret (matches .sops.yaml). The private key never
# leaves the operator's machine / the hosts' /etc/thermograph/age.key.
AGE_PUB="age1xx4dzs0dxlwvkv9sjuqzsphl7lfrxannkfken374yu2qvvcte9sqzktqt2"
SOPS_URL="https://github.com/getsops/sops/releases/download/v3.13.2/sops-v3.13.2.linux.amd64"
echo "==> Encrypting ${SSH_TARGET}:/etc/centralis.env on-host -> ${OUT}"
ssh -i "$SSH_KEY" "$SSH_TARGET" "AGE_PUB='$AGE_PUB' SOPS_URL='$SOPS_URL' bash -s" > "$OUT" <<'REMOTE'
set -euo pipefail
exec 3>&1 # real stdout carries ONLY the ciphertext
{ # setup noise -> stderr, so it can't corrupt the file
if ! command -v sops >/dev/null 2>&1; then
sudo curl -fsSL "$SOPS_URL" -o /usr/local/bin/sops
sudo chmod +x /usr/local/bin/sops
fi
} >&2
umask 077
tmp="$(mktemp)"; tmy="$(mktemp)"; tpy="$(mktemp)"
trap 'rm -f "$tmp" "$tmy" "$tpy"' EXIT
sudo cat /etc/centralis.env > "$tmp" # plaintext stays on this box only
# The declared assignment targets, in file order. `source` gives last-wins for a
# repeated key and we read the post-source environment, so the de-dup here only
# controls the ORDER of the emitted YAML, never which value is taken.
names="$(grep -oE '^[[:space:]]*(export[[:space:]]+)?[A-Za-z_][A-Za-z0-9_]*=' "$tmp" \
| sed -E 's/^[[:space:]]*(export[[:space:]]+)?//; s/=$//' | awk '!seen[$0]++')"
[ -n "$names" ] || { echo "!! no KEY=VALUE lines in /etc/centralis.env" >&2; exit 1; }
echo " capturing: $(echo "$names" | tr '\n' ' ')" >&2
cat > "$tpy" <<'PY'
import json, os, sys
for n in sys.argv[1:]:
v = os.environ.get(n)
if v is None:
sys.stderr.write("!! %s vanished after sourcing\n" % n)
sys.exit(1)
print("%s: %s" % (n, json.dumps(v))) # JSON-escaped scalar = valid YAML
PY
# Source in a CLEAN environment and read the values back, so what is stored is
# exactly what `set -a && . /etc/centralis.env` hands to docker compose. PATH is
# passed through only so python3 is findable. $names is deliberately unquoted:
# it is a whitespace-separated list of identifiers by construction.
# shellcheck disable=SC2086
env -i PATH="$PATH" bash -c \
'set -a; . "$1"; set +a; shift; exec python3 "$@"' _ "$tmp" "$tpy" $names > "$tmy"
[ -s "$tmy" ] || { echo "!! captured nothing" >&2; exit 1; }
cd /tmp # no .sops.yaml here, so --age is honored
sops -e --age "$AGE_PUB" --input-type yaml --output-type yaml "$tmy" >&3
REMOTE
if grep -q 'ENC\[AES256_GCM' "$OUT"; then
echo "==> OK: ${OUT} written and encrypted ($(grep -cE '^[A-Za-z_][A-Za-z0-9_]*: ' "$OUT") keys)"
echo " Next: infra/deploy/secrets/verify-centralis-render.sh ${ENV_NAME} ${SSH_TARGET}"
else
echo "!! ${OUT} is not encrypted (the on-host step failed) — removing" >&2
rm -f "$OUT"; exit 1
fi

152
infra/deploy/secrets/verify-centralis-render.sh Executable file
View file

@ -0,0 +1,152 @@
#!/usr/bin/env bash
# NON-DESTRUCTIVE go/no-go for the /etc/centralis.env cutover — the Centralis
# analogue of dry-run-render.sh, and the proof obligation for the migration:
# that rendering deploy/secrets/centralis.<env>.yaml produces a file whose
# EFFECTIVE VALUES are identical to the live hand-maintained one.
#
# deploy/secrets/verify-centralis-render.sh prod agent@169.58.46.181
#
# What it does NOT do: write /etc/centralis.env, restart anything, or print a
# single secret value. It renders to a 0700 scratch directory, sources both
# files in a clean shell, compares the resulting variables, and reports key
# NAMES and a verdict. The scratch directory is removed on every exit path.
#
# "Effective values" is the load-bearing phrase. /etc/centralis.env is consumed
# by `set -a && . /etc/centralis.env`, so the only comparison that means
# anything is between what BASH ends up with from each file — not between the
# two files' text. The live file quotes CENTRALIS_TOKENS and the rendered one
# quotes everything, so a textual diff would report a difference where there is
# none, and (much worse) would report a match if the vault had captured the
# quote characters as part of the value.
set -euo pipefail
cd "$(dirname "$(readlink -f "$0")")/../.."
ENV_NAME="${1:?usage: verify-centralis-render.sh <env> <ssh-target> [ssh-key]}"
SSH_TARGET="${2:?ssh target, e.g. agent@169.58.46.181}"
SSH_KEY="${3:-$HOME/.ssh/thermograph_agent_ed25519}"
VAULT="deploy/secrets/centralis.${ENV_NAME}.yaml"
[ -f "$VAULT" ] || { echo "!! no $VAULT — run seed-centralis-on-host.sh first" >&2; exit 1; }
SSH=(ssh -i "$SSH_KEY" "$SSH_TARGET")
SCP=(scp -q -i "$SSH_KEY")
scratch="$("${SSH[@]}" 'd=$(mktemp -d) && chmod 700 "$d" && mkdir -p "$d/deploy/secrets" && echo "$d"')"
[ -n "$scratch" ] || { echo "!! could not create a scratch dir on the host" >&2; exit 1; }
cleanup() { "${SSH[@]}" "rm -rf -- '$scratch'" >/dev/null 2>&1 || true; }
trap cleanup EXIT
# Only ciphertext and a shell script cross the wire. Nothing decrypted ever
# leaves the box; the render happens there, with the host's own age key.
"${SCP[@]}" deploy/render-secrets.sh "${SSH_TARGET}:${scratch}/deploy/render-secrets.sh"
"${SCP[@]}" "$VAULT" "${SSH_TARGET}:${scratch}/deploy/secrets/centralis.${ENV_NAME}.yaml"
"${SSH[@]}" "SCRATCH='$scratch' bash -s" <<'REMOTE'
set -euo pipefail
umask 077
. "$SCRATCH/deploy/render-secrets.sh"
CENTRALIS_RENDER_DEST="$SCRATCH/rendered.env" render_centralis_secrets "$SCRATCH" >&2
sudo cat /etc/centralis.env > "$SCRATCH/live.env"
# Ask bash what each file means, in a clean environment so no inherited variable
# can stand in for one the file failed to set.
dump() {
env -i PATH="$PATH" bash -c \
'set -a; . "$1"; set +a; exec python3 -c "
import json, os, sys
sys.stdout.write(json.dumps(dict(os.environ)))"' _ "$1"
}
dump "$SCRATCH/live.env" > "$SCRATCH/live.json"
dump "$SCRATCH/rendered.env" > "$SCRATCH/rend.json"
# The names each file actually assigns — so the shell's own PATH/PWD/SHLVL/_ are
# not mistaken for content.
names() {
grep -oE '^[[:space:]]*(export[[:space:]]+)?[A-Za-z_][A-Za-z0-9_]*=' "$1" \
| sed -E 's/^[[:space:]]*(export[[:space:]]+)?//; s/=$//' | sort -u
}
names "$SCRATCH/live.env" > "$SCRATCH/live.names"
names "$SCRATCH/rendered.env" > "$SCRATCH/rend.names"
python3 - "$SCRATCH" <<'PY'
import json, os, sys
d = sys.argv[1]
load = lambda n: json.load(open(os.path.join(d, n), encoding="utf-8"))
names = lambda n: [l.strip() for l in open(os.path.join(d, n), encoding="utf-8") if l.strip()]
live, rend = load("live.json"), load("rend.json")
ln, rn = set(names("live.names")), set(names("rend.names"))
lost, added = sorted(ln - rn), sorted(rn - ln)
changed = sorted(k for k in ln & rn if live.get(k) != rend.get(k))
print()
print(" live /etc/centralis.env : %d keys" % len(ln))
print(" rendered from the vault : %d keys" % len(rn))
print()
for k in sorted(ln & rn):
print(" %-34s %s" % (k, "same" if live.get(k) == rend.get(k) else "DIFFERENT"))
ok = not (lost or added or changed)
print()
if lost:
print(" MISSING from the render :", lost)
if added:
print(" ADDED by the render :", added)
if changed:
print(" VALUE CHANGED :", changed)
# The registry, by subject name. Values are never touched; `sorted(reg)` is a
# list of audit-log subjects, which is what the operator needs to see.
def subjects(env):
raw = env.get("CENTRALIS_TOKENS", "")
if not raw:
return None
try:
reg = json.loads(raw)
except ValueError as exc:
return "UNPARSEABLE (%s)" % exc
return sorted(reg) if isinstance(reg, dict) else "not a JSON object"
sl, sr = subjects(live), subjects(rend)
print(" CENTRALIS_TOKENS subjects live: %s" % (sl,))
print(" rendered: %s" % (sr,))
if sl != sr:
ok = False
print(" !! the token registry does not survive the migration")
print()
print(" VERDICT: %s" % ("PASS — the render is value-identical to the live file"
if ok else "FAIL — do NOT cut over"))
sys.exit(0 if ok else 1)
PY
# End-to-end: the two env files must produce the SAME docker compose model. This
# is the claim that actually matters — not "the files agree" but "the container
# Centralis would be started with is the same one it is running now". Compared
# by digest, on the box; the rendered model is full of secrets and is never
# printed, shipped, or kept.
compose=/opt/centralis/deploy/docker-compose.yml
if [ -f "$compose" ] && command -v docker >/dev/null 2>&1; then
model() {
env -i PATH="$PATH" HOME="$HOME" bash -c \
'set -a; . "$1"; set +a; cd /opt/centralis && \
docker compose -f deploy/docker-compose.yml config' _ "$1" 2>/dev/null \
| sha256sum | cut -d" " -f1
}
a="$(model "$SCRATCH/live.env")"
b="$(model "$SCRATCH/rendered.env")"
if [ -z "$a" ] || [ "$a" = "$(printf '' | sha256sum | cut -d' ' -f1)" ]; then
echo " compose model : could not be built from the LIVE env — check by hand"
elif [ "$a" = "$b" ]; then
echo " compose model : IDENTICAL (docker compose config digests match)"
else
echo " compose model : DIFFERENT — do NOT cut over"
exit 1
fi
else
echo " compose model : skipped (no /opt/centralis or no docker here)"
fi
REMOTE