diff --git a/infra/DEPLOY-DEV.md b/infra/DEPLOY-DEV.md index eba88bf..e66796b 100644 --- a/infra/DEPLOY-DEV.md +++ b/infra/DEPLOY-DEV.md @@ -47,6 +47,7 @@ old in-workflow `ci-cd.yml` (retired along with the rest of `.github/`). | `.forgejo/workflows/deploy-dev.yml` | build + deploy on pushes to `dev` (direct, or a PR merge) | | `.forgejo/workflows/build.yml` | shared build gate: deps, backend tests, JS syntax check, boot/health check | | `deploy/deploy-dev.sh` | pull `dev` into `~/thermograph-dev`, `docker compose up` the uncapped LAN stack, health check | +| `deploy/secrets/dev.yaml` | dev's SOPS vault — its own secrets and config, rendered to `/etc/thermograph.env` at deploy time (see "Config and secrets") | | `docker-compose.dev.yml` | dev overlay: no CPU limits, backend published on `0.0.0.0:8137` (frontend unpublished) | | `deploy/provision-dev-lan.sh` | one-time sudo-free bootstrap of the checkout + runner | | `deploy/forgejo/register-lan-runner.sh` | registers the Forgejo Actions runner on this machine | @@ -119,10 +120,54 @@ Bootstrap (or rebuild) it by hand: bash deploy/provision-dev-lan.sh ``` -Config: port/base path are baked into the unit from `deploy/deploy-dev.sh` -(`PORT`, `THERMOGRAPH_BASE`). Change them there and redeploy. The dedicated -checkout at `~/thermograph-dev` is separate from your working tree, so a deploy -never touches uncommitted edits in `~/Code/Thermograph`. +The dedicated checkout at `~/thermograph-dev` is separate from your working tree, +so a deploy never touches uncommitted edits in `~/Code/Thermograph`. + +## Config and secrets + +Dev's configuration lives in the same SOPS vault as prod's and beta's — +`deploy/secrets/dev.yaml`, encrypted to the same age recipient, rendered to +`/etc/thermograph.env` by the same `render-secrets.sh` at deploy time. Changing a +dev value is `sops deploy/secrets/dev.yaml` → commit → deploy, exactly as for the +VPS boxes. **`deploy/secrets/README.md` is the reference**; what follows is only +what is specific to this machine. + +Dev renders `dev.yaml` **alone**. It does *not* layer `common.yaml`, which is the +fleet's shared production credential set — registry token, both S3 keypairs (one +read-write pair, on the bucket that also holds prod's backups), the VAPID private +key that signs push to real subscribers, the IndexNow key, the metrics token. This +box is the CI runner and runs unreviewed `dev`-branch code; a plaintext render of +that set into `/etc/thermograph.env` would put all of it in every dev container's +environment. `deploy-dev.sh` exports `THERMOGRAPH_SECRETS_SKIP_COMMON=1` to prevent +it, and **aborts the deploy** if the renderer in the checkout cannot honour that. + +So dev holds its own `THERMOGRAPH_AUTH_SECRET` (its own, not prod's — the `daemon` +service refuses to start without one), its own `POSTGRES_PASSWORD`, a LAN +`THERMOGRAPH_BASE_URL`, `THERMOGRAPH_COOKIE_SECURE=0` (plain HTTP — the shared +value is `1` and would silently break login here), and the non-secret config it +would otherwise have inherited. It holds **no** S3 keys, no VAPID keypair, no +IndexNow key, no Discord or mail credentials: the lake falls through to the +Open-Meteo archive without bucket creds, and the app generates and persists its own +VAPID and IndexNow keys into the `appdata` volume. + +Two dev-only mechanics, both because this box has **no passwordless sudo** and the +runner is a `systemd --user` service with no tty: + +- the age key is read from `~/.config/sops/age/keys.txt`, not + `/etc/thermograph/age.key` — a root-owned `0400` key would send the renderer down + its `sudo cat` path, where it would hang on a prompt no CI job can answer; +- `/etc/thermograph.env` is **pre-created** owned by the deploying user, so the + renderer takes its in-place write path and no deploy needs `sudo` at all. + +Setting this up on a fresh dev machine is four typed commands — see **"Setting up a +new dev machine"** in `deploy/secrets/README.md`. Until the +`/etc/thermograph/secrets-env` marker exists the box renders nothing and falls back +to `deploy-dev.sh`'s built-in `POSTGRES_PASSWORD` default, which is the safe state +and the one to return to (delete the marker) if the vault ever gets in the way. + +`REGISTRY_TOKEN` is deliberately not in the vault: dev pulls with the persistent +`docker login` credential already in this host's docker config, like the prod/beta +CI paths do. ## Before monitoring a PR to land @@ -168,5 +213,8 @@ container/app logs under `host="dev"`, so LAN dev shows up alongside prod and beta in the same dashboards. This replaced the old `scripts/dashboard.py`. For a quick terminal check without Grafana, the app still serves raw counters at -the gated `GET /api/v2/metrics` route (loopback-only; set -`THERMOGRAPH_METRICS_TOKEN` to read it over an SSH tunnel). +the gated `GET /api/v2/metrics` route. Dev sets no `THERMOGRAPH_METRICS_TOKEN`, so +the route is direct-loopback-only — `curl localhost:8137/api/v2/metrics` from this +machine, which is where you already are. (A token exists on the VPS boxes to allow +reads through an SSH tunnel; dev has no reason to carry a copy, and the estate's +copy must not land here — see `deploy/secrets/README.md`.) diff --git a/infra/deploy/deploy-dev.sh b/infra/deploy/deploy-dev.sh index f4e3d70..08c501f 100755 --- a/infra/deploy/deploy-dev.sh +++ b/infra/deploy/deploy-dev.sh @@ -50,25 +50,82 @@ export COMPOSE_FILE="docker-compose.yml:docker-compose.dev.yml" # var wins over the compose-file `name:` key, preserving that split. export COMPOSE_PROJECT_NAME="${COMPOSE_PROJECT_NAME:-thermograph-dev}" -# deploy.sh needs POSTGRES_PASSWORD to interpolate the compose file (both to init -# the db container and to build backend's THERMOGRAPH_DATABASE_URL) -- normally -# supplied by /etc/thermograph.env via render-secrets.sh's SOPS render, but dev has -# no entry in deploy/secrets/ (only common/beta/prod.yaml exist), so -# render_thermograph_secrets degrades to a no-op here (see render-secrets.sh) and -# /etc/thermograph.env likely doesn't exist on a fresh dev box either. Postgres is -# loopback-only inside the compose network on dev (never published, no real -# credentials on this box), so a fixed default is fine -- same value the monorepo's -# deploy-dev.sh used historically. Override via the environment if you ever want a -# different one; this only sets it when nothing else already has. +# --- Secrets ----------------------------------------------------------------- +# dev renders from the SOPS vault the same way prod and beta do -- same files, same +# tool, same edit/commit/deploy loop -- with two dev-only adjustments. Both are +# expressed as environment that render-secrets.sh already parameterises, so nothing +# about the rendering logic is forked. See deploy/secrets/README.md. + +# (1) dev renders deploy/secrets/dev.yaml ALONE; it does NOT layer common.yaml. +# +# common.yaml is the internet-facing fleet's SHARED PRODUCTION credential set: the +# registry token, both S3 keypairs (the bucket keypair is read-write, and that +# bucket also holds prod's database backups), the VAPID private key that signs push +# to real subscribers, the IndexNow key, the metrics token. Layering it here would +# write all of them in plaintext to /etc/thermograph.env on the box that is also the +# Forgejo CI runner, and hand them to every container in the dev stack through +# `env_file:` -- that is, to whatever unreviewed branch dev happens to be running. +# It would also silently apply THERMOGRAPH_COOKIE_SECURE=1, which is correct for the +# TLS hosts and breaks login on dev's plain-HTTP LAN URL. +# +# dev.yaml is self-contained instead: it carries dev's own copy of everything dev +# genuinely needs, and nothing else. +export THERMOGRAPH_SECRETS_SKIP_COMMON=1 + +# (2) The age key is read from wherever it is READABLE, not necessarily +# /etc/thermograph/age.key. render-secrets.sh falls back to `sudo cat` for a +# root-owned 0400 key, and this box has no passwordless sudo -- under the Forgejo +# runner (a systemd --user service, no tty) that sudo cannot prompt, so the +# fleet-standard placement would make every dev deploy fail at the render. Prefer +# the standard path when it is readable; otherwise the operator's own keyring, which +# is where dev's key already lives (this box is the machine `sops` is run on), so +# provisioning dev needs no second copy of the estate's recovery root. +if [ -z "${THERMOGRAPH_AGE_KEY:-}" ] && [ ! -r /etc/thermograph/age.key ] \ + && [ -r "$HOME/.config/sops/age/keys.txt" ]; then + export THERMOGRAPH_AGE_KEY="$HOME/.config/sops/age/keys.txt" +fi + +# Fail closed rather than leak. If this box is provisioned to render (an env marker +# AND a key) but the render-secrets.sh in this checkout predates +# THERMOGRAPH_SECRETS_SKIP_COMMON, the render would quietly concatenate common.yaml +# into dev's /etc/thermograph.env -- exactly the leak (1) exists to prevent, with a +# success exit code. Checking a sibling script for the flag is blunt, but the two +# files are versioned together in this same checkout, and the alternative failure +# mode is unreviewed code running with the estate's object-storage keys. Remove this +# once the flag is unconditionally present in render-secrets.sh. +_render_sh="$(dirname "$0")/render-secrets.sh" +_secrets_marker="${THERMOGRAPH_SECRETS_ENV_FILE:-/etc/thermograph/secrets-env}" +_age_key="${THERMOGRAPH_AGE_KEY:-/etc/thermograph/age.key}" +if [ -s "$_secrets_marker" ] && [ -f "$_age_key" ] && [ -f "$_render_sh" ] \ + && ! grep -q 'THERMOGRAPH_SECRETS_SKIP_COMMON' "$_render_sh"; then + echo "!! This box is provisioned to render SOPS secrets (env marker + age key)," >&2 + echo "!! but $_render_sh does not honour" >&2 + echo "!! THERMOGRAPH_SECRETS_SKIP_COMMON, so it would render common.yaml -- the" >&2 + echo "!! fleet's shared PRODUCTION credentials -- into /etc/thermograph.env on the" >&2 + echo "!! CI-runner box. Refusing to deploy." >&2 + echo "!! Fix: update deploy/render-secrets.sh, or remove $_secrets_marker to fall" >&2 + echo "!! back to the un-vaulted dev path." >&2 + exit 1 +fi + +# deploy.sh needs POSTGRES_PASSWORD to interpolate the compose file (both to init the +# db container and to build backend's THERMOGRAPH_DATABASE_URL). On a provisioned dev +# box it arrives from dev.yaml via the render above -- deploy.sh sources +# /etc/thermograph.env AFTER calling the renderer, so the vault value wins over this +# line. This default is the fallback for the un-vaulted paths that remain: a fresh +# box before provisioning, and a bare `make dev-up`. It is the same value dev.yaml +# holds and the value dev's existing pgdata volume was initialized with -- keep the +# two in step, and note that changing it needs an ALTER ROLE against the running +# database as well (see deploy/secrets/README.md), not just an edit. export POSTGRES_PASSWORD="${POSTGRES_PASSWORD:-thermograph-dev}" -# REGISTRY_TOKEN (for `docker login` in deploy.sh) is a REAL credential -- read -# access to git.thermograph.org's registry -- and deliberately has NO default here. -# Until dev gets its own deploy/secrets/dev.yaml entry (see deploy/secrets/README.md) -# it must come from the calling environment: exported by hand for a manual run, or -# injected by whatever eventually SSHes in to trigger a dev deploy (analogous to -# how deploy.yml supplies it for prod/beta today). deploy.sh's own `docker login` -# step fails loudly if it's unset or wrong -- nothing silently no-ops. +# REGISTRY_TOKEN (for `docker login` in deploy.sh) is deliberately NOT in dev.yaml +# and has no default here. dev pulls with the persistent `docker login` credential +# already in this host's docker config -- the same arrangement the prod/beta CI +# deploy paths use, and the reason deploy.sh's login step is conditional. Putting a +# registry token in dev's vault would only add a second copy of a credential the box +# already has, in a file more processes can read. Export it by hand for a one-off run +# on a box that has never logged in; a genuine auth problem fails loudly at `pull`. echo "==> LAN-dev deploy: APP_DIR=$APP_DIR BRANCH=$BRANCH COMPOSE_FILE=$COMPOSE_FILE" exec "$(dirname "$0")/deploy.sh" "$@" diff --git a/infra/deploy/render-secrets.sh b/infra/deploy/render-secrets.sh index 3689162..834fa51 100755 --- a/infra/deploy/render-secrets.sh +++ b/infra/deploy/render-secrets.sh @@ -84,7 +84,21 @@ render_thermograph_secrets() { # Decrypt failures return 1 explicitly, so a partial env is never written and # correctness doesn't depend on the caller's shell options. common first, host # second, so a host value overrides a shared one (last-wins). - if [ -f "$repo/deploy/secrets/common.yaml" ]; then + # + # THERMOGRAPH_SECRETS_SKIP_COMMON=1 renders the host file ALONE. deploy-dev.sh + # sets it, and the reason is what common.yaml contains: eleven of its sixteen + # values are live production credentials — both S3 keypairs (one is read-write + # on the bucket holding prod's database backups), the VAPID private key that + # signs Web Push to real subscribers, REGISTRY_TOKEN, the IndexNow and metrics + # tokens. This render is a plaintext concatenation consumed via `env_file:`, so + # layering common on dev would put every one of those into the environment of + # every container on a desktop that is also the Forgejo CI runner, executing + # unreviewed dev-branch code with the docker socket mounted. + # + # An override in dev.yaml would not fix it: last-wins governs *consumers*, but + # the production value is still physically a line in the file. + if [ -f "$repo/deploy/secrets/common.yaml" ] \ + && [ "${THERMOGRAPH_SECRETS_SKIP_COMMON:-0}" != 1 ]; then env "${key_env[@]}" sops -d --input-type yaml --output-type dotenv \ "$repo/deploy/secrets/common.yaml" >> "$tmp" || { rm -f "$tmp"; return 1; } fi @@ -116,3 +130,280 @@ render_thermograph_secrets() { rm -f "$tmp" return "$rc" } + +# ============================================================================ +# Centralis — render /etc/centralis.env from deploy/secrets/centralis..yaml +# ============================================================================ +# +# Centralis (the estate's control plane, prod-only) keeps its own env file, +# deliberately separate from the app stack's /etc/thermograph.env. This function +# brings that file under the same vault, so its four credentials +# (CENTRALIS_AUTH_TOKEN, CENTRALIS_TOKENS, CENTRALIS_FORGEJO_TOKEN, +# DISCORD_TOKEN) stop being hand-edited on the box. +# +# --------------------------------------------------------------------------- +# WHY THIS IS A SEPARATE FUNCTION, NOT AN ARGUMENT TO THE ONE ABOVE +# --------------------------------------------------------------------------- +# The two files have DIFFERENT OUTPUT CONTRACTS, and that is the whole reason +# this migration is being done. +# +# /etc/thermograph.env is consumed by docker-compose `env_file:`, by the systemd +# unit's EnvironmentFile=, and by the stack containers' `. /host/thermograph.env`. +# It is written as raw `KEY=value` — sops' own dotenv output, unquoted. +# +# /etc/centralis.env is consumed by EXACTLY ONE thing, and it is a shell: +# +# sudo bash -c 'set -a && . /etc/centralis.env && set +a && docker compose up' +# +# `set -a; . file` runs every line through bash's full word-expansion pipeline: +# quote removal, parameter expansion, command substitution, field splitting. +# Raw dotenv output is therefore NOT a safe representation for this file, and +# sops cannot make it one — `sops -d --output-type dotenv` emits values verbatim. +# On a value of {"emi":"..."} that yields +# +# CENTRALIS_TOKENS={"emi":"..."} -> bash strips the quotes -> {emi:...} +# +# which is exactly the hand-edit that took the token registry out on 2026-07-24: +# Centralis rejected the malformed JSON, failed closed as designed, and served a +# single `shared` identity — indistinguishable from the file never having been +# written. A value containing a space would truncate; one containing a backtick +# or $( ) would EXECUTE. So this function does not reuse the dotenv path at all. +# It decrypts to JSON (lossless, unambiguous — dotenv escapes a newline to a +# literal \n and cannot be told apart from a literal backslash-n) and emits +# POSIX single-quoted assignments, then PROVES the result by sourcing it in a +# clean shell and comparing every value back against the plaintext before the +# file is allowed anywhere near /etc. +# +# Making render_thermograph_secrets take an output format would put a flag on +# the app stack's deploy path whose wrong value is a production outage, to save +# a few lines. It also could not be changed to emit quoted values for BOTH +# files: /etc/thermograph.env's consumers include Centralis's own secrets tools, +# which compare rendered values against the live file textually, so quoting it +# would report all 32 keys as value-changed. Two consumers, two formats. +# +# --------------------------------------------------------------------------- +# WHY common.yaml IS NOT READ HERE +# --------------------------------------------------------------------------- +# Not an oversight. Centralis shares no key with the app stack (verified: zero +# overlap between the two live env files). Concatenating common.yaml would put +# the VAPID private key, both S3 keypairs and REGISTRY_TOKEN into +# /etc/centralis.env and into the shell that runs Centralis's compose — 16 +# credentials handed to a service that wants none of them, for no benefit. +# +# --------------------------------------------------------------------------- +# WHY THIS ONE FAILS LOUDLY WITH NO "not configured here" PATH +# --------------------------------------------------------------------------- +# render_thermograph_secrets is called unconditionally by every deploy on every +# box, so it needs the (a) branch for hosts that have not been migrated. This +# function is called only by Centralis's own deploy, on the one host that runs +# Centralis. A caller asking for it has already asserted it should happen, so +# every missing prerequisite is an error. There is no silent no-op path, because +# a silent no-op here means deploying against a stale hand-edited file — the +# state this exists to end. +render_centralis_secrets() { + local repo="${1:-.}" # dir CONTAINING deploy/secrets + local key="${THERMOGRAPH_AGE_KEY:-/etc/thermograph/age.key}" + local marker="${THERMOGRAPH_SECRETS_ENV_FILE:-/etc/thermograph/secrets-env}" + # Overridable so the pre-cutover dry run can render somewhere harmless and diff + # it against the live file. Nothing in the deploy path sets it. + local dest="${CENTRALIS_RENDER_DEST:-/etc/centralis.env}" + local env_name vault + env_name=$(cat "$marker" 2>/dev/null || true) + + if [ -z "$env_name" ]; then + echo "!! no environment name at ${marker} — cannot tell which Centralis vault to render" >&2 + return 1 + fi + if [ ! -f "$key" ]; then + echo "!! no age key at ${key}; this host cannot decrypt the vault" >&2 + return 1 + fi + # Scoped by environment even though Centralis runs only on prod today. The + # alternative (one unscoped centralis.yaml) would render prod's bearer tokens + # onto any host that ran this, which is the wrong failure to leave lying around + # for the day Centralis gains a second home. Adding one is then a new file, + # not a redesign. + vault="$repo/deploy/secrets/centralis.${env_name}.yaml" + if [ ! -f "$vault" ]; then + echo "!! this host is configured for SOPS (env='${env_name}', age key present)" >&2 + echo "!! but no Centralis vault was found at ${vault}" >&2 + echo "!! pass the directory containing deploy/secrets — on the hosts that is" >&2 + echo "!! /opt/thermograph/infra — or add centralis.${env_name}.yaml to the vault." >&2 + return 1 + fi + command -v sops >/dev/null 2>&1 || { + echo "!! sops not installed but this host is configured for SOPS secrets" >&2; return 1; } + command -v python3 >/dev/null 2>&1 || { + echo "!! python3 not installed; needed to quote values safely for a sourced file" >&2; return 1; } + + echo "==> Rendering ${dest} from ${vault}" + # Same age-key handling as render_thermograph_secrets: read the 0400 root key + # directly when we can, else lift it through sudo into SOPS_AGE_KEY, so the key + # never has to be made readable by the deploy user. + local key_env=() + if [ -r "$key" ]; then + key_env=("SOPS_AGE_KEY_FILE=$key") + else + local keymat; keymat=$(sudo cat "$key" 2>/dev/null | grep '^AGE-SECRET-KEY-' || true) + [ -n "$keymat" ] || { echo "!! cannot read age key at $key (need sudo)" >&2; return 1; } + key_env=("SOPS_AGE_KEY=$keymat") + fi + + # ONE temp directory, 0700, holding plaintext — so cleanup is a single rm on + # every path. Deliberately NOT a `trap ... RETURN`: this file is SOURCED, and a + # RETURN trap set inside a sourced function persists into the caller's shell and + # re-fires when its next `source` completes (see the note in the function above). + local work rc=0 + work=$(mktemp -d) || { echo "!! mktemp -d failed" >&2; return 1; } + chmod 700 "$work" + + # The work happens in a subshell so that no failure path can skip the cleanup + # below, and so umask/cwd changes cannot leak into deploy.sh. + # + # Every step carries its own `|| exit 1` rather than relying on `set -e`. That + # is not belt-and-braces: `set -e` is DISABLED inside a compound command that + # is the left operand of `||`, which this subshell is (`( ... ) || rc=1`, needed + # so a failure here cannot abort a `set -e` caller mid-deploy). A `set -e` in + # here reads as protection and provides none — caught by a test that fed the + # renderer a nested mapping and watched it sail past the rejection. + ( + umask 077 + + # JSON, not dotenv. Lossless and unambiguous — see the header. + env "${key_env[@]}" sops -d --input-type yaml --output-type json "$vault" \ + > "$work/plain.json" || exit 1 + + python3 - "$work/plain.json" "$work/out.env" "$work/want.json" <<'EMIT' || exit 1 +import json, re, sys + +src, out_env, want_json = sys.argv[1], sys.argv[2], sys.argv[3] +NAME = re.compile(r"[A-Za-z_][A-Za-z0-9_]*\Z") + + +def shell_quote(v): + """POSIX single-quoting. Inside '...' every byte is literal except ' itself, + which is closed, backslash-escaped and reopened. There is no escape sequence + to get wrong and no character class to keep up to date: it is total, and + total is the only property worth having for a file bash will expand.""" + return "'" + v.replace("'", "'\\''") + "'" + + +def scalar(k, v): + if isinstance(v, str): + return v + if isinstance(v, bool): # before int: bool is an int in Python + return "true" if v else "false" + if isinstance(v, (int, float)): + return json.dumps(v) + if v is None: + return "" + sys.stderr.write( + "!! %s decrypts to a %s, but an env file holds only scalars\n" % (k, type(v).__name__)) + sys.exit(1) + + +data = json.load(open(src, encoding="utf-8")) +if not isinstance(data, dict): + sys.stderr.write("!! the vault did not decrypt to a mapping\n") + sys.exit(1) + +want, lines = {}, [] +for k, v in data.items(): + if not NAME.match(k): + sys.stderr.write("!! %r is not a usable shell variable name\n" % (k,)) + sys.exit(1) + want[k] = scalar(k, v) + lines.append("%s=%s\n" % (k, shell_quote(want[k]))) + +with open(out_env, "w", encoding="utf-8") as fh: + fh.write("# Rendered by infra/deploy/render-secrets.sh from the SOPS vault.\n") + fh.write("# DO NOT EDIT BY HAND: the next render overwrites this file, and a\n") + fh.write("# hand-edit is how the token registry was lost on 2026-07-24.\n") + fh.write("# Rotate by editing infra/deploy/secrets/centralis..yaml instead.\n") + fh.writelines(lines) +with open(want_json, "w", encoding="utf-8") as fh: + json.dump(want, fh) +sys.stderr.write(" %d keys quoted\n" % len(want)) +EMIT + + # PROVE IT. Source the rendered file the way the deploy does — in a clean + # environment, `set -a`, no inherited variables to mask a dropped key — and + # read every value back out. This is the test that the file cannot pass while + # carrying the 2026-07-24 bug, and it runs on every render, not once. + env -i PATH="$PATH" bash -c \ + 'set -a; . "$1"; set +a; exec python3 -c " +import json, os, sys +sys.stdout.write(json.dumps(dict(os.environ)))"' _ "$work/out.env" \ + > "$work/got.json" || exit 1 + + python3 - "$work/want.json" "$work/got.json" <<'VERIFY' || exit 1 +import json, sys + +want = json.load(open(sys.argv[1], encoding="utf-8")) +got = json.load(open(sys.argv[2], encoding="utf-8")) + +bad = [] +for k, v in want.items(): + if k not in got: + bad.append("%s: not set at all after sourcing" % k) + elif got[k] != v: + # NEVER print either value. The difference is the finding; the content is + # not, and this runs inside a deploy log. + bad.append("%s: survives sourcing as a DIFFERENT value (len %d -> %d)" + % (k, len(v), len(got[k]))) +if bad: + sys.stderr.write("!! the rendered file does not round-trip through bash:\n") + for b in bad: + sys.stderr.write("!! %s\n" % b) + sys.exit(1) + +# CENTRALIS_TOKENS is the reason for all of the above: JSON, full of double +# quotes, in a file bash expands. Centralis fails CLOSED on malformed JSON — it +# drops to the single `shared` identity, which looks exactly like the registry +# having never been configured. Refuse to write a file that would do that, +# rather than discovering it from a dashboard that is missing two people. +raw = got.get("CENTRALIS_TOKENS", "") +if raw: + try: + reg = json.loads(raw) + except ValueError as exc: + sys.stderr.write("!! CENTRALIS_TOKENS is not valid JSON after sourcing: %s\n" % exc) + sys.exit(1) + if not isinstance(reg, dict) or not reg: + sys.stderr.write("!! CENTRALIS_TOKENS must be a non-empty JSON object\n") + sys.exit(1) + if not all(isinstance(s, str) and isinstance(t, str) and t for s, t in reg.items()): + sys.stderr.write("!! CENTRALIS_TOKENS must map subject -> non-empty token string\n") + sys.exit(1) + # Subjects are names on audit lines, not credentials. Printing them is the + # whole point: it is how an operator sees at a glance that nobody was lost. + sys.stderr.write(" CENTRALIS_TOKENS parses: %d subject(s) — %s\n" + % (len(reg), ", ".join(sorted(reg)))) + +sys.stderr.write(" %d keys survive `set -a; . ` byte-for-byte\n" % len(want)) +VERIFY + ) || rc=1 + + # Only now, with a file that has been read back through bash and matched value + # for value, does anything touch $dest. + if [ "$rc" -eq 0 ]; then + # 0600, not 0640: this file is four bearer credentials for the estate's + # control plane and has no group that needs it. Prefer an in-place write so + # the existing owner/mode survive; fall back the same way the app renderer + # does, chowning on the sudo path so a non-root deploy user can still source + # what it just rendered. + if [ -f "$dest" ] && [ -w "$dest" ]; then + cat "$work/out.env" > "$dest" || rc=1 + elif install -m 0600 "$work/out.env" "$dest" 2>/dev/null; then : + elif sudo install -m 0600 -o "$(id -un)" -g "$(id -gn)" "$work/out.env" "$dest" 2>/dev/null; then : + else + echo "!! cannot write ${dest} (need file write access or passwordless sudo)" >&2 + rc=1 + fi + else + echo "!! render aborted; ${dest} left untouched" >&2 + fi + + rm -rf "$work" + return "$rc" +} diff --git a/infra/deploy/secrets/README.md b/infra/deploy/secrets/README.md index 129a5d7..7242fd3 100644 --- a/infra/deploy/secrets/README.md +++ b/infra/deploy/secrets/README.md @@ -14,9 +14,11 @@ no per-host duplication. | File | Holds | Encrypted? | |------|-------|------------| -| `common.yaml` | The 16 values identical on every host — VAPID keypair, metrics token, IndexNow key, `REGISTRY_TOKEN`, S3 endpoint/bucket and both S3 keypairs, plus shared non-secret config | ✅ | +| `common.yaml` | The 16 values identical on **prod and beta** — VAPID keypair, metrics token, IndexNow key, `REGISTRY_TOKEN`, S3 endpoint/bucket and both S3 keypairs, plus shared non-secret config. **Not layered under `dev`** — see below | ✅ | | `prod.yaml` | Prod's own — the three held-back credentials below, sizing (`APP_CPUS`, `DB_CPUS`, `DB_MEMORY`, `WORKERS`), `THERMOGRAPH_BASE_URL`, and the Discord + mail credentials that exist nowhere else | ✅ | | `beta.yaml` | Beta's own — the three held-back credentials, sizing, `THERMOGRAPH_BASE_URL` | ✅ | +| `dev.yaml` | The LAN dev box's own — **self-contained**, 12 values, no production credential among them | ✅ | +| `centralis.prod.yaml` | **Centralis's** nine variables, rendered to `/etc/centralis.env` on prod. A different service, a different output file, a different renderer — see below | ✅ | | `example.yaml` | Format reference / CI fixture (fake values) | ✅ | | `../../.sops.yaml` | Which age recipient files are encrypted to (plaintext config) | — | @@ -39,10 +41,179 @@ intentional and turn breaking it into a migration rather than an edit. If you ever *do* want them to differ, change one file. That is the whole point. +`dev.yaml` is the case that already differs: all three of its copies are its own, +generated on the desktop. The rule earned its keep the first time it was used. + At deploy the renderer concatenates `common.yaml` then `.yaml`, so a **host value wins** (env_file / `source` take the last occurrence of a duplicate key). `` comes from `/etc/thermograph/secrets-env` on the box (`prod`/`beta`/`dev`). -`dev` has no real secrets and is intentionally not wired in — see `deploy-dev.sh`. + +### `dev` is vaulted, but renders `dev.yaml` **alone** + +All three environments are managed here now. `dev` is the one that does **not** +get `common.yaml` layered under it: `deploy-dev.sh` exports +`THERMOGRAPH_SECRETS_SKIP_COMMON=1`, and the renderer skips the shared layer. + +That is not squeamishness about a dev box — it is what `common.yaml` contains. +Eleven of its sixteen values are live production credentials: `REGISTRY_TOKEN`, +`THERMOGRAPH_S3_ACCESS_KEY`/`_SECRET_KEY` and +`THERMOGRAPH_LAKE_S3_ACCESS_KEY`/`_SECRET_KEY` (one keypair, **read-write**, on the +bucket that also holds prod's database backups — see `../../ops/README.md`), +`THERMOGRAPH_VAPID_PRIVATE_KEY`/`_PUBLIC_KEY` (they sign Web Push to real +subscribers), `THERMOGRAPH_INDEXNOW_KEY`, `THERMOGRAPH_METRICS_TOKEN`. The render +is a **plaintext concatenation**, so layering it would put every one of those in +`/etc/thermograph.env` on the dev box and, through `env_file:`, into the +environment of every container in the dev stack. + +The dev box is the operator's desktop. It is also the Forgejo CI runner, whose +`docker`-labelled jobs get the host docker socket automounted, and the stack it +runs is the `dev` branch — code that has not been through the gate that guards +prod. Handing that the estate's read-write object-storage keys and the push +signing key is a strictly larger blast radius than anything the vault buys back. +An override in `dev.yaml` would not help: last-wins governs *consumers*, but the +production value is still physically a line in the file. + +One value in `common.yaml` is also simply **wrong** for dev: +`THERMOGRAPH_COOKIE_SECURE=1` is right for the TLS hosts and silently breaks login +over dev's plain-HTTP LAN URL. `dev.yaml` sets `0`. + +So `dev.yaml` is self-contained. Where the value is shared-but-harmless +(`PORT`, `THERMOGRAPH_BASE`, `TIMESCALEDB_TAG`) it carries its own copy — a +duplicated constant is the cheap half of this trade. + +**What `dev.yaml` holds** (12 keys): + +| Key | Why | +|-----|-----| +| `POSTGRES_PASSWORD`, `THERMOGRAPH_DATABASE_URL` | dev's own. Deliberately the weak, well-known `thermograph-dev` — the value dev's `pgdata` volume is already initialized with, and the DB is never published off the compose network. Kept per-host for the reason below, and *not* shared with prod | +| `THERMOGRAPH_AUTH_SECRET` | dev's **own**, freshly generated. Fixes a live crash loop: the `daemon` service refuses to start without it (it derives the `/internal/*` token from it), so dev's daemon had been restarting every 60s. Prod's value must never be here — it signs sessions and verification links | +| `THERMOGRAPH_BASE_URL` | `http://10.0.1.216:8137`. Unset, the app defaults to `https://thermograph.org`, so dev's IndexNow pings and Discord links claimed to be prod | +| `THERMOGRAPH_COOKIE_SECURE=0` | plain HTTP on the LAN | +| `THERMOGRAPH_MAIL_BACKEND=console`, `THERMOGRAPH_DISCORD_BOT=0` | explicit fail-safes. Both are already the code defaults; stating them means dev cannot start mailing or open a Discord gateway by inheriting a value | +| `PORT`, `THERMOGRAPH_BASE`, `TIMESCALEDB_TAG`, `DB_MEMORY`, `WORKERS` | non-secret config that would otherwise have come from `common.yaml` | + +**Deliberately absent, and why nothing breaks:** + +- **Both S3 keypairs.** Without them the `lake` service answers 503 and history + reads fall through to the Open-Meteo archive — an accelerator, never a + dependency (`docker-compose.yml`, `backend/data/climate.py`). +- **VAPID keypair.** `backend/notifications/push.py` resolves env → file → + generates once and persists to the `appdata` volume. Dev gets its own identity + for free; prod's private key stays off the box. +- **`THERMOGRAPH_INDEXNOW_KEY`.** Same env → file → self-generate ladder + (`backend/indexnow.py`). Prod's key is what authenticates URL submissions *for + thermograph.org*. +- **`THERMOGRAPH_METRICS_TOKEN`.** With no token `/api/v2/metrics` is + direct-loopback-only, which is exactly right on a LAN box; dev's Alloy agent + ships logs, not metrics, so nothing scrapes it. +- **`REGISTRY_TOKEN`.** dev pulls with the persistent `docker login` credential + already in the host's docker config, like the prod/beta CI paths. A vault copy + would only duplicate a credential the box already has into a more readable file. +- **All Discord and mail credentials.** Dev must not be able to act as the real + bot or send real mail. +- **`APP_CPUS`, `DB_CPUS`.** `docker-compose.dev.yml` `!reset`s both — dev runs + uncapped, so setting them would be a lie. (`DB_MEMORY` *is* live: the `db` + service's `mem_limit` is not reset.) + +If dev ever needs one of these for real, it gets **its own** — a dev Discord app, +a scoped read-only bucket key — never a copy of prod's. + +## Centralis: its own file, its own renderer + +`/etc/centralis.env` on prod is the control plane's env file — four real +credentials (`CENTRALIS_AUTH_TOKEN`, `CENTRALIS_TOKENS`, +`CENTRALIS_FORGEJO_TOKEN`, `DISCORD_TOKEN`) and five non-secret settings. It was +**hand-edited** until 2026-07-24, when a hand-edit wrote `CENTRALIS_TOKENS` as +bare JSON into a file that gets `source`d, bash stripped the double quotes, +`{"emi":"…"}` arrived as `{emi:…}`, and Centralis — correctly failing closed on +malformed JSON — dropped to the single `shared` identity. Nothing logged an +error. It looked exactly like the file had never been written. + +`centralis.prod.yaml` + `render_centralis_secrets()` in +[`../render-secrets.sh`](../render-secrets.sh) end that, by the only durable +route: **a human stops writing the file.** + +Seed / verify / rotate: + +```sh +# One-time seed from the live box (plaintext never leaves prod): +deploy/secrets/seed-centralis-on-host.sh prod agent@169.58.46.181 + +# Prove a render matches the live file before cutting over. Renders to a 0700 +# scratch dir on the host, sources both, compares — key NAMES and a verdict only: +deploy/secrets/verify-centralis-render.sh prod agent@169.58.46.181 + +# Thereafter, rotation is edit -> commit -> merge to main -> redeploy Centralis: +sops deploy/secrets/centralis.prod.yaml +``` + +Merging to `main` is what puts the new ciphertext on prod (`infra-sync.yml` +fast-forwards `/opt/thermograph`); the render happens when Centralis is +redeployed. See Centralis's own `deploy/README.md` §2 and §5. + +### Why `CENTRALIS_*` is not just added to `prod.yaml` + +It is the obvious move, and it hands the estate's control-plane bearer token to +every application container. + +`/etc/thermograph.env` is loaded into the app stack through `env_file:` and +through `. /host/thermograph.env` in the stack services' entrypoints. Every key +in it is in the environment of every container in the stack. `CENTRALIS_TOKENS` +and `CENTRALIS_AUTH_TOKEN` authenticate an endpoint that carries `sql_query` +with `write:true` and `run_on_host` over production, and `CENTRALIS_FORGEJO_TOKEN` +is repo-scope on the monorepo. Putting them in `prod.yaml` would mean that a +read-anything bug in the web backend, or a leaked container env dump, is a +foothold on the control plane. The two files are separate for a reason that +predates the vault, and the vault is not a reason to merge them. + +The converse — pointing Centralis's deploy at `/etc/thermograph.env` — has the +same problem from the other side, plus 32 unrelated variables in the shell that +runs its compose, plus a lifecycle coupling: every app deploy re-renders that +file, and Centralis and the app stack are deliberately deployed on different +cadences. + +### Why this file is fully quoted and `/etc/thermograph.env` is not + +Different consumers. + +`/etc/thermograph.env` is read by docker-compose `env_file:` and by systemd +`EnvironmentFile=`, which parse `KEY=value` literally. Raw `sops --output-type +dotenv` is the right representation for it, and it must stay that way: Centralis's +own `secrets_inventory` / `secrets_render` tools compare rendered values against +that file **textually**, so quoting it would report all 32 keys as value-changed. + +`/etc/centralis.env` is read by exactly one thing, and it is a shell: + +```sh +sudo bash -c 'set -a && . /etc/centralis.env && set +a && docker compose up -d --build' +``` + +`set -a; . file` runs every line through bash's full expansion pipeline. Raw +dotenv output is *unsafe* there and sops cannot make it safe — it emits values +verbatim. So `render_centralis_secrets` decrypts to **JSON** (lossless; dotenv +escapes a newline to a literal `\n` and cannot be told apart from a literal +backslash-n) and writes POSIX single-quoted assignments, in which every byte is +literal except `'` itself. + +Then it **proves the file** before letting it near `/etc`: it sources the +rendered file in a clean shell (`env -i`), reads every variable back, and +compares to the plaintext it started from. If any value does not survive, or if +`CENTRALIS_TOKENS` does not parse as a JSON object of subject → token, the +render aborts and the existing file is left untouched. The 2026-07-24 bug cannot +be written by this path, and cannot be written *through* it either. + +### Scoped by environment even though Centralis is prod-only + +The file is `centralis.prod.yaml`, not `centralis.yaml`, and the renderer takes +the `prod` from `/etc/thermograph/secrets-env` like everything else here. An +unscoped file would render prod's bearer tokens onto any host that ran the +function. If Centralis ever gains a second home, that is a new file rather than +a redesign — and the two boxes get different tokens, which is the point. + +`centralis.prod.yaml` is **not** layered under `common.yaml`. Centralis shares +no key with the app stack (verified: zero overlap between the two live env +files), so layering would only push the VAPID private key, both S3 keypairs and +`REGISTRY_TOKEN` into a file that wants none of them. ## The age key (the one thing that matters) @@ -50,9 +221,16 @@ value wins** (env_file / `source` take the last occurrence of a duplicate key). unrecoverable; leak it and every secret is compromised. - It is **never in the repo.** It lives at: - `~/.config/sops/age/keys.txt` on the operator's machine (for editing), and - - `/etc/thermograph/age.key` (`0400`) on each host that renders at deploy time. + - `/etc/thermograph/age.key` (`0400`) on each VPS that renders at deploy time. - **Back it up** in the password manager. The public recipient is in `.sops.yaml`. +On the **dev desktop** those two are the same file, on purpose. `render-secrets.sh` +falls back to `sudo cat` for a root-owned key, the desktop has no passwordless sudo, +and the Forgejo runner is a `systemd --user` service with no tty — a `/etc` copy +would make every CI-triggered dev deploy hang on a prompt it cannot answer. +`deploy-dev.sh` points `THERMOGRAPH_AGE_KEY` at the operator's keyring instead, so +the box keeps **one** copy of the recovery root rather than two. + ## Prerequisites (once per machine) Install `sops` + `age` (single static binaries). On a host, use @@ -73,9 +251,59 @@ ssh agent@169.58.46.181 'cd /opt/thermograph && git pull && deploy/deploy.sh' ``` A host-specific value (e.g. `POSTGRES_PASSWORD`) is the same, editing `prod.yaml` / -`beta.yaml` instead. **`POSTGRES_PASSWORD` is special**: the database only reads it on -a *fresh* volume, so also `ALTER ROLE thermograph PASSWORD '…'` inside the running -Postgres to match — the env change alone won't re-key an initialized DB. +`beta.yaml` / `dev.yaml` instead. **`POSTGRES_PASSWORD` is special**: the database +only reads it on a *fresh* volume, so also `ALTER ROLE thermograph PASSWORD '…'` +inside the running Postgres to match — the env change alone won't re-key an +initialized DB. (That applies to dev too. `dev.yaml` deliberately carries the value +dev's existing `pgdata` volume was initialized with, so nothing rotates the moment +dev is provisioned; rotating it later is the two-step above, plus the matching +default in `deploy-dev.sh`.) + +Dev is edited and deployed exactly like the others — `sops deploy/secrets/dev.yaml`, +commit, and the next `deploy-dev.sh` run (merge to `dev`, or by hand) renders it. + +## Setting up a new dev machine + +The desktop is hands-on: Centralis cannot SSH to it, so this is typed, not +automated. `provision-secrets.sh` is **not** the right tool here — it installs the +age key root-owned at `/etc/thermograph/age.key`, which the runner cannot read +(above). + +```sh +# 1. sops + age on PATH, and the age private key in the operator's keyring: +# ~/.config/sops/age/keys.txt (0600, restored from the password manager) +sops --version && age --version + +# 2. Name the environment. This marker is what switches rendering ON — until it +# exists the box renders nothing, which is the safe default. +sudo mkdir -p /etc/thermograph +printf 'dev\n' | sudo tee /etc/thermograph/secrets-env >/dev/null +sudo chmod 0644 /etc/thermograph/secrets-env + +# 3. Pre-create the render target owned by the deploying user, so no deploy ever +# needs sudo (the renderer writes in place when the file is writable — the same +# path beta's non-root `deploy` user takes). +sudo install -m 0600 -o "$USER" -g "$USER" /dev/null /etc/thermograph.env + +# 4. Dry-run the render before letting a deploy do it, and read the key names back. +# Nothing from common.yaml may appear here. +THERMOGRAPH_SECRETS_SKIP_COMMON=1 \ +SOPS_AGE_KEY_FILE=~/.config/sops/age/keys.txt \ + sops -d --input-type yaml --output-type dotenv deploy/secrets/dev.yaml | sed 's/=.*/=/' + +# 5. Deploy. deploy-dev.sh refuses to run if render-secrets.sh in the checkout +# cannot honour THERMOGRAPH_SECRETS_SKIP_COMMON, rather than leak. +APP_DIR=~/thermograph-dev bash deploy/deploy-dev.sh +``` + +To take a dev box **back** off the vault, delete `/etc/thermograph/secrets-env`: +presence detection turns rendering off and `deploy-dev.sh` falls back to its +built-in `POSTGRES_PASSWORD` default. + +A *different* dev machine (a second desktop, a laptop) needs its own values, not a +copy of this one's: give it its own `.yaml` and its own marker name, so the two +can never be confused for one host — the same argument as the three held-back +credentials above. ## Add a new secret @@ -83,6 +311,12 @@ Postgres to match — the env change alone won't re-key an initialized DB. 2. Add the app reader (`os.environ.get("THERMOGRAPH_NEW_THING")`). 3. Commit + deploy. It flows into `/etc/thermograph.env` automatically. +`common.yaml` reaches prod and beta only. If dev needs the new thing, decide +which it is and act on it: harmless config → copy it into `dev.yaml`; a real +credential → mint dev **its own** and put that in `dev.yaml`; neither → leave dev +without it and make sure the code degrades rather than crashes. Do not reach for +prod's copy. + ## Rotate the age identity itself (re-key) ```sh @@ -123,4 +357,10 @@ Order matters — values must match the live env exactly so `AUTH_SECRET` / VAPI - Rendering is **presence-detected**: a host without the age key + marker keeps its existing `/etc/thermograph.env`, so nothing breaks before a host is migrated. - Rendering happens **on the target box**, so CI/runners never see the age key or the - plaintext. + plaintext — **except on dev**, where the box *is* the runner. That is not a hole + this vault can close; it is the reason `dev.yaml` holds nothing that would matter + if the box were owned. Keep it that way: the day dev needs a real credential, + mint it a scoped one of its own. +- `deploy-dev.sh` **fails closed**: if the `render-secrets.sh` it finds cannot honour + `THERMOGRAPH_SECRETS_SKIP_COMMON`, it aborts the deploy rather than render the + shared production layer onto the dev box. diff --git a/infra/deploy/secrets/centralis.prod.yaml b/infra/deploy/secrets/centralis.prod.yaml new file mode 100644 index 0000000..9cb66eb --- /dev/null +++ b/infra/deploy/secrets/centralis.prod.yaml @@ -0,0 +1,24 @@ +CENTRALIS_AUTH_TOKEN: ENC[AES256_GCM,data:MoOrGKQoe0mFu0gk4koa0kX+I6SgcKSlDAh+motBvBy22kiWucumgrl9avCdepPP,iv:0Vn/8bld2+EZhcwEPepCnmn4ktXp63LJUrZ36uajZp8=,tag:cjM/AVlG21Mk36QB7TjfSw==,type:str] +CENTRALIS_FORGEJO_TOKEN: ENC[AES256_GCM,data:p1KKmZZHhB3IO9QdWe2CYuN+rho+lZf3aeADT2sxZrD5SYeA05O1VA==,iv:93QFRj4tNU44wo6eCn3oCGmLP2tm571S4IhVCXpSQk8=,tag:/yH0O5Y25RfLBmKOWWJkZQ==,type:str] +CENTRALIS_FORGEJO_URL: ENC[AES256_GCM,data:z3ZTFg/2F9ZeilolVXW2XlyjyQy9,iv:hX0lj94vXFG2lZCG3gykaE1oEjTsciREey7onyQgQsc=,tag:415VJUTErR9SG82mkS2jOg==,type:str] +CENTRALIS_LOKI_URL: ENC[AES256_GCM,data:cFzoMoUYhxEVIF/K8JArTB49GRAo,iv:BTgmKBOsTdnWcvYOww4Zve5dJd+ZnFnBbWv3y5V2Vi4=,tag:kZ6LKLIVXk4UOZYbAr483w==,type:str] +DISCORD_GUILD_ID: ENC[AES256_GCM,data:qUFkitxFHOZdA7h2xtNycUpdLA==,iv:RGfbTEdjWY4Wu9i8WQ0rAnDAwqLZZbTj/bsWavYEnjk=,tag:L2b2t+tLhGXpqmGcamBpNw==,type:str] +CENTRALIS_TRANSCRIPT_CHANNEL: ENC[AES256_GCM,data:poxdL903opQV5waTCnO4wM3eIg==,iv:m5VWJpWXb49qak0wYoS1/W7S22+BAl+AFzjTY2Zu0a8=,tag:8iILm7mM0S/9/aAG1DywkQ==,type:str] +CENTRALIS_VOICE_SUMMARY_CHANNEL: ENC[AES256_GCM,data:Ztvy9w8wpmEcROyRDJOosBJlWw==,iv:iugYisN38csVNIZamZ84mzXaPTABxU+I2yZ+CzL/ddQ=,tag:wNGJcoYN2obnoKY7uchYiw==,type:str] +DISCORD_TOKEN: ENC[AES256_GCM,data:hdrjzSqwKmUdmdLl8dOMUPzLxfZf3H4Z6D9b+PIn/nzapQAPECihrxHZswayinYV3nbcZKj2Qh3Aw2jzaP0OpYM6HFdZmosA,iv:nE4utRtwNCDEWOKjTaobpXJHWkXY6LQE1EvHGH+tUH0=,tag:+3UC7N8B1WCCPzBzmB4yog==,type:str] +CENTRALIS_TOKENS: ENC[AES256_GCM,data:nlDM6Y4rUuAiITo1iN/khDFiwCH8UlVA42SaB1d1a4CG7uVMZnl369MVTMWEXhe4GFOtXlSYgHt7XUMkVmCGQzfBwXed6VZcMPMD9XVkbdYz8+FtrOsf0KVQWvNOHQ6MnvOGMb/CVFBeo0A=,iv:OAw7hXOgAlUQWViCb/fkfqXaNEcyu7947ttbaqHYceM=,tag:sUoVL7Fac9c4dil/+bz7fw==,type:str] +sops: + age: + - enc: | + -----BEGIN AGE ENCRYPTED FILE----- + YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBGcitwT3BoSE9LWE5zejlh + WlZPYnJUWTlycC9nZnJaK2d3dmFLQWhsS1M4Ckx0c0hPaXVnaUZra3dFdFVJbm93 + N29XNXZtVUN6d3FjNFB5UDNseXNwMDgKLS0tIGZ4ckI1ejEyc2w5alJNSXF3YklW + cWZrUWdGd3BEdTYvZU5lV1BDcEE4Z0UK2Q/UGFJwBxLeSib1vJhfGAsR+RMLq144 + JEUSbfXvUHvuMyY94DhbXmh2WgqxWYQuvF9UrDLeDw14tNPIGoIpFg== + -----END AGE ENCRYPTED FILE----- + recipient: age1xx4dzs0dxlwvkv9sjuqzsphl7lfrxannkfken374yu2qvvcte9sqzktqt2 + lastmodified: "2026-07-25T01:02:49Z" + mac: ENC[AES256_GCM,data:HsxO525FU/AfZw6Y5nTrfPIefl1InTV4M7e03hGsxZRHS2Fwsdqiyl/swI2oUFt3tfEyn0RwNB9cEeG/yo4UPPY80E5wnBvv0kaV0gfibBNwBRuIbPJZR/vXm6k8+60shcCOrtvpby4zUmtuk84kEK4mHWpVzh4xs5eYY37la6I=,iv:yR1Zq5TusAdzt7JnarXqRU3X896JMXJzrJw7p6tTzWs=,tag:lK9ph8OJyF/W1E4CLX8AGw==,type:str] + unencrypted_suffix: _unencrypted + version: 3.13.2 diff --git a/infra/deploy/secrets/dev.yaml b/infra/deploy/secrets/dev.yaml new file mode 100644 index 0000000..eb84992 --- /dev/null +++ b/infra/deploy/secrets/dev.yaml @@ -0,0 +1,27 @@ +PORT: ENC[AES256_GCM,data:0LjaWw==,iv:3e8Q71QFlMAhQStgoiVh/c2ELI3+7UkWseAFv8BnElU=,tag:/wYfKmqErJpsLBjg1gs4Tw==,type:str] +THERMOGRAPH_BASE: ENC[AES256_GCM,data:Ww==,iv:e2yNIAAw4dtlahfbA+PCM/jyODKFwWFNfVpVJ1JoREc=,tag:QzoGwDgIwdA9OFAMIPej7Q==,type:str] +TIMESCALEDB_TAG: ENC[AES256_GCM,data:7Oj7OoxoaG51whg=,iv:PPbfdn4DHMxTPA5FUqv+lEY29wQsVX+g0aVuWvraYrU=,tag:NAyJZlRXU1iKo4jwBctUtw==,type:str] +DB_MEMORY: ENC[AES256_GCM,data:/3U=,iv:jFI/9NQaz8oIqBWAKydWUEEI7bI3SmLnsoBlVgfKw7I=,tag:oBoFANuTgZV0jXC9qR5IHQ==,type:str] +WORKERS: ENC[AES256_GCM,data:VQ==,iv:+d84ByEU22XVQi43NcRn8cHmbW9XudJMj0guidsHxBQ=,tag:XM//7y0VQCwini4CGL7pSg==,type:str] +THERMOGRAPH_BASE_URL: ENC[AES256_GCM,data:FK2kOaZG8ESwDi3VMOlo0XvBKvdADQ==,iv:c6GvCrAnAf+ZFh53xg3ZOlf934Ls+PyR/+CXRi933lE=,tag:cYLXvxP5hk7/6gmz/CCP0A==,type:str] +THERMOGRAPH_COOKIE_SECURE: ENC[AES256_GCM,data:zg==,iv:sVr33CvZoZmn0LcHlP+FVA8esjGy0aKe1A2iPrrrT0I=,tag:3YneNF7m05fCRGfMgYzByA==,type:str] +THERMOGRAPH_MAIL_BACKEND: ENC[AES256_GCM,data:pRBtOu6cTw==,iv:88QgyY6rwwp/6Y32r6Us1M+b7YpSLPfKIsCL2j/YuIY=,tag:FrchZZrMcCSeOFRWbePM4Q==,type:str] +THERMOGRAPH_DISCORD_BOT: ENC[AES256_GCM,data:NQ==,iv:Zl888EtEoQQd8FdqGb4/N7gGMT4yEymQfWd2fZZCNVA=,tag:Xspr8/taeQwh1qHhfFbLfw==,type:str] +POSTGRES_PASSWORD: ENC[AES256_GCM,data:l076hUutOAKw7+X6wboX,iv:9WVz3SVUikWREVjB6GsJ+EXWO3PvX+g+OaZNGN9f+jU=,tag:OkVuEqEbTSikpCNEySPX5A==,type:str] +THERMOGRAPH_DATABASE_URL: ENC[AES256_GCM,data:kXCYmZXewa8JSY5iJd0cHDtsKrxyKEl5tSVwWeQEd8u7XU+/dn4BExT4RDB3vUWVFVoHXYIXedMnDO8h0QfDCxSI13s=,iv:s1bDjdH4E4T+kF126saOrKlx3AnKEkRWF68KEPcCUe0=,tag:nCzD9U/7g28bJmUXkiYh3Q==,type:str] +THERMOGRAPH_AUTH_SECRET: ENC[AES256_GCM,data:EjwKGKh1od5vgvm8LLv1l6TXO7nshUwh5ihXDpETl+3ldtF8L4H3CWszsw==,iv:4mqwEXWpGpLYbIUotHj9Qf7FSD0ATwkxwhfuL7pG+Zc=,tag:0iLU0ee86H0CgmgPCCtatQ==,type:str] +sops: + age: + - enc: | + -----BEGIN AGE ENCRYPTED FILE----- + YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBzV0lrcUpjeU81TmJDRVpa + elppeU1mMUlpc3lKSTVtU3MxUi83bzN2dVVRCmxZbDJFUkJQVjhCSmljNFpPaWVu + dkNqTkJTd1ZEaHZsZitpTWlJdElCK28KLS0tIEdSTXN6S1Y2eE12U3o2RGxBNXNN + bDBlYkNxMDVTVEFLYVIyVkhIRnNwOGMKv/TMt307XqvzBLCOCA5C4kXFV9iJeVBn + 7gyzb5MRbHbDoNAY/5ckU3341uWUXUQ+IrPvVt2snAdXIlWghm+HwA== + -----END AGE ENCRYPTED FILE----- + recipient: age1xx4dzs0dxlwvkv9sjuqzsphl7lfrxannkfken374yu2qvvcte9sqzktqt2 + lastmodified: "2026-07-25T00:59:09Z" + mac: ENC[AES256_GCM,data:Ga1lMooQrBhU+YasAbN3THZvjrct21KRbuxOoXf+72K5mrgPmqZyFDdB71B8dOUzmz4cqTPw3+jQx2s2ZrZCEiR3qmb897I8hWbTiFWnURSC7rjytxIe6kLhDBjCtso8ThKUMqNjAglqjkytHXljP7Cg4zCuGLEgczoakEJVB7s=,iv:NFHljXdG619XJMjy+1OuUknu+QaG5nH68fvo/zAf/TE=,tag:MyFf/edujKzypOwloVUGAA==,type:str] + unencrypted_suffix: _unencrypted + version: 3.13.2 diff --git a/infra/deploy/secrets/seed-centralis-on-host.sh b/infra/deploy/secrets/seed-centralis-on-host.sh new file mode 100755 index 0000000..e856d55 --- /dev/null +++ b/infra/deploy/secrets/seed-centralis-on-host.sh @@ -0,0 +1,99 @@ +#!/usr/bin/env bash +# Seed deploy/secrets/centralis..yaml by encrypting prod's live +# /etc/centralis.env ENTIRELY ON THE BOX. The plaintext never leaves the box; +# only already-encrypted ciphertext is written back here. Encryption uses the +# age PUBLIC key (safe to hardcode), so the machine you run this from needs +# nothing but SSH access — no sops, no age, no private key. +# +# deploy/secrets/seed-centralis-on-host.sh prod agent@169.58.46.181 +# +# Then commit the resulting deploy/secrets/centralis..yaml. +# +# --------------------------------------------------------------------------- +# WHY THIS IS NOT seed-encrypt-on-host.sh WITH A DIFFERENT PATH +# --------------------------------------------------------------------------- +# `seed-encrypt-on-host.sh` captures the RAW TEXT to the right of the `=`. That +# is correct for /etc/thermograph.env, which is machine-rendered and whose 32 +# values happen to be bare tokens, but it is WRONG for a hand-maintained, +# shell-SOURCED file. /etc/centralis.env is consumed as +# +# set -a && . /etc/centralis.env && set +a && docker compose up +# +# so the value the program actually receives is the value AFTER bash's quote +# removal. `CENTRALIS_TOKENS` is stored there single-quoted, because it is JSON +# and contains double quotes. Capturing the raw text would put the surrounding +# single quotes INSIDE the vault; the renderer would then quote them again, and +# Centralis would receive `'{"emi":...}'` — literal quotes and all — and fail +# its JSON parse. Fails closed, so it would look like the token registry had +# been deleted. That is the same class of bug as the one this whole migration +# exists to remove, arriving from the other direction. +# +# So this script asks BASH what the values are, by sourcing the file in a clean +# environment (`env -i`) and reading the resulting variables. What goes into the +# vault is exactly what Centralis receives today, by construction. +set -euo pipefail +cd "$(dirname "$(readlink -f "$0")")/../.." + +ENV_NAME="${1:?usage: seed-centralis-on-host.sh [ssh-key]}" +SSH_TARGET="${2:?ssh target, e.g. agent@169.58.46.181}" +SSH_KEY="${3:-$HOME/.ssh/thermograph_agent_ed25519}" +OUT="deploy/secrets/centralis.${ENV_NAME}.yaml" + +# Public age recipient — NOT a secret (matches .sops.yaml). The private key never +# leaves the operator's machine / the hosts' /etc/thermograph/age.key. +AGE_PUB="age1xx4dzs0dxlwvkv9sjuqzsphl7lfrxannkfken374yu2qvvcte9sqzktqt2" +SOPS_URL="https://github.com/getsops/sops/releases/download/v3.13.2/sops-v3.13.2.linux.amd64" + +echo "==> Encrypting ${SSH_TARGET}:/etc/centralis.env on-host -> ${OUT}" +ssh -i "$SSH_KEY" "$SSH_TARGET" "AGE_PUB='$AGE_PUB' SOPS_URL='$SOPS_URL' bash -s" > "$OUT" <<'REMOTE' +set -euo pipefail +exec 3>&1 # real stdout carries ONLY the ciphertext +{ # setup noise -> stderr, so it can't corrupt the file + if ! command -v sops >/dev/null 2>&1; then + sudo curl -fsSL "$SOPS_URL" -o /usr/local/bin/sops + sudo chmod +x /usr/local/bin/sops + fi +} >&2 +umask 077 +tmp="$(mktemp)"; tmy="$(mktemp)"; tpy="$(mktemp)" +trap 'rm -f "$tmp" "$tmy" "$tpy"' EXIT +sudo cat /etc/centralis.env > "$tmp" # plaintext stays on this box only + +# The declared assignment targets, in file order. `source` gives last-wins for a +# repeated key and we read the post-source environment, so the de-dup here only +# controls the ORDER of the emitted YAML, never which value is taken. +names="$(grep -oE '^[[:space:]]*(export[[:space:]]+)?[A-Za-z_][A-Za-z0-9_]*=' "$tmp" \ + | sed -E 's/^[[:space:]]*(export[[:space:]]+)?//; s/=$//' | awk '!seen[$0]++')" +[ -n "$names" ] || { echo "!! no KEY=VALUE lines in /etc/centralis.env" >&2; exit 1; } +echo " capturing: $(echo "$names" | tr '\n' ' ')" >&2 + +cat > "$tpy" <<'PY' +import json, os, sys +for n in sys.argv[1:]: + v = os.environ.get(n) + if v is None: + sys.stderr.write("!! %s vanished after sourcing\n" % n) + sys.exit(1) + print("%s: %s" % (n, json.dumps(v))) # JSON-escaped scalar = valid YAML +PY + +# Source in a CLEAN environment and read the values back, so what is stored is +# exactly what `set -a && . /etc/centralis.env` hands to docker compose. PATH is +# passed through only so python3 is findable. $names is deliberately unquoted: +# it is a whitespace-separated list of identifiers by construction. +# shellcheck disable=SC2086 +env -i PATH="$PATH" bash -c \ + 'set -a; . "$1"; set +a; shift; exec python3 "$@"' _ "$tmp" "$tpy" $names > "$tmy" +[ -s "$tmy" ] || { echo "!! captured nothing" >&2; exit 1; } + +cd /tmp # no .sops.yaml here, so --age is honored +sops -e --age "$AGE_PUB" --input-type yaml --output-type yaml "$tmy" >&3 +REMOTE + +if grep -q 'ENC\[AES256_GCM' "$OUT"; then + echo "==> OK: ${OUT} written and encrypted ($(grep -cE '^[A-Za-z_][A-Za-z0-9_]*: ' "$OUT") keys)" + echo " Next: infra/deploy/secrets/verify-centralis-render.sh ${ENV_NAME} ${SSH_TARGET}" +else + echo "!! ${OUT} is not encrypted (the on-host step failed) — removing" >&2 + rm -f "$OUT"; exit 1 +fi diff --git a/infra/deploy/secrets/verify-centralis-render.sh b/infra/deploy/secrets/verify-centralis-render.sh new file mode 100755 index 0000000..5f1d28f --- /dev/null +++ b/infra/deploy/secrets/verify-centralis-render.sh @@ -0,0 +1,152 @@ +#!/usr/bin/env bash +# NON-DESTRUCTIVE go/no-go for the /etc/centralis.env cutover — the Centralis +# analogue of dry-run-render.sh, and the proof obligation for the migration: +# that rendering deploy/secrets/centralis..yaml produces a file whose +# EFFECTIVE VALUES are identical to the live hand-maintained one. +# +# deploy/secrets/verify-centralis-render.sh prod agent@169.58.46.181 +# +# What it does NOT do: write /etc/centralis.env, restart anything, or print a +# single secret value. It renders to a 0700 scratch directory, sources both +# files in a clean shell, compares the resulting variables, and reports key +# NAMES and a verdict. The scratch directory is removed on every exit path. +# +# "Effective values" is the load-bearing phrase. /etc/centralis.env is consumed +# by `set -a && . /etc/centralis.env`, so the only comparison that means +# anything is between what BASH ends up with from each file — not between the +# two files' text. The live file quotes CENTRALIS_TOKENS and the rendered one +# quotes everything, so a textual diff would report a difference where there is +# none, and (much worse) would report a match if the vault had captured the +# quote characters as part of the value. +set -euo pipefail +cd "$(dirname "$(readlink -f "$0")")/../.." + +ENV_NAME="${1:?usage: verify-centralis-render.sh [ssh-key]}" +SSH_TARGET="${2:?ssh target, e.g. agent@169.58.46.181}" +SSH_KEY="${3:-$HOME/.ssh/thermograph_agent_ed25519}" +VAULT="deploy/secrets/centralis.${ENV_NAME}.yaml" +[ -f "$VAULT" ] || { echo "!! no $VAULT — run seed-centralis-on-host.sh first" >&2; exit 1; } + +SSH=(ssh -i "$SSH_KEY" "$SSH_TARGET") +SCP=(scp -q -i "$SSH_KEY") + +scratch="$("${SSH[@]}" 'd=$(mktemp -d) && chmod 700 "$d" && mkdir -p "$d/deploy/secrets" && echo "$d"')" +[ -n "$scratch" ] || { echo "!! could not create a scratch dir on the host" >&2; exit 1; } +cleanup() { "${SSH[@]}" "rm -rf -- '$scratch'" >/dev/null 2>&1 || true; } +trap cleanup EXIT + +# Only ciphertext and a shell script cross the wire. Nothing decrypted ever +# leaves the box; the render happens there, with the host's own age key. +"${SCP[@]}" deploy/render-secrets.sh "${SSH_TARGET}:${scratch}/deploy/render-secrets.sh" +"${SCP[@]}" "$VAULT" "${SSH_TARGET}:${scratch}/deploy/secrets/centralis.${ENV_NAME}.yaml" + +"${SSH[@]}" "SCRATCH='$scratch' bash -s" <<'REMOTE' +set -euo pipefail +umask 077 + +. "$SCRATCH/deploy/render-secrets.sh" +CENTRALIS_RENDER_DEST="$SCRATCH/rendered.env" render_centralis_secrets "$SCRATCH" >&2 + +sudo cat /etc/centralis.env > "$SCRATCH/live.env" + +# Ask bash what each file means, in a clean environment so no inherited variable +# can stand in for one the file failed to set. +dump() { + env -i PATH="$PATH" bash -c \ + 'set -a; . "$1"; set +a; exec python3 -c " +import json, os, sys +sys.stdout.write(json.dumps(dict(os.environ)))"' _ "$1" +} +dump "$SCRATCH/live.env" > "$SCRATCH/live.json" +dump "$SCRATCH/rendered.env" > "$SCRATCH/rend.json" + +# The names each file actually assigns — so the shell's own PATH/PWD/SHLVL/_ are +# not mistaken for content. +names() { + grep -oE '^[[:space:]]*(export[[:space:]]+)?[A-Za-z_][A-Za-z0-9_]*=' "$1" \ + | sed -E 's/^[[:space:]]*(export[[:space:]]+)?//; s/=$//' | sort -u +} +names "$SCRATCH/live.env" > "$SCRATCH/live.names" +names "$SCRATCH/rendered.env" > "$SCRATCH/rend.names" + +python3 - "$SCRATCH" <<'PY' +import json, os, sys + +d = sys.argv[1] +load = lambda n: json.load(open(os.path.join(d, n), encoding="utf-8")) +names = lambda n: [l.strip() for l in open(os.path.join(d, n), encoding="utf-8") if l.strip()] + +live, rend = load("live.json"), load("rend.json") +ln, rn = set(names("live.names")), set(names("rend.names")) + +lost, added = sorted(ln - rn), sorted(rn - ln) +changed = sorted(k for k in ln & rn if live.get(k) != rend.get(k)) + +print() +print(" live /etc/centralis.env : %d keys" % len(ln)) +print(" rendered from the vault : %d keys" % len(rn)) +print() +for k in sorted(ln & rn): + print(" %-34s %s" % (k, "same" if live.get(k) == rend.get(k) else "DIFFERENT")) + +ok = not (lost or added or changed) +print() +if lost: + print(" MISSING from the render :", lost) +if added: + print(" ADDED by the render :", added) +if changed: + print(" VALUE CHANGED :", changed) + +# The registry, by subject name. Values are never touched; `sorted(reg)` is a +# list of audit-log subjects, which is what the operator needs to see. +def subjects(env): + raw = env.get("CENTRALIS_TOKENS", "") + if not raw: + return None + try: + reg = json.loads(raw) + except ValueError as exc: + return "UNPARSEABLE (%s)" % exc + return sorted(reg) if isinstance(reg, dict) else "not a JSON object" + +sl, sr = subjects(live), subjects(rend) +print(" CENTRALIS_TOKENS subjects live: %s" % (sl,)) +print(" rendered: %s" % (sr,)) +if sl != sr: + ok = False + print(" !! the token registry does not survive the migration") + +print() +print(" VERDICT: %s" % ("PASS — the render is value-identical to the live file" + if ok else "FAIL — do NOT cut over")) +sys.exit(0 if ok else 1) +PY + +# End-to-end: the two env files must produce the SAME docker compose model. This +# is the claim that actually matters — not "the files agree" but "the container +# Centralis would be started with is the same one it is running now". Compared +# by digest, on the box; the rendered model is full of secrets and is never +# printed, shipped, or kept. +compose=/opt/centralis/deploy/docker-compose.yml +if [ -f "$compose" ] && command -v docker >/dev/null 2>&1; then + model() { + env -i PATH="$PATH" HOME="$HOME" bash -c \ + 'set -a; . "$1"; set +a; cd /opt/centralis && \ + docker compose -f deploy/docker-compose.yml config' _ "$1" 2>/dev/null \ + | sha256sum | cut -d" " -f1 + } + a="$(model "$SCRATCH/live.env")" + b="$(model "$SCRATCH/rendered.env")" + if [ -z "$a" ] || [ "$a" = "$(printf '' | sha256sum | cut -d' ' -f1)" ]; then + echo " compose model : could not be built from the LIVE env — check by hand" + elif [ "$a" = "$b" ]; then + echo " compose model : IDENTICAL (docker compose config digests match)" + else + echo " compose model : DIFFERENT — do NOT cut over" + exit 1 + fi +else + echo " compose model : skipped (no /opt/centralis or no docker here)" +fi +REMOTE