Jinemi/thermograph

Fork 0

secrets: vault dev and Centralis; render dev without common.yaml #77

Merged

admin_emi merged 2 commits from feat/centralis-vault into dev

2026-07-25 01:14:07 +00:00

admin_emi commented

2026-07-25 01:11:18 +00:00

Owner

Brings the last two credential stores under SOPS. All three environments and the control plane are now managed the same way.

`dev.yaml` — dev wasn't "intentionally not wired in", it was broken

The vault README claimed dev had no real secrets. Both halves were false.

Dev needed secrets it didn't have. thermograph-dev-daemon-1 has been in a crash loop, restarting every ~60s: neither THERMOGRAPH_INTERNAL_TOKEN nor THERMOGRAPH_AUTH_SECRET is set; refusing to start. So dev has had no Discord gateway and no job timers. And with THERMOGRAPH_BASE_URL unset the app falls back to https://thermograph.org, so dev's IndexNow pings and verification links have been claiming to be prod.

Dev already held production credentials. backend-deploy-dev.yml injects S3_ACCESS_KEY/S3_SECRET_KEY into every dev deploy — and the only provisioned keypair for that bucket is read-write, on the bucket holding prod's database backups. Dev doesn't need them: without bucket creds the lake service 503s and history falls through to the archive. Those two workflow lines still need removing — outside this change's scope.

dev renders `dev.yaml` alone

Via THERMOGRAPH_SECRETS_SKIP_COMMON=1. Eleven of common.yaml's sixteen values are live production credentials, the render is a plaintext concatenation consumed through env_file:, and dev is the operator's desktop and the Forgejo CI runner — executing unreviewed dev-branch code with the docker socket mounted. An override in dev.yaml wouldn't help: last-wins governs consumers, but the production value would still physically be a line in the file.

Verified by running the real renderer in throwaway containers: current renderer leaks 28 lines onto dev including both S3 keypairs and the VAPID private key; patched gives exactly 12 keys, zero production credentials. Beta's render is byte-identical either way.

`centralis.prod.yaml` — its own file, its own renderer

/etc/centralis.env was hand-edited, which is how a JSON registry got written unquoted into a shell-sourced file and silently collapsed three identities to one. The renderer owns the file now.

Why not fold CENTRALIS_* into prod.yaml: /etc/thermograph.env is loaded into every container in the app stack. Folding these in would put CENTRALIS_AUTH_TOKEN and CENTRALIS_TOKENS — which authenticate an endpoint carrying run_on_host and sql_query(write:true) — into the web backend's environment. A read-anything bug in the app becomes a foothold on the control plane.

The finding that justifies the whole exercise

sops -d --output-type dotenv emits values verbatim. Reusing the existing render path would have reproduced the original bug exactly. The new function decrypts to JSON and emits POSIX single-quoted assignments. Then three guarantees rather than one:

It sources its own output in a clean shell (env -i, set -a) and compares every value to the plaintext before touching /etc.
It refuses to write unless CENTRALIS_TOKENS parses as a non-empty subject→token object after sourcing. Fed the literal {emi:x} shape it aborts and leaves the destination byte-identical.
Tested against 16 hostile values — embedded quotes, newlines, backticks, $(...), globs, ;rm -rf /, empty string. All round-trip exactly.

Value-identity proven, not asserted

Rendered vs live compared as effective values on prod — a textual diff would report a false difference, and would report a false match if the vault had captured quote characters as part of the value. Plus both env files producing a byte-identical docker compose config digest. 9 keys, both token subjects preserved. No value printed, logged, or shipped; only ciphertext crossed the wire.

Nothing is deployed. /etc/centralis.env is untouched; only a 0600 backup was placed alongside it.

Noted for later, not fixed here

/etc/thermograph.env is also shell-sourced, and survives on raw dotenv only because none of its 32 current values happens to contain a shell-special character. It is one quoted secret away from the identical bug.

Brings the last two credential stores under SOPS. All three environments and the control plane are now managed the same way. ## `dev.yaml` — dev wasn't "intentionally not wired in", it was broken The vault README claimed dev had no real secrets. **Both halves were false.** **Dev needed secrets it didn't have.** `thermograph-dev-daemon-1` has been in a crash loop, restarting every ~60s: `neither THERMOGRAPH_INTERNAL_TOKEN nor THERMOGRAPH_AUTH_SECRET is set; refusing to start`. So dev has had **no Discord gateway and no job timers**. And with `THERMOGRAPH_BASE_URL` unset the app falls back to `https://thermograph.org`, so dev's IndexNow pings and verification links have been claiming to be prod. **Dev already held production credentials.** `backend-deploy-dev.yml` injects `S3_ACCESS_KEY`/`S3_SECRET_KEY` into every dev deploy — and the only provisioned keypair for that bucket is **read-write, on the bucket holding prod's database backups**. Dev doesn't need them: without bucket creds the `lake` service 503s and history falls through to the archive. **Those two workflow lines still need removing** — outside this change's scope. ### dev renders `dev.yaml` alone Via `THERMOGRAPH_SECRETS_SKIP_COMMON=1`. Eleven of `common.yaml`'s sixteen values are live production credentials, the render is a **plaintext concatenation** consumed through `env_file:`, and dev is the operator's desktop *and* the Forgejo CI runner — executing unreviewed `dev`-branch code with the docker socket mounted. An override in `dev.yaml` wouldn't help: last-wins governs *consumers*, but the production value would still physically be a line in the file. Verified by running the real renderer in throwaway containers: **current renderer leaks 28 lines onto dev** including both S3 keypairs and the VAPID private key; patched gives **exactly 12 keys, zero production credentials**. Beta's render is byte-identical either way. ## `centralis.prod.yaml` — its own file, its own renderer `/etc/centralis.env` was hand-edited, which is how a JSON registry got written unquoted into a shell-sourced file and silently collapsed three identities to one. The renderer owns the file now. **Why not fold `CENTRALIS_*` into `prod.yaml`:** `/etc/thermograph.env` is loaded into *every container in the app stack*. Folding these in would put `CENTRALIS_AUTH_TOKEN` and `CENTRALIS_TOKENS` — which authenticate an endpoint carrying `run_on_host` and `sql_query(write:true)` — into the web backend's environment. A read-anything bug in the app becomes a foothold on the control plane. ### The finding that justifies the whole exercise **`sops -d --output-type dotenv` emits values verbatim.** Reusing the existing render path would have reproduced the original bug exactly. The new function decrypts to **JSON** and emits POSIX single-quoted assignments. Then three guarantees rather than one: 1. It **sources its own output** in a clean shell (`env -i`, `set -a`) and compares every value to the plaintext before touching `/etc`. 2. It **refuses to write** unless `CENTRALIS_TOKENS` parses as a non-empty subject→token object *after sourcing*. Fed the literal `{emi:x}` shape it aborts and leaves the destination byte-identical. 3. Tested against **16 hostile values** — embedded quotes, newlines, backticks, `$(...)`, globs, `;rm -rf /`, empty string. All round-trip exactly. ### Value-identity proven, not asserted Rendered vs live compared as **effective** values on prod — a textual diff would report a false difference, and would report a false *match* if the vault had captured quote characters as part of the value. Plus both env files producing a byte-identical `docker compose config` digest. 9 keys, both token subjects preserved. No value printed, logged, or shipped; only ciphertext crossed the wire. **Nothing is deployed.** `/etc/centralis.env` is untouched; only a `0600` backup was placed alongside it. ## Noted for later, not fixed here `/etc/thermograph.env` is **also** shell-sourced, and survives on raw dotenv only because none of its 32 current values happens to contain a shell-special character. It is one quoted secret away from the identical bug.

admin_emi added 1 commit 2026-07-25 01:11:20 +00:00

secrets: vault dev and Centralis; render dev without common.yaml

PR build (required check) / changes (pull_request) Successful in 7s

Details

secrets-guard / encrypted (pull_request) Successful in 5s

Details

PR build (required check) / build-backend (pull_request) Has been skipped

Details

PR build (required check) / build-frontend (pull_request) Has been skipped

Details

shell-lint / shellcheck (pull_request) Failing after 9s

Details

PR build (required check) / validate-observability (pull_request) Has been skipped

Details

PR build (required check) / gate (pull_request) Successful in 2s

Details

43c9e2052a

Brings the last two credential stores under SOPS, so all three environments and
the control plane are managed the same way.

dev.yaml — dev was not "intentionally not wired in", it was broken

The vault README claimed dev had no real secrets. Both halves were false.

Dev needed secrets it did not have: thermograph-dev-daemon-1 has been in a crash
loop, restarting every ~60s with "neither THERMOGRAPH_INTERNAL_TOKEN nor
THERMOGRAPH_AUTH_SECRET is set; refusing to start" — so dev has had no Discord
gateway and no job timers. With THERMOGRAPH_BASE_URL unset the app also falls
back to https://thermograph.org, so dev's IndexNow pings and verification links
claimed to be prod. dev.yaml gives it its own generated AUTH_SECRET and a real
base URL, and 12 values total.

And dev already held production credentials: backend-deploy-dev.yml injects
S3_ACCESS_KEY/S3_SECRET_KEY into every dev deploy, and the only provisioned
keypair for that bucket is read-write — on the bucket holding prod's backups.
Dev does not need them; without bucket creds the lake service 503s and history
falls through to the archive. Those workflow lines still need removing.

dev renders dev.yaml ALONE, via THERMOGRAPH_SECRETS_SKIP_COMMON=1. Eleven of
common.yaml's sixteen values are live production credentials, the render is a
plaintext concatenation consumed through env_file:, and dev is the operator's
desktop AND the Forgejo runner executing unreviewed dev-branch code with the
docker socket mounted. An override in dev.yaml would not help — last-wins
governs consumers, but prod's value is still physically a line in the file.
Verified: current renderer leaks 28 lines onto dev including both S3 keypairs
and the VAPID private key; patched gives exactly 12 keys and no prod credential.
Beta's render is byte-identical either way.

centralis.prod.yaml — its own file, its own renderer

/etc/centralis.env was hand-edited, which is how a JSON registry got written
unquoted into a shell-sourced file tonight and silently collapsed three
identities to one. The renderer now owns the file.

It is service- AND environment-scoped, not folded into prod.yaml, because
/etc/thermograph.env is loaded into every container in the app stack. Folding
these in would put CENTRALIS_AUTH_TOKEN and CENTRALIS_TOKENS — which
authenticate an endpoint carrying run_on_host and sql_query(write) — into the
web backend's environment, turning a read-anything bug in the app into a
foothold on the control plane.

Quoting is guaranteed three ways rather than assumed. Critically, `sops -d
--output-type dotenv` emits values verbatim, so reusing the existing render path
would have reproduced tonight's bug exactly. The new function decrypts to JSON
and emits POSIX single-quoted assignments; it then sources its own output in a
clean shell and compares every value before touching /etc; and it refuses to
write unless CENTRALIS_TOKENS parses as a non-empty subject->token object after
sourcing. Tested against 16 hostile values including embedded quotes, newlines
and ;rm -rf /.

Value-identity proven, not asserted: rendered vs live compared as *effective*
values on prod (a textual diff would report a false difference, and would report
a false match if the vault had captured quote characters as part of the value),
plus both env files producing a byte-identical `docker compose config` digest.
9 keys, both token subjects preserved. Nothing deployed.

Note for later: /etc/thermograph.env is also shell-sourced, and survives on raw
dotenv only because none of its 32 current values contains a shell-special
character. It is one quoted secret away from the same bug.

Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY

admin_emi added 1 commit 2026-07-25 01:13:13 +00:00

secrets: silence two SC2016s that are the intended behaviour

PR build (required check) / changes (pull_request) Successful in 8s

Details

secrets-guard / encrypted (pull_request) Successful in 6s

Details

shell-lint / shellcheck (pull_request) Successful in 6s

Details

PR build (required check) / build-backend (pull_request) Has been skipped

Details

PR build (required check) / build-frontend (pull_request) Has been skipped

Details

PR build (required check) / validate-observability (pull_request) Has been skipped

Details

PR build (required check) / gate (pull_request) Successful in 2s

Details

33cd69e44e

shell-lint failed on two single-quoted strings. Both are deliberate and
double-quoting either one would break it silently rather than loudly.

In render_centralis_secrets, the verification step runs `bash -c 'set -a; . "$1"'`
with the rendered file passed as an argument. The "$1" must be expanded by that
inner shell. Double-quoted, it would interpolate the OUTER $1 — the function's
own first argument, the repo root — so the check would source the wrong path and
still report success. That check exists precisely to catch a bad render, so a
false pass there is worse than no check.

In verify-centralis-render.sh, the string is executed by the REMOTE shell over
ssh. Double-quoted, mktemp would run locally and the script would send the host
a path that does not exist there.

Suppressed with the reason at each site rather than loosened globally.

Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY