Promote dev to main: dev + Centralis vaults #78

Merged

admin_emi merged 3 commits from dev into main

2026-07-25 01:16:06 +00:00

admin_emi commented

2026-07-25 01:14:24 +00:00

Owner

Promotes the dev and Centralis vaults to beta, and syncs infra/ to both hosts. Detail in #77.

What actually changes on the hosts when this merges. infra-sync.yml does git reset --hard and re-renders /etc/thermograph.env on prod and beta. Neither host's render changes: dev.yaml is only read by a box whose secrets-env says dev, and centralis.prod.yaml is read by a different function that infra-sync does not call.

Expected after sync — unchanged from baseline: prod 32 keys, beta 24, prod's /etc/centralis.env still 9. A different count means revert, not debug forward.

Centralis is deliberately not cut over by this merge. The vault lands in prod's checkout; /etc/centralis.env keeps its hand-written contents until render_centralis_secrets is run explicitly and Centralis is redeployed. That is a separate, reversible step with a byte-identical backup already in place, and it is sequenced after this so a bad render can't take the control plane down alongside an infra sync.

Dev gets nothing from this merge either — dev pulls its own checkout and runs deploy-dev.sh. The crash-looping thermograph-dev-daemon-1 is fixed the next time dev deploys, not by this.

The one behavioural change that reaches beta is the THERMOGRAPH_SECRETS_SKIP_COMMON guard, which beta never sets — verified byte-identical render either way.

Promotes the dev and Centralis vaults to **beta**, and syncs `infra/` to both hosts. Detail in #77. **What actually changes on the hosts when this merges.** `infra-sync.yml` does `git reset --hard` and re-renders `/etc/thermograph.env` on prod and beta. Neither host's render changes: `dev.yaml` is only read by a box whose `secrets-env` says `dev`, and `centralis.prod.yaml` is read by a *different* function that `infra-sync` does not call. **Expected after sync — unchanged from baseline:** prod 32 keys, beta 24, prod's `/etc/centralis.env` still 9. A different count means revert, not debug forward. **Centralis is deliberately not cut over by this merge.** The vault lands in prod's checkout; `/etc/centralis.env` keeps its hand-written contents until `render_centralis_secrets` is run explicitly and Centralis is redeployed. That is a separate, reversible step with a byte-identical backup already in place, and it is sequenced after this so a bad render can't take the control plane down alongside an infra sync. **Dev gets nothing from this merge either** — dev pulls its own checkout and runs `deploy-dev.sh`. The crash-looping `thermograph-dev-daemon-1` is fixed the next time dev deploys, not by this. The one behavioural change that reaches beta is the `THERMOGRAPH_SECRETS_SKIP_COMMON` guard, which beta never sets — verified byte-identical render either way.

admin_emi added 3 commits 2026-07-25 01:14:26 +00:00

secrets: vault dev and Centralis; render dev without common.yaml

PR build (required check) / changes (pull_request) Successful in 7s

Details

secrets-guard / encrypted (pull_request) Successful in 5s

Details

PR build (required check) / build-backend (pull_request) Has been skipped

Details

PR build (required check) / build-frontend (pull_request) Has been skipped

Details

shell-lint / shellcheck (pull_request) Failing after 9s

Details

PR build (required check) / validate-observability (pull_request) Has been skipped

Details

PR build (required check) / gate (pull_request) Successful in 2s

Details

43c9e2052a

Brings the last two credential stores under SOPS, so all three environments and
the control plane are managed the same way.

dev.yaml — dev was not "intentionally not wired in", it was broken

The vault README claimed dev had no real secrets. Both halves were false.

Dev needed secrets it did not have: thermograph-dev-daemon-1 has been in a crash
loop, restarting every ~60s with "neither THERMOGRAPH_INTERNAL_TOKEN nor
THERMOGRAPH_AUTH_SECRET is set; refusing to start" — so dev has had no Discord
gateway and no job timers. With THERMOGRAPH_BASE_URL unset the app also falls
back to https://thermograph.org, so dev's IndexNow pings and verification links
claimed to be prod. dev.yaml gives it its own generated AUTH_SECRET and a real
base URL, and 12 values total.

And dev already held production credentials: backend-deploy-dev.yml injects
S3_ACCESS_KEY/S3_SECRET_KEY into every dev deploy, and the only provisioned
keypair for that bucket is read-write — on the bucket holding prod's backups.
Dev does not need them; without bucket creds the lake service 503s and history
falls through to the archive. Those workflow lines still need removing.

dev renders dev.yaml ALONE, via THERMOGRAPH_SECRETS_SKIP_COMMON=1. Eleven of
common.yaml's sixteen values are live production credentials, the render is a
plaintext concatenation consumed through env_file:, and dev is the operator's
desktop AND the Forgejo runner executing unreviewed dev-branch code with the
docker socket mounted. An override in dev.yaml would not help — last-wins
governs consumers, but prod's value is still physically a line in the file.
Verified: current renderer leaks 28 lines onto dev including both S3 keypairs
and the VAPID private key; patched gives exactly 12 keys and no prod credential.
Beta's render is byte-identical either way.

centralis.prod.yaml — its own file, its own renderer

/etc/centralis.env was hand-edited, which is how a JSON registry got written
unquoted into a shell-sourced file tonight and silently collapsed three
identities to one. The renderer now owns the file.

It is service- AND environment-scoped, not folded into prod.yaml, because
/etc/thermograph.env is loaded into every container in the app stack. Folding
these in would put CENTRALIS_AUTH_TOKEN and CENTRALIS_TOKENS — which
authenticate an endpoint carrying run_on_host and sql_query(write) — into the
web backend's environment, turning a read-anything bug in the app into a
foothold on the control plane.

Quoting is guaranteed three ways rather than assumed. Critically, `sops -d
--output-type dotenv` emits values verbatim, so reusing the existing render path
would have reproduced tonight's bug exactly. The new function decrypts to JSON
and emits POSIX single-quoted assignments; it then sources its own output in a
clean shell and compares every value before touching /etc; and it refuses to
write unless CENTRALIS_TOKENS parses as a non-empty subject->token object after
sourcing. Tested against 16 hostile values including embedded quotes, newlines
and ;rm -rf /.

Value-identity proven, not asserted: rendered vs live compared as *effective*
values on prod (a textual diff would report a false difference, and would report
a false match if the vault had captured quote characters as part of the value),
plus both env files producing a byte-identical `docker compose config` digest.
9 keys, both token subjects preserved. Nothing deployed.

Note for later: /etc/thermograph.env is also shell-sourced, and survives on raw
dotenv only because none of its 32 current values contains a shell-special
character. It is one quoted secret away from the same bug.

Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY

secrets: silence two SC2016s that are the intended behaviour

PR build (required check) / changes (pull_request) Successful in 8s

Details

secrets-guard / encrypted (pull_request) Successful in 6s

Details

shell-lint / shellcheck (pull_request) Successful in 6s

Details

PR build (required check) / build-backend (pull_request) Has been skipped

Details

PR build (required check) / build-frontend (pull_request) Has been skipped

Details

PR build (required check) / validate-observability (pull_request) Has been skipped

Details

PR build (required check) / gate (pull_request) Successful in 2s

Details

33cd69e44e

shell-lint failed on two single-quoted strings. Both are deliberate and
double-quoting either one would break it silently rather than loudly.

In render_centralis_secrets, the verification step runs `bash -c 'set -a; . "$1"'`
with the rendered file passed as an argument. The "$1" must be expanded by that
inner shell. Double-quoted, it would interpolate the OUTER $1 — the function's
own first argument, the repo root — so the check would source the wrong path and
still report success. That check exists precisely to catch a bad render, so a
false pass there is worse than no check.

In verify-centralis-render.sh, the string is executed by the REMOTE shell over
ssh. Double-quoted, mktemp would run locally and the script would send the host
a path that does not exist there.

Suppressed with the reason at each site rather than loosened globally.

Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY