Promote main to release: vault consolidation, dev + Centralis vaults #79

Merged

admin_emi merged 7 commits from main into release 2026-07-25 01:19:22 +00:00

Conversation 0 Commits 7 Files changed 11 +1092 -105

admin_emi commented 2026-07-25 01:19:05 +00:00

Owner

Copy link

Promotes the vault work to prod. Detail in #75 and #77.

Already live on prod, ahead of this merge, because infra/ reaches the hosts via infra-sync.yml on main rather than through release:

• common.yaml consolidation — prod renders 32 keys, beta 24, both unchanged from baseline
• dev.yaml and centralis.prod.yaml present in prod's checkout
• The renderer's fail-closed guard and the THERMOGRAPH_SECRETS_SKIP_COMMON flag
• Centralis cut over to the vault: render_centralis_secrets run, effective values verified IDENTICAL against the pre-migration backup, redeployed, 3 identities intact, dashboard still mesh-only

So this hop carries the branch pointer rather than new behaviour for prod. Merging it keeps release from drifting behind main, which is what makes the next app deploy legible.

Verified after the Centralis cutover: 9 keys quoted, CENTRALIS_TOKENS parses: 2 subject(s) — emi, jin, 9 keys survive set -a; . <file> byte-for-byte, and the container came back with 3 identity(ies): emi, jin, shared.

Not done, deliberately: /etc/centralis.env.bak-pre-sops is still on prod. It is the rollback path and the cutover is minutes old; it holds live credentials and should be shredded once the dashboard has been exercised from a phone.

Promotes the vault work to **prod**. Detail in #75 and #77. **Already live on prod, ahead of this merge**, because `infra/` reaches the hosts via `infra-sync.yml` on `main` rather than through `release`: - `common.yaml` consolidation — prod renders 32 keys, beta 24, both unchanged from baseline - `dev.yaml` and `centralis.prod.yaml` present in prod's checkout - The renderer's fail-closed guard and the `THERMOGRAPH_SECRETS_SKIP_COMMON` flag - **Centralis cut over to the vault**: `render_centralis_secrets` run, effective values verified `IDENTICAL` against the pre-migration backup, redeployed, 3 identities intact, dashboard still mesh-only So this hop carries the branch pointer rather than new behaviour for prod. Merging it keeps `release` from drifting behind `main`, which is what makes the next app deploy legible. **Verified after the Centralis cutover:** `9 keys quoted`, `CENTRALIS_TOKENS parses: 2 subject(s) — emi, jin`, `9 keys survive set -a; . <file> byte-for-byte`, and the container came back with `3 identity(ies): emi, jin, shared`. **Not done, deliberately:** `/etc/centralis.env.bak-pre-sops` is still on prod. It is the rollback path and the cutover is minutes old; it holds live credentials and should be shredded once the dashboard has been exercised from a phone.

admin_emi added 7 commits 2026-07-25 01:19:07 +00:00

secrets: factor shared values into common.yaml; stop the renderer failing open ... cb69a22c0f

Two changes to how credentials are stored and distributed.

1. common.yaml now exists.

The renderer has always concatenated common.yaml then <env>.yaml, host winning,
and the README has always documented common.yaml with a checkmark. The file was
never created. So every value shared by prod and beta was duplicated in both
vaults, free to drift apart with nothing to detect it.

16 values move to common.yaml: VAPID keypair, metrics token, IndexNow key,
REGISTRY_TOKEN, the S3 endpoint/bucket and both S3 keypairs, plus shared config.
5 stay per-host because their values genuinely differ (APP_CPUS, DB_CPUS,
DB_MEMORY, WORKERS, THERMOGRAPH_BASE_URL). 8 exist only on prod — the Discord
and mail credentials, which beta does not have at all.

Three more are held back deliberately despite being identical today:
POSTGRES_PASSWORD, THERMOGRAPH_AUTH_SECRET, THERMOGRAPH_DATABASE_URL. These are
the credentials that let one environment act as another, and beta is the more
exposed box — it serves public Forgejo and Grafana. They match only because beta
was seeded from prod. Keeping them per-host costs one line each and preserves
the ability to diverge; putting them in common.yaml would encode the equivalence
as intentional and make breaking it a migration rather than an edit. Reasoning
recorded in the vault README, since a future reader will otherwise "fix" it.

Done without any plaintext leaving prod: ciphertext was shipped up, decrypted
against the host's age key, recombined, re-encrypted, and shipped back. The
consolidation refuses to write unless it has proved merged(common + env) equals
the original env vault exactly — same keys, same values — and re-verifies from
the written files afterwards. Both checks passed for prod and beta.

2. render_thermograph_secrets no longer fails open.

One `return 0` covered two different situations: "this host is not configured
for SOPS" (true of the LAN dev box, and correct to succeed) and "this host IS
configured but its vault is not where we looked". The second is a failure, and
returning 0 made it a silent no-op wearing a success code — a deploy would
report success having rendered nothing, and the host would keep serving whatever
/etc/thermograph.env already held, including after a rotation.

The likely cause is passing the wrong root: the function wants the directory
containing deploy/secrets, which on the hosts is /opt/thermograph/infra, not
/opt/thermograph. That mistake looked exactly like "not configured here", which
is why it went unnoticed. It now exits 1 and names the probable cause.

Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY

Merge pull request 'secrets: factor shared values into common.yaml; stop the renderer failing open' (#75) from fix/key-gaps-digit-regex into dev a00b2c016b

Merge pull request 'Promote dev to main: vault consolidation + renderer fail-closed' (#76) from dev into main 9350045866

secrets: vault dev and Centralis; render dev without common.yaml ... 43c9e2052a

Brings the last two credential stores under SOPS, so all three environments and
the control plane are managed the same way.

dev.yaml — dev was not "intentionally not wired in", it was broken

The vault README claimed dev had no real secrets. Both halves were false.

Dev needed secrets it did not have: thermograph-dev-daemon-1 has been in a crash
loop, restarting every ~60s with "neither THERMOGRAPH_INTERNAL_TOKEN nor
THERMOGRAPH_AUTH_SECRET is set; refusing to start" — so dev has had no Discord
gateway and no job timers. With THERMOGRAPH_BASE_URL unset the app also falls
back to https://thermograph.org, so dev's IndexNow pings and verification links
claimed to be prod. dev.yaml gives it its own generated AUTH_SECRET and a real
base URL, and 12 values total.

And dev already held production credentials: backend-deploy-dev.yml injects
S3_ACCESS_KEY/S3_SECRET_KEY into every dev deploy, and the only provisioned
keypair for that bucket is read-write — on the bucket holding prod's backups.
Dev does not need them; without bucket creds the lake service 503s and history
falls through to the archive. Those workflow lines still need removing.

dev renders dev.yaml ALONE, via THERMOGRAPH_SECRETS_SKIP_COMMON=1. Eleven of
common.yaml's sixteen values are live production credentials, the render is a
plaintext concatenation consumed through env_file:, and dev is the operator's
desktop AND the Forgejo runner executing unreviewed dev-branch code with the
docker socket mounted. An override in dev.yaml would not help — last-wins
governs consumers, but prod's value is still physically a line in the file.
Verified: current renderer leaks 28 lines onto dev including both S3 keypairs
and the VAPID private key; patched gives exactly 12 keys and no prod credential.
Beta's render is byte-identical either way.

centralis.prod.yaml — its own file, its own renderer

/etc/centralis.env was hand-edited, which is how a JSON registry got written
unquoted into a shell-sourced file tonight and silently collapsed three
identities to one. The renderer now owns the file.

It is service- AND environment-scoped, not folded into prod.yaml, because
/etc/thermograph.env is loaded into every container in the app stack. Folding
these in would put CENTRALIS_AUTH_TOKEN and CENTRALIS_TOKENS — which
authenticate an endpoint carrying run_on_host and sql_query(write) — into the
web backend's environment, turning a read-anything bug in the app into a
foothold on the control plane.

Quoting is guaranteed three ways rather than assumed. Critically, `sops -d
--output-type dotenv` emits values verbatim, so reusing the existing render path
would have reproduced tonight's bug exactly. The new function decrypts to JSON
and emits POSIX single-quoted assignments; it then sources its own output in a
clean shell and compares every value before touching /etc; and it refuses to
write unless CENTRALIS_TOKENS parses as a non-empty subject->token object after
sourcing. Tested against 16 hostile values including embedded quotes, newlines
and ;rm -rf /.

Value-identity proven, not asserted: rendered vs live compared as *effective*
values on prod (a textual diff would report a false difference, and would report
a false match if the vault had captured quote characters as part of the value),
plus both env files producing a byte-identical `docker compose config` digest.
9 keys, both token subjects preserved. Nothing deployed.

Note for later: /etc/thermograph.env is also shell-sourced, and survives on raw
dotenv only because none of its 32 current values contains a shell-special
character. It is one quoted secret away from the same bug.

Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY

secrets: silence two SC2016s that are the intended behaviour ... 33cd69e44e

shell-lint failed on two single-quoted strings. Both are deliberate and
double-quoting either one would break it silently rather than loudly.

In render_centralis_secrets, the verification step runs `bash -c 'set -a; . "$1"'`
with the rendered file passed as an argument. The "$1" must be expanded by that
inner shell. Double-quoted, it would interpolate the OUTER $1 — the function's
own first argument, the repo root — so the check would source the wrong path and
still report success. That check exists precisely to catch a bad render, so a
false pass there is worse than no check.

In verify-centralis-render.sh, the string is executed by the REMOTE shell over
ssh. Double-quoted, mktemp would run locally and the script would send the host
a path that does not exist there.

Suppressed with the reason at each site rather than loosened globally.

Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY

Merge pull request 'secrets: vault dev and Centralis; render dev without common.yaml' (#77) from feat/centralis-vault into dev 72ad6f300f

Merge pull request 'Promote dev to main: dev + Centralis vaults' (#78) from dev into main 6c6842ab9e

admin_emi merged commit 4fce9e1540 into release 2026-07-25 01:19:22 +00:00

admin_emi referenced this pull request from a commit 2026-07-25 01:19:24 +00:00

Merge pull request 'Promote main to release: vault consolidation, dev + Centralis vaults' (#79) from main into release

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

  

Compat/Breaking

Breaking change that won't be backward compatible

  

Kind/Bug

Something is not working

  

Kind/Documentation

Documentation changes

  

Kind/Enhancement

Improve existing functionality

  

Kind/Feature

New functionality

  

Kind/Security

This is security issue

  

Kind/Testing

Issue or pull request related to testing

  

Priority

Critical

The priority is critical

  

Priority

High

The priority is high

  

Priority

Low

The priority is low

  

Priority

Medium

The priority is medium

  

Reviewed

Confirmed

Issue has been confirmed

  

Reviewed

Duplicate

This issue or pull request already exists

  

Reviewed

Invalid

Invalid issue

  

Reviewed

Won't Fix

This issue won't be fixed

  

Status

Abandoned

Somebody has started to work on this but abandoned work

  

Status

Blocked

Something is blocking this issue or pull request

  

Status

Need More Info

Feedback is required to reproduce issue or to continue work

No labels

Compat/Breaking

Kind/Bug

Kind/Documentation

Kind/Enhancement

Kind/Feature

Kind/Security

Kind/Testing

Priority

Critical

Priority

High

Priority

Low

Priority

Medium

Reviewed

Confirmed

Reviewed

Duplicate

Reviewed

Invalid

Reviewed

Won't Fix

Status

Abandoned

Status

Blocked

Status

Need More Info

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Jinemi/thermograph#79

No description provided.