promote: dev → main #139

Merged

admin_emi merged 7 commits from dev into main 2026-07-31 05:18:04 +00:00

Conversation 0 Commits 7 Files changed 11 +642 -66

7 commits

Author	SHA1	Message	Date
admin_emi	4ae50ff759	Merge pull request 'secrets: make the two-host age key quorum a checked policy' (#137) from feat/age-recovery-quorum into dev	2026-07-31 05:14:22 +00:00
Emi Griffith	b2077b5294	secrets: make the two-host age key quorum a checked policy ... The copies of /etc/thermograph/age.key on vps1 and vps2 have always read as provisioning residue — a side effect of provision-secrets.sh rather than a decision. They are in fact the estate's recovery quorum: that key is the single recovery root for all five SOPS vaults AND for every off-box backup, since ops-cron.yml:142,197 pipe those dumps through `age -r` to the same recipient. On 2026-07-30 the operator's copy was shredded from the desktop before it had been copied to its replacement machine, leaving zero operator-side copies. These two were the only surviving material and the key was restored from vps2. Documenting the ordering rule that would have prevented it, and adding the check that makes the quorum verifiable rather than assumed. verify-age-quorum.sh asserts presence, permissions and hash equality across both hosts. Divergence is treated as seriously as absence: two hosts holding different keys means one renders secrets the other cannot read, and a restore silently picks the wrong one. Prints SHA-256 digests and modes only — a digest of a 32-byte random key is not reversible, so it is safe in a CI log, and the key is never read or copied anywhere. Exit status makes it usable as a cron gate. Also replaces the "wasn't re-verified for this pass" hedge in the README with what was actually measured: vps1 0440 root:deploy, vps2 0400 root:root, digests equal. The dev render does not need vps1's group bit — it runs as `agent`, which is not in group `deploy` and reaches the key through passwordless sudo, with /opt/thermograph-dev owned agent:agent. Tightening vps1 to 0400 root:root is therefore expected to be safe, and is called out as the follow-up rather than done here because it changes a live host on the deploy path. The check reports that group read as an advisory rather than a failure. It is the current provisioned state, and a check that fails on day one is a check that is ignored by day three.	2026-07-31 05:12:58 +00:00
admin_emi	b32c7b6b11	Merge pull request 'rotate Contabo S3 credentials' (#138) from fix/rotate-contabo-s3 into dev	2026-07-31 05:12:41 +00:00
Emi Griffith	11f1275998	rotate Contabo S3 credentials	2026-07-31 05:10:41 +00:00
Emi Griffith	657e27ba9f	openbao: render Centralis from the vault ... Centralis could not migrate at all. `seed-from-sops.sh --all` refused centralis.prod with `CENTRALIS_TOKENS: contains a shell metacharacter`, and render-secrets-openbao.sh had no Centralis path whatsoever — so the seeded data would have had nothing to read it. Both halves come from applying the app stack's rules to a file that does not share its shape. /etc/thermograph.env is raw unquoted KEY=value. /etc/centralis.env is consumed by exactly one thing and that thing is a shell: sudo bash -c 'set -a && . /etc/centralis.env && set +a && docker compose up' CENTRALIS_TOKENS is JSON, so it can never satisfy a no-metacharacters rule, and rendering it unquoted is precisely the 2026-07-24 failure: bash strips the quotes, Centralis fails closed on the malformed registry and serves a single `shared` identity, indistinguishable from the file never being written. So: - seed-from-sops.sh validates against the renderer that will consume the path. The dotenv rules still apply to common and env/*; centralis/* is exempt because it is single-quoted downstream. - render-secrets-openbao.sh gains thermograph_render_openbao_centralis, mirroring render_centralis_secrets: fetch as JSON, emit POSIX single-quoted assignments, then PROVE the file by sourcing it under `env -i` with `set -a` and comparing every value back before anything touches the destination. Mirrored rather than shared, per the existing note above thermograph_render_openbao — consolidate when the SOPS path is deleted. - render_centralis_secrets gains the same early-return backend dispatch the app path already uses, so the new function is reachable. Without it the SOPS path was the only path regardless of TG_SECRETS_BACKEND. Verified against a synthetic registry containing a JSON token map, an apostrophe, and `$(...)`/backtick/backslash/double-quote payloads: 5 keys round-trip byte-for-byte through `set -a; . file`, CENTRALIS_TOKENS parses to the right subjects, and the command substitution does not execute. The negative case is covered too — fed the unquoted 2026-07-24 shape, the round-trip check reports the value as changed (by length, never content) and refuses to write. Also corrects the claim in render-secrets-openbao.sh that the SOPS path never tripped the metacharacter hazard "because every current value happens to be alphanumeric". CENTRALIS_TOKENS is the counter-example; it never tripped those paths because it was never in them.	2026-07-30 19:55:17 -07:00
Emi Griffith	5001269b37	openbao: derive the AppRole credential path per environment ... render-secrets-openbao.sh resolved the AppRole file to a single host-wide default, /etc/thermograph/openbao-approle — which is prod's. On vps2 that made beta's render authenticate as tg-prod, and tg-host-prod.hcl denies thermograph/data/env/beta, so beta could never render from OpenBao. It surfaces as `cannot read thermograph/env/beta`: the policy working correctly against the wrong identity. env-topology.sh exists precisely because vps2 runs two environments and one host-wide marker cannot answer "which environment is this". The AppRole path is the same question and had been given the same host-wide answer. It is now derived per environment alongside host, checkout, branch, ports and DB role, and the renderer resolves THERMOGRAPH_BAO_APPROLE, then TG_BAO_APPROLE, then the conventional path — the same precedence render-secrets.sh:44 already uses for the backend selector. Only beta differs. prod and dev keep the conventional path; dev is alone on vps1 so there is no collision there. Beta parity passes when the path is supplied by hand (24 keys = 8 + 16 shared); this makes it automatic. Derivation confirmed by sourcing env-topology.sh for all three environments.	2026-07-30 19:49:28 -07:00
Emi Griffith	702bcd920c	openbao: fix bootstrap so it completes on 2.6.x ... bootstrap.sh could not run to completion on OpenBao 2.6.1: - Release asset names were wrong. `bao_<ver>_linux_amd64.tar.gz` and `bao_<ver>_SHA256SUMS` both 404; the assets are `openbao_<ver>_...` and `checksums.txt`. The tarball is now saved under its real name and the checksum grep anchored to end-of-line, because checksums.txt also lists a .sbom.json and `sha256sum -c` resolves each line by the filename inside it. - `openssl rand -base64 32` appends a newline and the static seal reads the key file raw, so the service failed with `Error configuring seal "static": unknown encoding for AES-256 key`. - The audit stanza relied on the block label being the device type. 2.6.1 requires explicit `type` and `path` with device settings under `options`. Worth noting the intermediate state: with type and path but a bare `file_path`, the server starts and silently ignores the log location, which is the worse failure given a wedged audit device stops OpenBao answering at all. - `disable_mlock` is unsupported in 2.6.1 and warned on every start. - Nothing opened port 8200 and ufw defaults to deny(incoming), so vps1 could not reach the vault. dev renders on vps1, so this would have surfaced as a timeout during cutover rather than here. bootstrap-policies.sh installed beta's credentials as `deploy:deploy`, but vps2 has no `deploy` user and both environments deploy as `agent` (env-topology.sh sets TG_SSH_TARGET=agent@ for both; deploy.yml uses one VPS2_SSH_USER). install_creds also printed a success line after a failed chown: the call site wrapped it in `|| echo`, which suppresses `set -e` for everything inside the function, so it fell through to chmod and reported an ownership it never applied. It now removes the half-written file and returns non-zero. Corrects the isolation rationale accordingly — separate credentials buy audit attribution and a policy boundary against mistakes, not deploy-user isolation. Documents the 2.6.0 root-token change (HCSEC-2026-08): `generate-root` now targets an authenticated endpoint, so recovery keys cannot mint a replacement token and revoking the last root token before a second admin identity exists is a one-way door. Also corrects the runbook's `verify-parity.sh --all`, which cannot work from a single host given the AppRole CIDR bindings.	2026-07-30 19:47:22 -07:00