6 commits

Author	SHA1	Message	Date
Emi Griffith	75968980f9	openbao: make verify-parity --all host-aware ... --all expanded to (dev beta prod) and then tried to check all three from wherever it was run. No host carries all three: dev is on vps1, beta and prod are on vps2. So --all could not succeed anywhere, and the way it failed was misleading rather than obvious -- on vps1, prod resolves TG_BAO_APPROLE to /etc/thermograph/openbao-approle, which on THAT box holds dev's credential, so prod's check authenticated as tg-dev and took a 403 from tg-host-prod.hcl. That reads like a policy bug. It is "wrong box". --all now means "every environment that lives on THIS host". Residency is decided by the environment's own address being bound here (TG_SSH_HOST against the local interfaces), and skips are announced rather than silent. Residency is deliberately NOT decided by TG_APP_DIR existing. That was the first attempt and it was wrong: vps1 still carries a stale /opt/thermograph from before the estate split (d5357d0, #106), so prod looked resident there and got checked anyway. Verified against both live hosts. A leftover directory is exactly the kind of thing that outlives the arrangement it belonged to. Zero environments checked is now a FAILURE. Previously a run that skipped everything would exit 0, and a green run is precisely what the 7-day cutover gate counts -- "nothing was wrong" and "nothing was examined" must not look alike. The success line reports how many were checked and says plainly that a green run on one host says nothing about the other. Live-verified on both boxes: vps1 --all -> dev PASS, beta and prod skipped, exit 0 vps2 --all -> dev skipped, beta and prod checked (both currently failing on the three known stale common.yaml keys)	2026-08-01 09:18:05 -07:00
Emi Griffith	7b14b9062c	openbao: make the parity gate actually work, and run it nightly ... Three fixes, all on the path between "the migration looks ready" and "the migration is ready". verify-parity.sh never sourced env-topology.sh, so TG_BAO_APPROLE was unset and render-secrets-openbao.sh fell through to the bare default -- prod's credential -- for every environment. On vps2 that made `--env beta` authenticate as tg-prod and take a 403 from tg-host-prod.hcl on thermograph/data/env/beta. `--all` failed the same way, taking dev with it. deploy.sh and deploy-stack.sh always sourced it; only the verifier did not, which is the worst place for the omission: the tool whose job is to notice divergence was itself diverging from the path it verifies. It now calls thermograph_topology per environment and takes TG_SKIP_COMMON from there rather than re-deriving it. infra-sync.yml renders all three env files on every push touching infra/**, and none of its three jobs sourced env-topology.sh either. TG_SECRETS_BACKEND was therefore unset and render-secrets.sh:44 defaulted to sops. Flipping the selector would have changed deploy.sh and deploy-stack.sh but not this workflow, which would have kept re-rendering from SOPS -- silent while parity holds, and a hard failure the moment the SOPS files are retired. The one-line cutover the README describes was never sufficient on its own. ops-cron.yml gains a secrets-parity job: prod and beta from vps2, dev from vps1, nightly, failing loudly on a mismatch. README.md called this the gate for a cutover; nothing implemented it. A gate that exists only in prose gets satisfied by assertion. It grants CI no vault access -- the host renders, CI only asks it to. First measured run, immediately after the verifier fix: dev 12 keys PASS, byte-identical beta 24 keys FAIL, 3 values differ prod 32 keys FAIL, 3 values differ The same three keys on both: THERMOGRAPH_S3_SECRET_KEY, THERMOGRAPH_LAKE_S3_SECRET_KEY, THERMOGRAPH_VAPID_CONTACT. All three live in common.yaml, seeded 2026-07-31 02:30Z -- before the Contabo rotation landed. dev passes because dev never layers common. One `seed-from-sops.sh --env common` fixes beta and prod together, and the 7-day clock starts after that.	2026-08-01 08:36:35 -07:00
Emi Griffith	657e27ba9f	openbao: render Centralis from the vault ... Centralis could not migrate at all. `seed-from-sops.sh --all` refused centralis.prod with `CENTRALIS_TOKENS: contains a shell metacharacter`, and render-secrets-openbao.sh had no Centralis path whatsoever — so the seeded data would have had nothing to read it. Both halves come from applying the app stack's rules to a file that does not share its shape. /etc/thermograph.env is raw unquoted KEY=value. /etc/centralis.env is consumed by exactly one thing and that thing is a shell: sudo bash -c 'set -a && . /etc/centralis.env && set +a && docker compose up' CENTRALIS_TOKENS is JSON, so it can never satisfy a no-metacharacters rule, and rendering it unquoted is precisely the 2026-07-24 failure: bash strips the quotes, Centralis fails closed on the malformed registry and serves a single `shared` identity, indistinguishable from the file never being written. So: - seed-from-sops.sh validates against the renderer that will consume the path. The dotenv rules still apply to common and env/*; centralis/* is exempt because it is single-quoted downstream. - render-secrets-openbao.sh gains thermograph_render_openbao_centralis, mirroring render_centralis_secrets: fetch as JSON, emit POSIX single-quoted assignments, then PROVE the file by sourcing it under `env -i` with `set -a` and comparing every value back before anything touches the destination. Mirrored rather than shared, per the existing note above thermograph_render_openbao — consolidate when the SOPS path is deleted. - render_centralis_secrets gains the same early-return backend dispatch the app path already uses, so the new function is reachable. Without it the SOPS path was the only path regardless of TG_SECRETS_BACKEND. Verified against a synthetic registry containing a JSON token map, an apostrophe, and `$(...)`/backtick/backslash/double-quote payloads: 5 keys round-trip byte-for-byte through `set -a; . file`, CENTRALIS_TOKENS parses to the right subjects, and the command substitution does not execute. The negative case is covered too — fed the unquoted 2026-07-24 shape, the round-trip check reports the value as changed (by length, never content) and refuses to write. Also corrects the claim in render-secrets-openbao.sh that the SOPS path never tripped the metacharacter hazard "because every current value happens to be alphanumeric". CENTRALIS_TOKENS is the counter-example; it never tripped those paths because it was never in them.	2026-07-30 19:55:17 -07:00
Emi Griffith	5001269b37	openbao: derive the AppRole credential path per environment ... render-secrets-openbao.sh resolved the AppRole file to a single host-wide default, /etc/thermograph/openbao-approle — which is prod's. On vps2 that made beta's render authenticate as tg-prod, and tg-host-prod.hcl denies thermograph/data/env/beta, so beta could never render from OpenBao. It surfaces as `cannot read thermograph/env/beta`: the policy working correctly against the wrong identity. env-topology.sh exists precisely because vps2 runs two environments and one host-wide marker cannot answer "which environment is this". The AppRole path is the same question and had been given the same host-wide answer. It is now derived per environment alongside host, checkout, branch, ports and DB role, and the renderer resolves THERMOGRAPH_BAO_APPROLE, then TG_BAO_APPROLE, then the conventional path — the same precedence render-secrets.sh:44 already uses for the backend selector. Only beta differs. prod and dev keep the conventional path; dev is alone on vps1 so there is no collision there. Beta parity passes when the path is supplied by hand (24 keys = 8 + 16 shared); this makes it automatic. Derivation confirmed by sourcing env-topology.sh for all three environments.	2026-07-30 19:49:28 -07:00
Emi Griffith	702bcd920c	openbao: fix bootstrap so it completes on 2.6.x ... bootstrap.sh could not run to completion on OpenBao 2.6.1: - Release asset names were wrong. `bao_<ver>_linux_amd64.tar.gz` and `bao_<ver>_SHA256SUMS` both 404; the assets are `openbao_<ver>_...` and `checksums.txt`. The tarball is now saved under its real name and the checksum grep anchored to end-of-line, because checksums.txt also lists a .sbom.json and `sha256sum -c` resolves each line by the filename inside it. - `openssl rand -base64 32` appends a newline and the static seal reads the key file raw, so the service failed with `Error configuring seal "static": unknown encoding for AES-256 key`. - The audit stanza relied on the block label being the device type. 2.6.1 requires explicit `type` and `path` with device settings under `options`. Worth noting the intermediate state: with type and path but a bare `file_path`, the server starts and silently ignores the log location, which is the worse failure given a wedged audit device stops OpenBao answering at all. - `disable_mlock` is unsupported in 2.6.1 and warned on every start. - Nothing opened port 8200 and ufw defaults to deny(incoming), so vps1 could not reach the vault. dev renders on vps1, so this would have surfaced as a timeout during cutover rather than here. bootstrap-policies.sh installed beta's credentials as `deploy:deploy`, but vps2 has no `deploy` user and both environments deploy as `agent` (env-topology.sh sets TG_SSH_TARGET=agent@ for both; deploy.yml uses one VPS2_SSH_USER). install_creds also printed a success line after a failed chown: the call site wrapped it in `|| echo`, which suppresses `set -e` for everything inside the function, so it fell through to chmod and reported an ownership it never applied. It now removes the half-written file and returns non-zero. Corrects the isolation rationale accordingly — separate credentials buy audit attribution and a policy boundary against mistakes, not deploy-user isolation. Documents the 2.6.0 root-token change (HCSEC-2026-08): `generate-root` now targets an authenticated endpoint, so recovery keys cannot mint a replacement token and revoking the last root token before a second admin identity exists is a one-way door. Also corrects the runbook's `verify-parity.sh --all`, which cannot work from a single host given the AppRole CIDR bindings.	2026-07-30 19:47:22 -07:00
Emi Griffith	6bd07d33cb	openbao: add a dormant OpenBao backend alongside SOPS ... Phase 1 of moving the secret store to OpenBao. Nothing is cut over: TG_SECRETS_BACKEND defaults to sops for all three environments, so SOPS remains authoritative until an environment is flipped in a reviewed PR. The reason for the migration is one specific failure class. infra/CLAUDE.md requires that vps1 never hold prod credentials, but that rule is currently enforced by a caller-side shell flag (THERMOGRAPH_SECRETS_SKIP_COMMON) at three call sites. A fourth call site, or one by-hand render, writes both S3 keypairs, the VAPID private key, REGISTRY_TOKEN and the IndexNow/metrics tokens onto the Forgejo CI-runner box and exits 0. policies/tg-host-dev.hcl makes that a 403 instead: dev's identity cannot read the common path at all. Shape: - vps2 only, native systemd (not a container: a Swarm service is circular, and a plain docker run is one `prune -a` from gone), raft storage (2.6.0 deprecated the file backend), static auto-unseal so a reboot cannot block deploys, mesh-only self-signed TLS (not ACME — that would put DNS in the boot chain of the secret store). - AppRole per environment, CIDR-bound, 5-minute tokens. Each environment's secret-id is owned by its own deploy user, which is a boundary that does not exist today since prod and beta both reach the same root-owned age key. - Render still produces the same raw unquoted dotenv: Centralis compares /etc/thermograph.env textually and the Swarm entrypoint parses it by line. The backend dispatch in render-secrets.sh returns early rather than refactoring the shared write path, so the SOPS path stays byte-identical. Verified by rendering dev (12 keys) and prod (32 keys) through it, matching the live hosts. The duplicated write path is noted for consolidation once prod has been on OpenBao for a release cycle. The OpenBao renderer rejects values containing a newline or a shell metacharacter. /etc/thermograph.env is sourced by bash in six places, so such a value is a command-execution path on the deploy host; the SOPS path has always had this hazard and avoids it only because every current value happens to be alphanumeric. Also records that the age keypair is NOT retired by this migration: it is the backup encryption key for every off-box dump (ops-cron.yml:142,197, 30-day retention), so deleting it with the vault files would silently destroy a month of database recoverability. verify-parity.sh is the cutover gate — it renders both backends and diffs them, reporting key names and counts only, never values.	2026-07-29 21:35:20 -07:00