openbao: make verify-parity --all host-aware #149

Merged

admin_emi merged 1 commit from fix/parity-all-host-aware into dev

2026-08-01 16:22:38 +00:00

admin_emi commented

2026-08-01 16:18:28 +00:00

Owner

--all expanded to (dev beta prod) and then tried to check all three from wherever it ran. No host carries all three — dev is on vps1, beta and prod are on vps2 — so --all could not succeed anywhere, and the way it failed was misleading rather than obvious: on vps1, prod resolves TG_BAO_APPROLE to /etc/thermograph/openbao-approle, which on that box holds dev's credential, so prod's check authenticated as tg-dev and took a 403 from tg-host-prod.hcl. That reads like a policy bug. It is "wrong box".

The README has shown verify-parity.sh --all as usable since it was written.

--all now means "every environment that lives on this host". Residency is decided by the environment's own address being bound locally (TG_SSH_HOST against the local interfaces), and skips are announced, never silent.

Residency is deliberately not `TG_APP_DIR`

That was my first attempt and it was wrong — caught by testing on the real hosts rather than reasoning about them. vps1 still carries a stale /opt/thermograph from before the estate split (d5357d0, #106), so prod looked resident there and got checked anyway. A leftover directory is exactly the kind of thing that outlives the arrangement it belonged to. Both hosts bind their public IP directly, so the address is authoritative where the directory is not.

(That stale checkout is worth removing separately — it is a full tree with its own infra/openbao/ and ops-cron.yml on the one box tg-host-dev.hcl exists to keep away from prod's secrets.)

Zero checked is now a failure

Previously a run that skipped everything exited 0. A green run is precisely what the 7-day cutover gate counts, so "nothing was wrong" and "nothing was examined" must not look alike. The success line now reports how many were checked and states plainly that a green run on one host says nothing about the other.

Live-verified on both boxes

vps1 --all -> dev PASS; beta, prod SKIPPED (live on vps2); exit 0
vps2 --all -> dev SKIPPED (lives on vps1); beta FAIL; prod FAIL

Both failures are the three known stale common.yaml keys awaiting seed-from-sops.sh --env common, not a regression.

Note the nightly secrets-parity job added in #143 already uses explicit --env per host, so it was never affected — this fixes the by-hand path an operator would reach for.

`--all` expanded to `(dev beta prod)` and then tried to check all three from wherever it ran. **No host carries all three** — dev is on vps1, beta and prod are on vps2 — so `--all` could not succeed anywhere, and the way it failed was misleading rather than obvious: on vps1, prod resolves `TG_BAO_APPROLE` to `/etc/thermograph/openbao-approle`, which on *that* box holds **dev's** credential, so prod's check authenticated as `tg-dev` and took a 403 from `tg-host-prod.hcl`. That reads like a policy bug. It is "wrong box". The README has shown `verify-parity.sh --all` as usable since it was written. `--all` now means **"every environment that lives on this host"**. Residency is decided by the environment's own address being bound locally (`TG_SSH_HOST` against the local interfaces), and skips are announced, never silent. ### Residency is deliberately not `TG_APP_DIR` That was my first attempt and it was wrong — caught by testing on the real hosts rather than reasoning about them. **vps1 still carries a stale `/opt/thermograph`** from before the estate split (`d5357d0`, #106), so prod looked resident there and got checked anyway. A leftover directory is exactly the kind of thing that outlives the arrangement it belonged to. Both hosts bind their public IP directly, so the address is authoritative where the directory is not. (That stale checkout is worth removing separately — it is a full tree with its own `infra/openbao/` and `ops-cron.yml` on the one box `tg-host-dev.hcl` exists to keep away from prod's secrets.) ### Zero checked is now a failure Previously a run that skipped everything exited 0. A green run is precisely what the 7-day cutover gate counts, so **"nothing was wrong" and "nothing was examined" must not look alike**. The success line now reports how many were checked and states plainly that a green run on one host says nothing about the other. ### Live-verified on both boxes ``` vps1 --all -> dev PASS; beta, prod SKIPPED (live on vps2); exit 0 vps2 --all -> dev SKIPPED (lives on vps1); beta FAIL; prod FAIL ``` Both failures are the three known stale `common.yaml` keys awaiting `seed-from-sops.sh --env common`, not a regression. Note the nightly `secrets-parity` job added in #143 already uses explicit `--env` per host, so it was never affected — this fixes the by-hand path an operator would reach for.

admin_emi added 1 commit 2026-08-01 16:18:29 +00:00

openbao: make verify-parity --all host-aware

secrets-guard / encrypted (pull_request) Successful in 4s

Details

PR build (required check) / changes (pull_request) Successful in 7s

Details

PR build (required check) / build-backend (pull_request) Has been skipped

Details

PR build (required check) / build-frontend (pull_request) Has been skipped

Details

shell-lint / shellcheck (pull_request) Successful in 9s

Details

PR build (required check) / validate-observability (pull_request) Has been skipped

Details

PR build (required check) / gate (pull_request) Successful in 2s

Details

Sync infra to hosts / sync-beta (push) Has been skipped

Details

Sync infra to hosts / sync-prod (push) Has been skipped

Details

shell-lint / shellcheck (push) Successful in 6s

Details

Sync infra to hosts / sync-dev (push) Successful in 5s

Details

Sync infra to hosts / sync-centralis (push) Has been skipped

Details

secrets-guard / encrypted (push) Successful in 4s

Details

75968980f9

--all expanded to (dev beta prod) and then tried to check all three from
wherever it was run. No host carries all three: dev is on vps1, beta and prod
are on vps2. So --all could not succeed anywhere, and the way it failed was
misleading rather than obvious -- on vps1, prod resolves TG_BAO_APPROLE to
/etc/thermograph/openbao-approle, which on THAT box holds dev's credential, so
prod's check authenticated as tg-dev and took a 403 from tg-host-prod.hcl.
That reads like a policy bug. It is "wrong box".

--all now means "every environment that lives on THIS host". Residency is
decided by the environment's own address being bound here (TG_SSH_HOST against
the local interfaces), and skips are announced rather than silent.

Residency is deliberately NOT decided by TG_APP_DIR existing. That was the
first attempt and it was wrong: vps1 still carries a stale /opt/thermograph
from before the estate split (d5357d0, #106), so prod looked resident there and
got checked anyway. Verified against both live hosts. A leftover directory is
exactly the kind of thing that outlives the arrangement it belonged to.

Zero environments checked is now a FAILURE. Previously a run that skipped
everything would exit 0, and a green run is precisely what the 7-day cutover
gate counts -- "nothing was wrong" and "nothing was examined" must not look
alike. The success line reports how many were checked and says plainly that a
green run on one host says nothing about the other.

Live-verified on both boxes:
  vps1 --all -> dev PASS, beta and prod skipped, exit 0
  vps2 --all -> dev skipped, beta and prod checked (both currently failing on
                the three known stale common.yaml keys)