openbao: make the parity gate actually work, and run it nightly #143

Merged

admin_emi merged 1 commit from fix/openbao-parity-gate into dev

2026-08-01 15:52:19 +00:00

admin_emi commented

2026-08-01 15:36:58 +00:00

Owner

Three fixes on the path between "the migration looks ready" and "the migration is ready".

verify-parity.sh could not verify beta or dev. It never sourced env-topology.sh, so TG_BAO_APPROLE was unset and render-secrets-openbao.sh:41 fell through to the bare default — prod's credential — for every environment. On vps2 that made --env beta authenticate as tg-prod and take a 403 from tg-host-prod.hcl on thermograph/data/env/beta; --all failed the same way and took dev with it. deploy.sh and deploy-stack.sh always sourced it. Only the verifier did not — the worst place for the omission, since the tool whose job is to notice divergence was itself diverging from the path it verifies. README.md asserted the property was already true.

infra-sync.yml would not have followed a cutover. It re-renders all three env files on every push touching infra/**, and none of its three jobs sourced env-topology.sh, so TG_SECRETS_BACKEND was unset and render-secrets.sh:44 defaulted to sops. Flipping the selector would have changed deploy.sh and deploy-stack.sh but not this workflow — silent while parity holds, hard failure once the SOPS files are retired. The one-line cutover README.md describes was never sufficient alone.

The nightly gate did not exist. README.md called 7 consecutive green parity runs the gate for a cutover and said they should run from ops-cron.yml. Nothing implemented it. Added as a secrets-parity job: prod and beta from vps2, dev from vps1, failing loudly on mismatch. It grants CI no vault access — the host renders, CI only asks it to.

First measured run, immediately after the fix

env	keys	result
dev	12	PASS — byte-identical
beta	24	FAIL — 3 values differ
prod	32	FAIL — 3 values differ

The same three keys on both: THERMOGRAPH_S3_SECRET_KEY, THERMOGRAPH_LAKE_S3_SECRET_KEY, THERMOGRAPH_VAPID_CONTACT. All three live in common.yaml, which was seeded 2026-07-31 02:30Z — before the Contabo rotation landed. dev passes because dev never layers common.

Before merging this, or at least before the first nightly run: seed-from-sops.sh --env common needs running by the operator (it reads production secrets in plaintext and inherits the not-for-an-agent rule). One re-seed fixes beta and prod together. The 7-day clock starts after that, not before — otherwise the job is scheduled red on day one.

Three fixes on the path between "the migration looks ready" and "the migration is ready". **`verify-parity.sh` could not verify beta or dev.** It never sourced `env-topology.sh`, so `TG_BAO_APPROLE` was unset and `render-secrets-openbao.sh:41` fell through to the bare default — prod's credential — for every environment. On vps2 that made `--env beta` authenticate as `tg-prod` and take a 403 from `tg-host-prod.hcl` on `thermograph/data/env/beta`; `--all` failed the same way and took dev with it. `deploy.sh` and `deploy-stack.sh` always sourced it. Only the verifier did not — the worst place for the omission, since the tool whose job is to notice divergence was itself diverging from the path it verifies. README.md asserted the property was already true. **`infra-sync.yml` would not have followed a cutover.** It re-renders all three env files on every push touching `infra/**`, and none of its three jobs sourced `env-topology.sh`, so `TG_SECRETS_BACKEND` was unset and `render-secrets.sh:44` defaulted to `sops`. Flipping the selector would have changed `deploy.sh` and `deploy-stack.sh` but not this workflow — silent while parity holds, hard failure once the SOPS files are retired. The one-line cutover README.md describes was never sufficient alone. **The nightly gate did not exist.** README.md called 7 consecutive green parity runs the gate for a cutover and said they should run from `ops-cron.yml`. Nothing implemented it. Added as a `secrets-parity` job: prod and beta from vps2, dev from vps1, failing loudly on mismatch. It grants CI no vault access — the host renders, CI only asks it to. ### First measured run, immediately after the fix | env | keys | result | |---|---|---| | dev | 12 | **PASS** — byte-identical | | beta | 24 | FAIL — 3 values differ | | prod | 32 | FAIL — 3 values differ | The same three keys on both: `THERMOGRAPH_S3_SECRET_KEY`, `THERMOGRAPH_LAKE_S3_SECRET_KEY`, `THERMOGRAPH_VAPID_CONTACT`. All three live in `common.yaml`, which was seeded 2026-07-31 02:30Z — before the Contabo rotation landed. dev passes because dev never layers `common`. **Before merging this, or at least before the first nightly run: `seed-from-sops.sh --env common` needs running by the operator** (it reads production secrets in plaintext and inherits the not-for-an-agent rule). One re-seed fixes beta and prod together. The 7-day clock starts after that, not before — otherwise the job is scheduled red on day one.

admin_emi added 1 commit 2026-08-01 15:36:59 +00:00

openbao: make the parity gate actually work, and run it nightly

PR build (required check) / changes (pull_request) Successful in 7s

Details

secrets-guard / encrypted (pull_request) Successful in 5s

Details

PR build (required check) / build-backend (pull_request) Has been skipped

Details

shell-lint / shellcheck (pull_request) Successful in 7s

Details

PR build (required check) / build-frontend (pull_request) Has been skipped

Details

PR build (required check) / validate-observability (pull_request) Has been skipped

Details

PR build (required check) / gate (pull_request) Successful in 2s

Details

Sync infra to hosts / sync-beta (push) Has been skipped

Details

Sync infra to hosts / sync-prod (push) Has been skipped

Details

Sync infra to hosts / sync-dev (push) Successful in 6s

Details

secrets-guard / encrypted (push) Successful in 5s

Details

shell-lint / shellcheck (push) Successful in 7s

Details

7b14b9062c

Three fixes, all on the path between "the migration looks ready" and "the
migration is ready".

verify-parity.sh never sourced env-topology.sh, so TG_BAO_APPROLE was unset and
render-secrets-openbao.sh fell through to the bare default -- prod's credential
-- for every environment. On vps2 that made `--env beta` authenticate as
tg-prod and take a 403 from tg-host-prod.hcl on thermograph/data/env/beta.
`--all` failed the same way, taking dev with it. deploy.sh and
deploy-stack.sh always sourced it; only the verifier did not, which is the
worst place for the omission: the tool whose job is to notice divergence was
itself diverging from the path it verifies. It now calls thermograph_topology
per environment and takes TG_SKIP_COMMON from there rather than re-deriving it.

infra-sync.yml renders all three env files on every push touching infra/**, and
none of its three jobs sourced env-topology.sh either. TG_SECRETS_BACKEND was
therefore unset and render-secrets.sh:44 defaulted to sops. Flipping the
selector would have changed deploy.sh and deploy-stack.sh but not this
workflow, which would have kept re-rendering from SOPS -- silent while parity
holds, and a hard failure the moment the SOPS files are retired. The one-line
cutover the README describes was never sufficient on its own.

ops-cron.yml gains a secrets-parity job: prod and beta from vps2, dev from
vps1, nightly, failing loudly on a mismatch. README.md called this the gate for
a cutover; nothing implemented it. A gate that exists only in prose gets
satisfied by assertion. It grants CI no vault access -- the host renders, CI
only asks it to.

First measured run, immediately after the verifier fix:

  dev   12 keys  PASS, byte-identical
  beta  24 keys  FAIL, 3 values differ
  prod  32 keys  FAIL, 3 values differ

The same three keys on both: THERMOGRAPH_S3_SECRET_KEY,
THERMOGRAPH_LAKE_S3_SECRET_KEY, THERMOGRAPH_VAPID_CONTACT. All three live in
common.yaml, seeded 2026-07-31 02:30Z -- before the Contabo rotation landed.
dev passes because dev never layers common. One `seed-from-sops.sh --env
common` fixes beta and prod together, and the 7-day clock starts after that.