openbao: make the parity gate actually work, and run it nightly
All checks were successful
PR build (required check) / changes (pull_request) Successful in 7s
secrets-guard / encrypted (pull_request) Successful in 5s
PR build (required check) / build-backend (pull_request) Has been skipped
shell-lint / shellcheck (pull_request) Successful in 7s
PR build (required check) / build-frontend (pull_request) Has been skipped
PR build (required check) / validate-observability (pull_request) Has been skipped
PR build (required check) / gate (pull_request) Successful in 2s
Sync infra to hosts / sync-beta (push) Has been skipped
Sync infra to hosts / sync-prod (push) Has been skipped
Sync infra to hosts / sync-dev (push) Successful in 6s
secrets-guard / encrypted (push) Successful in 5s
shell-lint / shellcheck (push) Successful in 7s

Three fixes, all on the path between "the migration looks ready" and "the
migration is ready".

verify-parity.sh never sourced env-topology.sh, so TG_BAO_APPROLE was unset and
render-secrets-openbao.sh fell through to the bare default -- prod's credential
-- for every environment. On vps2 that made `--env beta` authenticate as
tg-prod and take a 403 from tg-host-prod.hcl on thermograph/data/env/beta.
`--all` failed the same way, taking dev with it. deploy.sh and
deploy-stack.sh always sourced it; only the verifier did not, which is the
worst place for the omission: the tool whose job is to notice divergence was
itself diverging from the path it verifies. It now calls thermograph_topology
per environment and takes TG_SKIP_COMMON from there rather than re-deriving it.

infra-sync.yml renders all three env files on every push touching infra/**, and
none of its three jobs sourced env-topology.sh either. TG_SECRETS_BACKEND was
therefore unset and render-secrets.sh:44 defaulted to sops. Flipping the
selector would have changed deploy.sh and deploy-stack.sh but not this
workflow, which would have kept re-rendering from SOPS -- silent while parity
holds, and a hard failure the moment the SOPS files are retired. The one-line
cutover the README describes was never sufficient on its own.

ops-cron.yml gains a secrets-parity job: prod and beta from vps2, dev from
vps1, nightly, failing loudly on a mismatch. README.md called this the gate for
a cutover; nothing implemented it. A gate that exists only in prose gets
satisfied by assertion. It grants CI no vault access -- the host renders, CI
only asks it to.

First measured run, immediately after the verifier fix:

  dev   12 keys  PASS, byte-identical
  beta  24 keys  FAIL, 3 values differ
  prod  32 keys  FAIL, 3 values differ

The same three keys on both: THERMOGRAPH_S3_SECRET_KEY,
THERMOGRAPH_LAKE_S3_SECRET_KEY, THERMOGRAPH_VAPID_CONTACT. All three live in
common.yaml, seeded 2026-07-31 02:30Z -- before the Contabo rotation landed.
dev passes because dev never layers common. One `seed-from-sops.sh --env
common` fixes beta and prod together, and the 7-day clock starts after that.
This commit is contained in:
Emi Griffith 2026-08-01 08:36:35 -07:00
parent 001e6b1365
commit 7b14b9062c
4 changed files with 137 additions and 13 deletions

View file

@ -86,6 +86,15 @@ jobs:
exit 0
fi
if [ -f infra/deploy/render-secrets.sh ]; then
# env-topology.sh FIRST: it is what sets TG_SECRETS_BACKEND, and
# render-secrets.sh:44 defaults to `sops` when it is unset. Without
# this line the cutover would be only half-applied — deploy.sh would
# read the new backend while THIS workflow, which runs on every push
# touching infra/**, kept re-rendering the same file from SOPS. That
# is silent while parity holds and a hard failure the moment the SOPS
# files are retired.
. infra/deploy/env-topology.sh
thermograph_topology dev
. infra/deploy/render-secrets.sh
# SKIP_COMMON is dev's standing rule, not a preference: common.yaml
# is the fleet's shared production credential set, and vps1 also
@ -114,6 +123,13 @@ jobs:
git fetch --prune origin main
git reset --hard origin/main
if [ -f infra/deploy/render-secrets.sh ]; then
# See the note in sync-dev: env-topology.sh is what sets
# TG_SECRETS_BACKEND, and for beta it also sets TG_BAO_APPROLE to
# the -beta credential. Beta shares a filesystem with prod, so the
# bare default is prod's file and tg-host-prod.hcl denies beta's
# own path — a 403 at render time on the box that runs prod.
. infra/deploy/env-topology.sh
thermograph_topology beta
. infra/deploy/render-secrets.sh
render_thermograph_secrets /opt/thermograph-beta/infra beta /etc/thermograph-beta.env
fi
@ -139,6 +155,10 @@ jobs:
git fetch --prune origin main
git reset --hard origin/main
if [ -f infra/deploy/render-secrets.sh ]; then
# See the note in sync-dev — without env-topology.sh this job keeps
# rendering from SOPS regardless of what the backend selector says.
. infra/deploy/env-topology.sh
thermograph_topology prod
. infra/deploy/render-secrets.sh
render_thermograph_secrets /opt/thermograph/infra prod /etc/thermograph.env
fi

View file

@ -216,6 +216,65 @@ jobs:
rclone delete --min-age 30d "archive:$base/db/" 2>/dev/null || true
rclone delete --min-age 30d "archive:$base/data/" 2>/dev/null || true
secrets-parity:
name: OpenBao/SOPS parity gate
runs-on: docker
# The gate infra/openbao/README.md specifies for a cutover -- "7 consecutive
# green parity runs" -- and which, until now, nothing implemented. A gate
# that exists only in prose gets satisfied by assertion.
#
# verify-parity.sh renders each environment through BOTH backends and diffs
# the KEY=value sets after a last-wins collapse. It prints key NAMES and
# counts only, never values, so these logs are safe to read and to paste.
#
# A mismatch FAILS the job, deliberately and loudly. The failure modes on
# the other side of a bad cutover are all silent: a missing
# THERMOGRAPH_AUTH_SECRET makes every worker mint its own, so logins start
# working intermittently; a half-rendered VAPID pair makes push.py generate a
# new keypair and delete every subscription row as "gone". Red here is the
# only warning that arrives before those.
#
# Runs under sudo because the age key is 0400 root and the AppRole files are
# 0400 agent/root -- the SSH user cannot read them directly. This grants CI
# no vault access of its own: the HOST renders, CI only asks it to.
#
# The run history in Forgejo IS the record of consecutive green days. Do not
# count a day this job was skipped or the runner was down as green.
concurrency:
group: ops-secrets-parity
cancel-in-progress: false
steps:
- name: prod + beta (vps2)
uses: https://github.com/appleboy/ssh-action@v1.2.0
with:
host: ${{ secrets.VPS2_SSH_HOST }}
username: ${{ secrets.VPS2_SSH_USER }}
key: ${{ secrets.VPS2_SSH_KEY }}
port: ${{ secrets.VPS2_SSH_PORT }}
script: |
set -euo pipefail
# Each environment is checked from its OWN checkout. The trees are
# identical, but running beta's from beta's directory also proves
# that checkout is current rather than assuming it.
sudo /opt/thermograph/infra/openbao/verify-parity.sh --env prod
sudo /opt/thermograph-beta/infra/openbao/verify-parity.sh --env beta
- name: dev (vps1)
uses: https://github.com/appleboy/ssh-action@v1.2.0
with:
host: ${{ secrets.VPS1_SSH_HOST }}
username: ${{ secrets.VPS1_SSH_USER }}
key: ${{ secrets.VPS1_SSH_KEY }}
port: ${{ secrets.VPS1_SSH_PORT }}
script: |
set -euo pipefail
# dev renders dev.yaml ALONE -- verify-parity.sh takes that from
# env-topology.sh's TG_SKIP_COMMON rather than re-deriving it, so the
# comparison matches what deploy-dev.sh actually does. vps1 must
# never see common.yaml: it runs Forgejo, its CI, and dev's
# unreviewed branch.
sudo /opt/thermograph-dev/infra/openbao/verify-parity.sh --env dev
indexnow:
name: IndexNow ping
runs-on: docker

View file

@ -206,9 +206,16 @@ infra/openbao/verify-parity.sh --env beta # expect 24 k
cd /opt/thermograph-dev && infra/openbao/verify-parity.sh --env dev # expect 12 keys
```
Beta needs no special handling: `env-topology.sh` derives `TG_BAO_APPROLE` per
environment, so beta authenticates as `tg-beta` rather than falling back to prod's
credentials and being denied its own path by `tg-host-prod.hcl`.
Beta needs no special handling: `verify-parity.sh` sources `env-topology.sh` and calls
`thermograph_topology` per environment, so `TG_BAO_APPROLE` points at
`/etc/thermograph/openbao-approle-beta` and beta authenticates as `tg-beta` rather than
falling back to prod's credentials and being denied its own path by `tg-host-prod.hcl`.
That sourcing is load-bearing and was missing until 2026-08-01: this document asserted
the property while the verifier alone did not have it, so `--env beta` and `--all` both
took a 403 and the gate could not check the one environment whose isolation it exists
to prove. If you refactor `verify-parity.sh`, the `thermograph_topology` call is not
boilerplate.
`seed-from-sops.sh` reads every production secret in plaintext. Per `infra/CLAUDE.md`
the equivalent `seed-from-live.sh` is explicitly *not for an agent to run*; this
@ -224,10 +231,27 @@ One PR per hop, following the estate's own promotion model.
+ TG_SECRETS_BACKEND=openbao
```
Order: **dev → beta → prod**, with `verify-parity.sh` green throughout. Recommended
gate before prod: **7 consecutive green parity runs on all three environments**, run
nightly from `ops-cron.yml` over SSH (which gives continuous evidence without granting
CI any vault access).
Order: **dev → beta → prod**, with `verify-parity.sh` green throughout. The gate before
prod is **7 consecutive green parity runs on all three environments**. That runs
nightly from the `secrets-parity` job in `ops-cron.yml` over SSH, which gives
continuous evidence without granting CI any vault access — the host renders, CI only
asks it to. Forgejo's run history for that job is the record; a night the job was
skipped, or the runner was down, does not count as green.
Measured 2026-08-01, immediately after the verifier was fixed:
| env | keys | result |
|---|---|---|
| dev | 12 | **PASS** — byte-identical |
| beta | 24 | FAIL — 3 values differ |
| prod | 32 | FAIL — 3 values differ |
Identical three keys on both: `THERMOGRAPH_S3_SECRET_KEY`,
`THERMOGRAPH_LAKE_S3_SECRET_KEY`, `THERMOGRAPH_VAPID_CONTACT`. All three live in
`common.yaml`, which was seeded on 2026-07-31 at 02:30Z — *before* the Contabo
rotation landed. dev passes because dev never layers `common`. One
`seed-from-sops.sh --env common` fixes beta and prod together; the 7-day clock starts
after that, not before.
`TG_SECRETS_BACKEND=sops` remains a working two-way door for the whole period. Keep the
SOPS path for a full release cycle after prod flips — it is the best mitigation

View file

@ -52,6 +52,19 @@ while [ $# -gt 0 ]; do
done
[ ${#TARGETS[@]} -gt 0 ] || usage
# env-topology.sh FIRST, and this is not cosmetic. It is the only thing that sets
# TG_BAO_APPROLE, and without it render-secrets-openbao.sh:41 falls through to the
# bare default — prod's credential — for every environment. On vps2 that made
# `--env beta` authenticate as tg-prod and take a 403 from tg-host-prod.hcl on
# thermograph/data/env/beta, so the gate could not verify the one environment whose
# isolation it exists to prove. `--all` failed the same way, and dev with it.
#
# deploy.sh and deploy-stack.sh always sourced this; only the verifier did not,
# which is the worst place for the omission to be: the tool whose whole job is to
# notice a divergence was itself diverging from the path it verifies.
#
# shellcheck source=/dev/null
. "$INFRA_DIR/deploy/env-topology.sh"
# shellcheck source=/dev/null
. "$INFRA_DIR/deploy/render-secrets-openbao.sh"
@ -59,13 +72,21 @@ overall=0
for env_name in "${TARGETS[@]}"; do
echo "== parity check: $env_name"
# Per-environment topology: TG_BAO_APPROLE (beta's differs, because beta shares a
# filesystem with prod) and TG_SKIP_COMMON. Called inside the loop because --all
# walks three environments and the values differ per environment.
if ! thermograph_topology "$env_name"; then
echo "!! unknown environment: $env_name" >&2
overall=1; continue
fi
# dev renders WITHOUT common on both backends. On the SOPS side that is a caller
# flag; on the OpenBao side the policy also forbids it. Setting the flag here keeps
# the comparison apples-to-apples — and if the flag were wrong, the OpenBao side
# would fail closed with a 403 rather than quietly diverge, which is exactly the
# improvement being verified.
skip_common=0
[ "$env_name" = dev ] && skip_common=1
# flag; on the OpenBao side the policy also forbids it. Taking the flag from the
# topology rather than re-deriving it here keeps the comparison apples-to-apples
# AND keeps this script from drifting from deploy.sh:67, which reads the same
# variable. If the flag were wrong, the OpenBao side would fail closed with a 403
# rather than quietly diverge — which is exactly the improvement being verified.
skip_common="${TG_SKIP_COMMON:-0}"
sops_out=$(mktemp); bao_out=$(mktemp)
# Both temp files hold PLAINTEXT SECRETS. Removed on every exit path below; not a