render_centralis_secrets has existed since the OpenBao work and had no caller
on any deploy path -- only verify-centralis-render.sh, a check. A renderer
nothing calls is not a migration, it is a file. /etc/centralis.env has stayed
hand-maintained the whole time, with the vault holding a copy nothing consumed.
That gap is why turning on Google sign-in for Centralis had no safe home: the
only way to set CENTRALIS_GOOGLE_CLIENT_ID was to hand-edit the live file,
which is precisely what render-secrets.sh's header warns against.
THE GUARD IS THE POINT. On 2026-07-24 this estate lost Centralis' token
registry by writing a file missing a key the live one had. The renderer proves
its own output round-trips through bash, but it cannot know about a key that
exists only in the live file. So this job renders to a scratch path first,
compares KEY SETS (names only, both sides sourced in `env -i` so the
comparison is over what bash actually ends up with), and refuses to write --
failing loudly -- if the live file has anything the vault does not. Rendering
turns from a way to lose a hand-edit into the thing that catches one.
Verified the guard against a live file carrying a hand-added
CENTRALIS_GOOGLE_CLIENT_ID: correctly identified as a key that would be
deleted, write refused. Also verified the empty/unreadable-live-file path,
which needs `|| true` to survive pipefail.
It recreates the container where the other three jobs deliberately roll
nothing, and the difference is real rather than an exception: Centralis'
compose file interpolates /etc/centralis.env at PARSE time, so rendering
without recreating changes nothing -- the silent no-op this estate keeps
getting bitten by. `docker restart` would not do it either. One container, no
replicas, a two-second recreate. It also skips the recreate entirely when the
rendered file is byte-identical to the live one, so a no-op push does not
bounce a healthy container.
Three fixes, all on the path between "the migration looks ready" and "the
migration is ready".
verify-parity.sh never sourced env-topology.sh, so TG_BAO_APPROLE was unset and
render-secrets-openbao.sh fell through to the bare default -- prod's credential
-- for every environment. On vps2 that made `--env beta` authenticate as
tg-prod and take a 403 from tg-host-prod.hcl on thermograph/data/env/beta.
`--all` failed the same way, taking dev with it. deploy.sh and
deploy-stack.sh always sourced it; only the verifier did not, which is the
worst place for the omission: the tool whose job is to notice divergence was
itself diverging from the path it verifies. It now calls thermograph_topology
per environment and takes TG_SKIP_COMMON from there rather than re-deriving it.
infra-sync.yml renders all three env files on every push touching infra/**, and
none of its three jobs sourced env-topology.sh either. TG_SECRETS_BACKEND was
therefore unset and render-secrets.sh:44 defaulted to sops. Flipping the
selector would have changed deploy.sh and deploy-stack.sh but not this
workflow, which would have kept re-rendering from SOPS -- silent while parity
holds, and a hard failure the moment the SOPS files are retired. The one-line
cutover the README describes was never sufficient on its own.
ops-cron.yml gains a secrets-parity job: prod and beta from vps2, dev from
vps1, nightly, failing loudly on a mismatch. README.md called this the gate for
a cutover; nothing implemented it. A gate that exists only in prose gets
satisfied by assertion. It grants CI no vault access -- the host renders, CI
only asks it to.
First measured run, immediately after the verifier fix:
dev 12 keys PASS, byte-identical
beta 24 keys FAIL, 3 values differ
prod 32 keys FAIL, 3 values differ
The same three keys on both: THERMOGRAPH_S3_SECRET_KEY,
THERMOGRAPH_LAKE_S3_SECRET_KEY, THERMOGRAPH_VAPID_CONTACT. All three live in
common.yaml, seeded 2026-07-31 02:30Z -- before the Contabo rotation landed.
dev passes because dev never layers common. One `seed-from-sops.sh --env
common` fixes beta and prod together, and the 7-day clock starts after that.
The estate had exactly one registered runner, on the desktop. It went offline
at 2026-07-31 16:31Z; for the next 21 hours no PR could satisfy a required
check, no deploy could run, and the 03:00Z ops-cron -- the only backup for both
application databases and for Forgejo -- did not fire. Forgejo queued that
scheduled run rather than dropping it, so it completed on reconnect and nothing
was lost. A longer outage would have meant real gaps.
Three files claimed an "always-on Swarm-hosted runner" existed and that the
estate therefore no longer depended on the desktop. It did not exist: an early
revision of docker-stack.yml ran one as a Docker-in-Docker sidecar and it was
removed. That claim is why a single point of failure sat unnoticed. Corrected
in docker-stack.yml, forgejo/README.md and register-lan-runner.sh.
The new runner is a plain restart:always container, not a Swarm service: a
Swarm-scheduled runner cannot redeploy the Swarm that schedules it, so CI would
be gone exactly when the cluster is what is broken. It runs from
/opt/forgejo-runner rather than in place, because the checkout is reset on
every prod deploy and one `git clean -fdx` there would destroy the
registration.
vps2 runs prod, so the socket mount is bounded rather than assumed benign:
capacity 1, --cpus=2/--memory=4g on job containers, valid_volumes empty so no
job can bind-mount /etc/thermograph.env, and no thermograph network joined.
This defends against accident, not against a hostile workflow author -- stated
plainly in the compose header rather than implied.
The desktop runner stays registered as extra capacity. Nothing may assume it
is up.
The copies of /etc/thermograph/age.key on vps1 and vps2 have always read as
provisioning residue — a side effect of provision-secrets.sh rather than a
decision. They are in fact the estate's recovery quorum: that key is the single
recovery root for all five SOPS vaults AND for every off-box backup, since
ops-cron.yml:142,197 pipe those dumps through `age -r` to the same recipient.
On 2026-07-30 the operator's copy was shredded from the desktop before it had been
copied to its replacement machine, leaving zero operator-side copies. These two were
the only surviving material and the key was restored from vps2. Documenting the
ordering rule that would have prevented it, and adding the check that makes the
quorum verifiable rather than assumed.
verify-age-quorum.sh asserts presence, permissions and hash equality across both
hosts. Divergence is treated as seriously as absence: two hosts holding different
keys means one renders secrets the other cannot read, and a restore silently picks
the wrong one. Prints SHA-256 digests and modes only — a digest of a 32-byte random
key is not reversible, so it is safe in a CI log, and the key is never read or
copied anywhere. Exit status makes it usable as a cron gate.
Also replaces the "wasn't re-verified for this pass" hedge in the README with what
was actually measured: vps1 0440 root:deploy, vps2 0400 root:root, digests equal.
The dev render does not need vps1's group bit — it runs as `agent`, which is not in
group `deploy` and reaches the key through passwordless sudo, with
/opt/thermograph-dev owned agent:agent. Tightening vps1 to 0400 root:root is
therefore expected to be safe, and is called out as the follow-up rather than done
here because it changes a live host on the deploy path.
The check reports that group read as an advisory rather than a failure. It is the
current provisioned state, and a check that fails on day one is a check that is
ignored by day three.
Centralis could not migrate at all. `seed-from-sops.sh --all` refused
centralis.prod with `CENTRALIS_TOKENS: contains a shell metacharacter`, and
render-secrets-openbao.sh had no Centralis path whatsoever — so the seeded data
would have had nothing to read it.
Both halves come from applying the app stack's rules to a file that does not share
its shape. /etc/thermograph.env is raw unquoted KEY=value. /etc/centralis.env is
consumed by exactly one thing and that thing is a shell:
sudo bash -c 'set -a && . /etc/centralis.env && set +a && docker compose up'
CENTRALIS_TOKENS is JSON, so it can never satisfy a no-metacharacters rule, and
rendering it unquoted is precisely the 2026-07-24 failure: bash strips the quotes,
Centralis fails closed on the malformed registry and serves a single `shared`
identity, indistinguishable from the file never being written.
So:
- seed-from-sops.sh validates against the renderer that will consume the path. The
dotenv rules still apply to common and env/*; centralis/* is exempt because it is
single-quoted downstream.
- render-secrets-openbao.sh gains thermograph_render_openbao_centralis, mirroring
render_centralis_secrets: fetch as JSON, emit POSIX single-quoted assignments,
then PROVE the file by sourcing it under `env -i` with `set -a` and comparing
every value back before anything touches the destination. Mirrored rather than
shared, per the existing note above thermograph_render_openbao — consolidate when
the SOPS path is deleted.
- render_centralis_secrets gains the same early-return backend dispatch the app
path already uses, so the new function is reachable. Without it the SOPS path was
the only path regardless of TG_SECRETS_BACKEND.
Verified against a synthetic registry containing a JSON token map, an apostrophe,
and `$(...)`/backtick/backslash/double-quote payloads: 5 keys round-trip
byte-for-byte through `set -a; . file`, CENTRALIS_TOKENS parses to the right
subjects, and the command substitution does not execute. The negative case is
covered too — fed the unquoted 2026-07-24 shape, the round-trip check reports the
value as changed (by length, never content) and refuses to write.
Also corrects the claim in render-secrets-openbao.sh that the SOPS path never
tripped the metacharacter hazard "because every current value happens to be
alphanumeric". CENTRALIS_TOKENS is the counter-example; it never tripped those
paths because it was never in them.
render-secrets-openbao.sh resolved the AppRole file to a single host-wide default,
/etc/thermograph/openbao-approle — which is prod's. On vps2 that made beta's render
authenticate as tg-prod, and tg-host-prod.hcl denies thermograph/data/env/beta, so
beta could never render from OpenBao. It surfaces as `cannot read
thermograph/env/beta`: the policy working correctly against the wrong identity.
env-topology.sh exists precisely because vps2 runs two environments and one
host-wide marker cannot answer "which environment is this". The AppRole path is the
same question and had been given the same host-wide answer. It is now derived per
environment alongside host, checkout, branch, ports and DB role, and the renderer
resolves THERMOGRAPH_BAO_APPROLE, then TG_BAO_APPROLE, then the conventional path —
the same precedence render-secrets.sh:44 already uses for the backend selector.
Only beta differs. prod and dev keep the conventional path; dev is alone on vps1 so
there is no collision there.
Beta parity passes when the path is supplied by hand (24 keys = 8 + 16 shared);
this makes it automatic. Derivation confirmed by sourcing env-topology.sh for all
three environments.
bootstrap.sh could not run to completion on OpenBao 2.6.1:
- Release asset names were wrong. `bao_<ver>_linux_amd64.tar.gz` and
`bao_<ver>_SHA256SUMS` both 404; the assets are `openbao_<ver>_...` and
`checksums.txt`. The tarball is now saved under its real name and the checksum
grep anchored to end-of-line, because checksums.txt also lists a .sbom.json and
`sha256sum -c` resolves each line by the filename inside it.
- `openssl rand -base64 32` appends a newline and the static seal reads the key
file raw, so the service failed with `Error configuring seal "static": unknown
encoding for AES-256 key`.
- The audit stanza relied on the block label being the device type. 2.6.1 requires
explicit `type` and `path` with device settings under `options`. Worth noting the
intermediate state: with type and path but a bare `file_path`, the server starts
and silently ignores the log location, which is the worse failure given a wedged
audit device stops OpenBao answering at all.
- `disable_mlock` is unsupported in 2.6.1 and warned on every start.
- Nothing opened port 8200 and ufw defaults to deny(incoming), so vps1 could not
reach the vault. dev renders on vps1, so this would have surfaced as a timeout
during cutover rather than here.
bootstrap-policies.sh installed beta's credentials as `deploy:deploy`, but vps2 has
no `deploy` user and both environments deploy as `agent` (env-topology.sh sets
TG_SSH_TARGET=agent@ for both; deploy.yml uses one VPS2_SSH_USER). install_creds
also printed a success line after a failed chown: the call site wrapped it in
`|| echo`, which suppresses `set -e` for everything inside the function, so it fell
through to chmod and reported an ownership it never applied. It now removes the
half-written file and returns non-zero.
Corrects the isolation rationale accordingly — separate credentials buy audit
attribution and a policy boundary against mistakes, not deploy-user isolation.
Documents the 2.6.0 root-token change (HCSEC-2026-08): `generate-root` now targets
an authenticated endpoint, so recovery keys cannot mint a replacement token and
revoking the last root token before a second admin identity exists is a one-way
door. Also corrects the runbook's `verify-parity.sh --all`, which cannot work from
a single host given the AppRole CIDR bindings.
forgejo_db has been at 0/1 replicas since 2026-07-29 01:24 UTC. Postgres hit
an invalid data-directory lock file ("could not open file postmaster.pid ...
performing immediate shutdown because data directory lock file is invalid")
and exited 0. With restart_policy.condition=on-failure, Swarm read the zero
status as successful completion, marked the task Complete, and never
rescheduled it.
The forgejo service itself stayed Up and kept serving its homepage, so the
outage presented as every repository page, the whole API and all CI returning
500 with "dial tcp: lookup db on 127.0.0.11:53: no such host" — including the
auth path, which is why API calls reported "user does not exist [uid: 0]"
rather than a database error.
on-failure cannot distinguish "finished successfully" from "shut itself down
and should be restarted", and Postgres exits 0 on several such paths, so it is
the wrong policy for an always-on stateful service.
This is the durable fix; it does not restart the currently stopped task.
forgejo_db has been at 0/1 replicas since 2026-07-29 01:24 UTC. Postgres hit
an invalid data-directory lock file ("could not open file postmaster.pid ...
performing immediate shutdown because data directory lock file is invalid")
and exited 0. With restart_policy.condition=on-failure, Swarm read the zero
status as successful completion, marked the task Complete, and never
rescheduled it.
The forgejo service itself stayed Up and kept serving its homepage, so the
outage presented as every repository page, the whole API and all CI returning
500 with "dial tcp: lookup db on 127.0.0.11:53: no such host" — including the
auth path, which is why API calls reported "user does not exist [uid: 0]"
rather than a database error.
on-failure cannot distinguish "finished successfully" from "shut itself down
and should be restarted", and Postgres exits 0 on several such paths, so it is
the wrong policy for an always-on stateful service.
This is the durable fix; it does not restart the currently stopped task.
Phase 1 of moving the secret store to OpenBao. Nothing is cut over:
TG_SECRETS_BACKEND defaults to sops for all three environments, so SOPS
remains authoritative until an environment is flipped in a reviewed PR.
The reason for the migration is one specific failure class. infra/CLAUDE.md
requires that vps1 never hold prod credentials, but that rule is currently
enforced by a caller-side shell flag (THERMOGRAPH_SECRETS_SKIP_COMMON) at
three call sites. A fourth call site, or one by-hand render, writes both S3
keypairs, the VAPID private key, REGISTRY_TOKEN and the IndexNow/metrics
tokens onto the Forgejo CI-runner box and exits 0. policies/tg-host-dev.hcl
makes that a 403 instead: dev's identity cannot read the common path at all.
Shape:
- vps2 only, native systemd (not a container: a Swarm service is circular,
and a plain docker run is one `prune -a` from gone), raft storage (2.6.0
deprecated the file backend), static auto-unseal so a reboot cannot block
deploys, mesh-only self-signed TLS (not ACME — that would put DNS in the
boot chain of the secret store).
- AppRole per environment, CIDR-bound, 5-minute tokens. Each environment's
secret-id is owned by its own deploy user, which is a boundary that does
not exist today since prod and beta both reach the same root-owned age key.
- Render still produces the same raw unquoted dotenv: Centralis compares
/etc/thermograph.env textually and the Swarm entrypoint parses it by line.
The backend dispatch in render-secrets.sh returns early rather than
refactoring the shared write path, so the SOPS path stays byte-identical.
Verified by rendering dev (12 keys) and prod (32 keys) through it, matching
the live hosts. The duplicated write path is noted for consolidation once
prod has been on OpenBao for a release cycle.
The OpenBao renderer rejects values containing a newline or a shell
metacharacter. /etc/thermograph.env is sourced by bash in six places, so
such a value is a command-execution path on the deploy host; the SOPS path
has always had this hazard and avoids it only because every current value
happens to be alphanumeric.
Also records that the age keypair is NOT retired by this migration: it is
the backup encryption key for every off-box dump (ops-cron.yml:142,197,
30-day retention), so deleting it with the vault files would silently
destroy a month of database recoverability.
verify-parity.sh is the cutover gate — it renders both backends and diffs
them, reporting key names and counts only, never values.
Protection an admin walks through is documentation rather than a control, so
apply_to_admins goes on — but that removes the direct-push route for everyone,
including in an emergency, and the way back in should be written down before it
is needed rather than improvised.
There is no force flag; the supported route is lift the rule, act, restore it,
with both halves audited. Names the failure mode that actually justifies it: a
renamed status context makes the required check unsatisfiable, and with
apply_to_admins on that blocks every merge into dev and main with no exemption
for anyone.
.claude/BRANCHING.md — how the trunk-based flow is actually run: the serialised
merge queue (Forgejo has no native one, so whoever orchestrates is it), the two
promotions and who decides each, and the hotfix down-merge that has to happen in
the same session or it becomes a regression scheduled for the next promotion.
Two things it states because getting them wrong is expensive here. CI does not
gate PRs into `release` — pr-build.yml fires only for dev and main — so a
required check on that branch would make every production promotion unmergeable
with a 405 naming no cause; closing that gap means changing the workflow first
and turning the requirement on second. And the chain cannot fast-forward: every
promotion leaves a merge commit on the target the source never receives, so
promotions are merge commits until someone deliberately reconciles the branches.
.claude/ownership.md — which tasks may run concurrently, seamed along the four
domains, plus the shared spine that has to be serialised: the workflows (CI
lives only in the root .forgejo/), env-topology.sh, the deploy entry points and
the CLAUDE.md files. Names the recurring collisions too — a schema change spans
alembic and whatever reads the column, and if the payload shape moves it is
PAYLOAD_VER and the frontend as well, which is one task and not two.
