thermograph/infra/openbao

OpenBao — the secret store, phase 1

Status: dormant. Every file here is committed but nothing is cut over. SOPS is still authoritative for dev, beta and prod, because TG_SECRETS_BACKEND defaults to sops for all three in infra/deploy/env-topology.sh. Flipping an environment is a one-line reviewed change, and it should not happen until verify-parity.sh passes.

Verified against OpenBao v2.6.1 (2026-07-22). The binary is bao.

---

Why do this at all

Not for encryption, and not for audit — though the audit log is a real gain. The decisive reason is one specific failure class that a file-based vault cannot close.

infra/CLAUDE.md states the rule: "vps1 must never hold prod credentials. It's the box that runs Forgejo, its CI runner, and dev's unreviewed branch — the opposite of an isolation boundary."

Today that rule is enforced by a shell flag — THERMOGRAPH_SECRETS_SKIP_COMMON=1 — set by the caller, at three separate call sites (env-topology.sh:196 → deploy.sh:67, deploy-dev.sh:86, infra-sync.yml:93-95). The renderer itself does not know that env=dev means never-common. A fourth call site, or one by-hand render_thermograph_secrets /opt/thermograph-dev/infra dev, writes both S3 keypairs (one read-write on the bucket holding prod's database backups), the VAPID private key that signs Web Push to real subscribers, REGISTRY_TOKEN, and the IndexNow and metrics tokens onto the CI-runner box — and exits 0.

Under policies/tg-host-dev.hcl that is not possible to get wrong. dev's identity cannot read thermograph/data/common. A misconfigured caller gets a 403 instead of a silent success. The failure class is gone, not guarded. That is what this migration buys, and it is worth the cost of running a stateful service.

The estate has already hardened this once by hand (commit 7583258, "infra-sync: refuse to render dev's vault onto a host that is not dev"). That is the shape of a problem that wants an ACL, not another guard.

What it does not buy — stated plainly

• Rotation of provider-issued credentials. Roughly half the credential count is Discord, Contabo S3, the Forgejo registry token, VAPID, IndexNow. OpenBao cannot rotate any of them; they stay static KV entries. Contabo Object Storage is S3-compatible but exposes no IAM API, so the AWS secrets engine cannot issue against it either.
• Dynamic database credentials. Inapplicable here for two independent reasons: the apps read THERMOGRAPH_DATABASE_URL once at import and hold a pool, so a TTL'd credential kills the pool at expiry; and there is no reload path anywhere in the codebase — no SIGHUP handler, no config re-read.
• Better review than git. This is a real regression. Today a secret change is a PR with a diff, backstopped by secrets-guard CI. Under OpenBao it is an out-of-band API call with no diff and no review. Mitigated by committing a key-name manifest so CI can still assert no key vanished, plus the audit log and bao kv metadata as the change record — but not fully.
• Fewer moving parts. SOPS needs no server. This adds a stateful service to keep alive, upgrade, back up and TLS-rotate, for ~40 secrets that change rarely.

---

Shape of the deployment

Decision	Choice	Why
Host	vps2 only	vps1 must never hold prod credentials, and with a static seal the box also holds the key that decrypts everything. Cost: vps1 needs vps2 up to deploy dev — acceptable, since when vps2 is down prod is down anyway.
Process	native systemd, not a container	A Swarm service is circular (deploy-stack.sh renders secrets to deploy the stack). A plain docker run makes the vault depend on the daemon deploys restart, and is one prune -a from gone. Native = one dependency, the disk.
Storage	raft, single node	2.6.0 deprecated the file backend for removal in 2.7.0. Raft also has the only backup primitive (operator raft snapshot save). A two-node raft would be worse than one: quorum of two means losing either node loses writes.
Seal	static auto-unseal	See below.
TLS	self-signed, 10y, on disk	Must not come from Caddy/ACME — that would put DNS and the public internet in the boot chain of the secret store.
Auth	AppRole per environment, CIDR-bound	5-minute tokens; secret-ids that only work from the right mesh address.
Consumption	keep rendering a dotenv file	Preserves every existing seam and keeps OpenBao a deploy-time dependency, never a runtime one.

The unseal decision, which is the crux

Static seal, key at /etc/openbao/unseal.key (0400), plus an off-box copy.

Shamir (-key-shares=5 -key-threshold=3) means the vault is sealed after every process restart — reboot, package upgrade, OOM kill — and a human must type three shares. render-secrets.sh is on the deploy path for all three environments, so a sealed vault blocks every deploy and every hotfix. With one operator the N-of-M threshold is theatre: one person holds all the shares, so it buys nothing against the real threat while costing an availability property the estate has today for free. A Contabo reboot currently needs no human at all.

OpenBao's docs hedge static seal — "carefully evaluate" — but that caveat is aimed at multi-tenant enterprises. The argument here is that static seal is not a downgrade from the status quo: /etc/thermograph/age.key is already one 0400 file per host that decrypts everything forever with no audit trail. A static seal key is the identical trust model. What changes is everything layered above it.

The one new single point of failure: lose that key and the raft snapshots are unrestorable. It must exist in ≥2 places, one not on vps2. Hard operator obligation.

A transit seal against a second OpenBao on vps1 is genuinely better — vps2's disk would no longer contain the unseal key — but the vps1 unsealer needs unsealing too, so the circularity moves rather than dissolves, and it adds "vps1 must be up before vps2's vault unseals" as a new failure mode. Revisit once the primary migration is boring.

---

Layout

thermograph/data/common          the 16 shared beta+prod values   (= common.yaml)
thermograph/data/env/prod        prod's 16                        (= prod.yaml)
thermograph/data/env/beta        beta's 8                         (= beta.yaml)
thermograph/data/env/dev         dev's 12  — inherits NOTHING     (= dev.yaml)
thermograph/data/centralis/prod  Centralis' 9              (= centralis.prod.yaml)
thermograph/data/ops/backup      S3 creds for ops-cron + the backup age recipient
thermograph/data/legacy/age      the age PRIVATE key — see "the age key survives"

Inheritance lives in the render order (common then env/<name>, last-wins); isolation lives in the policy. Today both live in one shell variable.

centralis/<env> renders differently from everything above it, and must. The app stack's /etc/thermograph.env is raw unquoted KEY=value; /etc/centralis.env is consumed by set -a && . /etc/centralis.env, i.e. bash's full expansion pipeline, so it is emitted as POSIX single-quoted assignments and then verified by sourcing it in a clean shell and comparing every value back before the file is written. CENTRALIS_TOKENS is JSON and cannot survive the unquoted form: bash strips its quotes, Centralis fails closed on the malformed result and serves a single shared identity, which is indistinguishable from the file never having been written. That is the 2026-07-24 incident, and the round-trip check exists so a render carrying it cannot reach /etc.

Consequence for seeding: seed-from-sops.sh applies the raw-dotenv rules (no shell metacharacters, no newlines) only to common and env/*. Applying them to centralis/* would reject data that renders perfectly well — the validator matches the renderer that will consume the path, not a single global rule.

common is a top-level path rather than env/common on purpose: it makes the dev deny rule expressible as exactly one path, with no wildcard over env/* able to accidentally re-include it.

~20 of the 61 keys are not secrets (PORT, WORKERS, APP_CPUS, TIMESCALEDB_TAG, THERMOGRAPH_BASE_URL, the Discord channel IDs, the VAPID public key, …). They stay in the vault for the cutover, because byte-identical parity is the entire safety property and splitting them would destroy it. Moving pure config back into the repo is a clean follow-on that shrinks the vault-outage blast radius.

---

The age key survives this migration

This is the trap that would cause silent data loss.

ops-cron.yml:142 and :197 stream every off-box Postgres and Forgejo dump through pg_dump | age -r <recipient> | rclone rcat, with 30-day S3 retention. Restore uses /etc/thermograph/age.key. So the age keypair is not just the SOPS transport — it is the backup encryption key.

Deleting it along with the SOPS vault files would destroy up to 30 days of database recoverability, and nothing would notice until someone attempted a restore.

So: keep the age private key at thermograph/data/legacy/age and in the password manager, and keep it on disk until either the newest age-encrypted object in S3 has aged out, or the backup path is re-pointed at a dedicated backup recipient. Retiring SOPS and retiring age are two different projects.

---

Runbook

Stand it up (once, by the operator, on vps2)

sudo bash infra/openbao/bootstrap.sh          # binary, TLS, seal key, ufw, unit, init
# custody the recovery keys + /etc/openbao/unseal.key OFF the box, then:
export BAO_ADDR=https://127.0.0.1:8200 BAO_CACERT=/etc/thermograph/openbao-ca.crt
export BAO_TOKEN=<root token>
sudo -E bash infra/openbao/bootstrap-policies.sh   # mount, policies, approles

Do not revoke the root token here. OpenBao 2.6.0 replaced the unauthenticated /sys/generate-root with an authenticated /sys/generate-root-token (HCSEC-2026-08), and bao operator generate-root now targets the new one — so minting a root token requires a token you already have. The recovery keys are not a way back in; they rekey, they do not authenticate. Unless disable_unauthed_generate_root_endpoints = false is set in config.hcl, revoking the last root token locks you out of an otherwise healthy, unsealed vault, and the only remedy is re-initialising from scratch. Revoke only once a second admin identity exists.

Use /etc/thermograph/openbao-ca.crt (mode 0444) rather than /etc/openbao/tls/bao.crt — same certificate, but /etc/openbao/tls/ is 0700 openbao, so only root can read the latter. Without BAO_CACERT every command fails with certificate signed by unknown authority.

Seed and prove (on vps2 — the age key is already there)

export SOPS_AGE_KEY_FILE=/etc/thermograph/age.key
infra/openbao/seed-from-sops.sh --dry-run     # key names + counts, writes nothing
infra/openbao/seed-from-sops.sh --all

Then prove parity — not --all from one box. Each AppRole is secret_id_bound_cidrs to the host that legitimately renders it, so an environment can only be verified from its own host:

# on vps2
infra/openbao/verify-parity.sh --env prod                          # expect 32 keys
infra/openbao/verify-parity.sh --env beta                          # expect 24 keys
# on vps1
cd /opt/thermograph-dev && infra/openbao/verify-parity.sh --env dev  # expect 12 keys

Beta needs no special handling: verify-parity.sh sources env-topology.sh and calls thermograph_topology per environment, so TG_BAO_APPROLE points at /etc/thermograph/openbao-approle-beta and beta authenticates as tg-beta rather than falling back to prod's credentials and being denied its own path by tg-host-prod.hcl.

That sourcing is load-bearing and was missing until 2026-08-01: this document asserted the property while the verifier alone did not have it, so --env beta and --all both took a 403 and the gate could not check the one environment whose isolation it exists to prove. If you refactor verify-parity.sh, the thermograph_topology call is not boilerplate.

seed-from-sops.sh reads every production secret in plaintext. Per infra/CLAUDE.md the equivalent seed-from-live.sh is explicitly not for an agent to run; this inherits that rule.

Cut an environment over

One PR per hop, following the estate's own promotion model.

   dev)
-      TG_SECRETS_BACKEND=sops
+      TG_SECRETS_BACKEND=openbao

Order: dev → beta → prod, with verify-parity.sh green throughout. The gate before prod is 7 consecutive green parity runs on all three environments. That runs nightly from the secrets-parity job in ops-cron.yml over SSH, which gives continuous evidence without granting CI any vault access — the host renders, CI only asks it to. Forgejo's run history for that job is the record; a night the job was skipped, or the runner was down, does not count as green.

Measured 2026-08-01, immediately after the verifier was fixed:

env	keys	result
dev	12	PASS — byte-identical
beta	24	FAIL — 3 values differ
prod	32	FAIL — 3 values differ

Identical three keys on both: THERMOGRAPH_S3_SECRET_KEY, THERMOGRAPH_LAKE_S3_SECRET_KEY, THERMOGRAPH_VAPID_CONTACT. All three live in common.yaml, which was seeded on 2026-07-31 at 02:30Z — before the Contabo rotation landed. dev passes because dev never layers common. One seed-from-sops.sh --env common fixes beta and prod together; the 7-day clock starts after that, not before.

TG_SECRETS_BACKEND=sops remains a working two-way door for the whole period. Keep the SOPS path for a full release cycle after prod flips — it is the best mitigation available for anything unforeseen.

Rollback

Revert the one-line PR and redeploy. The SOPS path is untouched by this work — verified by rendering dev (12 keys) and prod (32 keys) through it after the dispatch was added, matching the live hosts exactly.

---

Failure modes

Failure	Effect	Mitigation
Vault down/sealed at deploy	Render returns 1, deploy aborts, running stack unaffected — temp-then-install means /etc/thermograph.env is never truncated	Restart=always + static seal so a reboot self-heals; alert on /v1/sys/health; TG_SECRETS_BACKEND=sops is a working revert
Audit device wedges	⚠️ OpenBao stops answering requests entirely when no enabled audit device can record them — a full /var on vps2 blocks every deploy	Two devices (file + syslog); logrotate signals SIGHUP or bao writes to an unlinked inode; disk alert on vps2
Static seal key lost	Raft snapshots unrestorable	≥2 copies, one off-box. The one new SPOF
Raft corruption / disk loss	Total vault loss	Nightly operator raft snapshot save, age-encrypted to S3 beside the DB dumps. Verify a restore end-to-end before any consumer depends on it
Cold boot of vps2	Nothing needs the vault	Swarm restarts from persisted specs; stack.env / thermograph.env / centralis.env all persist. This property exists because we render a file, and would be destroyed by app-native reads
Secret-id leak	Useless off the mesh	secret_id_bound_cidrs; revoke by accessor without knowing the value
beta credential reaches prod	Cross-env compromise	Separate AppRoles, separate policies, secret-ids owned by the separate deploy users (agent vs deploy) — a boundary that does not exist today, since both currently reach the same root-owned age key. Honest limit: root on vps2 defeats it, exactly as it defeats the single age key now
Upgrade to 2.7.0	Removes the file backend and built-in cloud KMS seals	Already avoided by choosing raft + static. Pin the version

---

Deliberately out of scope

For ~40 credentials and one operator the real risk is over-engineering. Not adopted: HA/multi-node raft, namespaces, dynamic database credentials, bao agent, cert auth, PKI, identity groups, transit seal (revisit later), OIDC for the operator.

OIDC for CI is out of scope for a reason worth recording. Forgejo Actions does support OIDC — shipped in Forgejo v15.0, needing Runner > v12.5.0, enabled via enable-openid-connect: true rather than GitHub's permissions: id-token: write. But this estate runs codeberg.org/forgejo/forgejo:9-rootless (infra/deploy/forgejo/docker-stack.yml:59). That is a six-major-version upgrade, and coupling it to this migration would mean two large independent migrations at once with no way to tell which one broke. Revisit after Forgejo reaches v15; then role_type=jwt with bound_claims removes the last CI-side bearer credential.

CI keeps its SSH keys and REGISTRY_TOKEN as Forgejo secrets. CI gets no vault access at all — the SSH-in architecture already means the host renders, never the runner, which is the least-privilege topology and should be preserved rather than "upgraded".

Secret zero, honestly

Nothing eliminates it; you choose where it sits and how little of it there is. After this phase the chain terminates at 9 Forgejo Actions secrets + 1 static seal key + 4 AppRole secret-ids, versus today's 13 CI secrets + 1 omnipotent age key per host.

---

Verification gaps to close on the box

Flagged rather than guessed, because these could not be checked from a workstation:

1. Whether a raft snapshot is restorable under a different seal key. Docs don't say. Test in phase 0, before anything depends on it.
2. Exact -wrap-ttl flag spelling on bao write — documented in the response-wrapping concept page but absent from the commands index. Check bao write -h.
3. Debian/Ubuntu package repo URL. The install docs claim .deb packages exist but give no repository; bootstrap.sh uses the GitHub release tarball with checksum verification, which is fine and also fixes a standing weakness — provision-secrets.sh installs sops and age with a bare sudo curl and no verification at all.
4. Vault-era naming persists inside OpenBao config (vault { } stanza, X-Vault-Wrap-TTL header). Expect it; don't "fix" it.