WIP: openbao: add a dormant OpenBao backend alongside SOPS #129
No reviewers
Labels
No labels
Compat/Breaking
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Security
Kind/Testing
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Status
Abandoned
Status
Blocked
Status
Need More Info
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference: Jinemi/thermograph#129
Loading…
Reference in a new issue
No description provided.
Delete branch "worktree-openbao-migration"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Marked WIP deliberately — see "what still needs a human" at the bottom. Nothing here is cut over.
Status: dormant
TG_SECRETS_BACKENDdefaults tosopsfor all three environments inenv-topology.sh, so SOPS remains authoritative for dev, beta and prod. Flipping an environment is a one-line reviewed change and should not happen untilverify-parity.shpasses.Verified against OpenBao v2.6.1. The binary is
bao.Why bother — one specific failure class
Not encryption, and not audit, though the audit log is a real gain.
infra/CLAUDE.mdrequires that "vps1 must never hold prod credentials. It's the box that runs Forgejo, its CI runner, and dev's unreviewed branch — the opposite of an isolation boundary."Today that rule is a shell flag —
THERMOGRAPH_SECRETS_SKIP_COMMON— set by the caller, at three separate sites (env-topology.sh:196→deploy.sh:67,deploy-dev.sh:86,infra-sync.yml:93-95). The renderer does not know thatenv=devmeans never-common. A fourth call site, or one by-handrender_thermograph_secrets /opt/thermograph-dev/infra dev, writes both S3 keypairs (one read-write on the bucket holding prod's database backups), the VAPID private key that signs Web Push to real subscribers,REGISTRY_TOKEN, and the IndexNow and metrics tokens onto the CI-runner box — and exits 0.policies/tg-host-dev.hclmakes that a 403. The failure class is gone, not guarded. The estate already hardened this once by hand (7583258), which is the shape of a problem that wants an ACL.Shape
deploy-stack.shrenders secrets to deploy the stack). A plaindocker runmakes the vault depend on the daemon deploys restart, and is oneprune -afrom gonefilebackend for removal in 2.7.0. Raft has the only backup primitive. A two-node raft would be worse than one — quorum of two loses writes if either diesagentvsdeploy) — a boundary that does not exist today, since both reach the same root-owned age keyRisk containment
The dispatch in
render-secrets.shreturns early rather than refactoring the shared write path, so the SOPS path stays byte-identical. That file is on the deploy path for all three environments with no test suite in front of it. Verified after the change:baoinstalled → rc=1, fails closed.shin the treeThe duplicated write path is documented with a note to consolidate once prod has been on OpenBao for a release cycle.
Two things this fixes on the way past
A latent code-execution path.
/etc/thermograph.envis raw unquoted dotenv but issourced by bash in six places (deploy.sh:133,deploy-stack.sh:98, both daemon entrypoints,ops-cron.yml:85, terraform). Any value containing a space, quote, backtick,$(or backslash would truncate, corrupt, or execute. It works today only because every current value happens to be alphanumeric — so "rotate to a stronger password" is currently a live RCE risk on the deploy host. The OpenBao renderer rejects such values at both render and seed time.The age key is not retired by this.
ops-cron.yml:142,197stream every off-box dump throughpg_dump | age -r <recipient> | rclone rcatwith 30-day retention, and restore uses/etc/thermograph/age.key. So the age keypair is the backup encryption key, not just the SOPS transport. Deleting it with the vault files would silently destroy a month of database recoverability. (This became load-bearing within hours: the Forgejo recovery in #128 depended on exactly that key.)Honest about what this does not buy
THERMOGRAPH_DATABASE_URLonce at import and hold a pool, and there is no reload path anywhere in the codebase.secrets-guard. Under OpenBao it is an out-of-band API call with no diff and no review. This is a genuine regression and a decision to make deliberately, not a detail. Partially mitigated by a committed key-name manifest plus the audit log.Not in scope, with reasons
OIDC for CI is deferioned: Forgejo Actions OIDC shipped in v15.0, but this estate runs
forgejo:9-rootless(deploy/forgejo/docker-stack.yml:59). That is a six-major-version upgrade, and coupling it here would mean two large migrations at once with no way to tell which broke. CI keeps its SSH keys andREGISTRY_TOKENas Forgejo secrets and gets no vault access at all — the SSH-in architecture already means the host renders, never the runner, which is the least-privilege topology.Also out: HA raft, namespaces,
bao agent, cert auth, PKI, transit seal (revisit later).What still needs a human
bootstrap.shstops and waits for this on purpose. Any place a script could store them is worse custody than the age key this is meant to improve on.seed-from-sops.sh. It reads every production secret in plaintext;infra/CLAUDE.mdsays the equivalentseed-from-live.shis explicitly not for an agent to run, and this inherits that.Cutover order once those are done:
verify-parity.sh --allgreen, then dev → beta → prod, one PR per hop, 7 consecutive green parity runs before prod.forgejo_db has been at 0/1 replicas since 2026-07-29 01:24 UTC. Postgres hit an invalid data-directory lock file ("could not open file postmaster.pid ... performing immediate shutdown because data directory lock file is invalid") and exited 0. With restart_policy.condition=on-failure, Swarm read the zero status as successful completion, marked the task Complete, and never rescheduled it. The forgejo service itself stayed Up and kept serving its homepage, so the outage presented as every repository page, the whole API and all CI returning 500 with "dial tcp: lookup db on 127.0.0.11:53: no such host" — including the auth path, which is why API calls reported "user does not exist [uid: 0]" rather than a database error. on-failure cannot distinguish "finished successfully" from "shut itself down and should be restarted", and Postgres exits 0 on several such paths, so it is the wrong policy for an always-on stateful service. This is the durable fix; it does not restart the currently stopped task.View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.