openbao: add a dormant OpenBao backend alongside SOPS #130

Merged

admin_emi merged 18 commits from openbao-backend into dev 2026-07-30 18:13:23 +00:00

Conversation 0 Commits 18 Files changed 15 +1620

admin_emi commented 2026-07-30 18:12:50 +00:00

Owner

Copy link

Supersedes #129, which is identical in content but was titled with a WIP: prefix — Forgejo treats that as a work-in-progress marker and blocks merging, surfacing as a misleading mergeable: false despite the branch being a clean fast-forward onto dev. Please close #129.

Merges dormant — SOPS stays authoritative

TG_SECRETS_BACKEND defaults to sops for all three environments in env-topology.sh, verified at merge time:

dev  -> TG_SECRETS_BACKEND=sops
beta -> TG_SECRETS_BACKEND=sops
prod -> TG_SECRETS_BACKEND=sops

So the new dispatch branch in render-secrets.sh is never taken. Merging this ships files plus an unused code path. Nothing about how secrets are resolved changes anywhere.

Verified against OpenBao v2.6.1. The binary is bao.

Why — one specific failure class

infra/CLAUDE.md requires that "vps1 must never hold prod credentials. It's the box that runs Forgejo, its CI runner, and dev's unreviewed branch — the opposite of an isolation boundary."

Today that rule is a shell flag — THERMOGRAPH_SECRETS_SKIP_COMMON — set by the caller, at three separate sites (env-topology.sh:196 → deploy.sh:67, deploy-dev.sh:86, infra-sync.yml:93-95). The renderer does not know that env=dev means never-common. A fourth call site, or one by-hand render_thermograph_secrets /opt/thermograph-dev/infra dev, writes both S3 keypairs (one read-write on the bucket holding prod's database backups), the VAPID private key that signs Web Push to real subscribers, REGISTRY_TOKEN, and the IndexNow and metrics tokens onto the CI-runner box — and exits 0.

policies/tg-host-dev.hcl makes that a 403. The failure class is gone, not guarded. The estate already hardened this once by hand (7583258), which is the shape of a problem that wants an ACL rather than another guard.

Shape

Decision	Choice	Why
Host	vps2 only	vps1 must never hold prod credentials; with a static seal that box would also hold the decryption key
Process	native systemd, not a container	A Swarm service is circular (deploy-stack.sh renders secrets to deploy the stack). A plain docker run makes the vault depend on the daemon deploys restart, and is one prune -a from gone
Storage	raft, single node	2.6.0 deprecated the file backend for removal in 2.7.0. Raft has the only backup primitive. A two-node raft is worse than one — quorum of two loses writes if either dies
Seal	static auto-unseal	A sealed vault blocks every deploy. Shamir needs a human after every reboot — an availability regression the estate does not have today. Static seal is the same trust model as the existing 0400 age key, with ACLs, audit and short tokens layered on
TLS	self-signed, on disk	Must not come from Caddy/ACME — that would put DNS and the public internet in the boot chain of the secret store
Auth	AppRole per env, CIDR-bound, 5m tokens	Each env's secret-id owned by its own deploy user (agent vs deploy) — a boundary that does not exist today, since both reach the same root-owned age key
Consumption	keep rendering a dotenv file	Keeps OpenBao a deploy-time dependency, never a runtime one, so a cold boot needs nothing from it

Risk containment

The dispatch in render-secrets.sh returns early rather than refactoring the shared write path, so the SOPS path stays byte-identical. That file is on the deploy path for all three environments with no test suite in front of it. Tested after the change, against the live hosts' actual key counts:

• SOPS render — dev 12 keys, beta 24, prod 32. Exact match to what each host has today.
• unknown backend → rc=1, output file untouched at 0 bytes
• openbao backend with no bao installed → rc=1, fails closed
• shellcheck clean across every .sh in the tree (shell-lint discovers new scripts automatically)

The duplicated write path is documented with a note to consolidate once prod has run on OpenBao for a release cycle.

Two things fixed on the way past

A latent code-execution path. /etc/thermograph.env is raw unquoted dotenv but is sourced by bash in six places (deploy.sh:133, deploy-stack.sh:98, both daemon entrypoints, ops-cron.yml:85, terraform). Any value containing a space, quote, backtick, $( or backslash would truncate, corrupt, or execute. It works today only because every current value happens to be alphanumeric — meaning "rotate to a stronger password" is currently a live RCE risk on the deploy host. The OpenBao renderer rejects such values at both render and seed time.

The age key is not retired by this. ops-cron.yml:142,197 stream every off-box dump through pg_dump | age -r <recipient> | rclone rcat with 30-day retention, and restore uses /etc/thermograph/age.key. The age keypair is the backup encryption key, not just the SOPS transport. Deleting it with the vault files would silently destroy a month of database recoverability — which became load-bearing within hours, since the Forgejo recovery in #128 depended on exactly that key.

Honest about what this does not buy

• No rotation of provider-issued credentials — Discord, Contabo S3, the registry token, VAPID, IndexNow. OpenBao cannot rotate any of them; Contabo exposes no IAM API.
• Dynamic DB credentials are inapplicable — apps read THERMOGRAPH_DATABASE_URL once at import and hold a pool, and there is no reload path anywhere in the codebase.
• Worse review than git. Today a secret change is a PR with a diff, backstopped by secrets-guard. Under OpenBao it is an out-of-band API call with no diff and no review. A genuine regression, to be decided deliberately. Partially mitigated by a committed key-name manifest plus the audit log.
• More moving parts. SOPS needs no server; this adds a stateful service to keep alive, upgrade, back up and TLS-rotate, for ~40 rarely-changing secrets.

Not in scope, with reasons

OIDC for CI is deferred: Forgejo Actions OIDC shipped in v15.0, but this estate runs forgejo:9-rootless (deploy/forgejo/docker-stack.yml:59). That is a six-major-version upgrade, and coupling it here would mean two large migrations at once with no way to tell which broke. CI keeps its SSH keys and REGISTRY_TOKEN as Forgejo secrets and gets no vault access at all — the SSH-in architecture already means the host renders, never the runner, which is the least-privilege topology.

Also out: HA raft, namespaces, bao agent, cert auth, PKI, transit seal (revisit later).

Still needs a human before any cutover

1. Custody of the unseal key and recovery shares. bootstrap.sh stops and waits for this on purpose.
2. The PR-review-for-secrets decision above.
3. Running seed-from-sops.sh — it reads every production secret in plaintext, and infra/CLAUDE.md says the equivalent seed-from-live.sh is explicitly not for an agent to run.
4. Verifying a raft snapshot restores under a different seal key — undocumented upstream, must be tested before anything depends on it.

Cutover order once done: verify-parity.sh --all green, then dev → beta → prod, one PR per hop, 7 consecutive green parity runs before prod.

Supersedes #129, which is identical in content but was titled with a `WIP:` prefix — Forgejo treats that as a work-in-progress marker and blocks merging, surfacing as a misleading `mergeable: false` despite the branch being a clean fast-forward onto `dev`. **Please close #129.** ## Merges dormant — SOPS stays authoritative `TG_SECRETS_BACKEND` defaults to `sops` for **all three** environments in `env-topology.sh`, verified at merge time: ``` dev -> TG_SECRETS_BACKEND=sops beta -> TG_SECRETS_BACKEND=sops prod -> TG_SECRETS_BACKEND=sops ``` So the new dispatch branch in `render-secrets.sh` is never taken. Merging this ships files plus an unused code path. Nothing about how secrets are resolved changes anywhere. Verified against **OpenBao v2.6.1**. The binary is `bao`. ## Why — one specific failure class `infra/CLAUDE.md` requires that *"vps1 must never hold prod credentials. It's the box that runs Forgejo, its CI runner, and dev's unreviewed branch — the opposite of an isolation boundary."* Today that rule is a **shell flag** — `THERMOGRAPH_SECRETS_SKIP_COMMON` — set by the *caller*, at three separate sites (`env-topology.sh:196` → `deploy.sh:67`, `deploy-dev.sh:86`, `infra-sync.yml:93-95`). The renderer does not know that `env=dev` means never-common. A fourth call site, or one by-hand `render_thermograph_secrets /opt/thermograph-dev/infra dev`, writes both S3 keypairs (one read-write on the bucket holding prod's database backups), the VAPID private key that signs Web Push to real subscribers, `REGISTRY_TOKEN`, and the IndexNow and metrics tokens onto the CI-runner box — **and exits 0**. `policies/tg-host-dev.hcl` makes that a 403. The failure class is gone, not guarded. The estate already hardened this once by hand (`7583258`), which is the shape of a problem that wants an ACL rather than another guard. ## Shape | Decision | Choice | Why | |---|---|---| | Host | vps2 only | vps1 must never hold prod credentials; with a static seal that box would also hold the decryption key | | Process | **native systemd**, not a container | A Swarm service is circular (`deploy-stack.sh` renders secrets to deploy the stack). A plain `docker run` makes the vault depend on the daemon deploys restart, and is one `prune -a` from gone | | Storage | raft, single node | 2.6.0 **deprecated the `file` backend** for removal in 2.7.0. Raft has the only backup primitive. A *two*-node raft is worse than one — quorum of two loses writes if either dies | | Seal | static auto-unseal | A sealed vault blocks every deploy. Shamir needs a human after every reboot — an availability regression the estate does not have today. Static seal is the *same* trust model as the existing 0400 age key, with ACLs, audit and short tokens layered on | | TLS | self-signed, on disk | Must not come from Caddy/ACME — that would put DNS and the public internet in the boot chain of the secret store | | Auth | AppRole per env, CIDR-bound, 5m tokens | Each env's secret-id owned by its own deploy user (`agent` vs `deploy`) — a boundary that does **not** exist today, since both reach the same root-owned age key | | Consumption | keep rendering a dotenv file | Keeps OpenBao a *deploy-time* dependency, never a runtime one, so a cold boot needs nothing from it | ## Risk containment The dispatch in `render-secrets.sh` **returns early** rather than refactoring the shared write path, so the SOPS path stays byte-identical. That file is on the deploy path for all three environments with no test suite in front of it. Tested after the change, against the live hosts' actual key counts: - SOPS render — dev **12 keys**, beta **24**, prod **32**. Exact match to what each host has today. - unknown backend → rc=1, output file untouched at 0 bytes - openbao backend with no `bao` installed → rc=1, fails closed - shellcheck clean across every `.sh` in the tree (`shell-lint` discovers new scripts automatically) The duplicated write path is documented with a note to consolidate once prod has run on OpenBao for a release cycle. ## Two things fixed on the way past **A latent code-execution path.** `/etc/thermograph.env` is raw unquoted dotenv but is `source`d by bash in six places (`deploy.sh:133`, `deploy-stack.sh:98`, both daemon entrypoints, `ops-cron.yml:85`, terraform). Any value containing a space, quote, backtick, `$(` or backslash would truncate, corrupt, or **execute**. It works today only because every current value happens to be alphanumeric — meaning "rotate to a stronger password" is currently a live RCE risk on the deploy host. The OpenBao renderer rejects such values at both render and seed time. **The age key is *not* retired by this.** `ops-cron.yml:142,197` stream every off-box dump through `pg_dump | age -r <recipient> | rclone rcat` with 30-day retention, and restore uses `/etc/thermograph/age.key`. The age keypair is the **backup encryption key**, not just the SOPS transport. Deleting it with the vault files would silently destroy a month of database recoverability — which became load-bearing within hours, since the Forgejo recovery in #128 depended on exactly that key. ## Honest about what this does not buy - **No rotation of provider-issued credentials** — Discord, Contabo S3, the registry token, VAPID, IndexNow. OpenBao cannot rotate any of them; Contabo exposes no IAM API. - **Dynamic DB credentials are inapplicable** — apps read `THERMOGRAPH_DATABASE_URL` once at import and hold a pool, and there is no reload path anywhere in the codebase. - **Worse review than git.** Today a secret change is a PR with a diff, backstopped by `secrets-guard`. Under OpenBao it is an out-of-band API call with no diff and no review. A genuine regression, to be decided deliberately. Partially mitigated by a committed key-name manifest plus the audit log. - **More moving parts.** SOPS needs no server; this adds a stateful service to keep alive, upgrade, back up and TLS-rotate, for ~40 rarely-changing secrets. ## Not in scope, with reasons OIDC for CI is deferred: Forgejo Actions OIDC shipped in **v15.0**, but this estate runs `forgejo:9-rootless` (`deploy/forgejo/docker-stack.yml:59`). That is a six-major-version upgrade, and coupling it here would mean two large migrations at once with no way to tell which broke. CI keeps its SSH keys and `REGISTRY_TOKEN` as Forgejo secrets and gets **no vault access at all** — the SSH-in architecture already means the host renders, never the runner, which is the least-privilege topology. Also out: HA raft, namespaces, `bao agent`, cert auth, PKI, transit seal (revisit later). ## Still needs a human before any cutover 1. **Custody of the unseal key and recovery shares.** `bootstrap.sh` stops and waits for this on purpose. 2. **The PR-review-for-secrets decision** above. 3. **Running `seed-from-sops.sh`** — it reads every production secret in plaintext, and `infra/CLAUDE.md` says the equivalent `seed-from-live.sh` is explicitly not for an agent to run. 4. **Verifying a raft snapshot restores under a different seal key** — undocumented upstream, must be tested before anything depends on it. Cutover order once done: `verify-parity.sh --all` green, then dev → beta → prod, one PR per hop, 7 consecutive green parity runs before prod.

admin_emi added 18 commits 2026-07-30 18:12:51 +00:00

Merge pull request 'promote: dev → main (CI consolidation, 15 workflows → 9)' (#90) from dev into main aa4ee7f4c4

Merge pull request 'promote: dev → main (alerting liveness rules onto the app heartbeat)' (#95) from dev into main c67d22b152

Merge pull request 'promote: dev → main (alert template: omit empty value line)' (#97) from dev into main 0766fd6bcb

Merge pull request 'promote: dev → main' (#100) from dev into main ba1085bf39

Merge pull request 'promote: dev → main (Weekly UTC-today date fix)' (#105) from dev into main 4d6ce5b70b

Merge pull request 'promote: dev → main (vps1/vps2 estate split)' (#106) from dev into main d5357d0670

Merge pull request 'promote: dev → main (cutover fixes: sync-dev guard, owner-db hardening, post-deploy gate)' (#110) from dev into main e6d9b36742

Merge pull request 'promote: dev → main (dev behind basic auth at dev.thermograph.org)' (#112) from dev into main c86489b207

Merge pull request 'promote: dev → main' (#114) from dev into main c1ab9410d3

Merge pull request 'promote: dev → main (sign in with Discord)' (#116) from dev into main efcf32defc

Merge pull request 'promote: dev → main (location bookmarks)' (#119) from dev into main 4f21e47750

frontend: load bookmarks-ui.js from the Go SSR home template too (#120) 2a4f516798

bookmarks: don't fabricate a local save when the server rejects it (#121) beb5fa5259

promote: dev → main ad4d2da120

Merge pull request 'promote: dev → main (sign in with Google; shared OAuth engine)' (#127) from dev into main 94de933cab

openbao: add a dormant OpenBao backend alongside SOPS ... 6bd07d33cb

Phase 1 of moving the secret store to OpenBao. Nothing is cut over:
TG_SECRETS_BACKEND defaults to sops for all three environments, so SOPS
remains authoritative until an environment is flipped in a reviewed PR.

The reason for the migration is one specific failure class. infra/CLAUDE.md
requires that vps1 never hold prod credentials, but that rule is currently
enforced by a caller-side shell flag (THERMOGRAPH_SECRETS_SKIP_COMMON) at
three call sites. A fourth call site, or one by-hand render, writes both S3
keypairs, the VAPID private key, REGISTRY_TOKEN and the IndexNow/metrics
tokens onto the Forgejo CI-runner box and exits 0. policies/tg-host-dev.hcl
makes that a 403 instead: dev's identity cannot read the common path at all.

Shape:
- vps2 only, native systemd (not a container: a Swarm service is circular,
  and a plain docker run is one `prune -a` from gone), raft storage (2.6.0
  deprecated the file backend), static auto-unseal so a reboot cannot block
  deploys, mesh-only self-signed TLS (not ACME — that would put DNS in the
  boot chain of the secret store).
- AppRole per environment, CIDR-bound, 5-minute tokens. Each environment's
  secret-id is owned by its own deploy user, which is a boundary that does
  not exist today since prod and beta both reach the same root-owned age key.
- Render still produces the same raw unquoted dotenv: Centralis compares
  /etc/thermograph.env textually and the Swarm entrypoint parses it by line.

The backend dispatch in render-secrets.sh returns early rather than
refactoring the shared write path, so the SOPS path stays byte-identical.
Verified by rendering dev (12 keys) and prod (32 keys) through it, matching
the live hosts. The duplicated write path is noted for consolidation once
prod has been on OpenBao for a release cycle.

The OpenBao renderer rejects values containing a newline or a shell
metacharacter. /etc/thermograph.env is sourced by bash in six places, so
such a value is a command-execution path on the deploy host; the SOPS path
has always had this hazard and avoids it only because every current value
happens to be alphanumeric.

Also records that the age keypair is NOT retired by this migration: it is
the backup encryption key for every off-box dump (ops-cron.yml:142,197,
30-day retention), so deleting it with the vault files would silently
destroy a month of database recoverability.

verify-parity.sh is the cutover gate — it renders both backends and diffs
them, reporting key names and counts only, never values.

forgejo: restart the db service on a clean exit, not just on failure ... 7878763b66

forgejo_db has been at 0/1 replicas since 2026-07-29 01:24 UTC. Postgres hit
an invalid data-directory lock file ("could not open file postmaster.pid ...
performing immediate shutdown because data directory lock file is invalid")
and exited 0. With restart_policy.condition=on-failure, Swarm read the zero
status as successful completion, marked the task Complete, and never
rescheduled it.

The forgejo service itself stayed Up and kept serving its homepage, so the
outage presented as every repository page, the whole API and all CI returning
500 with "dial tcp: lookup db on 127.0.0.11:53: no such host" — including the
auth path, which is why API calls reported "user does not exist [uid: 0]"
rather than a database error.

on-failure cannot distinguish "finished successfully" from "shut itself down
and should be restarted", and Postgres exits 0 on several such paths, so it is
the wrong policy for an always-on stateful service.

This is the durable fix; it does not restart the currently stopped task.

Merge dev into openbao branch (dev moved when #128 landed) 1941b184e5

admin_emi merged commit 2776a6cdb2 into dev 2026-07-30 18:13:23 +00:00

admin_emi deleted branch openbao-backend 2026-07-30 18:13:23 +00:00

admin_emi referenced this pull request from a commit 2026-07-30 18:13:24 +00:00

Merge pull request 'openbao: add a dormant OpenBao backend alongside SOPS' (#130) from openbao-backend into dev

admin_emi referenced this pull request 2026-07-30 18:19:00 +00:00

promote: dev → main #131

admin_emi referenced this pull request 2026-07-30 18:31:03 +00:00

promote: main → release #132

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

  

Compat/Breaking

Breaking change that won't be backward compatible

  

Kind/Bug

Something is not working

  

Kind/Documentation

Documentation changes

  

Kind/Enhancement

Improve existing functionality

  

Kind/Feature

New functionality

  

Kind/Security

This is security issue

  

Kind/Testing

Issue or pull request related to testing

  

Priority

Critical

The priority is critical

  

Priority

High

The priority is high

  

Priority

Low

The priority is low

  

Priority

Medium

The priority is medium

  

Reviewed

Confirmed

Issue has been confirmed

  

Reviewed

Duplicate

This issue or pull request already exists

  

Reviewed

Invalid

Invalid issue

  

Reviewed

Won't Fix

This issue won't be fixed

  

Status

Abandoned

Somebody has started to work on this but abandoned work

  

Status

Blocked

Something is blocking this issue or pull request

  

Status

Need More Info

Feedback is required to reproduce issue or to continue work

No labels

Compat/Breaking

Kind/Bug

Kind/Documentation

Kind/Enhancement

Kind/Feature

Kind/Security

Kind/Testing

Priority

Critical

Priority

High

Priority

Low

Priority

Medium

Reviewed

Confirmed

Reviewed

Duplicate

Reviewed

Invalid

Reviewed

Won't Fix

Status

Abandoned

Status

Blocked

Status

Need More Info

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Jinemi/thermograph#130

No description provided.