promote: dev → main #131

Merged
admin_emi merged 5 commits from dev into main 2026-07-30 18:19:28 +00:00
Owner

Promotes devmain, deploying to beta (beta.thermograph.org).

Who merges. Centralis, once the required check is green — under the promotion policy main is the agent's call, and a promotion PR left open is worse than one never opened.

Divergence. 5 ahead, 0 behind. 5 commit(s) to carry, and the target has nothing of its own
Fast-forwardable: the target has no commits of its own to preserve.

Predicted conflicts. None: no file was touched on both sides since the branches parted.

Commits (5).

  • 2776a6cd Merge pull request 'openbao: add a dormant OpenBao backend alongside SOPS' (#130) from openbao-backend into dev
  • 1941b184 Merge dev into openbao branch (dev moved when #128 landed)
  • 8a2c8386 forgejo: restart the db service on a clean exit, not just on failure
  • 7878763b forgejo: restart the db service on a clean exit, not just on failure
  • 6bd07d33 openbao: add a dormant OpenBao backend alongside SOPS

Merge with a merge commit, not a squash or a fast-forward. main runs ahead of dev from earlier promotions, so --ff-only fails with "diverging branches", and squashing would rewrite the shared history so the next promotion sees every prior commit as new. (Making promotions fast-forward-only is an open question, not current policy — it needs a one-time history reconciliation on the protected branches first.)

Promotes `dev` → `main`, deploying to beta (beta.thermograph.org). **Who merges.** Centralis, once the required check is green — under the promotion policy `main` is the agent's call, and a promotion PR left open is worse than one never opened. **Divergence.** 5 ahead, 0 behind. 5 commit(s) to carry, and the target has nothing of its own Fast-forwardable: the target has no commits of its own to preserve. **Predicted conflicts.** None: no file was touched on both sides since the branches parted. **Commits (5).** - 2776a6cd Merge pull request 'openbao: add a dormant OpenBao backend alongside SOPS' (#130) from openbao-backend into dev - 1941b184 Merge dev into openbao branch (dev moved when #128 landed) - 8a2c8386 forgejo: restart the db service on a clean exit, not just on failure - 7878763b forgejo: restart the db service on a clean exit, not just on failure - 6bd07d33 openbao: add a dormant OpenBao backend alongside SOPS **Merge with a merge commit, not a squash or a fast-forward.** `main` runs ahead of `dev` from earlier promotions, so `--ff-only` fails with "diverging branches", and squashing would rewrite the shared history so the next promotion sees every prior commit as new. (Making promotions fast-forward-only is an open question, not current policy — it needs a one-time history reconciliation on the protected branches first.)
admin_emi added 5 commits 2026-07-30 18:19:01 +00:00
Phase 1 of moving the secret store to OpenBao. Nothing is cut over:
TG_SECRETS_BACKEND defaults to sops for all three environments, so SOPS
remains authoritative until an environment is flipped in a reviewed PR.

The reason for the migration is one specific failure class. infra/CLAUDE.md
requires that vps1 never hold prod credentials, but that rule is currently
enforced by a caller-side shell flag (THERMOGRAPH_SECRETS_SKIP_COMMON) at
three call sites. A fourth call site, or one by-hand render, writes both S3
keypairs, the VAPID private key, REGISTRY_TOKEN and the IndexNow/metrics
tokens onto the Forgejo CI-runner box and exits 0. policies/tg-host-dev.hcl
makes that a 403 instead: dev's identity cannot read the common path at all.

Shape:
- vps2 only, native systemd (not a container: a Swarm service is circular,
  and a plain docker run is one `prune -a` from gone), raft storage (2.6.0
  deprecated the file backend), static auto-unseal so a reboot cannot block
  deploys, mesh-only self-signed TLS (not ACME — that would put DNS in the
  boot chain of the secret store).
- AppRole per environment, CIDR-bound, 5-minute tokens. Each environment's
  secret-id is owned by its own deploy user, which is a boundary that does
  not exist today since prod and beta both reach the same root-owned age key.
- Render still produces the same raw unquoted dotenv: Centralis compares
  /etc/thermograph.env textually and the Swarm entrypoint parses it by line.

The backend dispatch in render-secrets.sh returns early rather than
refactoring the shared write path, so the SOPS path stays byte-identical.
Verified by rendering dev (12 keys) and prod (32 keys) through it, matching
the live hosts. The duplicated write path is noted for consolidation once
prod has been on OpenBao for a release cycle.

The OpenBao renderer rejects values containing a newline or a shell
metacharacter. /etc/thermograph.env is sourced by bash in six places, so
such a value is a command-execution path on the deploy host; the SOPS path
has always had this hazard and avoids it only because every current value
happens to be alphanumeric.

Also records that the age keypair is NOT retired by this migration: it is
the backup encryption key for every off-box dump (ops-cron.yml:142,197,
30-day retention), so deleting it with the vault files would silently
destroy a month of database recoverability.

verify-parity.sh is the cutover gate — it renders both backends and diffs
them, reporting key names and counts only, never values.
forgejo: restart the db service on a clean exit, not just on failure
All checks were successful
PR build (required check) / changes (pull_request) Successful in 5s
secrets-guard / encrypted (pull_request) Successful in 4s
PR build (required check) / build-backend (pull_request) Has been skipped
shell-lint / shellcheck (pull_request) Successful in 6s
PR build (required check) / build-frontend (pull_request) Has been skipped
PR build (required check) / validate-observability (pull_request) Has been skipped
PR build (required check) / gate (pull_request) Successful in 1s
7878763b66
forgejo_db has been at 0/1 replicas since 2026-07-29 01:24 UTC. Postgres hit
an invalid data-directory lock file ("could not open file postmaster.pid ...
performing immediate shutdown because data directory lock file is invalid")
and exited 0. With restart_policy.condition=on-failure, Swarm read the zero
status as successful completion, marked the task Complete, and never
rescheduled it.

The forgejo service itself stayed Up and kept serving its homepage, so the
outage presented as every repository page, the whole API and all CI returning
500 with "dial tcp: lookup db on 127.0.0.11:53: no such host" — including the
auth path, which is why API calls reported "user does not exist [uid: 0]"
rather than a database error.

on-failure cannot distinguish "finished successfully" from "shut itself down
and should be restarted", and Postgres exits 0 on several such paths, so it is
the wrong policy for an always-on stateful service.

This is the durable fix; it does not restart the currently stopped task.
forgejo: restart the db service on a clean exit, not just on failure
All checks were successful
shell-lint / shellcheck (pull_request) Successful in 6s
PR build (required check) / gate (pull_request) Successful in 1s
secrets-guard / encrypted (pull_request) Successful in 4s
PR build (required check) / changes (pull_request) Successful in 8s
PR build (required check) / build-backend (pull_request) Has been skipped
PR build (required check) / build-frontend (pull_request) Has been skipped
PR build (required check) / validate-observability (pull_request) Has been skipped
Sync infra to hosts / sync-beta (push) Has been skipped
Sync infra to hosts / sync-prod (push) Has been skipped
Sync infra to hosts / sync-dev (push) Successful in 5s
secrets-guard / encrypted (push) Successful in 4s
shell-lint / shellcheck (push) Successful in 6s
8a2c838663
forgejo_db has been at 0/1 replicas since 2026-07-29 01:24 UTC. Postgres hit
an invalid data-directory lock file ("could not open file postmaster.pid ...
performing immediate shutdown because data directory lock file is invalid")
and exited 0. With restart_policy.condition=on-failure, Swarm read the zero
status as successful completion, marked the task Complete, and never
rescheduled it.

The forgejo service itself stayed Up and kept serving its homepage, so the
outage presented as every repository page, the whole API and all CI returning
500 with "dial tcp: lookup db on 127.0.0.11:53: no such host" — including the
auth path, which is why API calls reported "user does not exist [uid: 0]"
rather than a database error.

on-failure cannot distinguish "finished successfully" from "shut itself down
and should be restarted", and Postgres exits 0 on several such paths, so it is
the wrong policy for an always-on stateful service.

This is the durable fix; it does not restart the currently stopped task.
Merge dev into openbao branch (dev moved when #128 landed)
All checks were successful
PR build (required check) / changes (pull_request) Successful in 6s
secrets-guard / encrypted (pull_request) Successful in 4s
PR build (required check) / build-backend (pull_request) Has been skipped
PR build (required check) / build-frontend (pull_request) Has been skipped
shell-lint / shellcheck (pull_request) Successful in 7s
PR build (required check) / validate-observability (pull_request) Has been skipped
PR build (required check) / gate (pull_request) Successful in 1s
1941b184e5
Merge pull request 'openbao: add a dormant OpenBao backend alongside SOPS' (#130) from openbao-backend into dev
All checks were successful
Sync infra to hosts / sync-beta (push) Has been skipped
Sync infra to hosts / sync-dev (push) Successful in 5s
secrets-guard / encrypted (push) Successful in 4s
Sync infra to hosts / sync-prod (push) Has been skipped
shell-lint / shellcheck (push) Successful in 6s
secrets-guard / encrypted (pull_request) Successful in 5s
PR build (required check) / build-backend (pull_request) Has been skipped
PR build (required check) / build-frontend (pull_request) Has been skipped
PR build (required check) / gate (pull_request) Successful in 1s
PR build (required check) / changes (pull_request) Successful in 6s
shell-lint / shellcheck (pull_request) Successful in 6s
PR build (required check) / validate-observability (pull_request) Has been skipped
2776a6cdb2
admin_emi merged commit dbc79351a2 into main 2026-07-30 18:19:28 +00:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Jinemi/thermograph#131
No description provided.