thermograph/.forgejo/workflows/infra-sync.yml

166 lines
7.7 KiB
YAML
Raw Normal View History

name: Sync infra to hosts
# The infra DOMAIN's own pipeline (new with the monorepo -- the split era had
# no infra deploy workflow at all; infra changes rode along lazily with the
# next app deploy's `git reset`). On a push touching infra/**, SSH to every
# host, fast-forward each ENVIRONMENT's monorepo checkout and re-render that
# environment's env file from the SOPS vault -- so a compose edit or a secret
# rotation lands push-button instead of waiting for the next app deploy.
#
# Deliberately does NOT roll any service: image tags are the app domains'
# axis, not infra's. A compose or stack change that must recreate containers
# takes effect on the next app deploy, or a by-hand
# `SERVICE=all ... infra/deploy/deploy.sh` run (see that script's header).
#
# THREE CHECKOUTS, TWO HOSTS. This is the shape the vps1/vps2 split forces:
#
# vps1 /opt/thermograph-dev branch dev -> /etc/thermograph.env
# vps2 /opt/thermograph-beta branch main -> /etc/thermograph-beta.env
# vps2 /opt/thermograph branch main -> /etc/thermograph.env
#
# The two checkouts on vps2 are separate directories on purpose: a `git reset
# --hard` for beta must not be able to move prod's tree, and each environment
# renders its own env file from its own vault file. Because the paths differ,
# the beta and prod jobs below can safely run at the same time on one box.
#
# Beta and prod both track `main` (infra is not environment-staged -- only app
# IMAGES are, via tags). Dev tracks `dev`, which is why this workflow now
# triggers on both branches: a dev-branch infra change has to reach the dev
# checkout, and only that one.
#
# Each `render_thermograph_secrets` call passes the environment name and the
# output path EXPLICITLY. It used to rely on the host's
# /etc/thermograph/secrets-env marker, which cannot answer the question on vps2
# -- one marker, two environments. Passing them here means a beta sync can
# never render prod's vault, and vice versa.
on:
push:
branches: [dev, main]
paths: ['infra/**']
workflow_dispatch: {}
jobs:
sync-dev:
# Only the dev branch feeds the dev checkout.
if: github.ref_name == 'dev'
runs-on: docker
concurrency:
group: infra-sync-dev
cancel-in-progress: false
steps:
- name: Sync dev checkout on vps1 + re-render secrets
uses: https://github.com/appleboy/ssh-action@v1.2.0
with:
host: ${{ secrets.VPS1_SSH_HOST }}
username: ${{ secrets.VPS1_SSH_USER }}
key: ${{ secrets.VPS1_SSH_KEY }}
port: ${{ secrets.VPS1_SSH_PORT }}
script: |
set -euo pipefail
cd /opt/thermograph-dev
git fetch --prune origin dev
git reset --hard origin/dev
echo "synced to $(git log --oneline -1)"
# RENDER ONLY IF THIS HOST IS ACTUALLY DEV.
#
# dev's env file is /etc/thermograph.env — the same path beta uses,
# and during the vps1/vps2 cutover beta is STILL ON VPS1. Rendering
# unconditionally would overwrite the live env file of the
# environment currently serving beta.thermograph.org with dev's
# credentials: beta's running containers would survive (env_file is
# read at container start) and then come up wrong on its next
# deploy, pointed at dev's database password.
#
# The host marker is the box's own statement of which environment it
# is, and it is the right authority for "is it safe to write this
# file here". It flips to `dev` in provision-dev.sh, which is run
# only after beta has vacated. Until then this skips loudly rather
# than failing: the checkout sync above is still useful and correct.
marker=$(cat /etc/thermograph/secrets-env 2>/dev/null || true)
if [ "$marker" != dev ]; then
echo "!! /etc/thermograph/secrets-env says '${marker:-<unset>}', not 'dev'."
echo "!! Skipping the secrets render — this host is not dev yet, and"
echo "!! /etc/thermograph.env here belongs to '${marker:-another environment}'."
exit 0
fi
if [ -f infra/deploy/render-secrets.sh ]; then
openbao: make the parity gate actually work, and run it nightly Three fixes, all on the path between "the migration looks ready" and "the migration is ready". verify-parity.sh never sourced env-topology.sh, so TG_BAO_APPROLE was unset and render-secrets-openbao.sh fell through to the bare default -- prod's credential -- for every environment. On vps2 that made `--env beta` authenticate as tg-prod and take a 403 from tg-host-prod.hcl on thermograph/data/env/beta. `--all` failed the same way, taking dev with it. deploy.sh and deploy-stack.sh always sourced it; only the verifier did not, which is the worst place for the omission: the tool whose job is to notice divergence was itself diverging from the path it verifies. It now calls thermograph_topology per environment and takes TG_SKIP_COMMON from there rather than re-deriving it. infra-sync.yml renders all three env files on every push touching infra/**, and none of its three jobs sourced env-topology.sh either. TG_SECRETS_BACKEND was therefore unset and render-secrets.sh:44 defaulted to sops. Flipping the selector would have changed deploy.sh and deploy-stack.sh but not this workflow, which would have kept re-rendering from SOPS -- silent while parity holds, and a hard failure the moment the SOPS files are retired. The one-line cutover the README describes was never sufficient on its own. ops-cron.yml gains a secrets-parity job: prod and beta from vps2, dev from vps1, nightly, failing loudly on a mismatch. README.md called this the gate for a cutover; nothing implemented it. A gate that exists only in prose gets satisfied by assertion. It grants CI no vault access -- the host renders, CI only asks it to. First measured run, immediately after the verifier fix: dev 12 keys PASS, byte-identical beta 24 keys FAIL, 3 values differ prod 32 keys FAIL, 3 values differ The same three keys on both: THERMOGRAPH_S3_SECRET_KEY, THERMOGRAPH_LAKE_S3_SECRET_KEY, THERMOGRAPH_VAPID_CONTACT. All three live in common.yaml, seeded 2026-07-31 02:30Z -- before the Contabo rotation landed. dev passes because dev never layers common. One `seed-from-sops.sh --env common` fixes beta and prod together, and the 7-day clock starts after that.
2026-08-01 15:36:35 +00:00
# env-topology.sh FIRST: it is what sets TG_SECRETS_BACKEND, and
# render-secrets.sh:44 defaults to `sops` when it is unset. Without
# this line the cutover would be only half-applied — deploy.sh would
# read the new backend while THIS workflow, which runs on every push
# touching infra/**, kept re-rendering the same file from SOPS. That
# is silent while parity holds and a hard failure the moment the SOPS
# files are retired.
. infra/deploy/env-topology.sh
thermograph_topology dev
. infra/deploy/render-secrets.sh
# SKIP_COMMON is dev's standing rule, not a preference: common.yaml
# is the fleet's shared production credential set, and vps1 also
# runs Forgejo and its CI. See render-secrets.sh.
THERMOGRAPH_SECRETS_SKIP_COMMON=1 \
render_thermograph_secrets /opt/thermograph-dev/infra dev /etc/thermograph.env
fi
sync-beta:
if: github.ref_name == 'main'
runs-on: docker
concurrency:
group: infra-sync-beta
cancel-in-progress: false
steps:
- name: Sync beta checkout on vps2 + re-render secrets
uses: https://github.com/appleboy/ssh-action@v1.2.0
with:
host: ${{ secrets.VPS2_SSH_HOST }}
username: ${{ secrets.VPS2_SSH_USER }}
key: ${{ secrets.VPS2_SSH_KEY }}
port: ${{ secrets.VPS2_SSH_PORT }}
script: |
set -euo pipefail
cd /opt/thermograph-beta
git fetch --prune origin main
git reset --hard origin/main
if [ -f infra/deploy/render-secrets.sh ]; then
openbao: make the parity gate actually work, and run it nightly Three fixes, all on the path between "the migration looks ready" and "the migration is ready". verify-parity.sh never sourced env-topology.sh, so TG_BAO_APPROLE was unset and render-secrets-openbao.sh fell through to the bare default -- prod's credential -- for every environment. On vps2 that made `--env beta` authenticate as tg-prod and take a 403 from tg-host-prod.hcl on thermograph/data/env/beta. `--all` failed the same way, taking dev with it. deploy.sh and deploy-stack.sh always sourced it; only the verifier did not, which is the worst place for the omission: the tool whose job is to notice divergence was itself diverging from the path it verifies. It now calls thermograph_topology per environment and takes TG_SKIP_COMMON from there rather than re-deriving it. infra-sync.yml renders all three env files on every push touching infra/**, and none of its three jobs sourced env-topology.sh either. TG_SECRETS_BACKEND was therefore unset and render-secrets.sh:44 defaulted to sops. Flipping the selector would have changed deploy.sh and deploy-stack.sh but not this workflow, which would have kept re-rendering from SOPS -- silent while parity holds, and a hard failure the moment the SOPS files are retired. The one-line cutover the README describes was never sufficient on its own. ops-cron.yml gains a secrets-parity job: prod and beta from vps2, dev from vps1, nightly, failing loudly on a mismatch. README.md called this the gate for a cutover; nothing implemented it. A gate that exists only in prose gets satisfied by assertion. It grants CI no vault access -- the host renders, CI only asks it to. First measured run, immediately after the verifier fix: dev 12 keys PASS, byte-identical beta 24 keys FAIL, 3 values differ prod 32 keys FAIL, 3 values differ The same three keys on both: THERMOGRAPH_S3_SECRET_KEY, THERMOGRAPH_LAKE_S3_SECRET_KEY, THERMOGRAPH_VAPID_CONTACT. All three live in common.yaml, seeded 2026-07-31 02:30Z -- before the Contabo rotation landed. dev passes because dev never layers common. One `seed-from-sops.sh --env common` fixes beta and prod together, and the 7-day clock starts after that.
2026-08-01 15:36:35 +00:00
# See the note in sync-dev: env-topology.sh is what sets
# TG_SECRETS_BACKEND, and for beta it also sets TG_BAO_APPROLE to
# the -beta credential. Beta shares a filesystem with prod, so the
# bare default is prod's file and tg-host-prod.hcl denies beta's
# own path — a 403 at render time on the box that runs prod.
. infra/deploy/env-topology.sh
thermograph_topology beta
. infra/deploy/render-secrets.sh
render_thermograph_secrets /opt/thermograph-beta/infra beta /etc/thermograph-beta.env
fi
echo "synced to $(git log --oneline -1)"
sync-prod:
if: github.ref_name == 'main'
runs-on: docker
concurrency:
group: infra-sync-prod
cancel-in-progress: false
steps:
- name: Sync prod checkout on vps2 + re-render secrets
uses: https://github.com/appleboy/ssh-action@v1.2.0
with:
host: ${{ secrets.VPS2_SSH_HOST }}
username: ${{ secrets.VPS2_SSH_USER }}
key: ${{ secrets.VPS2_SSH_KEY }}
port: ${{ secrets.VPS2_SSH_PORT }}
script: |
set -euo pipefail
cd /opt/thermograph
git fetch --prune origin main
git reset --hard origin/main
if [ -f infra/deploy/render-secrets.sh ]; then
openbao: make the parity gate actually work, and run it nightly Three fixes, all on the path between "the migration looks ready" and "the migration is ready". verify-parity.sh never sourced env-topology.sh, so TG_BAO_APPROLE was unset and render-secrets-openbao.sh fell through to the bare default -- prod's credential -- for every environment. On vps2 that made `--env beta` authenticate as tg-prod and take a 403 from tg-host-prod.hcl on thermograph/data/env/beta. `--all` failed the same way, taking dev with it. deploy.sh and deploy-stack.sh always sourced it; only the verifier did not, which is the worst place for the omission: the tool whose job is to notice divergence was itself diverging from the path it verifies. It now calls thermograph_topology per environment and takes TG_SKIP_COMMON from there rather than re-deriving it. infra-sync.yml renders all three env files on every push touching infra/**, and none of its three jobs sourced env-topology.sh either. TG_SECRETS_BACKEND was therefore unset and render-secrets.sh:44 defaulted to sops. Flipping the selector would have changed deploy.sh and deploy-stack.sh but not this workflow, which would have kept re-rendering from SOPS -- silent while parity holds, and a hard failure the moment the SOPS files are retired. The one-line cutover the README describes was never sufficient on its own. ops-cron.yml gains a secrets-parity job: prod and beta from vps2, dev from vps1, nightly, failing loudly on a mismatch. README.md called this the gate for a cutover; nothing implemented it. A gate that exists only in prose gets satisfied by assertion. It grants CI no vault access -- the host renders, CI only asks it to. First measured run, immediately after the verifier fix: dev 12 keys PASS, byte-identical beta 24 keys FAIL, 3 values differ prod 32 keys FAIL, 3 values differ The same three keys on both: THERMOGRAPH_S3_SECRET_KEY, THERMOGRAPH_LAKE_S3_SECRET_KEY, THERMOGRAPH_VAPID_CONTACT. All three live in common.yaml, seeded 2026-07-31 02:30Z -- before the Contabo rotation landed. dev passes because dev never layers common. One `seed-from-sops.sh --env common` fixes beta and prod together, and the 7-day clock starts after that.
2026-08-01 15:36:35 +00:00
# See the note in sync-dev — without env-topology.sh this job keeps
# rendering from SOPS regardless of what the backend selector says.
. infra/deploy/env-topology.sh
thermograph_topology prod
. infra/deploy/render-secrets.sh
render_thermograph_secrets /opt/thermograph/infra prod /etc/thermograph.env
fi
echo "synced to $(git log --oneline -1)"