2026-07-23 05:11:33 +00:00
|
|
|
name: Sync infra to hosts
|
|
|
|
|
|
|
|
|
|
# The infra DOMAIN's own pipeline (new with the monorepo -- the split era had
|
|
|
|
|
# no infra deploy workflow at all; infra changes rode along lazily with the
|
2026-07-26 06:56:38 +00:00
|
|
|
# next app deploy's `git reset`). On a push touching infra/**, SSH to every
|
|
|
|
|
# host, fast-forward each ENVIRONMENT's monorepo checkout and re-render that
|
|
|
|
|
# environment's env file from the SOPS vault -- so a compose edit or a secret
|
|
|
|
|
# rotation lands push-button instead of waiting for the next app deploy.
|
2026-07-23 05:11:33 +00:00
|
|
|
#
|
|
|
|
|
# Deliberately does NOT roll any service: image tags are the app domains'
|
2026-07-26 06:56:38 +00:00
|
|
|
# axis, not infra's. A compose or stack change that must recreate containers
|
|
|
|
|
# takes effect on the next app deploy, or a by-hand
|
2026-07-23 05:11:33 +00:00
|
|
|
# `SERVICE=all ... infra/deploy/deploy.sh` run (see that script's header).
|
|
|
|
|
#
|
2026-07-26 06:56:38 +00:00
|
|
|
# THREE CHECKOUTS, TWO HOSTS. This is the shape the vps1/vps2 split forces:
|
|
|
|
|
#
|
|
|
|
|
# vps1 /opt/thermograph-dev branch dev -> /etc/thermograph.env
|
|
|
|
|
# vps2 /opt/thermograph-beta branch main -> /etc/thermograph-beta.env
|
|
|
|
|
# vps2 /opt/thermograph branch main -> /etc/thermograph.env
|
|
|
|
|
#
|
|
|
|
|
# The two checkouts on vps2 are separate directories on purpose: a `git reset
|
|
|
|
|
# --hard` for beta must not be able to move prod's tree, and each environment
|
|
|
|
|
# renders its own env file from its own vault file. Because the paths differ,
|
|
|
|
|
# the beta and prod jobs below can safely run at the same time on one box.
|
|
|
|
|
#
|
|
|
|
|
# Beta and prod both track `main` (infra is not environment-staged -- only app
|
|
|
|
|
# IMAGES are, via tags). Dev tracks `dev`, which is why this workflow now
|
|
|
|
|
# triggers on both branches: a dev-branch infra change has to reach the dev
|
|
|
|
|
# checkout, and only that one.
|
|
|
|
|
#
|
|
|
|
|
# Each `render_thermograph_secrets` call passes the environment name and the
|
|
|
|
|
# output path EXPLICITLY. It used to rely on the host's
|
|
|
|
|
# /etc/thermograph/secrets-env marker, which cannot answer the question on vps2
|
|
|
|
|
# -- one marker, two environments. Passing them here means a beta sync can
|
|
|
|
|
# never render prod's vault, and vice versa.
|
2026-07-23 05:11:33 +00:00
|
|
|
|
|
|
|
|
on:
|
|
|
|
|
push:
|
2026-07-26 06:56:38 +00:00
|
|
|
branches: [dev, main]
|
2026-07-23 05:11:33 +00:00
|
|
|
paths: ['infra/**']
|
|
|
|
|
workflow_dispatch: {}
|
|
|
|
|
|
|
|
|
|
jobs:
|
2026-07-26 06:56:38 +00:00
|
|
|
sync-dev:
|
|
|
|
|
# Only the dev branch feeds the dev checkout.
|
|
|
|
|
if: github.ref_name == 'dev'
|
|
|
|
|
runs-on: docker
|
|
|
|
|
concurrency:
|
|
|
|
|
group: infra-sync-dev
|
|
|
|
|
cancel-in-progress: false
|
|
|
|
|
steps:
|
|
|
|
|
- name: Sync dev checkout on vps1 + re-render secrets
|
|
|
|
|
uses: https://github.com/appleboy/ssh-action@v1.2.0
|
|
|
|
|
with:
|
|
|
|
|
host: ${{ secrets.VPS1_SSH_HOST }}
|
|
|
|
|
username: ${{ secrets.VPS1_SSH_USER }}
|
|
|
|
|
key: ${{ secrets.VPS1_SSH_KEY }}
|
|
|
|
|
port: ${{ secrets.VPS1_SSH_PORT }}
|
|
|
|
|
script: |
|
|
|
|
|
set -euo pipefail
|
|
|
|
|
cd /opt/thermograph-dev
|
|
|
|
|
git fetch --prune origin dev
|
|
|
|
|
git reset --hard origin/dev
|
2026-07-26 07:05:09 +00:00
|
|
|
echo "synced to $(git log --oneline -1)"
|
|
|
|
|
|
|
|
|
|
# RENDER ONLY IF THIS HOST IS ACTUALLY DEV.
|
|
|
|
|
#
|
|
|
|
|
# dev's env file is /etc/thermograph.env — the same path beta uses,
|
|
|
|
|
# and during the vps1/vps2 cutover beta is STILL ON VPS1. Rendering
|
|
|
|
|
# unconditionally would overwrite the live env file of the
|
|
|
|
|
# environment currently serving beta.thermograph.org with dev's
|
|
|
|
|
# credentials: beta's running containers would survive (env_file is
|
|
|
|
|
# read at container start) and then come up wrong on its next
|
|
|
|
|
# deploy, pointed at dev's database password.
|
|
|
|
|
#
|
|
|
|
|
# The host marker is the box's own statement of which environment it
|
|
|
|
|
# is, and it is the right authority for "is it safe to write this
|
|
|
|
|
# file here". It flips to `dev` in provision-dev.sh, which is run
|
|
|
|
|
# only after beta has vacated. Until then this skips loudly rather
|
|
|
|
|
# than failing: the checkout sync above is still useful and correct.
|
|
|
|
|
marker=$(cat /etc/thermograph/secrets-env 2>/dev/null || true)
|
|
|
|
|
if [ "$marker" != dev ]; then
|
|
|
|
|
echo "!! /etc/thermograph/secrets-env says '${marker:-<unset>}', not 'dev'."
|
|
|
|
|
echo "!! Skipping the secrets render — this host is not dev yet, and"
|
|
|
|
|
echo "!! /etc/thermograph.env here belongs to '${marker:-another environment}'."
|
|
|
|
|
exit 0
|
|
|
|
|
fi
|
2026-07-26 06:56:38 +00:00
|
|
|
if [ -f infra/deploy/render-secrets.sh ]; then
|
openbao: make the parity gate actually work, and run it nightly
Three fixes, all on the path between "the migration looks ready" and "the
migration is ready".
verify-parity.sh never sourced env-topology.sh, so TG_BAO_APPROLE was unset and
render-secrets-openbao.sh fell through to the bare default -- prod's credential
-- for every environment. On vps2 that made `--env beta` authenticate as
tg-prod and take a 403 from tg-host-prod.hcl on thermograph/data/env/beta.
`--all` failed the same way, taking dev with it. deploy.sh and
deploy-stack.sh always sourced it; only the verifier did not, which is the
worst place for the omission: the tool whose job is to notice divergence was
itself diverging from the path it verifies. It now calls thermograph_topology
per environment and takes TG_SKIP_COMMON from there rather than re-deriving it.
infra-sync.yml renders all three env files on every push touching infra/**, and
none of its three jobs sourced env-topology.sh either. TG_SECRETS_BACKEND was
therefore unset and render-secrets.sh:44 defaulted to sops. Flipping the
selector would have changed deploy.sh and deploy-stack.sh but not this
workflow, which would have kept re-rendering from SOPS -- silent while parity
holds, and a hard failure the moment the SOPS files are retired. The one-line
cutover the README describes was never sufficient on its own.
ops-cron.yml gains a secrets-parity job: prod and beta from vps2, dev from
vps1, nightly, failing loudly on a mismatch. README.md called this the gate for
a cutover; nothing implemented it. A gate that exists only in prose gets
satisfied by assertion. It grants CI no vault access -- the host renders, CI
only asks it to.
First measured run, immediately after the verifier fix:
dev 12 keys PASS, byte-identical
beta 24 keys FAIL, 3 values differ
prod 32 keys FAIL, 3 values differ
The same three keys on both: THERMOGRAPH_S3_SECRET_KEY,
THERMOGRAPH_LAKE_S3_SECRET_KEY, THERMOGRAPH_VAPID_CONTACT. All three live in
common.yaml, seeded 2026-07-31 02:30Z -- before the Contabo rotation landed.
dev passes because dev never layers common. One `seed-from-sops.sh --env
common` fixes beta and prod together, and the 7-day clock starts after that.
2026-08-01 15:36:35 +00:00
|
|
|
# env-topology.sh FIRST: it is what sets TG_SECRETS_BACKEND, and
|
|
|
|
|
# render-secrets.sh:44 defaults to `sops` when it is unset. Without
|
|
|
|
|
# this line the cutover would be only half-applied — deploy.sh would
|
|
|
|
|
# read the new backend while THIS workflow, which runs on every push
|
|
|
|
|
# touching infra/**, kept re-rendering the same file from SOPS. That
|
|
|
|
|
# is silent while parity holds and a hard failure the moment the SOPS
|
|
|
|
|
# files are retired.
|
|
|
|
|
. infra/deploy/env-topology.sh
|
|
|
|
|
thermograph_topology dev
|
2026-07-26 06:56:38 +00:00
|
|
|
. infra/deploy/render-secrets.sh
|
|
|
|
|
# SKIP_COMMON is dev's standing rule, not a preference: common.yaml
|
|
|
|
|
# is the fleet's shared production credential set, and vps1 also
|
|
|
|
|
# runs Forgejo and its CI. See render-secrets.sh.
|
|
|
|
|
THERMOGRAPH_SECRETS_SKIP_COMMON=1 \
|
|
|
|
|
render_thermograph_secrets /opt/thermograph-dev/infra dev /etc/thermograph.env
|
|
|
|
|
fi
|
|
|
|
|
|
2026-07-23 05:11:33 +00:00
|
|
|
sync-beta:
|
2026-07-26 06:56:38 +00:00
|
|
|
if: github.ref_name == 'main'
|
2026-07-23 05:11:33 +00:00
|
|
|
runs-on: docker
|
|
|
|
|
concurrency:
|
|
|
|
|
group: infra-sync-beta
|
|
|
|
|
cancel-in-progress: false
|
|
|
|
|
steps:
|
2026-07-26 06:56:38 +00:00
|
|
|
- name: Sync beta checkout on vps2 + re-render secrets
|
2026-07-23 05:11:33 +00:00
|
|
|
uses: https://github.com/appleboy/ssh-action@v1.2.0
|
|
|
|
|
with:
|
2026-07-26 06:56:38 +00:00
|
|
|
host: ${{ secrets.VPS2_SSH_HOST }}
|
|
|
|
|
username: ${{ secrets.VPS2_SSH_USER }}
|
|
|
|
|
key: ${{ secrets.VPS2_SSH_KEY }}
|
|
|
|
|
port: ${{ secrets.VPS2_SSH_PORT }}
|
2026-07-23 05:11:33 +00:00
|
|
|
script: |
|
|
|
|
|
set -euo pipefail
|
2026-07-26 06:56:38 +00:00
|
|
|
cd /opt/thermograph-beta
|
2026-07-23 05:11:33 +00:00
|
|
|
git fetch --prune origin main
|
|
|
|
|
git reset --hard origin/main
|
|
|
|
|
if [ -f infra/deploy/render-secrets.sh ]; then
|
openbao: make the parity gate actually work, and run it nightly
Three fixes, all on the path between "the migration looks ready" and "the
migration is ready".
verify-parity.sh never sourced env-topology.sh, so TG_BAO_APPROLE was unset and
render-secrets-openbao.sh fell through to the bare default -- prod's credential
-- for every environment. On vps2 that made `--env beta` authenticate as
tg-prod and take a 403 from tg-host-prod.hcl on thermograph/data/env/beta.
`--all` failed the same way, taking dev with it. deploy.sh and
deploy-stack.sh always sourced it; only the verifier did not, which is the
worst place for the omission: the tool whose job is to notice divergence was
itself diverging from the path it verifies. It now calls thermograph_topology
per environment and takes TG_SKIP_COMMON from there rather than re-deriving it.
infra-sync.yml renders all three env files on every push touching infra/**, and
none of its three jobs sourced env-topology.sh either. TG_SECRETS_BACKEND was
therefore unset and render-secrets.sh:44 defaulted to sops. Flipping the
selector would have changed deploy.sh and deploy-stack.sh but not this
workflow, which would have kept re-rendering from SOPS -- silent while parity
holds, and a hard failure the moment the SOPS files are retired. The one-line
cutover the README describes was never sufficient on its own.
ops-cron.yml gains a secrets-parity job: prod and beta from vps2, dev from
vps1, nightly, failing loudly on a mismatch. README.md called this the gate for
a cutover; nothing implemented it. A gate that exists only in prose gets
satisfied by assertion. It grants CI no vault access -- the host renders, CI
only asks it to.
First measured run, immediately after the verifier fix:
dev 12 keys PASS, byte-identical
beta 24 keys FAIL, 3 values differ
prod 32 keys FAIL, 3 values differ
The same three keys on both: THERMOGRAPH_S3_SECRET_KEY,
THERMOGRAPH_LAKE_S3_SECRET_KEY, THERMOGRAPH_VAPID_CONTACT. All three live in
common.yaml, seeded 2026-07-31 02:30Z -- before the Contabo rotation landed.
dev passes because dev never layers common. One `seed-from-sops.sh --env
common` fixes beta and prod together, and the 7-day clock starts after that.
2026-08-01 15:36:35 +00:00
|
|
|
# See the note in sync-dev: env-topology.sh is what sets
|
|
|
|
|
# TG_SECRETS_BACKEND, and for beta it also sets TG_BAO_APPROLE to
|
|
|
|
|
# the -beta credential. Beta shares a filesystem with prod, so the
|
|
|
|
|
# bare default is prod's file and tg-host-prod.hcl denies beta's
|
|
|
|
|
# own path — a 403 at render time on the box that runs prod.
|
|
|
|
|
. infra/deploy/env-topology.sh
|
|
|
|
|
thermograph_topology beta
|
2026-07-23 05:11:33 +00:00
|
|
|
. infra/deploy/render-secrets.sh
|
2026-07-26 06:56:38 +00:00
|
|
|
render_thermograph_secrets /opt/thermograph-beta/infra beta /etc/thermograph-beta.env
|
2026-07-23 05:11:33 +00:00
|
|
|
fi
|
|
|
|
|
echo "synced to $(git log --oneline -1)"
|
|
|
|
|
|
|
|
|
|
sync-prod:
|
2026-07-26 06:56:38 +00:00
|
|
|
if: github.ref_name == 'main'
|
2026-07-23 05:11:33 +00:00
|
|
|
runs-on: docker
|
|
|
|
|
concurrency:
|
|
|
|
|
group: infra-sync-prod
|
|
|
|
|
cancel-in-progress: false
|
|
|
|
|
steps:
|
2026-07-26 06:56:38 +00:00
|
|
|
- name: Sync prod checkout on vps2 + re-render secrets
|
2026-07-23 05:11:33 +00:00
|
|
|
uses: https://github.com/appleboy/ssh-action@v1.2.0
|
|
|
|
|
with:
|
2026-07-26 06:56:38 +00:00
|
|
|
host: ${{ secrets.VPS2_SSH_HOST }}
|
|
|
|
|
username: ${{ secrets.VPS2_SSH_USER }}
|
|
|
|
|
key: ${{ secrets.VPS2_SSH_KEY }}
|
|
|
|
|
port: ${{ secrets.VPS2_SSH_PORT }}
|
2026-07-23 05:11:33 +00:00
|
|
|
script: |
|
|
|
|
|
set -euo pipefail
|
|
|
|
|
cd /opt/thermograph
|
|
|
|
|
git fetch --prune origin main
|
|
|
|
|
git reset --hard origin/main
|
|
|
|
|
if [ -f infra/deploy/render-secrets.sh ]; then
|
openbao: make the parity gate actually work, and run it nightly
Three fixes, all on the path between "the migration looks ready" and "the
migration is ready".
verify-parity.sh never sourced env-topology.sh, so TG_BAO_APPROLE was unset and
render-secrets-openbao.sh fell through to the bare default -- prod's credential
-- for every environment. On vps2 that made `--env beta` authenticate as
tg-prod and take a 403 from tg-host-prod.hcl on thermograph/data/env/beta.
`--all` failed the same way, taking dev with it. deploy.sh and
deploy-stack.sh always sourced it; only the verifier did not, which is the
worst place for the omission: the tool whose job is to notice divergence was
itself diverging from the path it verifies. It now calls thermograph_topology
per environment and takes TG_SKIP_COMMON from there rather than re-deriving it.
infra-sync.yml renders all three env files on every push touching infra/**, and
none of its three jobs sourced env-topology.sh either. TG_SECRETS_BACKEND was
therefore unset and render-secrets.sh:44 defaulted to sops. Flipping the
selector would have changed deploy.sh and deploy-stack.sh but not this
workflow, which would have kept re-rendering from SOPS -- silent while parity
holds, and a hard failure the moment the SOPS files are retired. The one-line
cutover the README describes was never sufficient on its own.
ops-cron.yml gains a secrets-parity job: prod and beta from vps2, dev from
vps1, nightly, failing loudly on a mismatch. README.md called this the gate for
a cutover; nothing implemented it. A gate that exists only in prose gets
satisfied by assertion. It grants CI no vault access -- the host renders, CI
only asks it to.
First measured run, immediately after the verifier fix:
dev 12 keys PASS, byte-identical
beta 24 keys FAIL, 3 values differ
prod 32 keys FAIL, 3 values differ
The same three keys on both: THERMOGRAPH_S3_SECRET_KEY,
THERMOGRAPH_LAKE_S3_SECRET_KEY, THERMOGRAPH_VAPID_CONTACT. All three live in
common.yaml, seeded 2026-07-31 02:30Z -- before the Contabo rotation landed.
dev passes because dev never layers common. One `seed-from-sops.sh --env
common` fixes beta and prod together, and the 7-day clock starts after that.
2026-08-01 15:36:35 +00:00
|
|
|
# See the note in sync-dev — without env-topology.sh this job keeps
|
|
|
|
|
# rendering from SOPS regardless of what the backend selector says.
|
|
|
|
|
. infra/deploy/env-topology.sh
|
|
|
|
|
thermograph_topology prod
|
2026-07-23 05:11:33 +00:00
|
|
|
. infra/deploy/render-secrets.sh
|
2026-07-26 06:56:38 +00:00
|
|
|
render_thermograph_secrets /opt/thermograph/infra prod /etc/thermograph.env
|
2026-07-23 05:11:33 +00:00
|
|
|
fi
|
|
|
|
|
echo "synced to $(git log --oneline -1)"
|