b62eb60121
All checks were successful
PR build (required check) / changes (pull_request) Successful in 7s
secrets-guard / encrypted (pull_request) Successful in 6s
PR build (required check) / build-backend (pull_request) Has been skipped
shell-lint / shellcheck (pull_request) Successful in 7s
PR build (required check) / build-frontend (pull_request) Has been skipped
PR build (required check) / validate-observability (pull_request) Has been skipped
PR build (required check) / gate (pull_request) Successful in 2s
shell-lint / shellcheck (push) Successful in 8s
secrets-guard / encrypted (push) Successful in 7s
render_centralis_secrets has existed since the OpenBao work and had no caller on any deploy path -- only verify-centralis-render.sh, a check. A renderer nothing calls is not a migration, it is a file. /etc/centralis.env has stayed hand-maintained the whole time, with the vault holding a copy nothing consumed. That gap is why turning on Google sign-in for Centralis had no safe home: the only way to set CENTRALIS_GOOGLE_CLIENT_ID was to hand-edit the live file, which is precisely what render-secrets.sh's header warns against. THE GUARD IS THE POINT. On 2026-07-24 this estate lost Centralis' token registry by writing a file missing a key the live one had. The renderer proves its own output round-trips through bash, but it cannot know about a key that exists only in the live file. So this job renders to a scratch path first, compares KEY SETS (names only, both sides sourced in `env -i` so the comparison is over what bash actually ends up with), and refuses to write -- failing loudly -- if the live file has anything the vault does not. Rendering turns from a way to lose a hand-edit into the thing that catches one. Verified the guard against a live file carrying a hand-added CENTRALIS_GOOGLE_CLIENT_ID: correctly identified as a key that would be deleted, write refused. Also verified the empty/unreadable-live-file path, which needs `|| true` to survive pipefail. It recreates the container where the other three jobs deliberately roll nothing, and the difference is real rather than an exception: Centralis' compose file interpolates /etc/centralis.env at PARSE time, so rendering without recreating changes nothing -- the silent no-op this estate keeps getting bitten by. `docker restart` would not do it either. One container, no replicas, a two-second recreate. It also skips the recreate entirely when the rendered file is byte-identical to the live one, so a no-op push does not bounce a healthy container.
267 lines
13 KiB
YAML
267 lines
13 KiB
YAML
name: Sync infra to hosts
|
|
|
|
# The infra DOMAIN's own pipeline (new with the monorepo -- the split era had
|
|
# no infra deploy workflow at all; infra changes rode along lazily with the
|
|
# next app deploy's `git reset`). On a push touching infra/**, SSH to every
|
|
# host, fast-forward each ENVIRONMENT's monorepo checkout and re-render that
|
|
# environment's env file from the SOPS vault -- so a compose edit or a secret
|
|
# rotation lands push-button instead of waiting for the next app deploy.
|
|
#
|
|
# Deliberately does NOT roll any service: image tags are the app domains'
|
|
# axis, not infra's. A compose or stack change that must recreate containers
|
|
# takes effect on the next app deploy, or a by-hand
|
|
# `SERVICE=all ... infra/deploy/deploy.sh` run (see that script's header).
|
|
#
|
|
# THREE CHECKOUTS, TWO HOSTS. This is the shape the vps1/vps2 split forces:
|
|
#
|
|
# vps1 /opt/thermograph-dev branch dev -> /etc/thermograph.env
|
|
# vps2 /opt/thermograph-beta branch main -> /etc/thermograph-beta.env
|
|
# vps2 /opt/thermograph branch main -> /etc/thermograph.env
|
|
#
|
|
# The two checkouts on vps2 are separate directories on purpose: a `git reset
|
|
# --hard` for beta must not be able to move prod's tree, and each environment
|
|
# renders its own env file from its own vault file. Because the paths differ,
|
|
# the beta and prod jobs below can safely run at the same time on one box.
|
|
#
|
|
# Beta and prod both track `main` (infra is not environment-staged -- only app
|
|
# IMAGES are, via tags). Dev tracks `dev`, which is why this workflow now
|
|
# triggers on both branches: a dev-branch infra change has to reach the dev
|
|
# checkout, and only that one.
|
|
#
|
|
# Each `render_thermograph_secrets` call passes the environment name and the
|
|
# output path EXPLICITLY. It used to rely on the host's
|
|
# /etc/thermograph/secrets-env marker, which cannot answer the question on vps2
|
|
# -- one marker, two environments. Passing them here means a beta sync can
|
|
# never render prod's vault, and vice versa.
|
|
|
|
on:
|
|
push:
|
|
branches: [dev, main]
|
|
paths: ['infra/**']
|
|
workflow_dispatch: {}
|
|
|
|
jobs:
|
|
sync-dev:
|
|
# Only the dev branch feeds the dev checkout.
|
|
if: github.ref_name == 'dev'
|
|
runs-on: docker
|
|
concurrency:
|
|
group: infra-sync-dev
|
|
cancel-in-progress: false
|
|
steps:
|
|
- name: Sync dev checkout on vps1 + re-render secrets
|
|
uses: https://github.com/appleboy/ssh-action@v1.2.0
|
|
with:
|
|
host: ${{ secrets.VPS1_SSH_HOST }}
|
|
username: ${{ secrets.VPS1_SSH_USER }}
|
|
key: ${{ secrets.VPS1_SSH_KEY }}
|
|
port: ${{ secrets.VPS1_SSH_PORT }}
|
|
script: |
|
|
set -euo pipefail
|
|
cd /opt/thermograph-dev
|
|
git fetch --prune origin dev
|
|
git reset --hard origin/dev
|
|
echo "synced to $(git log --oneline -1)"
|
|
|
|
# RENDER ONLY IF THIS HOST IS ACTUALLY DEV.
|
|
#
|
|
# dev's env file is /etc/thermograph.env — the same path beta uses,
|
|
# and during the vps1/vps2 cutover beta is STILL ON VPS1. Rendering
|
|
# unconditionally would overwrite the live env file of the
|
|
# environment currently serving beta.thermograph.org with dev's
|
|
# credentials: beta's running containers would survive (env_file is
|
|
# read at container start) and then come up wrong on its next
|
|
# deploy, pointed at dev's database password.
|
|
#
|
|
# The host marker is the box's own statement of which environment it
|
|
# is, and it is the right authority for "is it safe to write this
|
|
# file here". It flips to `dev` in provision-dev.sh, which is run
|
|
# only after beta has vacated. Until then this skips loudly rather
|
|
# than failing: the checkout sync above is still useful and correct.
|
|
marker=$(cat /etc/thermograph/secrets-env 2>/dev/null || true)
|
|
if [ "$marker" != dev ]; then
|
|
echo "!! /etc/thermograph/secrets-env says '${marker:-<unset>}', not 'dev'."
|
|
echo "!! Skipping the secrets render — this host is not dev yet, and"
|
|
echo "!! /etc/thermograph.env here belongs to '${marker:-another environment}'."
|
|
exit 0
|
|
fi
|
|
if [ -f infra/deploy/render-secrets.sh ]; then
|
|
# env-topology.sh FIRST: it is what sets TG_SECRETS_BACKEND, and
|
|
# render-secrets.sh:44 defaults to `sops` when it is unset. Without
|
|
# this line the cutover would be only half-applied — deploy.sh would
|
|
# read the new backend while THIS workflow, which runs on every push
|
|
# touching infra/**, kept re-rendering the same file from SOPS. That
|
|
# is silent while parity holds and a hard failure the moment the SOPS
|
|
# files are retired.
|
|
. infra/deploy/env-topology.sh
|
|
thermograph_topology dev
|
|
. infra/deploy/render-secrets.sh
|
|
# SKIP_COMMON is dev's standing rule, not a preference: common.yaml
|
|
# is the fleet's shared production credential set, and vps1 also
|
|
# runs Forgejo and its CI. See render-secrets.sh.
|
|
THERMOGRAPH_SECRETS_SKIP_COMMON=1 \
|
|
render_thermograph_secrets /opt/thermograph-dev/infra dev /etc/thermograph.env
|
|
fi
|
|
|
|
sync-beta:
|
|
if: github.ref_name == 'main'
|
|
runs-on: docker
|
|
concurrency:
|
|
group: infra-sync-beta
|
|
cancel-in-progress: false
|
|
steps:
|
|
- name: Sync beta checkout on vps2 + re-render secrets
|
|
uses: https://github.com/appleboy/ssh-action@v1.2.0
|
|
with:
|
|
host: ${{ secrets.VPS2_SSH_HOST }}
|
|
username: ${{ secrets.VPS2_SSH_USER }}
|
|
key: ${{ secrets.VPS2_SSH_KEY }}
|
|
port: ${{ secrets.VPS2_SSH_PORT }}
|
|
script: |
|
|
set -euo pipefail
|
|
cd /opt/thermograph-beta
|
|
git fetch --prune origin main
|
|
git reset --hard origin/main
|
|
if [ -f infra/deploy/render-secrets.sh ]; then
|
|
# See the note in sync-dev: env-topology.sh is what sets
|
|
# TG_SECRETS_BACKEND, and for beta it also sets TG_BAO_APPROLE to
|
|
# the -beta credential. Beta shares a filesystem with prod, so the
|
|
# bare default is prod's file and tg-host-prod.hcl denies beta's
|
|
# own path — a 403 at render time on the box that runs prod.
|
|
. infra/deploy/env-topology.sh
|
|
thermograph_topology beta
|
|
. infra/deploy/render-secrets.sh
|
|
render_thermograph_secrets /opt/thermograph-beta/infra beta /etc/thermograph-beta.env
|
|
fi
|
|
echo "synced to $(git log --oneline -1)"
|
|
|
|
sync-prod:
|
|
if: github.ref_name == 'main'
|
|
runs-on: docker
|
|
concurrency:
|
|
group: infra-sync-prod
|
|
cancel-in-progress: false
|
|
steps:
|
|
- name: Sync prod checkout on vps2 + re-render secrets
|
|
uses: https://github.com/appleboy/ssh-action@v1.2.0
|
|
with:
|
|
host: ${{ secrets.VPS2_SSH_HOST }}
|
|
username: ${{ secrets.VPS2_SSH_USER }}
|
|
key: ${{ secrets.VPS2_SSH_KEY }}
|
|
port: ${{ secrets.VPS2_SSH_PORT }}
|
|
script: |
|
|
set -euo pipefail
|
|
cd /opt/thermograph
|
|
git fetch --prune origin main
|
|
git reset --hard origin/main
|
|
if [ -f infra/deploy/render-secrets.sh ]; then
|
|
# See the note in sync-dev — without env-topology.sh this job keeps
|
|
# rendering from SOPS regardless of what the backend selector says.
|
|
. infra/deploy/env-topology.sh
|
|
thermograph_topology prod
|
|
. infra/deploy/render-secrets.sh
|
|
render_thermograph_secrets /opt/thermograph/infra prod /etc/thermograph.env
|
|
fi
|
|
echo "synced to $(git log --oneline -1)"
|
|
|
|
sync-centralis:
|
|
# Centralis' /etc/centralis.env has been HAND-MAINTAINED. The vault has held a
|
|
# copy since deploy/secrets/centralis.prod.yaml was seeded, and
|
|
# render_centralis_secrets has existed since the OpenBao work -- but nothing
|
|
# on any deploy path ever called it. A renderer with no caller is not a
|
|
# migration, it is a file.
|
|
#
|
|
# That gap is why enabling Google sign-in on Centralis had no safe home: the
|
|
# only way to set CENTRALIS_GOOGLE_CLIENT_ID was to hand-edit the live file,
|
|
# which is exactly what render-secrets.sh's header warns against.
|
|
#
|
|
# WHY THIS JOB RECREATES A CONTAINER WHEN THE OTHER THREE DELIBERATELY DO NOT.
|
|
# This workflow's header says it never rolls a service, because image tags are
|
|
# the app domains' axis. Centralis is different in kind: its compose file
|
|
# interpolates /etc/centralis.env at PARSE time (`set -a && . /etc/centralis.env
|
|
# && docker compose up -d`), so rendering the file without recreating the
|
|
# container changes nothing at all. Rendering and not recreating would be the
|
|
# silent no-op this estate keeps getting bitten by. It is one container with no
|
|
# replicas -- a two-second recreate, not a deploy.
|
|
if: github.ref_name == 'main'
|
|
runs-on: docker
|
|
concurrency:
|
|
group: infra-sync-centralis
|
|
cancel-in-progress: false
|
|
steps:
|
|
- name: Render /etc/centralis.env from the vault, then recreate Centralis
|
|
uses: https://github.com/appleboy/ssh-action@v1.2.0
|
|
with:
|
|
host: ${{ secrets.VPS2_SSH_HOST }}
|
|
username: ${{ secrets.VPS2_SSH_USER }}
|
|
key: ${{ secrets.VPS2_SSH_KEY }}
|
|
port: ${{ secrets.VPS2_SSH_PORT }}
|
|
script: |
|
|
set -euo pipefail
|
|
umask 077
|
|
cd /opt/thermograph
|
|
[ -f infra/deploy/secrets/centralis.prod.yaml ] || { echo "no Centralis vault — nothing to do"; exit 0; }
|
|
|
|
. infra/deploy/env-topology.sh
|
|
thermograph_topology prod
|
|
. infra/deploy/render-secrets.sh
|
|
|
|
scratch=$(mktemp -d); chmod 700 "$scratch"
|
|
trap 'rm -rf -- "$scratch"' EXIT
|
|
|
|
# Render to scratch FIRST. CENTRALIS_RENDER_DEST is the override the
|
|
# renderer already carries for exactly this: look before writing.
|
|
CENTRALIS_RENDER_DEST="$scratch/rendered.env" \
|
|
render_centralis_secrets /opt/thermograph/infra
|
|
|
|
# THE GUARD. On 2026-07-24 this estate lost Centralis' token registry by
|
|
# writing a file that was missing a key the live one had. The renderer
|
|
# proves its OWN output round-trips through bash, but it cannot know
|
|
# about a key that exists only in the live file -- someone hand-editing
|
|
# /etc/centralis.env to add, say, CENTRALIS_GOOGLE_CLIENT_ID is invisible
|
|
# to it. So: the vault must be a SUPERSET of what is live, or nothing is
|
|
# written and the job fails loudly.
|
|
#
|
|
# Names only. Both files are sourced in a clean environment so the
|
|
# comparison is over what bash actually ends up with, not over text.
|
|
# The `|| true` is load-bearing under `set -o pipefail`: grep exits 1 when
|
|
# a file yields no matching keys, which would otherwise abort the job on
|
|
# the exact edge case this guard exists to handle -- an empty or
|
|
# unreadable live file, i.e. a first run.
|
|
keys() { env -i bash -c 'set -a; . "$1" >/dev/null 2>&1; set +a; compgen -v' _ "$1" \
|
|
| { grep -E '^(CENTRALIS_|DISCORD_)' || true; } | sort -u; }
|
|
keys "$scratch/rendered.env" > "$scratch/new.keys"
|
|
if [ -r /etc/centralis.env ]; then keys /etc/centralis.env > "$scratch/live.keys"; else : > "$scratch/live.keys"; fi
|
|
|
|
missing=$(comm -23 "$scratch/live.keys" "$scratch/new.keys" || true)
|
|
if [ -n "$missing" ]; then
|
|
echo "!! REFUSING to write /etc/centralis.env."
|
|
echo "!! The live file has keys the vault does not, so rendering would DELETE them:"
|
|
printf '!! %s\n' $missing
|
|
echo "!! Someone hand-edited the live file. Put those keys in the vault instead:"
|
|
echo "!! sops edit infra/deploy/secrets/centralis.prod.yaml"
|
|
exit 1
|
|
fi
|
|
|
|
added=$(comm -13 "$scratch/live.keys" "$scratch/new.keys" || true)
|
|
[ -n "$added" ] && printf ' new key from the vault: %s\n' $added
|
|
|
|
# Same file, byte for byte? Then there is nothing to do and no reason to
|
|
# bounce a healthy container.
|
|
if [ -r /etc/centralis.env ] && cmp -s "$scratch/rendered.env" /etc/centralis.env; then
|
|
echo "==> /etc/centralis.env already matches the vault — not recreating"
|
|
exit 0
|
|
fi
|
|
|
|
install -m 0600 -o "$(stat -c %U /etc/centralis.env 2>/dev/null || echo "$USER")" \
|
|
-g "$(stat -c %G /etc/centralis.env 2>/dev/null || echo "$USER")" \
|
|
"$scratch/rendered.env" /etc/centralis.env 2>/dev/null \
|
|
|| sudo install -m 0600 -o agent -g agent "$scratch/rendered.env" /etc/centralis.env
|
|
echo "==> wrote /etc/centralis.env from the vault"
|
|
|
|
# Recreate so compose re-reads it. `docker restart` would NOT: the values
|
|
# are interpolated when the compose file is parsed, not read at runtime.
|
|
cd /opt/centralis/deploy
|
|
set -a; . /etc/centralis.env; set +a
|
|
docker compose up -d
|
|
docker compose ps --format '{{.Name}}\t{{.State}}\t{{.Status}}'
|