Compare commits

..

1 commit

Author SHA1 Message Date
Emi Griffith
a4bcae5bf9 lb: trust the host Caddy's X-Forwarded-Proto
All checks were successful
PR build (required check) / changes (pull_request) Successful in 7s
secrets-guard / encrypted (pull_request) Successful in 5s
PR build (required check) / build-backend (pull_request) Has been skipped
PR build (required check) / build-frontend (pull_request) Has been skipped
shell-lint / shellcheck (pull_request) Successful in 7s
PR build (required check) / validate-observability (pull_request) Has been skipped
PR build (required check) / gate (pull_request) Successful in 2s
Every OAuth redirect URI the backend builds is currently http://, whatever
scheme the user actually arrived with.

There are two Caddy hops in front of the app: the host Caddy terminates TLS
and sets X-Forwarded-Proto: https, then proxies to 127.0.0.1:8137 over plain
HTTP; the stack LB is the second hop. Caddy preserves an incoming
X-Forwarded-* header only when the immediate peer is a trusted proxy, and
trusted_proxies was never set here -- so the LB overwrote the header with the
scheme of the connection it had just received, which is http.

accounts/oauth.py:_redirect_uri builds the callback from x-forwarded-proto,
and that URI must match what is registered in the provider's console.

Measured on vps2, same request to each hop:

  direct to web, XFP=https      -> https://thermograph.org/api/v2/...
  through the LB, XFP=https     -> http://thermograph.org/api/v2/...
  through the PATCHED LB        -> https://thermograph.org/api/v2/...
  through the PATCHED LB, no XFP-> http://thermograph.org/api/v2/...

The last line is the point of using trusted_proxies rather than hardcoding
`header_up X-Forwarded-Proto https`: the LB still reports the truth when
nothing in front of it claims otherwise.

private_ranges covers 127.0.0.1/8 and the Docker bridge ranges, which is where
the host Caddy reaches this container from. The LB binds loopback only,
precisely so it cannot be reached un-fronted, so the only party that can set
these headers is the host Caddy -- trusting the local peer widens nothing.

Applied to beta's LB too. Beta is where a provider change gets tested before
it reaches thermograph.org, so it needs the same behaviour or the test is not
a test.
2026-08-01 09:06:11 -07:00
2 changed files with 8 additions and 153 deletions

View file

@ -163,105 +163,3 @@ jobs:
render_thermograph_secrets /opt/thermograph/infra prod /etc/thermograph.env render_thermograph_secrets /opt/thermograph/infra prod /etc/thermograph.env
fi fi
echo "synced to $(git log --oneline -1)" echo "synced to $(git log --oneline -1)"
sync-centralis:
# Centralis' /etc/centralis.env has been HAND-MAINTAINED. The vault has held a
# copy since deploy/secrets/centralis.prod.yaml was seeded, and
# render_centralis_secrets has existed since the OpenBao work -- but nothing
# on any deploy path ever called it. A renderer with no caller is not a
# migration, it is a file.
#
# That gap is why enabling Google sign-in on Centralis had no safe home: the
# only way to set CENTRALIS_GOOGLE_CLIENT_ID was to hand-edit the live file,
# which is exactly what render-secrets.sh's header warns against.
#
# WHY THIS JOB RECREATES A CONTAINER WHEN THE OTHER THREE DELIBERATELY DO NOT.
# This workflow's header says it never rolls a service, because image tags are
# the app domains' axis. Centralis is different in kind: its compose file
# interpolates /etc/centralis.env at PARSE time (`set -a && . /etc/centralis.env
# && docker compose up -d`), so rendering the file without recreating the
# container changes nothing at all. Rendering and not recreating would be the
# silent no-op this estate keeps getting bitten by. It is one container with no
# replicas -- a two-second recreate, not a deploy.
if: github.ref_name == 'main'
runs-on: docker
concurrency:
group: infra-sync-centralis
cancel-in-progress: false
steps:
- name: Render /etc/centralis.env from the vault, then recreate Centralis
uses: https://github.com/appleboy/ssh-action@v1.2.0
with:
host: ${{ secrets.VPS2_SSH_HOST }}
username: ${{ secrets.VPS2_SSH_USER }}
key: ${{ secrets.VPS2_SSH_KEY }}
port: ${{ secrets.VPS2_SSH_PORT }}
script: |
set -euo pipefail
umask 077
cd /opt/thermograph
[ -f infra/deploy/secrets/centralis.prod.yaml ] || { echo "no Centralis vault — nothing to do"; exit 0; }
. infra/deploy/env-topology.sh
thermograph_topology prod
. infra/deploy/render-secrets.sh
scratch=$(mktemp -d); chmod 700 "$scratch"
trap 'rm -rf -- "$scratch"' EXIT
# Render to scratch FIRST. CENTRALIS_RENDER_DEST is the override the
# renderer already carries for exactly this: look before writing.
CENTRALIS_RENDER_DEST="$scratch/rendered.env" \
render_centralis_secrets /opt/thermograph/infra
# THE GUARD. On 2026-07-24 this estate lost Centralis' token registry by
# writing a file that was missing a key the live one had. The renderer
# proves its OWN output round-trips through bash, but it cannot know
# about a key that exists only in the live file -- someone hand-editing
# /etc/centralis.env to add, say, CENTRALIS_GOOGLE_CLIENT_ID is invisible
# to it. So: the vault must be a SUPERSET of what is live, or nothing is
# written and the job fails loudly.
#
# Names only. Both files are sourced in a clean environment so the
# comparison is over what bash actually ends up with, not over text.
# The `|| true` is load-bearing under `set -o pipefail`: grep exits 1 when
# a file yields no matching keys, which would otherwise abort the job on
# the exact edge case this guard exists to handle -- an empty or
# unreadable live file, i.e. a first run.
keys() { env -i bash -c 'set -a; . "$1" >/dev/null 2>&1; set +a; compgen -v' _ "$1" \
| { grep -E '^(CENTRALIS_|DISCORD_)' || true; } | sort -u; }
keys "$scratch/rendered.env" > "$scratch/new.keys"
if [ -r /etc/centralis.env ]; then keys /etc/centralis.env > "$scratch/live.keys"; else : > "$scratch/live.keys"; fi
missing=$(comm -23 "$scratch/live.keys" "$scratch/new.keys" || true)
if [ -n "$missing" ]; then
echo "!! REFUSING to write /etc/centralis.env."
echo "!! The live file has keys the vault does not, so rendering would DELETE them:"
printf '!! %s\n' $missing
echo "!! Someone hand-edited the live file. Put those keys in the vault instead:"
echo "!! sops edit infra/deploy/secrets/centralis.prod.yaml"
exit 1
fi
added=$(comm -13 "$scratch/live.keys" "$scratch/new.keys" || true)
[ -n "$added" ] && printf ' new key from the vault: %s\n' $added
# Same file, byte for byte? Then there is nothing to do and no reason to
# bounce a healthy container.
if [ -r /etc/centralis.env ] && cmp -s "$scratch/rendered.env" /etc/centralis.env; then
echo "==> /etc/centralis.env already matches the vault — not recreating"
exit 0
fi
install -m 0600 -o "$(stat -c %U /etc/centralis.env 2>/dev/null || echo "$USER")" \
-g "$(stat -c %G /etc/centralis.env 2>/dev/null || echo "$USER")" \
"$scratch/rendered.env" /etc/centralis.env 2>/dev/null \
|| sudo install -m 0600 -o agent -g agent "$scratch/rendered.env" /etc/centralis.env
echo "==> wrote /etc/centralis.env from the vault"
# Recreate so compose re-reads it. `docker restart` would NOT: the values
# are interpolated when the compose file is parsed, not read at runtime.
cd /opt/centralis/deploy
set -a; . /etc/centralis.env; set +a
docker compose up -d
docker compose ps --format '{{.Name}}\t{{.State}}\t{{.Status}}'

View file

@ -36,13 +36,8 @@ usage: verify-parity.sh (--all | --env <dev|beta|prod> ...)
Renders each environment through BOTH backends and compares the resulting Renders each environment through BOTH backends and compares the resulting
KEY=value sets. Prints key names and counts only, never values. KEY=value sets. Prints key names and counts only, never values.
--all means "every environment that lives on THIS host", not all three: dev is Exit status: 0 = every requested environment is at parity; 1 = a mismatch.
on vps1, beta and prod are on vps2, so no single box can check them all. An Suitable as a CI / cron gate.
environment with no checkout here is skipped out loud. Skipping every one of
them is an error, not a pass.
Exit status: 0 = every environment checked here is at parity; 1 = a mismatch,
or nothing was checkable. Suitable as a CI / cron gate.
EOF EOF
exit 2 exit 2
} }
@ -74,43 +69,17 @@ done
. "$INFRA_DIR/deploy/render-secrets-openbao.sh" . "$INFRA_DIR/deploy/render-secrets-openbao.sh"
overall=0 overall=0
checked=0
for env_name in "${TARGETS[@]}"; do for env_name in "${TARGETS[@]}"; do
echo "== parity check: $env_name"
# Per-environment topology: TG_BAO_APPROLE (beta's differs, because beta shares a # Per-environment topology: TG_BAO_APPROLE (beta's differs, because beta shares a
# filesystem with prod), TG_SKIP_COMMON and TG_APP_DIR. Called before the header # filesystem with prod) and TG_SKIP_COMMON. Called inside the loop because --all
# line because --all walks three environments and may skip some of them. # walks three environments and the values differ per environment.
if ! thermograph_topology "$env_name"; then if ! thermograph_topology "$env_name"; then
echo "!! unknown environment: $env_name" >&2 echo "!! unknown environment: $env_name" >&2
overall=1; continue overall=1; continue
fi fi
# NO ENVIRONMENT CAN BE CHECKED FROM A HOST IT DOES NOT LIVE ON, and --all used to
# pretend otherwise. dev is on vps1; beta and prod are on vps2. Running --all
# anywhere therefore guaranteed a failure on whichever environments are elsewhere,
# and the failure was actively misleading: on vps1, prod resolves TG_BAO_APPROLE to
# /etc/thermograph/openbao-approle, which on THAT box holds dev's credential, so
# prod's check authenticated as tg-dev and took a 403. Fails closed, but reads like
# a policy bug rather than "wrong box".
#
# Identity comes from the ENVIRONMENT'S OWN ADDRESS being bound here, not from its
# checkout existing. The first version of this check tested for TG_APP_DIR and was
# wrong: vps1 still carries a stale /opt/thermograph from before the estate split,
# so prod looked resident there and was checked anyway. A leftover directory is
# exactly the kind of thing that outlives the arrangement it belonged to.
#
# Skipping is announced, never silent. A gate that quietly narrows what it checked
# is worse than one that fails, because the run still goes green.
if [ -n "${TG_SSH_HOST:-}" ] && command -v ip >/dev/null 2>&1; then
if ! ip -4 -o addr show 2>/dev/null | awk '{print $4}' | cut -d/ -f1 \
| grep -qxF "$TG_SSH_HOST"; then
echo "== parity check: $env_name — SKIPPED, lives on ${TG_HOST:-another host} (${TG_SSH_HOST}), not here"
continue
fi
fi
echo "== parity check: $env_name"
checked=$((checked + 1))
# dev renders WITHOUT common on both backends. On the SOPS side that is a caller # dev renders WITHOUT common on both backends. On the SOPS side that is a caller
# flag; on the OpenBao side the policy also forbids it. Taking the flag from the # flag; on the OpenBao side the policy also forbids it. Taking the flag from the
# topology rather than re-deriving it here keeps the comparison apples-to-apples # topology rather than re-deriving it here keeps the comparison apples-to-apples
@ -198,23 +167,11 @@ for k in sorted(order):
cleanup cleanup
done done
# Zero environments checked is a FAILURE, not a pass. Otherwise a --all run on a
# host that carries none of them — a new box, a renamed checkout, a typo'd --env —
# exits 0 having verified nothing, and a green run is exactly what the 7-day gate
# counts. "Nothing was wrong" and "nothing was examined" must not look alike.
if [ "$checked" -eq 0 ]; then
echo
echo "!! nothing was checked: none of [${TARGETS[*]}] has a checkout on this host." >&2
echo "!! dev lives on vps1; beta and prod live on vps2. Run this where they are." >&2
exit 1
fi
if [ "$overall" = 0 ]; then if [ "$overall" = 0 ]; then
echo echo
echo "PARITY OK ${checked} environment(s) checked on this host." echo "PARITY OK for: ${TARGETS[*]}"
echo "Safe to consider flipping TG_SECRETS_BACKEND for these environments." echo "Safe to consider flipping TG_SECRETS_BACKEND for these environments."
echo "Gate before prod: 7 consecutive green runs covering ALL THREE environments." echo "Recommended gate before prod: 7 consecutive green runs on all three."
echo "That means both hosts — a green run here says nothing about the other box."
else else
echo echo
echo "PARITY FAILED — do NOT flip TG_SECRETS_BACKEND." >&2 echo "PARITY FAILED — do NOT flip TG_SECRETS_BACKEND." >&2