Merge pull request 'promote: main → release (jinemi registry namespace, Caddy trusted_proxies, CI login fix)' (#159) from main into release
All checks were successful
secrets-guard / encrypted (push) Successful in 7s
shell-lint / shellcheck (push) Successful in 9s
Build + push images (Forgejo registry) / build-push (backend) (push) Successful in 24s
Build + push images (Forgejo registry) / build-push (frontend) (push) Successful in 1m15s
Deploy / deploy (backend) (push) Successful in 1m30s
Deploy / deploy (frontend) (push) Successful in 1m44s
All checks were successful
secrets-guard / encrypted (push) Successful in 7s
shell-lint / shellcheck (push) Successful in 9s
Build + push images (Forgejo registry) / build-push (backend) (push) Successful in 24s
Build + push images (Forgejo registry) / build-push (frontend) (push) Successful in 1m15s
Deploy / deploy (backend) (push) Successful in 1m30s
Deploy / deploy (frontend) (push) Successful in 1m44s
This commit is contained in:
commit
1afe6fbaac
54 changed files with 985 additions and 156 deletions
|
|
@ -4,8 +4,8 @@ name: Build + push images (Forgejo registry)
|
||||||
# They were identical apart from the domain string: the paths filter, the
|
# They were identical apart from the domain string: the paths filter, the
|
||||||
# concurrency group, IMAGE_PATH, the tag key and the build context.
|
# concurrency group, IMAGE_PATH, the tag key and the build context.
|
||||||
#
|
#
|
||||||
# Images stay separate and independently deployable -- emi/thermograph/backend
|
# Images stay separate and independently deployable -- jinemi/thermograph/backend
|
||||||
# and emi/thermograph/frontend -- exactly as before. Build-once,
|
# and jinemi/thermograph/frontend -- exactly as before. Build-once,
|
||||||
# deploy-everywhere; nothing about the published artefacts changes here.
|
# deploy-everywhere; nothing about the published artefacts changes here.
|
||||||
#
|
#
|
||||||
# SEMVER EXCEPTION, preserved: a v*.*.* tag push carries no paths context, so a
|
# SEMVER EXCEPTION, preserved: a v*.*.* tag push carries no paths context, so a
|
||||||
|
|
@ -110,13 +110,49 @@ jobs:
|
||||||
|
|
||||||
- name: Log in to the Forgejo registry
|
- name: Log in to the Forgejo registry
|
||||||
if: steps.image.outputs.changed == 'true'
|
if: steps.image.outputs.changed == 'true'
|
||||||
|
env:
|
||||||
|
REGISTRY_TOKEN: ${{ secrets.REGISTRY_TOKEN }}
|
||||||
run: |
|
run: |
|
||||||
# NOT secrets.GITHUB_TOKEN -- Forgejo's per-job auto-injected token can
|
# NOT secrets.GITHUB_TOKEN -- Forgejo's per-job auto-injected token can
|
||||||
# never push to the container registry (a known Forgejo limitation,
|
# never push to the container registry (a known Forgejo limitation,
|
||||||
# independent of any permission setting). Needs a manually provisioned
|
# independent of any permission setting). Needs a manually provisioned
|
||||||
# PAT with write:package scope, stored as REGISTRY_TOKEN.
|
# PAT with write:package scope, stored as REGISTRY_TOKEN.
|
||||||
echo "${{ secrets.REGISTRY_TOKEN }}" | docker login "${{ steps.image.outputs.host }}" \
|
#
|
||||||
--username emi --password-stdin
|
# WHITESPACE IS THE FAILURE THIS GUARDS AGAINST, and it is a nasty one.
|
||||||
|
# `docker login --password-stdin` strips exactly ONE trailing newline.
|
||||||
|
# `echo "$T"` adds one. So a secret pasted WITH a trailing newline --
|
||||||
|
# which is what happens if you copy from a terminal or an editor that
|
||||||
|
# ends files with one -- arrives as "<token>\n" and the registry answers
|
||||||
|
#
|
||||||
|
# Error response from daemon: Get "https://.../v2/": denied:
|
||||||
|
#
|
||||||
|
# with no further detail. That is indistinguishable from a revoked or
|
||||||
|
# wrong token, and it cost an afternoon on 2026-08-01 chasing a
|
||||||
|
# credential that was in fact valid. Strip first, echo never.
|
||||||
|
#
|
||||||
|
# The token also goes through the ENVIRONMENT rather than being
|
||||||
|
# interpolated into the script text. An expression is substituted before bash
|
||||||
|
# parses the line, so a value containing a quote or newline would change
|
||||||
|
# the shape of the command itself rather than just its arguments.
|
||||||
|
tok=$(printf '%s' "$REGISTRY_TOKEN" | tr -d ' \t\r\n')
|
||||||
|
if [ -z "$tok" ]; then
|
||||||
|
echo "::error::REGISTRY_TOKEN is empty or unset. Set it at" \
|
||||||
|
"Settings -> Actions -> Secrets on this repository."
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
# Shape check is a WARNING, not a failure: a hard assertion on the token
|
||||||
|
# format would block every build the day Forgejo changes it. But saying
|
||||||
|
# so up front turns the registry's opaque "denied:" into a diagnosis.
|
||||||
|
# Length and character class only -- the value is never printed.
|
||||||
|
case "$tok" in
|
||||||
|
*[!0-9a-f]*) echo "::warning::REGISTRY_TOKEN contains characters outside" \
|
||||||
|
"[0-9a-f]; a Forgejo PAT is 40 hex characters. If the" \
|
||||||
|
"login below fails, re-paste the secret." ;;
|
||||||
|
esac
|
||||||
|
[ "${#tok}" -eq 40 ] || echo "::warning::REGISTRY_TOKEN is ${#tok} characters," \
|
||||||
|
"expected 40. If the login below fails, re-paste the secret."
|
||||||
|
printf '%s' "$tok" | docker login "${{ steps.image.outputs.host }}" \
|
||||||
|
--username admin_emi --password-stdin
|
||||||
|
|
||||||
- name: Build
|
- name: Build
|
||||||
if: steps.image.outputs.changed == 'true'
|
if: steps.image.outputs.changed == 'true'
|
||||||
|
|
|
||||||
|
|
@ -86,6 +86,15 @@ jobs:
|
||||||
exit 0
|
exit 0
|
||||||
fi
|
fi
|
||||||
if [ -f infra/deploy/render-secrets.sh ]; then
|
if [ -f infra/deploy/render-secrets.sh ]; then
|
||||||
|
# env-topology.sh FIRST: it is what sets TG_SECRETS_BACKEND, and
|
||||||
|
# render-secrets.sh:44 defaults to `sops` when it is unset. Without
|
||||||
|
# this line the cutover would be only half-applied — deploy.sh would
|
||||||
|
# read the new backend while THIS workflow, which runs on every push
|
||||||
|
# touching infra/**, kept re-rendering the same file from SOPS. That
|
||||||
|
# is silent while parity holds and a hard failure the moment the SOPS
|
||||||
|
# files are retired.
|
||||||
|
. infra/deploy/env-topology.sh
|
||||||
|
thermograph_topology dev
|
||||||
. infra/deploy/render-secrets.sh
|
. infra/deploy/render-secrets.sh
|
||||||
# SKIP_COMMON is dev's standing rule, not a preference: common.yaml
|
# SKIP_COMMON is dev's standing rule, not a preference: common.yaml
|
||||||
# is the fleet's shared production credential set, and vps1 also
|
# is the fleet's shared production credential set, and vps1 also
|
||||||
|
|
@ -114,6 +123,13 @@ jobs:
|
||||||
git fetch --prune origin main
|
git fetch --prune origin main
|
||||||
git reset --hard origin/main
|
git reset --hard origin/main
|
||||||
if [ -f infra/deploy/render-secrets.sh ]; then
|
if [ -f infra/deploy/render-secrets.sh ]; then
|
||||||
|
# See the note in sync-dev: env-topology.sh is what sets
|
||||||
|
# TG_SECRETS_BACKEND, and for beta it also sets TG_BAO_APPROLE to
|
||||||
|
# the -beta credential. Beta shares a filesystem with prod, so the
|
||||||
|
# bare default is prod's file and tg-host-prod.hcl denies beta's
|
||||||
|
# own path — a 403 at render time on the box that runs prod.
|
||||||
|
. infra/deploy/env-topology.sh
|
||||||
|
thermograph_topology beta
|
||||||
. infra/deploy/render-secrets.sh
|
. infra/deploy/render-secrets.sh
|
||||||
render_thermograph_secrets /opt/thermograph-beta/infra beta /etc/thermograph-beta.env
|
render_thermograph_secrets /opt/thermograph-beta/infra beta /etc/thermograph-beta.env
|
||||||
fi
|
fi
|
||||||
|
|
@ -139,7 +155,113 @@ jobs:
|
||||||
git fetch --prune origin main
|
git fetch --prune origin main
|
||||||
git reset --hard origin/main
|
git reset --hard origin/main
|
||||||
if [ -f infra/deploy/render-secrets.sh ]; then
|
if [ -f infra/deploy/render-secrets.sh ]; then
|
||||||
|
# See the note in sync-dev — without env-topology.sh this job keeps
|
||||||
|
# rendering from SOPS regardless of what the backend selector says.
|
||||||
|
. infra/deploy/env-topology.sh
|
||||||
|
thermograph_topology prod
|
||||||
. infra/deploy/render-secrets.sh
|
. infra/deploy/render-secrets.sh
|
||||||
render_thermograph_secrets /opt/thermograph/infra prod /etc/thermograph.env
|
render_thermograph_secrets /opt/thermograph/infra prod /etc/thermograph.env
|
||||||
fi
|
fi
|
||||||
echo "synced to $(git log --oneline -1)"
|
echo "synced to $(git log --oneline -1)"
|
||||||
|
|
||||||
|
sync-centralis:
|
||||||
|
# Centralis' /etc/centralis.env has been HAND-MAINTAINED. The vault has held a
|
||||||
|
# copy since deploy/secrets/centralis.prod.yaml was seeded, and
|
||||||
|
# render_centralis_secrets has existed since the OpenBao work -- but nothing
|
||||||
|
# on any deploy path ever called it. A renderer with no caller is not a
|
||||||
|
# migration, it is a file.
|
||||||
|
#
|
||||||
|
# That gap is why enabling Google sign-in on Centralis had no safe home: the
|
||||||
|
# only way to set CENTRALIS_GOOGLE_CLIENT_ID was to hand-edit the live file,
|
||||||
|
# which is exactly what render-secrets.sh's header warns against.
|
||||||
|
#
|
||||||
|
# WHY THIS JOB RECREATES A CONTAINER WHEN THE OTHER THREE DELIBERATELY DO NOT.
|
||||||
|
# This workflow's header says it never rolls a service, because image tags are
|
||||||
|
# the app domains' axis. Centralis is different in kind: its compose file
|
||||||
|
# interpolates /etc/centralis.env at PARSE time (`set -a && . /etc/centralis.env
|
||||||
|
# && docker compose up -d`), so rendering the file without recreating the
|
||||||
|
# container changes nothing at all. Rendering and not recreating would be the
|
||||||
|
# silent no-op this estate keeps getting bitten by. It is one container with no
|
||||||
|
# replicas -- a two-second recreate, not a deploy.
|
||||||
|
if: github.ref_name == 'main'
|
||||||
|
runs-on: docker
|
||||||
|
concurrency:
|
||||||
|
group: infra-sync-centralis
|
||||||
|
cancel-in-progress: false
|
||||||
|
steps:
|
||||||
|
- name: Render /etc/centralis.env from the vault, then recreate Centralis
|
||||||
|
uses: https://github.com/appleboy/ssh-action@v1.2.0
|
||||||
|
with:
|
||||||
|
host: ${{ secrets.VPS2_SSH_HOST }}
|
||||||
|
username: ${{ secrets.VPS2_SSH_USER }}
|
||||||
|
key: ${{ secrets.VPS2_SSH_KEY }}
|
||||||
|
port: ${{ secrets.VPS2_SSH_PORT }}
|
||||||
|
script: |
|
||||||
|
set -euo pipefail
|
||||||
|
umask 077
|
||||||
|
cd /opt/thermograph
|
||||||
|
[ -f infra/deploy/secrets/centralis.prod.yaml ] || { echo "no Centralis vault — nothing to do"; exit 0; }
|
||||||
|
|
||||||
|
. infra/deploy/env-topology.sh
|
||||||
|
thermograph_topology prod
|
||||||
|
. infra/deploy/render-secrets.sh
|
||||||
|
|
||||||
|
scratch=$(mktemp -d); chmod 700 "$scratch"
|
||||||
|
trap 'rm -rf -- "$scratch"' EXIT
|
||||||
|
|
||||||
|
# Render to scratch FIRST. CENTRALIS_RENDER_DEST is the override the
|
||||||
|
# renderer already carries for exactly this: look before writing.
|
||||||
|
CENTRALIS_RENDER_DEST="$scratch/rendered.env" \
|
||||||
|
render_centralis_secrets /opt/thermograph/infra
|
||||||
|
|
||||||
|
# THE GUARD. On 2026-07-24 this estate lost Centralis' token registry by
|
||||||
|
# writing a file that was missing a key the live one had. The renderer
|
||||||
|
# proves its OWN output round-trips through bash, but it cannot know
|
||||||
|
# about a key that exists only in the live file -- someone hand-editing
|
||||||
|
# /etc/centralis.env to add, say, CENTRALIS_GOOGLE_CLIENT_ID is invisible
|
||||||
|
# to it. So: the vault must be a SUPERSET of what is live, or nothing is
|
||||||
|
# written and the job fails loudly.
|
||||||
|
#
|
||||||
|
# Names only. Both files are sourced in a clean environment so the
|
||||||
|
# comparison is over what bash actually ends up with, not over text.
|
||||||
|
# The `|| true` is load-bearing under `set -o pipefail`: grep exits 1 when
|
||||||
|
# a file yields no matching keys, which would otherwise abort the job on
|
||||||
|
# the exact edge case this guard exists to handle -- an empty or
|
||||||
|
# unreadable live file, i.e. a first run.
|
||||||
|
keys() { env -i bash -c 'set -a; . "$1" >/dev/null 2>&1; set +a; compgen -v' _ "$1" \
|
||||||
|
| { grep -E '^(CENTRALIS_|DISCORD_)' || true; } | sort -u; }
|
||||||
|
keys "$scratch/rendered.env" > "$scratch/new.keys"
|
||||||
|
if [ -r /etc/centralis.env ]; then keys /etc/centralis.env > "$scratch/live.keys"; else : > "$scratch/live.keys"; fi
|
||||||
|
|
||||||
|
missing=$(comm -23 "$scratch/live.keys" "$scratch/new.keys" || true)
|
||||||
|
if [ -n "$missing" ]; then
|
||||||
|
echo "!! REFUSING to write /etc/centralis.env."
|
||||||
|
echo "!! The live file has keys the vault does not, so rendering would DELETE them:"
|
||||||
|
printf '!! %s\n' $missing
|
||||||
|
echo "!! Someone hand-edited the live file. Put those keys in the vault instead:"
|
||||||
|
echo "!! sops edit infra/deploy/secrets/centralis.prod.yaml"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
added=$(comm -13 "$scratch/live.keys" "$scratch/new.keys" || true)
|
||||||
|
[ -n "$added" ] && printf ' new key from the vault: %s\n' $added
|
||||||
|
|
||||||
|
# Same file, byte for byte? Then there is nothing to do and no reason to
|
||||||
|
# bounce a healthy container.
|
||||||
|
if [ -r /etc/centralis.env ] && cmp -s "$scratch/rendered.env" /etc/centralis.env; then
|
||||||
|
echo "==> /etc/centralis.env already matches the vault — not recreating"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
install -m 0600 -o "$(stat -c %U /etc/centralis.env 2>/dev/null || echo "$USER")" \
|
||||||
|
-g "$(stat -c %G /etc/centralis.env 2>/dev/null || echo "$USER")" \
|
||||||
|
"$scratch/rendered.env" /etc/centralis.env 2>/dev/null \
|
||||||
|
|| sudo install -m 0600 -o agent -g agent "$scratch/rendered.env" /etc/centralis.env
|
||||||
|
echo "==> wrote /etc/centralis.env from the vault"
|
||||||
|
|
||||||
|
# Recreate so compose re-reads it. `docker restart` would NOT: the values
|
||||||
|
# are interpolated when the compose file is parsed, not read at runtime.
|
||||||
|
cd /opt/centralis/deploy
|
||||||
|
set -a; . /etc/centralis.env; set +a
|
||||||
|
docker compose up -d
|
||||||
|
docker compose ps --format '{{.Name}}\t{{.State}}\t{{.Status}}'
|
||||||
|
|
|
||||||
|
|
@ -8,8 +8,15 @@ name: Ops cron (backup + IndexNow)
|
||||||
#
|
#
|
||||||
# The jobs SSH into a host and run inside the already-running stack (docker
|
# The jobs SSH into a host and run inside the already-running stack (docker
|
||||||
# exec), the same way deploy.sh runs its own post-deploy IndexNow ping -- no new
|
# exec), the same way deploy.sh runs its own post-deploy IndexNow ping -- no new
|
||||||
# network exposure, no separate dependency install. Runs on the `docker` label
|
# network exposure, no separate dependency install. Runs on the `docker` label,
|
||||||
# (the always-on Swarm-hosted runner deploy/forgejo/ stood up).
|
# served by the vps2 runner (infra/deploy/forgejo/runner-vps2/) and, when it is
|
||||||
|
# up, the desktop.
|
||||||
|
#
|
||||||
|
# THIS WORKFLOW IS THE ESTATE'S ONLY BACKUP, so its dependency on a runner is
|
||||||
|
# load-bearing. On 2026-07-31 the desktop was the only registered runner and
|
||||||
|
# went offline for 21 hours; this schedule did not fire at 03:00Z. Forgejo held
|
||||||
|
# the run in `waiting` and it completed on reconnect, so nothing was lost that
|
||||||
|
# time. The vps2 runner exists so the next outage is not a coin flip.
|
||||||
#
|
#
|
||||||
# WHICH BOX EACH JOB TALKS TO, and why the secret names say so:
|
# WHICH BOX EACH JOB TALKS TO, and why the secret names say so:
|
||||||
#
|
#
|
||||||
|
|
@ -209,6 +216,65 @@ jobs:
|
||||||
rclone delete --min-age 30d "archive:$base/db/" 2>/dev/null || true
|
rclone delete --min-age 30d "archive:$base/db/" 2>/dev/null || true
|
||||||
rclone delete --min-age 30d "archive:$base/data/" 2>/dev/null || true
|
rclone delete --min-age 30d "archive:$base/data/" 2>/dev/null || true
|
||||||
|
|
||||||
|
secrets-parity:
|
||||||
|
name: OpenBao/SOPS parity gate
|
||||||
|
runs-on: docker
|
||||||
|
# The gate infra/openbao/README.md specifies for a cutover -- "7 consecutive
|
||||||
|
# green parity runs" -- and which, until now, nothing implemented. A gate
|
||||||
|
# that exists only in prose gets satisfied by assertion.
|
||||||
|
#
|
||||||
|
# verify-parity.sh renders each environment through BOTH backends and diffs
|
||||||
|
# the KEY=value sets after a last-wins collapse. It prints key NAMES and
|
||||||
|
# counts only, never values, so these logs are safe to read and to paste.
|
||||||
|
#
|
||||||
|
# A mismatch FAILS the job, deliberately and loudly. The failure modes on
|
||||||
|
# the other side of a bad cutover are all silent: a missing
|
||||||
|
# THERMOGRAPH_AUTH_SECRET makes every worker mint its own, so logins start
|
||||||
|
# working intermittently; a half-rendered VAPID pair makes push.py generate a
|
||||||
|
# new keypair and delete every subscription row as "gone". Red here is the
|
||||||
|
# only warning that arrives before those.
|
||||||
|
#
|
||||||
|
# Runs under sudo because the age key is 0400 root and the AppRole files are
|
||||||
|
# 0400 agent/root -- the SSH user cannot read them directly. This grants CI
|
||||||
|
# no vault access of its own: the HOST renders, CI only asks it to.
|
||||||
|
#
|
||||||
|
# The run history in Forgejo IS the record of consecutive green days. Do not
|
||||||
|
# count a day this job was skipped or the runner was down as green.
|
||||||
|
concurrency:
|
||||||
|
group: ops-secrets-parity
|
||||||
|
cancel-in-progress: false
|
||||||
|
steps:
|
||||||
|
- name: prod + beta (vps2)
|
||||||
|
uses: https://github.com/appleboy/ssh-action@v1.2.0
|
||||||
|
with:
|
||||||
|
host: ${{ secrets.VPS2_SSH_HOST }}
|
||||||
|
username: ${{ secrets.VPS2_SSH_USER }}
|
||||||
|
key: ${{ secrets.VPS2_SSH_KEY }}
|
||||||
|
port: ${{ secrets.VPS2_SSH_PORT }}
|
||||||
|
script: |
|
||||||
|
set -euo pipefail
|
||||||
|
# Each environment is checked from its OWN checkout. The trees are
|
||||||
|
# identical, but running beta's from beta's directory also proves
|
||||||
|
# that checkout is current rather than assuming it.
|
||||||
|
sudo /opt/thermograph/infra/openbao/verify-parity.sh --env prod
|
||||||
|
sudo /opt/thermograph-beta/infra/openbao/verify-parity.sh --env beta
|
||||||
|
|
||||||
|
- name: dev (vps1)
|
||||||
|
uses: https://github.com/appleboy/ssh-action@v1.2.0
|
||||||
|
with:
|
||||||
|
host: ${{ secrets.VPS1_SSH_HOST }}
|
||||||
|
username: ${{ secrets.VPS1_SSH_USER }}
|
||||||
|
key: ${{ secrets.VPS1_SSH_KEY }}
|
||||||
|
port: ${{ secrets.VPS1_SSH_PORT }}
|
||||||
|
script: |
|
||||||
|
set -euo pipefail
|
||||||
|
# dev renders dev.yaml ALONE -- verify-parity.sh takes that from
|
||||||
|
# env-topology.sh's TG_SKIP_COMMON rather than re-deriving it, so the
|
||||||
|
# comparison matches what deploy-dev.sh actually does. vps1 must
|
||||||
|
# never see common.yaml: it runs Forgejo, its CI, and dev's
|
||||||
|
# unreviewed branch.
|
||||||
|
sudo /opt/thermograph-dev/infra/openbao/verify-parity.sh --env dev
|
||||||
|
|
||||||
indexnow:
|
indexnow:
|
||||||
name: IndexNow ping
|
name: IndexNow ping
|
||||||
runs-on: docker
|
runs-on: docker
|
||||||
|
|
|
||||||
6
.gitignore
vendored
6
.gitignore
vendored
|
|
@ -2,3 +2,9 @@
|
||||||
# Claude Code worktrees. These are other sessions' checkouts living inside this
|
# Claude Code worktrees. These are other sessions' checkouts living inside this
|
||||||
# repo; `git add -A` will otherwise commit them as embedded git repositories.
|
# repo; `git add -A` will otherwise commit them as embedded git repositories.
|
||||||
.claude/worktrees/
|
.claude/worktrees/
|
||||||
|
|
||||||
|
# Per-operator Claude Code settings — local permission grants, not team config.
|
||||||
|
# Untracked but not ignored meant one `git add -A` would publish one operator's
|
||||||
|
# allowlist (including which secrets they may decrypt) to everyone.
|
||||||
|
# `.claude/settings.json` IS tracked and stays the shared, reviewed config.
|
||||||
|
.claude/settings.local.json
|
||||||
|
|
|
||||||
|
|
@ -29,8 +29,10 @@ concurrently. Read both before dispatching parallel work.
|
||||||
|
|
||||||
**`dev` is a first-class hosted environment, not just an integration branch.**
|
**`dev` is a first-class hosted environment, not just an integration branch.**
|
||||||
It runs on vps1 — the same box as Forgejo and the monitoring stack — with its
|
It runs on vps1 — the same box as Forgejo and the monitoring stack — with its
|
||||||
own Postgres container, reachable only on the WireGuard mesh
|
own Postgres container, reachable only from vps1 itself (`127.0.0.1:8137` —
|
||||||
(`10.10.0.2:8137`): no public DNS record, no Caddy site, no TLS. It is deployed
|
`infra/docker-compose.yml` binds the port to loopback, so the mesh does not
|
||||||
|
reach it either; use an SSH tunnel): no public DNS record, no Caddy site, no
|
||||||
|
TLS. It is deployed
|
||||||
by CI like beta and prod. The desktop hosts no Thermograph environment at all;
|
by CI like beta and prod. The desktop hosts no Thermograph environment at all;
|
||||||
`make dev-up` there is a laptop convenience for running the stack locally, not
|
`make dev-up` there is a laptop convenience for running the stack locally, not
|
||||||
a deployment target.
|
a deployment target.
|
||||||
|
|
@ -77,7 +79,7 @@ cannot describe vps2 alone, since vps2 runs beta and prod side by side.
|
||||||
- Each service's live tag is persisted host-side, so a single-service roll never
|
- Each service's live tag is persisted host-side, so a single-service roll never
|
||||||
disturbs the sibling's running tag.
|
disturbs the sibling's running tag.
|
||||||
|
|
||||||
Images are `emi/thermograph/backend` and `emi/thermograph/frontend`, tagged
|
Images are `jinemi/thermograph/backend` and `jinemi/thermograph/frontend`, tagged
|
||||||
`sha-<12hex>` on every push and by semver on `v*.*.*` tags. They build and deploy
|
`sha-<12hex>` on every push and by semver on `v*.*.*` tags. They build and deploy
|
||||||
**independently** — a backend change ships without a frontend deploy and vice
|
**independently** — a backend change ships without a frontend deploy and vice
|
||||||
versa. That independence is the point of the FE/BE split and survived
|
versa. That independence is the point of the FE/BE split and survived
|
||||||
|
|
|
||||||
|
|
@ -25,7 +25,7 @@ Forgejo only reads root workflows, but dead copies are a trap):**
|
||||||
|
|
||||||
- `backend-build-push.yml` / `frontend-build-push.yml` — the old per-repo
|
- `backend-build-push.yml` / `frontend-build-push.yml` — the old per-repo
|
||||||
build-push, path-filtered (`paths: ['backend/**']` etc.), image paths now
|
build-push, path-filtered (`paths: ['backend/**']` etc.), image paths now
|
||||||
**explicit**: `emi/thermograph/backend|frontend` (the old
|
**explicit**: `jinemi/thermograph/backend|frontend` (the old
|
||||||
`${{ github.repository }}/app` derivation collides in a monorepo). Per-domain
|
`${{ github.repository }}/app` derivation collides in a monorepo). Per-domain
|
||||||
concurrency groups. Caveat: `v*.*.*` tag pushes carry no paths context, so a
|
concurrency groups. Caveat: `v*.*.*` tag pushes carry no paths context, so a
|
||||||
version tag builds **both** images (intended: a release names the set).
|
version tag builds **both** images (intended: a release names the set).
|
||||||
|
|
@ -65,7 +65,7 @@ Forgejo only reads root workflows, but dead copies are a trap):**
|
||||||
name, so no project-name concern there).
|
name, so no project-name concern there).
|
||||||
|
|
||||||
**Verified here:** every workflow + compose file parses (PyYAML); all three
|
**Verified here:** every workflow + compose file parses (PyYAML); all three
|
||||||
deploy scripts pass `bash -n`; no `emi/thermograph-{backend,frontend}/app`
|
deploy scripts pass `bash -n`; no `admin_emi/thermograph-{backend,frontend}/app`
|
||||||
stragglers in sh/yml. **Not verified (impossible locally):** Forgejo actually
|
stragglers in sh/yml. **Not verified (impossible locally):** Forgejo actually
|
||||||
honoring `on.push.paths` and `workflow_call` inputs on our version; any live
|
honoring `on.push.paths` and `workflow_call` inputs on our version; any live
|
||||||
deploy.
|
deploy.
|
||||||
|
|
@ -118,7 +118,7 @@ Checked every split-repo branch by dry-run merge; migrated everything real:
|
||||||
|
|
||||||
## Cutover runbook
|
## Cutover runbook
|
||||||
|
|
||||||
1. Create `emi/thermograph` on Forgejo (a **new** repo — do not resurrect the
|
1. Create `Jinemi/thermograph` on Forgejo (a **new** repo — do not resurrect the
|
||||||
archived monorepo) and push `main`.
|
archived monorepo) and push `main`.
|
||||||
2. Actions config, once: secrets `REGISTRY_TOKEN`, `SSH_HOST/USER/KEY/PORT`
|
2. Actions config, once: secrets `REGISTRY_TOKEN`, `SSH_HOST/USER/KEY/PORT`
|
||||||
(beta), `PROD_SSH_*` (prod); variable `REGISTRY_HOST=git.thermograph.org`.
|
(beta), `PROD_SSH_*` (prod); variable `REGISTRY_HOST=git.thermograph.org`.
|
||||||
|
|
@ -132,7 +132,7 @@ Checked every split-repo branch by dry-run merge; migrated everything real:
|
||||||
`/etc/thermograph*` files stay as-is.
|
`/etc/thermograph*` files stay as-is.
|
||||||
5. LAN box: re-point `~/thermograph-dev` at the monorepo the same way.
|
5. LAN box: re-point `~/thermograph-dev` at the monorepo the same way.
|
||||||
6. First push builds the **new** image paths — old
|
6. First push builds the **new** image paths — old
|
||||||
`emi/thermograph-{backend,frontend}/app` packages stay in the registry for
|
`admin_emi/thermograph-{backend,frontend}/app` packages stay in the registry for
|
||||||
rollback; GC after burn-in. deploy.sh's tag prune already targets the new
|
rollback; GC after burn-in. deploy.sh's tag prune already targets the new
|
||||||
paths only.
|
paths only.
|
||||||
7. One `workflow_dispatch` of `backend-deploy.yml` after its build-push
|
7. One `workflow_dispatch` of `backend-deploy.yml` after its build-push
|
||||||
|
|
|
||||||
|
|
@ -8,8 +8,8 @@ for: **per-domain images, per-domain deploys, and an async FE/BE contract**.
|
||||||
|
|
||||||
| Dir | What | CI |
|
| Dir | What | CI |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `backend/` | FastAPI graded-climate API, accounts, notifications (Discord bot, push, mail), data pipeline | `build-push` → image `emi/thermograph/backend`; `deploy` |
|
| `backend/` | FastAPI graded-climate API, accounts, notifications (Discord bot, push, mail), data pipeline | `build-push` → image `jinemi/thermograph/backend`; `deploy` |
|
||||||
| `frontend/` | Public client: static JS/CSS + SSR pages | same `build-push` / `deploy` workflows, matrixed by domain; image `emi/thermograph/frontend` |
|
| `frontend/` | Public client: static JS/CSS + SSR pages | same `build-push` / `deploy` workflows, matrixed by domain; image `jinemi/thermograph/frontend` |
|
||||||
| `infra/` | Compose (dev, on vps1) + two Swarm stacks co-resident on vps2 (beta, prod), deploy scripts, terraform, SOPS secrets vault, ops cron | `infra-sync` (host checkout + secrets render), `secrets-guard`, `ops-cron` |
|
| `infra/` | Compose (dev, on vps1) + two Swarm stacks co-resident on vps2 (beta, prod), deploy scripts, terraform, SOPS secrets vault, ops cron | `infra-sync` (host checkout + secrets render), `secrets-guard`, `ops-cron` |
|
||||||
| `observability/` | Loki + Grafana + Alloy stack | `observability-validate` |
|
| `observability/` | Loki + Grafana + Alloy stack | `observability-validate` |
|
||||||
|
|
||||||
|
|
@ -27,7 +27,7 @@ which docs in this repo are currently stale.
|
||||||
|
|
||||||
Every workflow in `.forgejo/workflows/` is **path-filtered to its domain**: a
|
Every workflow in `.forgejo/workflows/` is **path-filtered to its domain**: a
|
||||||
push touching only `frontend/**` builds/deploys nothing else. Images stay
|
push touching only `frontend/**` builds/deploys nothing else. Images stay
|
||||||
separate (`emi/thermograph/backend`, `emi/thermograph/frontend`, each tagged
|
separate (`jinemi/thermograph/backend`, `jinemi/thermograph/frontend`, each tagged
|
||||||
`sha-<12hex>`), deploys stay per-service (`infra/deploy/deploy.sh
|
`sha-<12hex>`), deploys stay per-service (`infra/deploy/deploy.sh
|
||||||
SERVICE=backend|frontend|all`), and the API version contract
|
SERVICE=backend|frontend|all`), and the API version contract
|
||||||
(`GET /api/version`, `PAYLOAD_VER`) still lets FE and BE ship out of lockstep.
|
(`GET /api/version`, `PAYLOAD_VER`) still lets FE and BE ship out of lockstep.
|
||||||
|
|
|
||||||
|
|
@ -45,7 +45,7 @@ for systemd/CI callers. Entry target is `app:app`.
|
||||||
that disagree with the labels the API returns.
|
that disagree with the labels the API returns.
|
||||||
- **Fahrenheit country set** — `api/content_payloads.py`'s `F_COUNTRIES` is
|
- **Fahrenheit country set** — `api/content_payloads.py`'s `F_COUNTRIES` is
|
||||||
canonical; `frontend`'s `server/internal/format::FCountries` and
|
canonical; `frontend`'s `server/internal/format::FCountries` and
|
||||||
`static/units.js`'s `F_REGIONS` must stay identical to it. Both are asserted
|
`frontend/static/units.js`'s `F_REGIONS` must stay identical to it. Both are asserted
|
||||||
by tests in `frontend/server/internal/format/format_test.go` — but note the
|
by tests in `frontend/server/internal/format/format_test.go` — but note the
|
||||||
backend cross-check **skips in CI** (the frontend image's build context is
|
backend cross-check **skips in CI** (the frontend image's build context is
|
||||||
`frontend/`, so this file is unreachable from the builder stage, which is the
|
`frontend/`, so this file is unreachable from the builder stage, which is the
|
||||||
|
|
|
||||||
|
|
@ -3,7 +3,7 @@
|
||||||
The Thermograph API service: grades recent local weather against ~45 years of
|
The Thermograph API service: grades recent local weather against ~45 years of
|
||||||
climate history, and hosts the accounts, notification, and SSR-content
|
climate history, and hosts the accounts, notification, and SSR-content
|
||||||
back-end that the rest of the split Thermograph stack (`thermograph-frontend`)
|
back-end that the rest of the split Thermograph stack (`thermograph-frontend`)
|
||||||
talks to over HTTP. Split from the `emi/thermograph` monorepo — this repo owns
|
talks to over HTTP. Split from the `Jinemi/thermograph` monorepo — this repo owns
|
||||||
the API/DB/accounts/notifications layer only; it renders no HTML/CSS/JS of its
|
the API/DB/accounts/notifications layer only; it renders no HTML/CSS/JS of its
|
||||||
own.
|
own.
|
||||||
|
|
||||||
|
|
@ -67,9 +67,9 @@ cities_flavor.json bundled reference data (generated by gen_cities.py /
|
||||||
`/content/...`; it negotiates compatibility via `GET /api/version`.
|
`/content/...`; it negotiates compatibility via `GET /api/version`.
|
||||||
- **`thermograph-infra`** owns the deploy/Compose/Terraform layer — this repo
|
- **`thermograph-infra`** owns the deploy/Compose/Terraform layer — this repo
|
||||||
only builds and publishes its own container image
|
only builds and publishes its own container image
|
||||||
(`git.thermograph.org/emi/thermograph-backend/app`) and hands infra a tag to
|
(`git.thermograph.org/admin_emi/thermograph-backend/app`) and hands infra a tag to
|
||||||
roll out (`SERVICE=backend` + `BACKEND_IMAGE_TAG` into infra's
|
roll out (`SERVICE=backend` + `BACKEND_IMAGE_TAG` into infra's
|
||||||
`deploy/deploy.sh`).
|
`infra/deploy/deploy.sh`).
|
||||||
- **`thermograph-docs`** holds the cross-repo architecture/runbook docs; this
|
- **`thermograph-docs`** holds the cross-repo architecture/runbook docs; this
|
||||||
README only covers what's local to this service.
|
README only covers what's local to this service.
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -22,11 +22,11 @@ services:
|
||||||
backend:
|
backend:
|
||||||
# Defaults to a locally-built `:local` image (scripts/smoke.sh builds it). In CI,
|
# Defaults to a locally-built `:local` image (scripts/smoke.sh builds it). In CI,
|
||||||
# set BACKEND_IMAGE_TAG=sha-<12hex> to smoke the exact published image.
|
# set BACKEND_IMAGE_TAG=sha-<12hex> to smoke the exact published image.
|
||||||
# The path must match what build-push.yml publishes -- emi/thermograph/backend.
|
# The path must match what build-push.yml publishes -- jinemi/thermograph/backend.
|
||||||
# It read emi/thermograph-backend/app (the retired split-era path) until this
|
# It read admin_emi/thermograph-backend/app (the retired split-era path) until this
|
||||||
# was corrected; harmless for the default `:local` build, wrong the moment
|
# was corrected; harmless for the default `:local` build, wrong the moment
|
||||||
# BACKEND_IMAGE_TAG names a real published tag.
|
# BACKEND_IMAGE_TAG names a real published tag.
|
||||||
image: ${REGISTRY_HOST:-git.thermograph.org}/${BACKEND_IMAGE_PATH:-emi/thermograph/backend}:${BACKEND_IMAGE_TAG:-local}
|
image: ${REGISTRY_HOST:-git.thermograph.org}/${BACKEND_IMAGE_PATH:-jinemi/thermograph/backend}:${BACKEND_IMAGE_TAG:-local}
|
||||||
depends_on:
|
depends_on:
|
||||||
db:
|
db:
|
||||||
condition: service_healthy
|
condition: service_healthy
|
||||||
|
|
|
||||||
|
|
@ -14,7 +14,7 @@ export POSTGRES_PASSWORD="${POSTGRES_PASSWORD:-smoke}"
|
||||||
# The image to smoke, pulled from the registry (no local build). CI passes the just-
|
# The image to smoke, pulled from the registry (no local build). CI passes the just-
|
||||||
# pushed sha (BACKEND_IMAGE_TAG=sha-<12hex>); locally it defaults to a published tag.
|
# pushed sha (BACKEND_IMAGE_TAG=sha-<12hex>); locally it defaults to a published tag.
|
||||||
export BACKEND_IMAGE_TAG="${BACKEND_IMAGE_TAG:-v0.0.2-split-ci}"
|
export BACKEND_IMAGE_TAG="${BACKEND_IMAGE_TAG:-v0.0.2-split-ci}"
|
||||||
IMG="${REGISTRY_HOST:-git.thermograph.org}/${BACKEND_IMAGE_PATH:-emi/thermograph-backend/app}:${BACKEND_IMAGE_TAG}"
|
IMG="${REGISTRY_HOST:-git.thermograph.org}/${BACKEND_IMAGE_PATH:-admin_emi/thermograph-backend/app}:${BACKEND_IMAGE_TAG}"
|
||||||
export SMOKE_HOST_PORT="${SMOKE_HOST_PORT:-18137}"
|
export SMOKE_HOST_PORT="${SMOKE_HOST_PORT:-18137}"
|
||||||
COMPOSE=(docker compose -f docker-compose.test.yml)
|
COMPOSE=(docker compose -f docker-compose.test.yml)
|
||||||
PORT_URL="http://127.0.0.1:${SMOKE_HOST_PORT}"
|
PORT_URL="http://127.0.0.1:${SMOKE_HOST_PORT}"
|
||||||
|
|
|
||||||
|
|
@ -12,8 +12,8 @@ thermograph/
|
||||||
├── README.md domain table + how CI stays decoupled
|
├── README.md domain table + how CI stays decoupled
|
||||||
├── .forgejo/workflows/ ALL CI. Nine files. Never add workflows elsewhere.
|
├── .forgejo/workflows/ ALL CI. Nine files. Never add workflows elsewhere.
|
||||||
├── .claude/ settings.json + enforcement hooks
|
├── .claude/ settings.json + enforcement hooks
|
||||||
├── backend/ FastAPI API + Go daemon → emi/thermograph/backend
|
├── backend/ FastAPI API + Go daemon → jinemi/thermograph/backend
|
||||||
├── frontend/ Go SSR + static assets → emi/thermograph/frontend
|
├── frontend/ Go SSR + static assets → jinemi/thermograph/frontend
|
||||||
├── infra/ compose, Swarm, deploy, secrets, terraform, ops
|
├── infra/ compose, Swarm, deploy, secrets, terraform, ops
|
||||||
├── observability/ Loki + Grafana + Alloy
|
├── observability/ Loki + Grafana + Alloy
|
||||||
└── docs/onboarding/ you are here
|
└── docs/onboarding/ you are here
|
||||||
|
|
|
||||||
|
|
@ -102,7 +102,7 @@ alembic + `/healthz` 200 + a real `/api/v2/place` 200.
|
||||||
|
|
||||||
Triggers on pushes to `dev`/`main`/`release` touching `backend/**` or
|
Triggers on pushes to `dev`/`main`/`release` touching `backend/**` or
|
||||||
`frontend/**`, and on `v*.*.*` tags. Publishes
|
`frontend/**`, and on `v*.*.*` tags. Publishes
|
||||||
`git.thermograph.org/emi/thermograph/{backend,frontend}:sha-<12hex>`.
|
`git.thermograph.org/Jinemi/thermograph/{backend,frontend}:sha-<12hex>`.
|
||||||
|
|
||||||
- **Serialised** (`max-parallel: 1`): one physical runner, and two whole image
|
- **Serialised** (`max-parallel: 1`): one physical runner, and two whole image
|
||||||
builds racing each other buys nothing.
|
builds racing each other buys nothing.
|
||||||
|
|
|
||||||
|
|
@ -148,7 +148,7 @@ pages nobody. **The only proof is a message actually arriving in
|
||||||
|
|
||||||
> ⚠️ **Merging is not deploying — and this one has already bitten.**
|
> ⚠️ **Merging is not deploying — and this one has already bitten.**
|
||||||
> `/opt/observability` on vps1 and vps2 is a checkout of the **archived**
|
> `/opt/observability` on vps1 and vps2 is a checkout of the **archived**
|
||||||
> `emi/thermograph-observability` repo. It can never `git pull` again; the
|
> `admin_emi/thermograph-observability` repo. It can never `git pull` again; the
|
||||||
> content now lives here under `observability/`. A previous observability PR
|
> content now lives here under `observability/`. A previous observability PR
|
||||||
> merged and had **zero live effect** for exactly this reason.
|
> merged and had **zero live effect** for exactly this reason.
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -77,7 +77,7 @@ mentally translate host names.
|
||||||
### `backend/README.md` is written for the split-repo era
|
### `backend/README.md` is written for the split-repo era
|
||||||
|
|
||||||
It describes `thermograph-backend` as its own repo with sibling repos, uses the
|
It describes `thermograph-backend` as its own repo with sibling repos, uses the
|
||||||
retired image path `git.thermograph.org/emi/thermograph-backend/app`, and says
|
retired image path `git.thermograph.org/admin_emi/thermograph-backend/app`, and says
|
||||||
*"there is no Makefile in this repo yet"* — there is (`make test`, `make smoke`,
|
*"there is no Makefile in this repo yet"* — there is (`make test`, `make smoke`,
|
||||||
`make clean-venv`).
|
`make clean-venv`).
|
||||||
|
|
||||||
|
|
@ -106,7 +106,7 @@ it — that's the standing instruction in the root `CLAUDE.md`.
|
||||||
### `observability/README.md`: merging is not deploying
|
### `observability/README.md`: merging is not deploying
|
||||||
|
|
||||||
`/opt/observability` on vps1 and vps2 is a checkout of the **archived**
|
`/opt/observability` on vps1 and vps2 is a checkout of the **archived**
|
||||||
`emi/thermograph-observability` repo. It can never `git pull` again. A previous
|
`admin_emi/thermograph-observability` repo. It can never `git pull` again. A previous
|
||||||
observability PR merged and had **zero live effect** for exactly this reason.
|
observability PR merged and had **zero live effect** for exactly this reason.
|
||||||
The README documents the hand-ship procedure; until the checkouts are re-pointed
|
The README documents the hand-ship procedure; until the checkouts are re-pointed
|
||||||
at the monorepo, that's the only path. Flagged again in
|
at the monorepo, that's the only path. Flagged again in
|
||||||
|
|
@ -118,7 +118,7 @@ at the monorepo, that's the only path. Flagged again in
|
||||||
|
|
||||||
### ~~The `docker-compose.test.yml` files pull retired image paths~~ — fixed
|
### ~~The `docker-compose.test.yml` files pull retired image paths~~ — fixed
|
||||||
|
|
||||||
**Fixed.** Both now use `emi/thermograph/backend`, the path `build-push.yml`
|
**Fixed.** Both now use `jinemi/thermograph/backend`, the path `build-push.yml`
|
||||||
actually publishes.
|
actually publishes.
|
||||||
|
|
||||||
The frontend harness had the worse half of this: it also pinned
|
The frontend harness had the worse half of this: it also pinned
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,6 @@
|
||||||
# Onboarding — becoming a full contributor to Thermograph
|
# Onboarding — becoming a full contributor to Thermograph
|
||||||
|
|
||||||
This is the developer onboarding path for the `emi/thermograph` monorepo: what
|
This is the developer onboarding path for the `Jinemi/thermograph` monorepo: what
|
||||||
the product is, how the code is shaped, how to run it, what will break silently
|
the product is, how the code is shaped, how to run it, what will break silently
|
||||||
if you get it wrong, and how a change actually travels from your editor to
|
if you get it wrong, and how a change actually travels from your editor to
|
||||||
`thermograph.org`.
|
`thermograph.org`.
|
||||||
|
|
@ -35,10 +35,10 @@ readings.
|
||||||
|
|
||||||
- **`backend/`** — Python 3.12 / FastAPI. Owns all climate data, grading, the
|
- **`backend/`** — Python 3.12 / FastAPI. Owns all climate data, grading, the
|
||||||
API, accounts, notifications, plus a Go daemon (`backend/daemon/`) that owns
|
API, accounts, notifications, plus a Go daemon (`backend/daemon/`) that owns
|
||||||
the Discord gateway and recurring timers. Ships as `emi/thermograph/backend`.
|
the Discord gateway and recurring timers. Ships as `jinemi/thermograph/backend`.
|
||||||
- **`frontend/`** — **Go** SSR service (`frontend/server/`) plus every static
|
- **`frontend/`** — **Go** SSR service (`frontend/server/`) plus every static
|
||||||
asset. No climate data, no database, no compute. Ships as
|
asset. No climate data, no database, no compute. Ships as
|
||||||
`emi/thermograph/frontend`. (The Python implementation at `frontend/*.py` is
|
`jinemi/thermograph/frontend`. (The Python implementation at `frontend/*.py` is
|
||||||
the superseded original — see [traps](11-traps.md).)
|
the superseded original — see [traps](11-traps.md).)
|
||||||
- **`infra/`** — compose and Swarm files, deploy scripts, the SOPS secrets
|
- **`infra/`** — compose and Swarm files, deploy scripts, the SOPS secrets
|
||||||
vault, Terraform, ops query tooling.
|
vault, Terraform, ops query tooling.
|
||||||
|
|
|
||||||
|
|
@ -10,7 +10,7 @@ names.
|
||||||
## The live service is Go — `server/`
|
## The live service is Go — `server/`
|
||||||
|
|
||||||
`server/` (module `thermograph/frontend`, Go 1.26) is what builds, tests, ships
|
`server/` (module `thermograph/frontend`, Go 1.26) is what builds, tests, ships
|
||||||
and runs. It is the only thing in the `emi/thermograph/frontend` image.
|
and runs. It is the only thing in the `jinemi/thermograph/frontend` image.
|
||||||
|
|
||||||
The Python files at this level — `app.py`, `content.py`, `api_client.py`,
|
The Python files at this level — `app.py`, `content.py`, `api_client.py`,
|
||||||
`format.py`, `content_loader.py`, `paths.py`, `templates/*.html.j2` — are the
|
`format.py`, `content_loader.py`, `paths.py`, `templates/*.html.j2` — are the
|
||||||
|
|
|
||||||
|
|
@ -5,7 +5,7 @@ static asset for [Thermograph](https://thermograph.org) — grades recent local
|
||||||
weather against ~45 years of climate history. This domain holds **no climate
|
weather against ~45 years of climate history. This domain holds **no climate
|
||||||
data and does no compute**; it renders pages and serves the JS/CSS/image assets
|
data and does no compute**; it renders pages and serves the JS/CSS/image assets
|
||||||
that call the backend's API from the browser. It builds its own image
|
that call the backend's API from the browser. It builds its own image
|
||||||
(`emi/thermograph/frontend`) and deploys independently of the backend.
|
(`jinemi/thermograph/frontend`) and deploys independently of the backend.
|
||||||
|
|
||||||
See [`CLAUDE.md`](CLAUDE.md) for the agent-facing detail on the deploy contract
|
See [`CLAUDE.md`](CLAUDE.md) for the agent-facing detail on the deploy contract
|
||||||
and the cross-service contracts, and [`DESIGN.md`](DESIGN.md) for the visual
|
and the cross-service contracts, and [`DESIGN.md`](DESIGN.md) for the visual
|
||||||
|
|
|
||||||
|
|
@ -27,9 +27,9 @@ services:
|
||||||
# domain-keyed rule build-push.yml and deploy.yml both use -- so the harness
|
# domain-keyed rule build-push.yml and deploy.yml both use -- so the harness
|
||||||
# tracks the tree instead of drifting behind a hardcoded pin. (It was pinned
|
# tracks the tree instead of drifting behind a hardcoded pin. (It was pinned
|
||||||
# to the split-era v0.0.2-split-ci, under the retired
|
# to the split-era v0.0.2-split-ci, under the retired
|
||||||
# emi/thermograph-backend/app path, long after both were superseded.)
|
# admin_emi/thermograph-backend/app path, long after both were superseded.)
|
||||||
# Override THERMOGRAPH_BACKEND_TEST_TAG to pin a specific build.
|
# Override THERMOGRAPH_BACKEND_TEST_TAG to pin a specific build.
|
||||||
image: ${REGISTRY_HOST:-git.thermograph.org}/${BACKEND_IMAGE_PATH:-emi/thermograph/backend}:${THERMOGRAPH_BACKEND_TEST_TAG:?set it, or run via scripts/backend-for-tests.sh which derives it}
|
image: ${REGISTRY_HOST:-git.thermograph.org}/${BACKEND_IMAGE_PATH:-jinemi/thermograph/backend}:${THERMOGRAPH_BACKEND_TEST_TAG:?set it, or run via scripts/backend-for-tests.sh which derives it}
|
||||||
depends_on:
|
depends_on:
|
||||||
db:
|
db:
|
||||||
condition: service_healthy
|
condition: service_healthy
|
||||||
|
|
|
||||||
|
|
@ -55,5 +55,5 @@ binary; static assets and the YAML copy are read from disk.
|
||||||
go build ./... && go vet ./... && go test ./...
|
go build ./... && go vet ./... && go test ./...
|
||||||
|
|
||||||
The deployed binary is `/usr/local/bin/thermograph-frontend` inside the
|
The deployed binary is `/usr/local/bin/thermograph-frontend` inside the
|
||||||
`emi/thermograph/frontend` image; the image name, `frontend-*` CI workflows and
|
`jinemi/thermograph/frontend` image; the image name, `frontend-*` CI workflows and
|
||||||
deploy path are unchanged from the Python service.
|
deploy path are unchanged from the Python service.
|
||||||
|
|
|
||||||
|
|
@ -58,7 +58,7 @@ THERMOGRAPH_INTERNAL_TOKEN=
|
||||||
# :local, which is the default here -- only set these to pull a specific
|
# :local, which is the default here -- only set these to pull a specific
|
||||||
# published build instead of building locally.
|
# published build instead of building locally.
|
||||||
# REGISTRY_HOST=git.thermograph.org
|
# REGISTRY_HOST=git.thermograph.org
|
||||||
# BACKEND_IMAGE_PATH=emi/thermograph-backend/app
|
# BACKEND_IMAGE_PATH=admin_emi/thermograph-backend/app
|
||||||
# BACKEND_IMAGE_TAG=local
|
# BACKEND_IMAGE_TAG=local
|
||||||
# FRONTEND_IMAGE_PATH=emi/thermograph-frontend/app
|
# FRONTEND_IMAGE_PATH=admin_emi/thermograph-frontend/app
|
||||||
# FRONTEND_IMAGE_TAG=local
|
# FRONTEND_IMAGE_TAG=local
|
||||||
|
|
|
||||||
|
|
@ -26,7 +26,7 @@ rather than which environment happens to live there:
|
||||||
|
|
||||||
| Host | Public IP | Mesh IP | Role |
|
| Host | Public IP | Mesh IP | Role |
|
||||||
|------|-----------|---------|------|
|
|------|-----------|---------|------|
|
||||||
| **vps1** | `75.119.132.91` | `10.10.0.2` | "Operational programs": Forgejo (git + CI + registry, `git.thermograph.org`), Grafana/Loki/Alloy (`dashboard.thermograph.org`), the `emigriffith.dev` portfolio, and the **dev** environment (mesh-only, `10.10.0.2:8137`, no public DNS/Caddy/TLS) |
|
| **vps1** | `75.119.132.91` | `10.10.0.2` | "Operational programs": Forgejo (git + CI + registry; `dev.jinemi.com` canonical, `git.thermograph.org` still served and still used by the registry/mesh pins/runners), Grafana/Loki/Alloy (`dashboard.thermograph.org`), the `emigriffith.dev` portfolio, and the **dev** environment (mesh-only, `10.10.0.2:8137`, no public DNS/Caddy/TLS) |
|
||||||
| **vps2** | `169.58.46.181` | `10.10.0.1` | "The deployed environment": **prod** and **beta**, as two separate Docker Swarm stacks, plus Centralis, Postfix, the backups, and the one shared TimescaleDB instance both stacks use |
|
| **vps2** | `169.58.46.181` | `10.10.0.1` | "The deployed environment": **prod** and **beta**, as two separate Docker Swarm stacks, plus Centralis, Postfix, the backups, and the one shared TimescaleDB instance both stacks use |
|
||||||
| **desktop** | — | `10.10.0.3` | AI-model hosting (voice-to-text, an upcoming-feature LLM) plus flex Swarm-worker capacity. Hosts **no** Thermograph environment — `make dev-up` still works there, but only as a laptop-local rehearsal |
|
| **desktop** | — | `10.10.0.3` | AI-model hosting (voice-to-text, an upcoming-feature LLM) plus flex Swarm-worker capacity. Hosts **no** Thermograph environment — `make dev-up` still works there, but only as a laptop-local rehearsal |
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -12,7 +12,10 @@ Two VPS boxes plus the operator's desktop, on one WireGuard mesh, named by
|
||||||
ROLE rather than by environment:
|
ROLE rather than by environment:
|
||||||
|
|
||||||
- **vps1** — `75.119.132.91`, mesh `10.10.0.2`. "Operational programs": Forgejo
|
- **vps1** — `75.119.132.91`, mesh `10.10.0.2`. "Operational programs": Forgejo
|
||||||
(git + CI + registry, `git.thermograph.org`), Grafana/Loki/Alloy
|
(git + CI + registry; `dev.jinemi.com` is canonical — Forgejo's `ROOT_URL` —
|
||||||
|
and `git.thermograph.org` is still served off the same Caddy site block and
|
||||||
|
still load-bearing for the registry, mesh pins and runner URLs, so don't
|
||||||
|
retire it; see `deploy/forgejo/README.md`), Grafana/Loki/Alloy
|
||||||
(`dashboard.thermograph.org`), the `emigriffith.dev` portfolio site, and the
|
(`dashboard.thermograph.org`), the `emigriffith.dev` portfolio site, and the
|
||||||
**dev** environment (`/opt/thermograph-dev`, tracks `dev`), with its own
|
**dev** environment (`/opt/thermograph-dev`, tracks `dev`), with its own
|
||||||
Postgres container. Dev is mesh-only here — published on `10.10.0.2:8137`,
|
Postgres container. Dev is mesh-only here — published on `10.10.0.2:8137`,
|
||||||
|
|
|
||||||
|
|
@ -3,7 +3,7 @@
|
||||||
Infrastructure for [Thermograph](https://thermograph.org): Terraform host
|
Infrastructure for [Thermograph](https://thermograph.org): Terraform host
|
||||||
provisioning, the SOPS+age secrets vault, WireGuard/Swarm networking, Forgejo,
|
provisioning, the SOPS+age secrets vault, WireGuard/Swarm networking, Forgejo,
|
||||||
Caddy, mail, and the deploy scripts that run the already-built app images on each
|
Caddy, mail, and the deploy scripts that run the already-built app images on each
|
||||||
host. This is a domain of the `emi/thermograph` monorepo — a host's checkout
|
host. This is a domain of the `Jinemi/thermograph` monorepo — a host's checkout
|
||||||
(`/opt/thermograph`, `/opt/thermograph-beta`, or `/opt/thermograph-dev`) is a
|
(`/opt/thermograph`, `/opt/thermograph-beta`, or `/opt/thermograph-dev`) is a
|
||||||
checkout of the whole monorepo, and `infra/` never builds app source; it only
|
checkout of the whole monorepo, and `infra/` never builds app source; it only
|
||||||
runs published images.
|
runs published images.
|
||||||
|
|
|
||||||
|
|
@ -2,7 +2,11 @@
|
||||||
#
|
#
|
||||||
# vps1 serves:
|
# vps1 serves:
|
||||||
# emigriffith.dev -> static portfolio site (from disk)
|
# emigriffith.dev -> static portfolio site (from disk)
|
||||||
# git.thermograph.org -> Forgejo (see deploy/forgejo/caddy-git.conf)
|
# dev.jinemi.com -> Forgejo, CANONICAL (Forgejo's ROOT_URL)
|
||||||
|
# git.thermograph.org -> Forgejo, same site block — still served and
|
||||||
|
# still needed: the registry host in image names,
|
||||||
|
# the mesh /etc/hosts pins and the runners' URLs
|
||||||
|
# all use it (see deploy/forgejo/caddy-git.conf)
|
||||||
# dashboard.thermograph.org -> Grafana (see observability/caddy-grafana.conf)
|
# dashboard.thermograph.org -> Grafana (see observability/caddy-grafana.conf)
|
||||||
#
|
#
|
||||||
# and hosts the **dev** environment at dev.thermograph.org (below).
|
# and hosts the **dev** environment at dev.thermograph.org (below).
|
||||||
|
|
|
||||||
|
|
@ -79,7 +79,7 @@ or 8 is done wrong, which is what the verification lines are for.
|
||||||
|
|
||||||
```
|
```
|
||||||
ssh agent@169.58.46.181
|
ssh agent@169.58.46.181
|
||||||
sudo git clone --branch main http://10.10.0.2:3080/emi/thermograph.git /opt/thermograph-beta
|
sudo git clone --branch main http://10.10.0.2:3080/Jinemi/thermograph.git /opt/thermograph-beta
|
||||||
sudo chown -R agent:agent /opt/thermograph-beta
|
sudo chown -R agent:agent /opt/thermograph-beta
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -16,7 +16,7 @@ The `db` service runs the stock **`timescale/timescaledb:latest-pg18`** image
|
||||||
TTL, and the `recent_stamp` token embedded in derived-payload validity.
|
TTL, and the `recent_stamp` token embedded in derived-payload validity.
|
||||||
|
|
||||||
The schema is created by Alembic (`backend/alembic/versions/0002_climate_hypertables.py`,
|
The schema is created by Alembic (`backend/alembic/versions/0002_climate_hypertables.py`,
|
||||||
run at app boot via `deploy/entrypoint.sh`). The app reads/writes it through
|
run at app boot via `backend/deploy/entrypoint.sh`). The app reads/writes it through
|
||||||
`backend/data/climate_store.py` (psycopg + polars). See
|
`backend/data/climate_store.py` (psycopg + polars). See
|
||||||
`deploy/POSTGRES-MIGRATION.md` for the parquet→hypertable cutover.
|
`deploy/POSTGRES-MIGRATION.md` for the parquet→hypertable cutover.
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -13,7 +13,7 @@
|
||||||
# ssh deploy@vps 'SERVICE=all BACKEND_IMAGE_TAG=sha-<a> FRONTEND_IMAGE_TAG=sha-<b> /opt/thermograph/infra/deploy/deploy.sh'
|
# ssh deploy@vps 'SERVICE=all BACKEND_IMAGE_TAG=sha-<a> FRONTEND_IMAGE_TAG=sha-<b> /opt/thermograph/infra/deploy/deploy.sh'
|
||||||
#
|
#
|
||||||
# FE/BE CI-CD split: backend and frontend are published from separate repos as
|
# FE/BE CI-CD split: backend and frontend are published from separate repos as
|
||||||
# separate images (emi/thermograph/backend, emi/thermograph/frontend),
|
# separate images (jinemi/thermograph/backend, jinemi/thermograph/frontend),
|
||||||
# so a deploy targets ONE service and leaves the other's running container +
|
# so a deploy targets ONE service and leaves the other's running container +
|
||||||
# tag untouched. Each service's live tag is persisted host-side in
|
# tag untouched. Each service's live tag is persisted host-side in
|
||||||
# deploy/.image-tags.env (untracked -- survives the git reset below) so a
|
# deploy/.image-tags.env (untracked -- survives the git reset below) so a
|
||||||
|
|
@ -282,7 +282,7 @@ esac
|
||||||
# host's existing cred; a genuine auth problem then fails loudly at `pull`.
|
# host's existing cred; a genuine auth problem then fails loudly at `pull`.
|
||||||
if [ -n "${REGISTRY_TOKEN:-}" ]; then
|
if [ -n "${REGISTRY_TOKEN:-}" ]; then
|
||||||
echo "==> Logging in to the registry ($REGISTRY_HOST)"
|
echo "==> Logging in to the registry ($REGISTRY_HOST)"
|
||||||
echo "$REGISTRY_TOKEN" | docker login "$REGISTRY_HOST" --username emi --password-stdin
|
echo "$REGISTRY_TOKEN" | docker login "$REGISTRY_HOST" --username admin_emi --password-stdin
|
||||||
else
|
else
|
||||||
echo "==> No REGISTRY_TOKEN in env; relying on the host's existing docker login to $REGISTRY_HOST"
|
echo "==> No REGISTRY_TOKEN in env; relying on the host's existing docker login to $REGISTRY_HOST"
|
||||||
fi
|
fi
|
||||||
|
|
@ -328,7 +328,7 @@ case " ${TARGETS[*]} " in
|
||||||
# then always found no daemon binary in a Postgres image and dropped
|
# then always found no daemon binary in a Postgres image and dropped
|
||||||
# daemon from EVERY backend deploy, regardless of what the real backend
|
# daemon from EVERY backend deploy, regardless of what the real backend
|
||||||
# image contained -- reproduced and confirmed against beta directly.
|
# image contained -- reproduced and confirmed against beta directly.
|
||||||
daemon_img="${REGISTRY_HOST:-git.thermograph.org}/${BACKEND_IMAGE_PATH:-emi/thermograph/backend}:${BACKEND_IMAGE_TAG}"
|
daemon_img="${REGISTRY_HOST:-git.thermograph.org}/${BACKEND_IMAGE_PATH:-jinemi/thermograph/backend}:${BACKEND_IMAGE_TAG}"
|
||||||
if ! docker run --rm --entrypoint sh "$daemon_img" -c 'test -x /usr/local/bin/thermograph-daemon' 2>/dev/null; then
|
if ! docker run --rm --entrypoint sh "$daemon_img" -c 'test -x /usr/local/bin/thermograph-daemon' 2>/dev/null; then
|
||||||
echo "==> $daemon_img predates the daemon binary; rolling without the daemon service this run"
|
echo "==> $daemon_img predates the daemon binary; rolling without the daemon service this run"
|
||||||
kept=()
|
kept=()
|
||||||
|
|
@ -408,8 +408,8 @@ fi
|
||||||
# roll never garbage-collects the image a rollback would need; docker also
|
# roll never garbage-collects the image a rollback would need; docker also
|
||||||
# refuses to remove an image any container still uses.
|
# refuses to remove an image any container still uses.
|
||||||
echo "==> Pruning old app-image tags"
|
echo "==> Pruning old app-image tags"
|
||||||
_be_repo="${REGISTRY_HOST}/${BACKEND_IMAGE_PATH:-emi/thermograph/backend}"
|
_be_repo="${REGISTRY_HOST}/${BACKEND_IMAGE_PATH:-jinemi/thermograph/backend}"
|
||||||
_fe_repo="${REGISTRY_HOST}/${FRONTEND_IMAGE_PATH:-emi/thermograph/frontend}"
|
_fe_repo="${REGISTRY_HOST}/${FRONTEND_IMAGE_PATH:-jinemi/thermograph/frontend}"
|
||||||
docker images --format '{{.Repository}}:{{.Tag}}' \
|
docker images --format '{{.Repository}}:{{.Tag}}' \
|
||||||
| grep -E "^(${_be_repo}|${_fe_repo}):" \
|
| grep -E "^(${_be_repo}|${_fe_repo}):" \
|
||||||
| grep -v -e "^${_be_repo}:${BACKEND_IMAGE_TAG}$" -e "^${_fe_repo}:${FRONTEND_IMAGE_TAG}$" \
|
| grep -v -e "^${_be_repo}:${BACKEND_IMAGE_TAG}$" -e "^${_fe_repo}:${FRONTEND_IMAGE_TAG}$" \
|
||||||
|
|
|
||||||
|
|
@ -3,16 +3,24 @@
|
||||||
Runs as `deploy/forgejo/docker-stack.yml` — the only Swarm-scheduled workload
|
Runs as `deploy/forgejo/docker-stack.yml` — the only Swarm-scheduled workload
|
||||||
this cluster carries (prod and beta's app stacks are separate `docker stack
|
this cluster carries (prod and beta's app stacks are separate `docker stack
|
||||||
deploy`s that happen to run on the manager node, vps2 — see
|
deploy`s that happen to run on the manager node, vps2 — see
|
||||||
`deploy/stack/README` context in `DEPLOY.md`). Pinned to the **vps1** node
|
`DEPLOY.md`). Pinned to the **vps1** node
|
||||||
(`75.119.132.91`) via the `role=forge` label from
|
(`75.119.132.91`) via the `role=forge` label from
|
||||||
`deploy/swarm/label-forge-node.sh`.
|
`deploy/swarm/label-forge-node.sh`.
|
||||||
|
|
||||||
The Actions **runner** is deliberately *not* part of this stack — see
|
The Actions **runner** is deliberately *not* part of this stack: a
|
||||||
`DEPLOY-DEV.md` and the note in `register-lan-runner.sh`'s own header for
|
Swarm-scheduled runner cannot redeploy the Swarm that schedules it, so CI would
|
||||||
where it runs today. That's the canonical placement per
|
be unavailable exactly when the cluster is what needs repairing. An earlier
|
||||||
`thermograph-docs/runbooks/implementation-handoff.md` Track B step 5; an
|
revision did run it as a Swarm-scheduled Docker-in-Docker sidecar pinned to the
|
||||||
earlier revision of this stack ran the runner as a Swarm-scheduled
|
Forgejo node; that is gone.
|
||||||
Docker-in-Docker sidecar pinned to the Forgejo node, which is gone now.
|
|
||||||
|
Runners live in two places, and only one of them counts:
|
||||||
|
|
||||||
|
- **`runner-vps2/`** — the always-on runner, a plain `restart: always` container
|
||||||
|
on vps2. This is the one CI depends on.
|
||||||
|
- **`register-lan-runner.sh`** — the desktop's systemd runner. Extra capacity.
|
||||||
|
**Nothing may assume it is up.** It was the only registered runner in the
|
||||||
|
estate until 2026-08-01, and its 21-hour outage on 2026-07-31 froze every
|
||||||
|
merge and deploy and stopped the nightly backup from firing.
|
||||||
|
|
||||||
## Prerequisites
|
## Prerequisites
|
||||||
|
|
||||||
|
|
@ -75,6 +83,58 @@ pattern as its other site blocks, same automatic-HTTPS.
|
||||||
reach the registry over the mesh, not the public internet — see "Registry
|
reach the registry over the mesh, not the public internet — see "Registry
|
||||||
access from mesh clients" below.
|
access from mesh clients" below.
|
||||||
|
|
||||||
|
## Migrating the domain (git.thermograph.org -> dev.jinemi.com)
|
||||||
|
|
||||||
|
**Web UI and OAuth: done.** `dev.jinemi.com` is Forgejo's `ROOT_URL`
|
||||||
|
(`FORGEJO_DOMAIN` in `docker-stack.yml`), so it is the name Forgejo generates in
|
||||||
|
clone URLs, the Google OAuth callback, webhook payload URLs and mail links.
|
||||||
|
|
||||||
|
**Registry: deliberately not done.** Renaming the registry host is a separate
|
||||||
|
migration from renaming the web UI, and the two need not happen together.
|
||||||
|
|
||||||
|
`caddy-git.conf` lists both names on a *single* site block. That is deliberate,
|
||||||
|
not cosmetic: the `/v2/*` mesh-only matcher is per-block, so a second block for
|
||||||
|
either name would re-expose the registry API publicly and undo hazard #15.
|
||||||
|
Whatever else changes, keep the two names in one block.
|
||||||
|
|
||||||
|
**`git.thermograph.org` must stay served.** It is not a courtesy redirect for
|
||||||
|
old bookmarks — CI resolves it:
|
||||||
|
|
||||||
|
- images are named by registry host, and the `git.thermograph.org/` prefix is
|
||||||
|
baked into the runner labels (`register-lan-runner.sh`,
|
||||||
|
`runner-vps2/README.md`), the CI-runner image (`ci-runner/Dockerfile`),
|
||||||
|
`REGISTRY_HOST` in `infra/.env.example`, and every already-pushed tag;
|
||||||
|
- every mesh client in "Registry access from mesh clients" below pins
|
||||||
|
`git.thermograph.org` to `10.10.0.2` in `/etc/hosts`, which is what makes
|
||||||
|
Caddy's `/v2/*` matcher see a mesh source IP instead of returning 403;
|
||||||
|
- registered runners hold the instance URL they registered with
|
||||||
|
(`https://git.thermograph.org`); they keep working while that name resolves.
|
||||||
|
|
||||||
|
### Before changing ROOT_URL again
|
||||||
|
|
||||||
|
Login is Google-SSO-only, so the Google OAuth client must already carry a
|
||||||
|
matching redirect URI, or nobody can reach the UI — including to undo the
|
||||||
|
change. Both of these are registered on client
|
||||||
|
`337247886376-4360kepi20f24pue62ptg0lnqhcht46l.apps.googleusercontent.com`:
|
||||||
|
|
||||||
|
https://dev.jinemi.com/user/oauth2/google/callback
|
||||||
|
https://git.thermograph.org/user/oauth2/google/callback
|
||||||
|
|
||||||
|
Verify rather than assume. Google answers this with no credentials: a
|
||||||
|
registered URI serves the sign-in page, an unregistered one returns
|
||||||
|
`redirect_uri_mismatch`. Substitute the host you intend to make canonical:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
CID=337247886376-4360kepi20f24pue62ptg0lnqhcht46l.apps.googleusercontent.com
|
||||||
|
curl -sL "https://accounts.google.com/o/oauth2/v2/auth?client_id=$CID&redirect_uri=https%3A%2F%2F<host>%2Fuser%2Foauth2%2Fgoogle%2Fcallback&response_type=code&scope=openid+email+profile&state=probe" \
|
||||||
|
| grep -q redirect_uri_mismatch && echo REJECTED || echo ACCEPTED
|
||||||
|
```
|
||||||
|
|
||||||
|
The stack has **no auto-deploy**: a `FORGEJO_DOMAIN` change here only takes
|
||||||
|
effect when someone re-runs `docker stack deploy` by hand on the manager (vps2)
|
||||||
|
— see "Deploy / update" above. Until then the running service keeps whatever
|
||||||
|
`ROOT_URL` it was last deployed with, regardless of what this file says.
|
||||||
|
|
||||||
## Registry access from mesh clients
|
## Registry access from mesh clients
|
||||||
|
|
||||||
Any node that needs `docker login`/push/pull against the registry (the CI
|
Any node that needs `docker login`/push/pull against the registry (the CI
|
||||||
|
|
@ -93,6 +153,28 @@ echo "10.10.0.2 git.thermograph.org" | sudo tee -a /etc/hosts
|
||||||
numbering — adjust if you assigned it differently.) The git/web UI keeps
|
numbering — adjust if you assigned it differently.) The git/web UI keeps
|
||||||
working normally for everyone else since only `/v2/*` is restricted.
|
working normally for everyone else since only `/v2/*` is restricted.
|
||||||
|
|
||||||
|
**Pin the canonical name too, not just the image host.** Forgejo derives the
|
||||||
|
registry's bearer-token realm from `ROOT_URL`, so a client that already
|
||||||
|
resolves `git.thermograph.org` over the mesh is still sent to
|
||||||
|
`https://<ROOT_URL host>/v2/token` to collect a token. When `ROOT_URL` became
|
||||||
|
`dev.jinemi.com`, that second request took the public route, the `/v2/*`
|
||||||
|
matcher above answered 403, and docker fell back to anonymous — every push then
|
||||||
|
failed with `unauthorized: reqPackageAccess`, which reads exactly like a
|
||||||
|
revoked token or a missing scope. It is neither. Pin both names:
|
||||||
|
|
||||||
|
```
|
||||||
|
echo "10.10.0.2 git.thermograph.org dev.jinemi.com" | sudo tee -a /etc/hosts
|
||||||
|
```
|
||||||
|
|
||||||
|
The realm moves whenever `ROOT_URL` does, so read it rather than assuming:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
curl -sI https://git.thermograph.org/v2/ | grep -i www-authenticate
|
||||||
|
```
|
||||||
|
|
||||||
|
These `/etc/hosts` pins are host state. Nothing in this repo writes them, so a
|
||||||
|
reprovisioned node has to be pinned again before it can pull.
|
||||||
|
|
||||||
## Register the Actions runner
|
## Register the Actions runner
|
||||||
|
|
||||||
Once Forgejo answers at its domain:
|
Once Forgejo answers at its domain:
|
||||||
|
|
@ -117,14 +199,48 @@ parallel, and even capacity 3 (an earlier, undocumented hand-tune) leaves
|
||||||
`build-backend`/`build-frontend`/`validate-observability` queued behind
|
`build-backend`/`build-frontend`/`validate-observability` queued behind
|
||||||
those three before they get a slot.
|
those three before they get a slot.
|
||||||
|
|
||||||
**Adding more runner capacity should mean raising this number, or adding a
|
**This paragraph used to say a runner must never go on vps2. That was reversed
|
||||||
second runner alongside it — not putting a runner on prod or beta (vps2).**
|
deliberately on 2026-08-01, and the reasoning is worth keeping rather than
|
||||||
`container.docker_host: automount` gives job containers the *host's* Docker
|
quietly deleting.**
|
||||||
socket; on vps2 that would mean any CI job has root-equivalent access to both
|
|
||||||
the prod and beta stacks running there. An earlier revision of this stack
|
The original objection stands on its merits: `container.docker_host: automount`
|
||||||
actually ran the runner as a Swarm-hosted container on the box Forgejo was
|
gives job containers the *host's* Docker socket, so a runner on vps2 means any
|
||||||
pinned to and was deliberately reverted for this same class of reason — see
|
CI job has root-equivalent access to both the prod and beta stacks running
|
||||||
the note at the top of `docker-stack.yml`.
|
there. An earlier revision of this stack ran the runner as a Swarm-hosted
|
||||||
|
container on the Forgejo node and was reverted for the same class of reason.
|
||||||
|
|
||||||
|
What changed is that the alternative turned out to be worse. The desktop was
|
||||||
|
the estate's **only** registered runner, and when it went offline on
|
||||||
|
2026-07-31 nothing merged, nothing deployed, and the nightly backup did not
|
||||||
|
fire for 21 hours — on a repo where every branch is protected and every change
|
||||||
|
is a PR, a single absent runner freezes the whole estate. Availability of CI
|
||||||
|
is not a convenience here; it is what the backup and the deploy path both
|
||||||
|
hang off.
|
||||||
|
|
||||||
|
So the runner on vps2 (`runner-vps2/`) is a considered trade, not an
|
||||||
|
oversight, and it is bounded rather than assumed benign:
|
||||||
|
|
||||||
|
- `capacity: 1` — one job at a time, so a CI burst cannot contend with
|
||||||
|
`thermograph_web` for the box's six cores.
|
||||||
|
- `--cpus=2 --memory=4g` on job containers, set in `container.options`. Job
|
||||||
|
containers are siblings of the runner rather than children, so the runner's
|
||||||
|
own compose limits do not reach them; that option is the only lever that
|
||||||
|
does.
|
||||||
|
- `valid_volumes: []` — a job cannot bind-mount a host path, which is the
|
||||||
|
difference between "a job can read `/etc/thermograph.env`" and "a job
|
||||||
|
cannot", on the box where that file is prod's.
|
||||||
|
- The compose project joins no `thermograph` network.
|
||||||
|
|
||||||
|
Be honest about what that buys: it is defence against **accident**, not
|
||||||
|
against a hostile workflow author. Forgejo's own database on vps1 already
|
||||||
|
stores `VPS2_SSH_KEY`, which is root on vps2, so the material was already
|
||||||
|
reachable — what changed is that it is now reachable by a *job* rather than
|
||||||
|
only at rest. On a two-person estate where both people can already SSH to that
|
||||||
|
box as root, that is the honest boundary.
|
||||||
|
|
||||||
|
**If you are adding capacity rather than redundancy, raise `capacity` or add a
|
||||||
|
runner on a box that hosts nothing.** The vps2 runner exists so that CI has a
|
||||||
|
second home, not because prod is a good place to run CI.
|
||||||
|
|
||||||
## Custom CI job image (`ci-runner/`)
|
## Custom CI job image (`ci-runner/`)
|
||||||
|
|
||||||
|
|
@ -143,14 +259,14 @@ bug hit during the frontend Go rewrite). `ci-runner` adds `docker-ce-cli` +
|
||||||
`git`/`python3`/`python3-yaml` for the other jobs that need them
|
`git`/`python3`/`python3-yaml` for the other jobs that need them
|
||||||
(`shell-lint`, `observability-validate`).
|
(`shell-lint`, `observability-validate`).
|
||||||
|
|
||||||
Current tag: `git.thermograph.org/emi/thermograph/ci-runner:v2` (`v1` is
|
Current tag: `git.thermograph.org/jinemi/thermograph/ci-runner:v2` (`v1` is
|
||||||
broken — do not register any runner against it). Rebuild/push (requires a PAT
|
broken — do not register any runner against it). Rebuild/push (requires a PAT
|
||||||
with `write:package` scope — the embedded git-remote token lacks it, same
|
with `write:package` scope — the embedded git-remote token lacks it, same
|
||||||
requirement documented in `build-push.yml`):
|
requirement documented in `build-push.yml`):
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker build -t git.thermograph.org/emi/thermograph/ci-runner:vN deploy/forgejo/ci-runner
|
docker build -t git.thermograph.org/jinemi/thermograph/ci-runner:vN deploy/forgejo/ci-runner
|
||||||
docker push git.thermograph.org/emi/thermograph/ci-runner:vN
|
docker push git.thermograph.org/jinemi/thermograph/ci-runner:vN
|
||||||
```
|
```
|
||||||
|
|
||||||
`register-lan-runner.sh`'s `LABELS` default points at the current tag, so
|
`register-lan-runner.sh`'s `LABELS` default points at the current tag, so
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,6 @@
|
||||||
# Only the block below (from "git.thermograph.org {") goes into the live
|
# Only the block below (from "git.thermograph.org, dev.jinemi.com {") goes into
|
||||||
# Caddyfile — append just that part, not this instructional header, or it
|
# the live Caddyfile — append just that part, not this instructional header, or
|
||||||
# ends up as a confusing orphaned comment in production config with no
|
# it ends up as a confusing orphaned comment in production config with no
|
||||||
# surrounding context (docker-stack.yml isn't visible from there).
|
# surrounding context (docker-stack.yml isn't visible from there).
|
||||||
#
|
#
|
||||||
# Target: /etc/caddy/Caddyfile on vps1 (the box `forgejo` is pinned to — see
|
# Target: /etc/caddy/Caddyfile on vps1 (the box `forgejo` is pinned to — see
|
||||||
|
|
@ -9,19 +9,31 @@
|
||||||
# Forgejo publishes its web UI to 127.0.0.1:3080 only (host-local), and this
|
# Forgejo publishes its web UI to 127.0.0.1:3080 only (host-local), and this
|
||||||
# block is the only thing that ever talks to it.
|
# block is the only thing that ever talks to it.
|
||||||
#
|
#
|
||||||
# Registry exposure (hazard #15, see docker-stack.yml's header comment):
|
# TWO hostnames, ONE block, and that is load-bearing — see "Registry exposure"
|
||||||
# git.thermograph.org/v2/* is the built-in OCI registry API. It's blocked
|
# below. Both names must share a site block so the /v2/* restriction covers
|
||||||
# from anywhere except the WireGuard mesh (10.10.0.0/24) — reachable to CI
|
# both. Splitting dev.jinemi.com into its own block serves the same Forgejo
|
||||||
# runners and Swarm nodes over wg0, not from the public internet. Everything
|
# with the registry API open to the public internet.
|
||||||
# else (web UI, git-over-HTTP, PR pages) stays public like the rest of the
|
|
||||||
# site.
|
|
||||||
#
|
#
|
||||||
# Update the domain below to match whatever you actually pointed at vps1's
|
# dev.jinemi.com is CANONICAL — it is Forgejo's ROOT_URL (docker-stack.yml,
|
||||||
# IP, if not git.thermograph.org.
|
# FORGEJO_DOMAIN), so it is the name Forgejo puts in clone URLs, the OAuth
|
||||||
|
# callback, webhook payload URLs and mail links.
|
||||||
|
#
|
||||||
|
# git.thermograph.org is kept as a served alias, NOT as dead weight: the
|
||||||
|
# registry host baked into image names, the mesh /etc/hosts pins and the
|
||||||
|
# runners' registered instance URL all still say git.thermograph.org. Removing
|
||||||
|
# this name from the block breaks CI, not just old bookmarks. Both names get
|
||||||
|
# their own Let's Encrypt cert automatically (HTTP-01); both A records point
|
||||||
|
# at vps1. See README.md, "Migrating the domain".
|
||||||
|
#
|
||||||
|
# Registry exposure (hazard #15, see docker-stack.yml's header comment):
|
||||||
|
# /v2/* is the built-in OCI registry API. It's blocked from anywhere except
|
||||||
|
# the WireGuard mesh (10.10.0.0/24) — reachable to CI runners and Swarm nodes
|
||||||
|
# over wg0, not from the public internet. Everything else (web UI,
|
||||||
|
# git-over-HTTP, PR pages) stays public like the rest of the site.
|
||||||
|
|
||||||
# --- copy from here down into /etc/caddy/Caddyfile ---
|
# --- copy from here down into /etc/caddy/Caddyfile ---
|
||||||
|
|
||||||
git.thermograph.org {
|
git.thermograph.org, dev.jinemi.com {
|
||||||
encode zstd gzip
|
encode zstd gzip
|
||||||
|
|
||||||
@registry_external {
|
@registry_external {
|
||||||
|
|
|
||||||
|
|
@ -19,9 +19,9 @@
|
||||||
# class, and skips the per-job install entirely.
|
# class, and skips the per-job install entirely.
|
||||||
#
|
#
|
||||||
# Build/push (manual — see ../README.md for when to rebuild):
|
# Build/push (manual — see ../README.md for when to rebuild):
|
||||||
# docker build -t git.thermograph.org/emi/thermograph/ci-runner:vN \
|
# docker build -t git.thermograph.org/jinemi/thermograph/ci-runner:vN \
|
||||||
# infra/deploy/forgejo/ci-runner
|
# infra/deploy/forgejo/ci-runner
|
||||||
# docker push git.thermograph.org/emi/thermograph/ci-runner:vN
|
# docker push git.thermograph.org/jinemi/thermograph/ci-runner:vN
|
||||||
#
|
#
|
||||||
# Cutting over the live runner to a new tag means editing the `labels` array
|
# Cutting over the live runner to a new tag means editing the `labels` array
|
||||||
# in ~/forgejo-runner/.runner on the runner host directly (same runner
|
# in ~/forgejo-runner/.runner on the runner host directly (same runner
|
||||||
|
|
|
||||||
|
|
@ -3,11 +3,16 @@
|
||||||
# from the Terraform-managed docker-compose.yml that runs the Thermograph app
|
# from the Terraform-managed docker-compose.yml that runs the Thermograph app
|
||||||
# itself: this stack's only job is Forgejo and its container registry.
|
# itself: this stack's only job is Forgejo and its container registry.
|
||||||
#
|
#
|
||||||
# The Actions RUNNER is deliberately NOT a service in this stack. Per
|
# The Actions RUNNER is deliberately NOT a service in this stack, and that is a
|
||||||
# thermograph-docs/runbooks/implementation-handoff.md (Track B step 5), the canonical
|
# reasoned choice rather than an omission: a Swarm-scheduled runner cannot
|
||||||
# design registers the runner on the desktop node as a plain systemd service
|
# redeploy the Swarm that schedules it, so CI would be gone exactly when the
|
||||||
# (deploy/forgejo/register-lan-runner.sh) — the same place the pre-Forgejo
|
# cluster is the thing that is broken.
|
||||||
# GitHub self-hosted runner already lived, not a Swarm-scheduled container.
|
#
|
||||||
|
# The runner that CI depends on runs on vps2 as a plain restart:always container
|
||||||
|
# — deploy/forgejo/runner-vps2/. The desktop's systemd runner
|
||||||
|
# (deploy/forgejo/register-lan-runner.sh) stays registered as extra capacity.
|
||||||
|
# Nothing may assume the desktop is up: it was the ONLY runner until 2026-08-01,
|
||||||
|
# and its 21-hour outage on 2026-07-31 froze every merge, deploy and backup.
|
||||||
#
|
#
|
||||||
# No Traefik here. Forgejo is pinned to vps1 (node.labels.role == forge)
|
# No Traefik here. Forgejo is pinned to vps1 (node.labels.role == forge)
|
||||||
# because that's what was chosen, and vps1 is ALSO the emigriffith.dev portfolio
|
# because that's what was chosen, and vps1 is ALSO the emigriffith.dev portfolio
|
||||||
|
|
@ -75,8 +80,25 @@ services:
|
||||||
FORGEJO__database__HOST: db:5432
|
FORGEJO__database__HOST: db:5432
|
||||||
FORGEJO__database__NAME: forgejo
|
FORGEJO__database__NAME: forgejo
|
||||||
FORGEJO__database__USER: forgejo
|
FORGEJO__database__USER: forgejo
|
||||||
FORGEJO__server__DOMAIN: "${FORGEJO_DOMAIN:-git.thermograph.org}"
|
# Canonical domain. Forgejo has exactly ONE ROOT_URL and derives every
|
||||||
FORGEJO__server__ROOT_URL: "https://${FORGEJO_DOMAIN:-git.thermograph.org}/"
|
# absolute URL from it — clone URLs, the Google OAuth callback, webhook
|
||||||
|
# payload URLs, mail links. git.thermograph.org is still served (Caddy
|
||||||
|
# answers for both names off one site block, deploy/forgejo/caddy-git.conf)
|
||||||
|
# and still works for browsing, git-over-HTTP and the API; it is simply no
|
||||||
|
# longer the name Forgejo generates.
|
||||||
|
#
|
||||||
|
# Changing this breaks login unless the matching redirect URI exists on
|
||||||
|
# the Google OAuth client FIRST — login is SSO-only, so a mismatch locks
|
||||||
|
# everyone out of the UI, including out of the ability to undo it. Both
|
||||||
|
# https://dev.jinemi.com/user/oauth2/google/callback and the
|
||||||
|
# git.thermograph.org one are registered; verify with a probe before
|
||||||
|
# changing this again (deploy/forgejo/README.md, "Migrating the domain").
|
||||||
|
#
|
||||||
|
# NOT changed by this: the registry host baked into image names, the mesh
|
||||||
|
# /etc/hosts pins, and the runners' registered instance URL all still say
|
||||||
|
# git.thermograph.org. That is a separate migration.
|
||||||
|
FORGEJO__server__DOMAIN: "${FORGEJO_DOMAIN:-dev.jinemi.com}"
|
||||||
|
FORGEJO__server__ROOT_URL: "https://${FORGEJO_DOMAIN:-dev.jinemi.com}/"
|
||||||
FORGEJO__server__SSH_PORT: "2222"
|
FORGEJO__server__SSH_PORT: "2222"
|
||||||
# Without this, Forgejo starts no SSH server at all despite SSH_PORT
|
# Without this, Forgejo starts no SSH server at all despite SSH_PORT
|
||||||
# being set — git@ clones get "connection refused", not a slow failure.
|
# being set — git@ clones get "connection refused", not a slow failure.
|
||||||
|
|
|
||||||
|
|
@ -18,11 +18,16 @@
|
||||||
# breaks by keeping it registered — no workflow requests it — but do not build
|
# breaks by keeping it registered — no workflow requests it — but do not build
|
||||||
# anything new on it.
|
# anything new on it.
|
||||||
#
|
#
|
||||||
# TODO(cutover): whether this machine keeps serving the `docker` label at all is
|
# This machine keeps serving the `docker` label as EXTRA capacity, and that is
|
||||||
# an open call. It is no longer the only runner (the Swarm-hosted one is
|
# now a settled call rather than an open one. The claim it used to make here —
|
||||||
# always-on), and the box's new job is AI model hosting plus flex Swarm-worker
|
# that a Swarm-hosted runner was always-on, so the estate no longer depended on
|
||||||
# capacity. Keeping it is fine — it is real CI capacity — but the estate no
|
# this box — was false: no such runner existed. This was the only registered
|
||||||
# longer depends on it, so decide deliberately rather than by inertia.
|
# runner in the estate until 2026-08-01, and when it went offline on 2026-07-31
|
||||||
|
# every merge, deploy and backup stopped for 21 hours.
|
||||||
|
#
|
||||||
|
# The runner the estate depends on is deploy/forgejo/runner-vps2/. Keeping this
|
||||||
|
# one is worthwhile (a second runner means a queue drains instead of stalling),
|
||||||
|
# but nothing may be built on the assumption that this box is up.
|
||||||
#
|
#
|
||||||
# bash deploy/forgejo/register-lan-runner.sh <forgejo_url> <registration_token>
|
# bash deploy/forgejo/register-lan-runner.sh <forgejo_url> <registration_token>
|
||||||
#
|
#
|
||||||
|
|
@ -35,7 +40,7 @@ set -euo pipefail
|
||||||
FORGEJO_URL="${1:?usage: $0 <forgejo_url> <registration_token>}"
|
FORGEJO_URL="${1:?usage: $0 <forgejo_url> <registration_token>}"
|
||||||
TOKEN="${2:?}"
|
TOKEN="${2:?}"
|
||||||
RUNNER_DIR="${RUNNER_DIR:-$HOME/forgejo-runner}"
|
RUNNER_DIR="${RUNNER_DIR:-$HOME/forgejo-runner}"
|
||||||
LABELS="${LABELS:-docker:docker://git.thermograph.org/emi/thermograph/ci-runner:v2,thermograph-lan}"
|
LABELS="${LABELS:-docker:docker://git.thermograph.org/jinemi/thermograph/ci-runner:v2,thermograph-lan}"
|
||||||
|
|
||||||
echo "==> Stopping and disabling the old GitHub Actions runner service, if present"
|
echo "==> Stopping and disabling the old GitHub Actions runner service, if present"
|
||||||
systemctl --user stop github-actions-runner 2>/dev/null || true
|
systemctl --user stop github-actions-runner 2>/dev/null || true
|
||||||
|
|
@ -88,6 +93,13 @@ cat > "$HOME/.config/systemd/user/forgejo-runner.service" <<EOF
|
||||||
[Unit]
|
[Unit]
|
||||||
Description=Forgejo Actions runner (docker + thermograph-lan)
|
Description=Forgejo Actions runner (docker + thermograph-lan)
|
||||||
After=network-online.target
|
After=network-online.target
|
||||||
|
# These bound the Restart=always loop below, so a genuinely broken config (bad
|
||||||
|
# token, docker unreachable) ends as a stopped unit rather than spinning
|
||||||
|
# forever. They belong in [Unit]: systemd accepts them in [Service] without
|
||||||
|
# complaint but IGNORES them, leaving the 10s default — \`systemctl --user show
|
||||||
|
# forgejo-runner -p StartLimitIntervalUSec\` is how you tell which one you got.
|
||||||
|
StartLimitIntervalSec=300
|
||||||
|
StartLimitBurst=5
|
||||||
|
|
||||||
[Service]
|
[Service]
|
||||||
WorkingDirectory=${RUNNER_DIR}
|
WorkingDirectory=${RUNNER_DIR}
|
||||||
|
|
@ -97,7 +109,16 @@ WorkingDirectory=${RUNNER_DIR}
|
||||||
# SOPS-configured) fails with "not installed" even though it plainly is.
|
# SOPS-configured) fails with "not installed" even though it plainly is.
|
||||||
Environment=PATH=%h/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
|
Environment=PATH=%h/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
|
||||||
ExecStart=${RUNNER_DIR}/forgejo-runner daemon -c config.yaml
|
ExecStart=${RUNNER_DIR}/forgejo-runner daemon -c config.yaml
|
||||||
Restart=on-failure
|
# \`always\`, NOT \`on-failure\` — the same distinction that took Forgejo down for
|
||||||
|
# 27 hours on 2026-07-29 (see docker-stack.yml's restart_policy comment): an
|
||||||
|
# always-on daemon that exits **0** is not "finished successfully", it is a
|
||||||
|
# daemon that stopped and must come back. \`on-failure\` cannot tell those apart,
|
||||||
|
# and forgejo-runner exits 0 on several paths — a lost instance connection it
|
||||||
|
# gives up on, or a SIGTERM from a Docker restart it interprets as a clean
|
||||||
|
# shutdown. The failure mode is silent: the unit sits \`inactive (dead)\`, the UI
|
||||||
|
# shows the runner offline, and every protected-branch merge blocks on a check
|
||||||
|
# that will never be produced.
|
||||||
|
Restart=always
|
||||||
RestartSec=5
|
RestartSec=5
|
||||||
|
|
||||||
[Install]
|
[Install]
|
||||||
|
|
|
||||||
3
infra/deploy/forgejo/runner-vps2/.gitignore
vendored
Normal file
3
infra/deploy/forgejo/runner-vps2/.gitignore
vendored
Normal file
|
|
@ -0,0 +1,3 @@
|
||||||
|
# The runner registration (data/.runner) holds a long-lived runner token, and
|
||||||
|
# the daemon writes caches and job workspaces beside it. None of it belongs in git.
|
||||||
|
data/
|
||||||
151
infra/deploy/forgejo/runner-vps2/README.md
Normal file
151
infra/deploy/forgejo/runner-vps2/README.md
Normal file
|
|
@ -0,0 +1,151 @@
|
||||||
|
# The always-on Actions runner (vps2)
|
||||||
|
|
||||||
|
The runner CI actually depends on. The desktop runner
|
||||||
|
(`../register-lan-runner.sh`) stays registered as extra capacity, but nothing in
|
||||||
|
the estate may assume it is up.
|
||||||
|
|
||||||
|
Labels: `docker` only. `thermograph-lan` is deliberately **not** registered here
|
||||||
|
— it was the host-native LAN-deploy label, dev moved to vps1 and is deployed
|
||||||
|
over SSH like beta and prod, and no workflow requests it any more.
|
||||||
|
|
||||||
|
## Why this exists
|
||||||
|
|
||||||
|
On 2026-07-31 16:31Z the desktop — then the estate's **only** registered runner
|
||||||
|
— went offline. For the next 21 hours nothing merged (every protected branch
|
||||||
|
requires a check that only a runner can produce), nothing deployed, and the
|
||||||
|
03:00Z `ops-cron` did not run. Forgejo held that scheduled run in `waiting`
|
||||||
|
rather than dropping it, so all three jobs completed on reconnect and no backup
|
||||||
|
was actually lost. A longer outage would not have been so kind.
|
||||||
|
|
||||||
|
`docker-stack.yml` and this directory's parent README both described an
|
||||||
|
"always-on Swarm-hosted runner" during that entire window. There wasn't one; an
|
||||||
|
early revision had it as a Docker-in-Docker sidecar and it was removed. That is
|
||||||
|
why the single point of failure went unnoticed for weeks, and it is the reason
|
||||||
|
the root `CLAUDE.md` treats a stale doc as a correctness bug rather than a
|
||||||
|
documentation one.
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
|
||||||
|
- vps2's docker daemon is logged in to `git.thermograph.org` (it is —
|
||||||
|
`/root/.docker/config.json`), so the daemon can pull the job image.
|
||||||
|
- `git.thermograph.org` resolves to the **mesh** address from vps2 for registry
|
||||||
|
traffic. Job containers get this via `--add-host` in `config.yaml`; the host
|
||||||
|
daemon already has it.
|
||||||
|
- The job image exists: `git.thermograph.org/jinemi/thermograph/ci-runner:v2`.
|
||||||
|
Built from `../ci-runner/Dockerfile`; see that file for when to rebuild.
|
||||||
|
|
||||||
|
## Where it is deployed, and why not from the checkout
|
||||||
|
|
||||||
|
**`/opt/forgejo-runner/` on vps2**, holding a copy of `docker-compose.yml` and
|
||||||
|
`config.yaml` from this directory, plus a `data/` the runner owns.
|
||||||
|
|
||||||
|
Deliberately *not* run in place from `/opt/thermograph/…/runner-vps2/`. That
|
||||||
|
checkout is `git reset --hard`'d on every prod deploy; untracked files do
|
||||||
|
survive that (it is the same property `deploy/.image-tags.env` relies on), but a
|
||||||
|
single `git clean -fdx` during troubleshooting would delete `data/.runner` and
|
||||||
|
de-register the runner. Putting the thing that repairs the estate inside the
|
||||||
|
tree the estate rewrites is the same mistake as scheduling it on the Swarm it
|
||||||
|
deploys.
|
||||||
|
|
||||||
|
So this directory is the source of truth; vps2 gets a copy:
|
||||||
|
|
||||||
|
The two files do **not** go to the same place, and getting it wrong fails in a
|
||||||
|
confusing way. `docker-compose.yml` mounts `./data:/data` and starts the daemon
|
||||||
|
with `--config /data/config.yaml`, so `/data` *is* `data/` — a `config.yaml`
|
||||||
|
sitting beside the compose file is invisible to the container. The daemon then
|
||||||
|
either refuses to start or silently runs on defaults (no `--add-host`, so every
|
||||||
|
registry push fails as if the credential were wrong).
|
||||||
|
|
||||||
|
```bash
|
||||||
|
install -d -m 0755 /opt/forgejo-runner/data
|
||||||
|
cp /opt/thermograph/infra/deploy/forgejo/runner-vps2/docker-compose.yml \
|
||||||
|
/opt/forgejo-runner/
|
||||||
|
cp /opt/thermograph/infra/deploy/forgejo/runner-vps2/config.yaml \
|
||||||
|
/opt/forgejo-runner/data/
|
||||||
|
```
|
||||||
|
|
||||||
|
`data/` is also where `register` writes `.runner`, so the runner's whole state —
|
||||||
|
config plus registration — lives in the one directory the compose file mounts.
|
||||||
|
|
||||||
|
Re-copy after changing either file here, then `docker compose up -d`. There is
|
||||||
|
no automation for that step and there should not be: a workflow that redeploys
|
||||||
|
the runner runs *on* the runner.
|
||||||
|
|
||||||
|
## One-time registration
|
||||||
|
|
||||||
|
The registration token is short-lived and single-purpose. Fetch it from the
|
||||||
|
Forgejo UI (**Site Administration → Actions → Runners → Create new Runner**) or
|
||||||
|
the API, and keep it out of shell history — it is a credential, however
|
||||||
|
briefly.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd /opt/forgejo-runner
|
||||||
|
mkdir -p data
|
||||||
|
cp config.yaml data/config.yaml
|
||||||
|
|
||||||
|
read -rs REG_TOKEN && export REG_TOKEN # paste; not echoed, not in history
|
||||||
|
|
||||||
|
docker run --rm -it \
|
||||||
|
-v "$PWD/data:/data" -w /data --user root \
|
||||||
|
code.forgejo.org/forgejo/runner:6.3.1 \
|
||||||
|
forgejo-runner register --no-interactive \
|
||||||
|
--instance https://git.thermograph.org \
|
||||||
|
--token "$REG_TOKEN" \
|
||||||
|
--name thermograph-vps2 \
|
||||||
|
--labels 'docker:docker://git.thermograph.org/jinemi/thermograph/ci-runner:v2'
|
||||||
|
|
||||||
|
unset REG_TOKEN
|
||||||
|
```
|
||||||
|
|
||||||
|
That writes `data/.runner`, which holds the **long-lived** runner token. It is
|
||||||
|
`0600` and must stay out of git — `data/` is gitignored for exactly this reason.
|
||||||
|
Re-running `register` with a fresh registration token replaces it; the old
|
||||||
|
registration then shows offline in the UI and should be deleted there.
|
||||||
|
|
||||||
|
## Run it
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker compose up -d
|
||||||
|
docker compose logs -f --tail=50 runner # expect "runner: ... successfully activated"
|
||||||
|
```
|
||||||
|
|
||||||
|
Confirm Forgejo agrees, rather than trusting the log:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# on vps1, where Forgejo's database lives
|
||||||
|
fdb=$(docker ps -qf name=forgejo_db | head -1)
|
||||||
|
docker exec "$fdb" sh -c 'psql -U $POSTGRES_USER -d $POSTGRES_DB -c \
|
||||||
|
"select name, to_timestamp(last_online) as last_online, agent_labels from action_runner order by id;"'
|
||||||
|
```
|
||||||
|
|
||||||
|
Two rows, both recent. `last_online` is the only honest liveness signal — a
|
||||||
|
runner process that is up but cannot reach Forgejo looks healthy locally and is
|
||||||
|
useless.
|
||||||
|
|
||||||
|
## Upgrading
|
||||||
|
|
||||||
|
Pin changes go in `docker-compose.yml` (`image:`) and are applied with
|
||||||
|
`docker compose up -d`. The runner token survives an image change; it lives in
|
||||||
|
`data/.runner`, not in the image.
|
||||||
|
|
||||||
|
Changing the **job** image (the `docker://…ci-runner:vN` label) is a
|
||||||
|
re-registration or a hand-edit of `data/.runner`'s `labels` array — the same
|
||||||
|
caveat `../ci-runner/Dockerfile` records for the desktop runner. Keep the two
|
||||||
|
runners on the same job image or a workflow's behaviour starts depending on
|
||||||
|
which runner happened to pick it up.
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
**Jobs queue forever with both runners online.** The workflow asked for a label
|
||||||
|
nobody registered. `runs-on: docker` is the only label this estate uses.
|
||||||
|
|
||||||
|
**`docker pull` fails inside a job, works on the host.** The `--add-host` line
|
||||||
|
in `config.yaml` is missing or wrong. vps1's Caddy serves `/v2/*` to the mesh
|
||||||
|
only, and the job container has no other way to learn the mesh address.
|
||||||
|
|
||||||
|
**Jobs fail with permission denied on the socket.** `user: root` was dropped
|
||||||
|
from the compose file.
|
||||||
|
|
||||||
|
**The runner is up but `last_online` is stale.** It cannot reach
|
||||||
|
`https://git.thermograph.org` — check egress from vps2 and that Forgejo on vps1
|
||||||
|
is serving.
|
||||||
53
infra/deploy/forgejo/runner-vps2/config.yaml
Normal file
53
infra/deploy/forgejo/runner-vps2/config.yaml
Normal file
|
|
@ -0,0 +1,53 @@
|
||||||
|
# Forgejo Actions runner config for the vps2 runner. Mounted at /data/config.yaml
|
||||||
|
# by runner-vps2/docker-compose.yml.
|
||||||
|
#
|
||||||
|
# Values that differ from `forgejo-runner generate-config` defaults are the only
|
||||||
|
# ones written out below; everything else is left at its default deliberately, so
|
||||||
|
# a future runner upgrade inherits new defaults instead of a frozen copy of the
|
||||||
|
# old ones.
|
||||||
|
|
||||||
|
log:
|
||||||
|
level: info
|
||||||
|
job_level: info
|
||||||
|
|
||||||
|
runner:
|
||||||
|
file: .runner
|
||||||
|
|
||||||
|
# ONE job at a time. The desktop runner uses capacity 8, which is right for a
|
||||||
|
# box whose whole job is CI. This runner shares a host with prod, so the
|
||||||
|
# binding constraint is not throughput, it is that a CI burst must never be
|
||||||
|
# able to contend with thermograph_web for CPU. One job at a time also makes
|
||||||
|
# the resource ceiling below exact rather than a per-job estimate multiplied
|
||||||
|
# by an unknown.
|
||||||
|
capacity: 1
|
||||||
|
|
||||||
|
timeout: 3h
|
||||||
|
shutdown_timeout: 3h
|
||||||
|
|
||||||
|
container:
|
||||||
|
# 1) --add-host: git.thermograph.org resolves publicly to vps1, but vps1's
|
||||||
|
# Caddy rejects the /v2/* registry API from off-mesh. Job containers that
|
||||||
|
# docker-pull or docker-push therefore need the MESH address, and DNS will
|
||||||
|
# not give it to them. Same line the desktop runner carries; the value is
|
||||||
|
# the same from vps2 because both boxes are on 10.10.0.0/24.
|
||||||
|
#
|
||||||
|
# 2) --cpus/--memory: vps2 runs prod. Without a ceiling a runaway job (a
|
||||||
|
# `docker build` that fans out, a test that leaks) competes with the
|
||||||
|
# production app for the same 6 cores. These bound the JOB container, which
|
||||||
|
# is a sibling of the runner rather than its child, so the runner's own
|
||||||
|
# compose limits do not cover it -- this is the only lever that does.
|
||||||
|
options: --add-host=git.thermograph.org:10.10.0.2 --cpus=2 --memory=4g
|
||||||
|
|
||||||
|
# Jobs may mount NOTHING from the host. The workflows here do not need to:
|
||||||
|
# they check out into the job container's own workspace and reach the estate
|
||||||
|
# over SSH. An empty list is the difference between "a job can read
|
||||||
|
# /etc/thermograph.env" and "a job cannot", on the box where that file is
|
||||||
|
# prod's.
|
||||||
|
valid_volumes: []
|
||||||
|
|
||||||
|
privileged: false
|
||||||
|
|
||||||
|
# Mounts the host docker socket into the job container. build-push.yml runs
|
||||||
|
# `docker build`, so it needs a daemon; this points it at vps2's. See the
|
||||||
|
# README for what that does and does not mean for trust.
|
||||||
|
docker_host: "automount"
|
||||||
83
infra/deploy/forgejo/runner-vps2/docker-compose.yml
Normal file
83
infra/deploy/forgejo/runner-vps2/docker-compose.yml
Normal file
|
|
@ -0,0 +1,83 @@
|
||||||
|
# The always-on Forgejo Actions runner, on vps2.
|
||||||
|
#
|
||||||
|
# cd /opt/forgejo-runner # a COPY of this file, not the checkout
|
||||||
|
# docker compose up -d
|
||||||
|
#
|
||||||
|
# It runs from /opt/forgejo-runner rather than in place, because the checkout is
|
||||||
|
# git reset --hard'd on every prod deploy and one `git clean -fdx` there would
|
||||||
|
# delete data/.runner. See README.md.
|
||||||
|
#
|
||||||
|
# Registration is a one-off and is NOT in this file, because it takes a
|
||||||
|
# short-lived token a human fetches. See README.md.
|
||||||
|
#
|
||||||
|
# ============================================================================
|
||||||
|
# WHY A SECOND RUNNER EXISTS AT ALL
|
||||||
|
# ============================================================================
|
||||||
|
# Until 2026-08-01 the estate had exactly one registered runner, on the desktop.
|
||||||
|
# It went offline at 2026-07-31 16:31Z and everything stopped for 21 hours: no
|
||||||
|
# merge could satisfy a required check, no deploy could run, and the 03:00Z
|
||||||
|
# ops-cron -- THE backup for both application databases and for Forgejo -- did
|
||||||
|
# not fire. Forgejo queued that run rather than dropping it, so it completed on
|
||||||
|
# reconnect, but a longer outage would have meant real backup gaps.
|
||||||
|
#
|
||||||
|
# So this runner is not extra capacity. It is the one that has to be up.
|
||||||
|
#
|
||||||
|
# ============================================================================
|
||||||
|
# WHY A PLAIN CONTAINER AND NOT A SWARM SERVICE
|
||||||
|
# ============================================================================
|
||||||
|
# Same argument as infra/openbao/config/openbao.hcl makes for OpenBao: the thing
|
||||||
|
# that repairs the estate must not depend on the estate. A Swarm-scheduled
|
||||||
|
# runner cannot redeploy the Swarm it is scheduled by, and if the cluster is the
|
||||||
|
# thing that is broken, CI is gone exactly when it is needed. `restart: always`
|
||||||
|
# on the local daemon has one dependency: the local daemon.
|
||||||
|
#
|
||||||
|
# An earlier revision of deploy/forgejo/docker-stack.yml did run a runner as a
|
||||||
|
# Swarm-scheduled Docker-in-Docker sidecar. It was removed, and the docs kept
|
||||||
|
# claiming an "always-on Swarm-hosted runner" for weeks afterwards -- which is
|
||||||
|
# how the single point of failure went unnoticed.
|
||||||
|
#
|
||||||
|
# ============================================================================
|
||||||
|
# WHAT MOUNTING THE DOCKER SOCKET ON THE PROD HOST MEANS
|
||||||
|
# ============================================================================
|
||||||
|
# Be plain about it: /var/run/docker.sock is root-equivalent on vps2, and vps2
|
||||||
|
# runs prod. A workflow that can start a container can mount / and read
|
||||||
|
# /etc/thermograph.env. This runner does not create that exposure from nothing
|
||||||
|
# -- Forgejo's own database already stores VPS2_SSH_KEY, which is root on this
|
||||||
|
# box -- but it does move the exposure from "a secret at rest" to "a secret a
|
||||||
|
# job can reach", and that is a real difference.
|
||||||
|
#
|
||||||
|
# What bounds it, in order of how much each is worth:
|
||||||
|
# 1. Only this repo's workflows run here, and only the owner can push to it.
|
||||||
|
# 2. config.yaml sets valid_volumes: [] -- jobs cannot bind-mount host paths.
|
||||||
|
# 3. capacity: 1 and --cpus/--memory keep a job from contending with prod.
|
||||||
|
# 4. This compose project joins no thermograph network, so a job container
|
||||||
|
# reaches prod's services no more easily than any other host process.
|
||||||
|
#
|
||||||
|
# It is defence against accident, not against a hostile workflow author. On a
|
||||||
|
# two-person estate where both people can already SSH to this box as root, that
|
||||||
|
# is the honest boundary.
|
||||||
|
|
||||||
|
name: forgejo-runner
|
||||||
|
|
||||||
|
services:
|
||||||
|
runner:
|
||||||
|
image: code.forgejo.org/forgejo/runner:6.3.1
|
||||||
|
restart: always
|
||||||
|
# The socket is root:docker on the host; the image's default user is not in
|
||||||
|
# that group. Running as root is what the socket mount requires.
|
||||||
|
user: root
|
||||||
|
working_dir: /data
|
||||||
|
command: forgejo-runner daemon --config /data/config.yaml
|
||||||
|
volumes:
|
||||||
|
- ./data:/data
|
||||||
|
- /var/run/docker.sock:/var/run/docker.sock
|
||||||
|
# Bounds the DAEMON. Job containers are siblings started through the socket,
|
||||||
|
# not children, so they are unaffected by these -- container.options in
|
||||||
|
# config.yaml is what bounds those.
|
||||||
|
cpus: 1.0
|
||||||
|
mem_limit: 1g
|
||||||
|
logging:
|
||||||
|
driver: json-file
|
||||||
|
options:
|
||||||
|
max-size: "10m"
|
||||||
|
max-file: "3"
|
||||||
|
|
@ -26,7 +26,7 @@ SELF_DIR=$(cd "$(dirname "$0")" && pwd)
|
||||||
. "$SELF_DIR/env-topology.sh"
|
. "$SELF_DIR/env-topology.sh"
|
||||||
thermograph_topology dev
|
thermograph_topology dev
|
||||||
|
|
||||||
REPO_URL="${REPO_URL:-http://10.10.0.2:3080/emi/thermograph.git}"
|
REPO_URL="${REPO_URL:-http://10.10.0.2:3080/Jinemi/thermograph.git}"
|
||||||
APP_DIR="${APP_DIR:-$TG_APP_DIR}"
|
APP_DIR="${APP_DIR:-$TG_APP_DIR}"
|
||||||
BRANCH="${BRANCH:-$TG_BRANCH}"
|
BRANCH="${BRANCH:-$TG_BRANCH}"
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -124,8 +124,8 @@ case "$SERVICE" in
|
||||||
esac
|
esac
|
||||||
export BACKEND_IMAGE_TAG="${BACKEND_IMAGE_TAG:-local}"
|
export BACKEND_IMAGE_TAG="${BACKEND_IMAGE_TAG:-local}"
|
||||||
export FRONTEND_IMAGE_TAG="${FRONTEND_IMAGE_TAG:-local}"
|
export FRONTEND_IMAGE_TAG="${FRONTEND_IMAGE_TAG:-local}"
|
||||||
BACKEND_IMAGE="$REGISTRY_HOST/${BACKEND_IMAGE_PATH:-emi/thermograph/backend}:$BACKEND_IMAGE_TAG"
|
BACKEND_IMAGE="$REGISTRY_HOST/${BACKEND_IMAGE_PATH:-jinemi/thermograph/backend}:$BACKEND_IMAGE_TAG"
|
||||||
FRONTEND_IMAGE="$REGISTRY_HOST/${FRONTEND_IMAGE_PATH:-emi/thermograph/frontend}:$FRONTEND_IMAGE_TAG"
|
FRONTEND_IMAGE="$REGISTRY_HOST/${FRONTEND_IMAGE_PATH:-jinemi/thermograph/frontend}:$FRONTEND_IMAGE_TAG"
|
||||||
|
|
||||||
# --- timescale image pin ---------------------------------------------------------
|
# --- timescale image pin ---------------------------------------------------------
|
||||||
# Hazard #7: the db image under an existing volume must never drift. Resolve
|
# Hazard #7: the db image under an existing volume must never drift. Resolve
|
||||||
|
|
@ -158,7 +158,7 @@ fi
|
||||||
|
|
||||||
# --- registry ------------------------------------------------------------------
|
# --- registry ------------------------------------------------------------------
|
||||||
if [ -n "${REGISTRY_TOKEN:-}" ]; then
|
if [ -n "${REGISTRY_TOKEN:-}" ]; then
|
||||||
echo "$REGISTRY_TOKEN" | docker login "$REGISTRY_HOST" --username emi --password-stdin
|
echo "$REGISTRY_TOKEN" | docker login "$REGISTRY_HOST" --username admin_emi --password-stdin
|
||||||
fi
|
fi
|
||||||
echo "==> Pulling images"
|
echo "==> Pulling images"
|
||||||
pull_ok=0
|
pull_ok=0
|
||||||
|
|
@ -290,8 +290,8 @@ FRONTEND_IMAGE_TAG=$FRONTEND_IMAGE_TAG
|
||||||
EOF
|
EOF
|
||||||
|
|
||||||
# GC superseded app-image tags (keep the running pair), same as deploy.sh.
|
# GC superseded app-image tags (keep the running pair), same as deploy.sh.
|
||||||
_be_repo="$REGISTRY_HOST/${BACKEND_IMAGE_PATH:-emi/thermograph/backend}"
|
_be_repo="$REGISTRY_HOST/${BACKEND_IMAGE_PATH:-jinemi/thermograph/backend}"
|
||||||
_fe_repo="$REGISTRY_HOST/${FRONTEND_IMAGE_PATH:-emi/thermograph/frontend}"
|
_fe_repo="$REGISTRY_HOST/${FRONTEND_IMAGE_PATH:-jinemi/thermograph/frontend}"
|
||||||
docker images --format '{{.Repository}}:{{.Tag}}' \
|
docker images --format '{{.Repository}}:{{.Tag}}' \
|
||||||
| grep -E "^(${_be_repo}|${_fe_repo}):" \
|
| grep -E "^(${_be_repo}|${_fe_repo}):" \
|
||||||
| grep -v -e "^${_be_repo}:${BACKEND_IMAGE_TAG}$" -e "^${_fe_repo}:${FRONTEND_IMAGE_TAG}$" \
|
| grep -v -e "^${_be_repo}:${BACKEND_IMAGE_TAG}$" -e "^${_fe_repo}:${FRONTEND_IMAGE_TAG}$" \
|
||||||
|
|
|
||||||
|
|
@ -13,6 +13,34 @@
|
||||||
{
|
{
|
||||||
auto_https off
|
auto_https off
|
||||||
admin off
|
admin off
|
||||||
|
|
||||||
|
# WITHOUT THIS, EVERY OAUTH REDIRECT URI THE BACKEND BUILDS IS http://.
|
||||||
|
#
|
||||||
|
# There are two Caddy hops in front of the app: the HOST Caddy terminates TLS
|
||||||
|
# and sets X-Forwarded-Proto: https, then proxies to 127.0.0.1:8137 over plain
|
||||||
|
# HTTP; this LB is the second hop. Caddy only preserves an incoming
|
||||||
|
# X-Forwarded-* header when the immediate peer is a TRUSTED proxy — otherwise
|
||||||
|
# it overwrites the header with the scheme of the connection it just received,
|
||||||
|
# which here is http. So the app saw http:// no matter how the user arrived.
|
||||||
|
#
|
||||||
|
# accounts/oauth.py:_redirect_uri builds the callback from x-forwarded-proto,
|
||||||
|
# and that URI must match what is registered in the provider's console.
|
||||||
|
# Discord tolerates the http:// form; GOOGLE REJECTS a non-HTTPS redirect URI
|
||||||
|
# for a web client outright, so Google sign-in could not work at all until
|
||||||
|
# this was fixed — independently of whether the credentials were configured.
|
||||||
|
#
|
||||||
|
# Measured on vps2 before the fix, same request to each hop:
|
||||||
|
# direct to web, XFP=https -> https://thermograph.org/api/v2/...
|
||||||
|
# through this LB, XFP=https -> http://thermograph.org/api/v2/...
|
||||||
|
#
|
||||||
|
# private_ranges covers 127.0.0.1/8 and the Docker bridge ranges, which is
|
||||||
|
# where the host Caddy reaches this container from. Nothing outside the box
|
||||||
|
# can connect here — the LB is bound to loopback precisely so it cannot be
|
||||||
|
# reached un-fronted (hazard #6, above) — so trusting the local peer does not
|
||||||
|
# widen anything: the only party that can set these headers is the host Caddy.
|
||||||
|
servers {
|
||||||
|
trusted_proxies static private_ranges
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
:8137 {
|
:8137 {
|
||||||
|
|
|
||||||
|
|
@ -17,6 +17,17 @@
|
||||||
{
|
{
|
||||||
auto_https off
|
auto_https off
|
||||||
admin off
|
admin off
|
||||||
|
|
||||||
|
# Same reason as ./Caddyfile — see the long note there. Without it this second
|
||||||
|
# Caddy hop overwrites the host Caddy's X-Forwarded-Proto: https with http,
|
||||||
|
# and every OAuth redirect URI the backend builds comes out non-HTTPS, which
|
||||||
|
# Google rejects outright for a web client.
|
||||||
|
#
|
||||||
|
# Beta needs it as much as prod does, and arguably first: beta is where an
|
||||||
|
# OAuth provider change gets tested before it reaches thermograph.org.
|
||||||
|
servers {
|
||||||
|
trusted_proxies static private_ranges
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
:8137 {
|
:8137 {
|
||||||
|
|
|
||||||
|
|
@ -62,7 +62,7 @@
|
||||||
|
|
||||||
services:
|
services:
|
||||||
beta-web:
|
beta-web:
|
||||||
image: ${REGISTRY_HOST:-git.thermograph.org}/${BACKEND_IMAGE_PATH:-emi/thermograph/backend}:${BACKEND_IMAGE_TAG:?required}
|
image: ${REGISTRY_HOST:-git.thermograph.org}/${BACKEND_IMAGE_PATH:-jinemi/thermograph/backend}:${BACKEND_IMAGE_TAG:?required}
|
||||||
entrypoint: ["/host/env-entrypoint.sh"]
|
entrypoint: ["/host/env-entrypoint.sh"]
|
||||||
environment:
|
environment:
|
||||||
# Beta's OWN role and OWN database on the shared instance. `db` resolves
|
# Beta's OWN role and OWN database on the shared instance. `db` resolves
|
||||||
|
|
@ -108,7 +108,7 @@ services:
|
||||||
failure_action: rollback
|
failure_action: rollback
|
||||||
|
|
||||||
beta-worker:
|
beta-worker:
|
||||||
image: ${REGISTRY_HOST:-git.thermograph.org}/${BACKEND_IMAGE_PATH:-emi/thermograph/backend}:${BACKEND_IMAGE_TAG:?required}
|
image: ${REGISTRY_HOST:-git.thermograph.org}/${BACKEND_IMAGE_PATH:-jinemi/thermograph/backend}:${BACKEND_IMAGE_TAG:?required}
|
||||||
entrypoint: ["/host/env-entrypoint.sh"]
|
entrypoint: ["/host/env-entrypoint.sh"]
|
||||||
environment:
|
environment:
|
||||||
THERMOGRAPH_DATABASE_URL: postgresql+asyncpg://thermograph_beta:${POSTGRES_PASSWORD}@db:5432/thermograph_beta
|
THERMOGRAPH_DATABASE_URL: postgresql+asyncpg://thermograph_beta:${POSTGRES_PASSWORD}@db:5432/thermograph_beta
|
||||||
|
|
@ -144,7 +144,7 @@ services:
|
||||||
condition: on-failure
|
condition: on-failure
|
||||||
|
|
||||||
beta-lake:
|
beta-lake:
|
||||||
image: ${REGISTRY_HOST:-git.thermograph.org}/${BACKEND_IMAGE_PATH:-emi/thermograph/backend}:${BACKEND_IMAGE_TAG:?required}
|
image: ${REGISTRY_HOST:-git.thermograph.org}/${BACKEND_IMAGE_PATH:-jinemi/thermograph/backend}:${BACKEND_IMAGE_TAG:?required}
|
||||||
entrypoint: ["/host/env-entrypoint.sh"]
|
entrypoint: ["/host/env-entrypoint.sh"]
|
||||||
environment:
|
environment:
|
||||||
THERMOGRAPH_ROLE: lake
|
THERMOGRAPH_ROLE: lake
|
||||||
|
|
@ -176,7 +176,7 @@ services:
|
||||||
failure_action: rollback
|
failure_action: rollback
|
||||||
|
|
||||||
beta-daemon:
|
beta-daemon:
|
||||||
image: ${REGISTRY_HOST:-git.thermograph.org}/${BACKEND_IMAGE_PATH:-emi/thermograph/backend}:${BACKEND_IMAGE_TAG:?required}
|
image: ${REGISTRY_HOST:-git.thermograph.org}/${BACKEND_IMAGE_PATH:-jinemi/thermograph/backend}:${BACKEND_IMAGE_TAG:?required}
|
||||||
# Same reasoning as prod's daemon: NOT env-entrypoint.sh, because that shim
|
# Same reasoning as prod's daemon: NOT env-entrypoint.sh, because that shim
|
||||||
# execs the image's own entrypoint (Alembic + uvicorn) and migrations belong
|
# execs the image's own entrypoint (Alembic + uvicorn) and migrations belong
|
||||||
# to the one-shot task. Source the host-rendered env and exec the binary.
|
# to the one-shot task. Source the host-rendered env and exec the binary.
|
||||||
|
|
@ -218,7 +218,7 @@ services:
|
||||||
failure_action: rollback
|
failure_action: rollback
|
||||||
|
|
||||||
beta-frontend:
|
beta-frontend:
|
||||||
image: ${REGISTRY_HOST:-git.thermograph.org}/${FRONTEND_IMAGE_PATH:-emi/thermograph/frontend}:${FRONTEND_IMAGE_TAG:?required}
|
image: ${REGISTRY_HOST:-git.thermograph.org}/${FRONTEND_IMAGE_PATH:-jinemi/thermograph/frontend}:${FRONTEND_IMAGE_TAG:?required}
|
||||||
entrypoint: ["/host/env-entrypoint.sh"]
|
entrypoint: ["/host/env-entrypoint.sh"]
|
||||||
# REQUIRED: overriding `entrypoint:` with no `command:` drops the image's
|
# REQUIRED: overriding `entrypoint:` with no `command:` drops the image's
|
||||||
# CMD entirely, and env-entrypoint.sh's fallback (`exec uvicorn app:app`)
|
# CMD entirely, and env-entrypoint.sh's fallback (`exec uvicorn app:app`)
|
||||||
|
|
|
||||||
|
|
@ -64,7 +64,7 @@ services:
|
||||||
condition: on-failure
|
condition: on-failure
|
||||||
|
|
||||||
web:
|
web:
|
||||||
image: ${REGISTRY_HOST:-git.thermograph.org}/${BACKEND_IMAGE_PATH:-emi/thermograph/backend}:${BACKEND_IMAGE_TAG:?required}
|
image: ${REGISTRY_HOST:-git.thermograph.org}/${BACKEND_IMAGE_PATH:-jinemi/thermograph/backend}:${BACKEND_IMAGE_TAG:?required}
|
||||||
entrypoint: ["/host/env-entrypoint.sh"]
|
entrypoint: ["/host/env-entrypoint.sh"]
|
||||||
environment:
|
environment:
|
||||||
THERMOGRAPH_DATABASE_URL: postgresql+asyncpg://thermograph:${POSTGRES_PASSWORD}@db:5432/thermograph
|
THERMOGRAPH_DATABASE_URL: postgresql+asyncpg://thermograph:${POSTGRES_PASSWORD}@db:5432/thermograph
|
||||||
|
|
@ -109,7 +109,7 @@ services:
|
||||||
failure_action: rollback
|
failure_action: rollback
|
||||||
|
|
||||||
worker:
|
worker:
|
||||||
image: ${REGISTRY_HOST:-git.thermograph.org}/${BACKEND_IMAGE_PATH:-emi/thermograph/backend}:${BACKEND_IMAGE_TAG:?required}
|
image: ${REGISTRY_HOST:-git.thermograph.org}/${BACKEND_IMAGE_PATH:-jinemi/thermograph/backend}:${BACKEND_IMAGE_TAG:?required}
|
||||||
entrypoint: ["/host/env-entrypoint.sh"]
|
entrypoint: ["/host/env-entrypoint.sh"]
|
||||||
environment:
|
environment:
|
||||||
THERMOGRAPH_DATABASE_URL: postgresql+asyncpg://thermograph:${POSTGRES_PASSWORD}@db:5432/thermograph
|
THERMOGRAPH_DATABASE_URL: postgresql+asyncpg://thermograph:${POSTGRES_PASSWORD}@db:5432/thermograph
|
||||||
|
|
@ -153,7 +153,7 @@ services:
|
||||||
# healthy and answers 503/404, and web falls through to NASA — the lake is
|
# healthy and answers 503/404, and web falls through to NASA — the lake is
|
||||||
# an accelerator, never a point of failure.
|
# an accelerator, never a point of failure.
|
||||||
lake:
|
lake:
|
||||||
image: ${REGISTRY_HOST:-git.thermograph.org}/${BACKEND_IMAGE_PATH:-emi/thermograph/backend}:${BACKEND_IMAGE_TAG:?required}
|
image: ${REGISTRY_HOST:-git.thermograph.org}/${BACKEND_IMAGE_PATH:-jinemi/thermograph/backend}:${BACKEND_IMAGE_TAG:?required}
|
||||||
entrypoint: ["/host/env-entrypoint.sh"]
|
entrypoint: ["/host/env-entrypoint.sh"]
|
||||||
environment:
|
environment:
|
||||||
THERMOGRAPH_ROLE: lake
|
THERMOGRAPH_ROLE: lake
|
||||||
|
|
@ -186,7 +186,7 @@ services:
|
||||||
# (/usr/local/bin/thermograph-daemon, built into the backend image) and the
|
# (/usr/local/bin/thermograph-daemon, built into the backend image) and the
|
||||||
# app share the /internal/* API contract, so one BACKEND_IMAGE_TAG rolls
|
# app share the /internal/* API contract, so one BACKEND_IMAGE_TAG rolls
|
||||||
# them together and they can never skew.
|
# them together and they can never skew.
|
||||||
image: ${REGISTRY_HOST:-git.thermograph.org}/${BACKEND_IMAGE_PATH:-emi/thermograph/backend}:${BACKEND_IMAGE_TAG:?required}
|
image: ${REGISTRY_HOST:-git.thermograph.org}/${BACKEND_IMAGE_PATH:-jinemi/thermograph/backend}:${BACKEND_IMAGE_TAG:?required}
|
||||||
# Not env-entrypoint.sh: that shim's handoff execs the image's
|
# Not env-entrypoint.sh: that shim's handoff execs the image's
|
||||||
# deploy/entrypoint.sh (Alembic + uvicorn) whenever it exists, which is
|
# deploy/entrypoint.sh (Alembic + uvicorn) whenever it exists, which is
|
||||||
# exactly what this service must never run — migrations belong to the
|
# exactly what this service must never run — migrations belong to the
|
||||||
|
|
@ -243,7 +243,7 @@ services:
|
||||||
failure_action: rollback
|
failure_action: rollback
|
||||||
|
|
||||||
frontend:
|
frontend:
|
||||||
image: ${REGISTRY_HOST:-git.thermograph.org}/${FRONTEND_IMAGE_PATH:-emi/thermograph/frontend}:${FRONTEND_IMAGE_TAG:?required}
|
image: ${REGISTRY_HOST:-git.thermograph.org}/${FRONTEND_IMAGE_PATH:-jinemi/thermograph/frontend}:${FRONTEND_IMAGE_TAG:?required}
|
||||||
entrypoint: ["/host/env-entrypoint.sh"]
|
entrypoint: ["/host/env-entrypoint.sh"]
|
||||||
# REQUIRED, not cosmetic: overriding `entrypoint:` with no `command:` drops
|
# REQUIRED, not cosmetic: overriding `entrypoint:` with no `command:` drops
|
||||||
# the image's own CMD entirely (Docker/Swarm semantics, not merged) --
|
# the image's own CMD entirely (Docker/Swarm semantics, not merged) --
|
||||||
|
|
|
||||||
|
|
@ -14,7 +14,7 @@ submission ($25 one-time) is deferred.
|
||||||
|
|
||||||
## What's in the repo
|
## What's in the repo
|
||||||
- `twa-manifest.json` — Bubblewrap config (reference values; see comment inside).
|
- `twa-manifest.json` — Bubblewrap config (reference values; see comment inside).
|
||||||
- `../../frontend/.well-known/assetlinks.json` — Digital Asset Links, served at
|
- `../../../frontend/static/.well-known/assetlinks.json` — Digital Asset Links, served at
|
||||||
`https://thermograph.org/.well-known/assetlinks.json`. Its fingerprint is a
|
`https://thermograph.org/.well-known/assetlinks.json`. Its fingerprint is a
|
||||||
**placeholder** until you generate the signing key (step 4).
|
**placeholder** until you generate the signing key (step 4).
|
||||||
|
|
||||||
|
|
@ -45,7 +45,7 @@ bubblewrap build
|
||||||
bubblewrap fingerprint list # or: keytool -list -v -keystore android-keystore.jks
|
bubblewrap fingerprint list # or: keytool -list -v -keystore android-keystore.jks
|
||||||
```
|
```
|
||||||
|
|
||||||
Copy the `SHA256` value into `frontend/.well-known/assetlinks.json`, replacing
|
Copy the `SHA256` value into `frontend/static/.well-known/assetlinks.json`, replacing
|
||||||
`REPLACE_WITH_TWA_SIGNING_KEY_SHA256_FINGERPRINT`, then **deploy the site** so the
|
`REPLACE_WITH_TWA_SIGNING_KEY_SHA256_FINGERPRINT`, then **deploy the site** so the
|
||||||
new file is live. Confirm:
|
new file is live. Confirm:
|
||||||
|
|
||||||
|
|
@ -80,6 +80,7 @@ install for every user. Store it out of the repo, backed up.
|
||||||
`.aab`; the store review for a well-formed TWA is light. Not part of this pass.
|
`.aab`; the store review for a well-formed TWA is light. Not part of this pass.
|
||||||
|
|
||||||
## Why no backend changes
|
## Why no backend changes
|
||||||
Android delegates the TWA's web notifications to the OS, so `backend/push.py`
|
Android delegates the TWA's web notifications to the OS, so
|
||||||
(VAPID) and `backend/notify.py` deliver push to the app unchanged. Nothing in
|
`backend/notifications/push.py` (VAPID) and `backend/notifications/notify.py`
|
||||||
|
deliver push to the app unchanged. Nothing in
|
||||||
`backend/` is touched by this track.
|
`backend/` is touched by this track.
|
||||||
|
|
|
||||||
|
|
@ -6,7 +6,7 @@
|
||||||
# build-push.yml -- backend = ${BACKEND_IMAGE_PATH}, frontend =
|
# build-push.yml -- backend = ${BACKEND_IMAGE_PATH}, frontend =
|
||||||
# ${FRONTEND_IMAGE_PATH}, tagged independently by BACKEND_IMAGE_TAG /
|
# ${FRONTEND_IMAGE_PATH}, tagged independently by BACKEND_IMAGE_TAG /
|
||||||
# FRONTEND_IMAGE_TAG. This replaces the earlier Stage-4 model where both
|
# FRONTEND_IMAGE_TAG. This replaces the earlier Stage-4 model where both
|
||||||
# containers shared the single emi/thermograph/app image and
|
# containers shared the single admin_emi/thermograph/app image and
|
||||||
# THERMOGRAPH_SERVICE_ROLE picked the process; the split Dockerfiles now start
|
# THERMOGRAPH_SERVICE_ROLE picked the process; the split Dockerfiles now start
|
||||||
# the right process directly (frontend the thermograph-frontend Go binary,
|
# the right process directly (frontend the thermograph-frontend Go binary,
|
||||||
# backend entrypoint.sh), so a backend deploy and a frontend deploy are fully
|
# backend entrypoint.sh), so a backend deploy and a frontend deploy are fully
|
||||||
|
|
@ -99,10 +99,10 @@ services:
|
||||||
# No `build:` here -- infra holds no Dockerfile; each service's Dockerfile
|
# No `build:` here -- infra holds no Dockerfile; each service's Dockerfile
|
||||||
# lives in its own app repo. deploy.sh sets BACKEND_IMAGE_TAG to the SHA
|
# lives in its own app repo. deploy.sh sets BACKEND_IMAGE_TAG to the SHA
|
||||||
# build-push.yml pushed for the backend commit being deployed; local dev
|
# build-push.yml pushed for the backend commit being deployed; local dev
|
||||||
# builds `emi/thermograph/backend:local` from the backend repo (see
|
# builds `jinemi/thermograph/backend:local` from the backend repo (see
|
||||||
# infra Makefile) and this pulls/uses it. BACKEND_IMAGE_PATH/TAG are
|
# infra Makefile) and this pulls/uses it. BACKEND_IMAGE_PATH/TAG are
|
||||||
# independent of the frontend's, so the two services deploy separately.
|
# independent of the frontend's, so the two services deploy separately.
|
||||||
image: ${REGISTRY_HOST:-git.thermograph.org}/${BACKEND_IMAGE_PATH:-emi/thermograph/backend}:${BACKEND_IMAGE_TAG:-local}
|
image: ${REGISTRY_HOST:-git.thermograph.org}/${BACKEND_IMAGE_PATH:-jinemi/thermograph/backend}:${BACKEND_IMAGE_TAG:-local}
|
||||||
depends_on:
|
depends_on:
|
||||||
db:
|
db:
|
||||||
condition: service_healthy
|
condition: service_healthy
|
||||||
|
|
@ -180,7 +180,7 @@ services:
|
||||||
# without them the service stays healthy and every read falls through to
|
# without them the service stays healthy and every read falls through to
|
||||||
# NASA. Never published on a host port: only the backend talks to it.
|
# NASA. Never published on a host port: only the backend talks to it.
|
||||||
lake:
|
lake:
|
||||||
image: ${REGISTRY_HOST:-git.thermograph.org}/${BACKEND_IMAGE_PATH:-emi/thermograph/backend}:${BACKEND_IMAGE_TAG:-local}
|
image: ${REGISTRY_HOST:-git.thermograph.org}/${BACKEND_IMAGE_PATH:-jinemi/thermograph/backend}:${BACKEND_IMAGE_TAG:-local}
|
||||||
environment:
|
environment:
|
||||||
THERMOGRAPH_ROLE: lake
|
THERMOGRAPH_ROLE: lake
|
||||||
PORT: 8141
|
PORT: 8141
|
||||||
|
|
@ -207,7 +207,7 @@ services:
|
||||||
# share the /internal/* API contract, so they must roll together — a
|
# share the /internal/* API contract, so they must roll together — a
|
||||||
# separate tag could skew them. It owns the Discord gateway websocket and
|
# separate tag could skew them. It owns the Discord gateway websocket and
|
||||||
# the recurring-job timers; anything needing data calls back into backend.
|
# the recurring-job timers; anything needing data calls back into backend.
|
||||||
image: ${REGISTRY_HOST:-git.thermograph.org}/${BACKEND_IMAGE_PATH:-emi/thermograph/backend}:${BACKEND_IMAGE_TAG:-local}
|
image: ${REGISTRY_HOST:-git.thermograph.org}/${BACKEND_IMAGE_PATH:-jinemi/thermograph/backend}:${BACKEND_IMAGE_TAG:-local}
|
||||||
# Bypass deploy/entrypoint.sh entirely: that script runs Alembic, and the
|
# Bypass deploy/entrypoint.sh entirely: that script runs Alembic, and the
|
||||||
# backend service's boot already owns migrations — two containers racing
|
# backend service's boot already owns migrations — two containers racing
|
||||||
# `alembic upgrade head` against one database is a real hazard, not a
|
# `alembic upgrade head` against one database is a real hazard, not a
|
||||||
|
|
@ -262,7 +262,7 @@ services:
|
||||||
# frontend/Dockerfile (which starts the thermograph-frontend Go binary
|
# frontend/Dockerfile (which starts the thermograph-frontend Go binary
|
||||||
# directly -- no THERMOGRAPH_SERVICE_ROLE process-picking). Independent
|
# directly -- no THERMOGRAPH_SERVICE_ROLE process-picking). Independent
|
||||||
# FRONTEND_IMAGE_TAG so a frontend deploy never disturbs the backend's tag.
|
# FRONTEND_IMAGE_TAG so a frontend deploy never disturbs the backend's tag.
|
||||||
image: ${REGISTRY_HOST:-git.thermograph.org}/${FRONTEND_IMAGE_PATH:-emi/thermograph/frontend}:${FRONTEND_IMAGE_TAG:-local}
|
image: ${REGISTRY_HOST:-git.thermograph.org}/${FRONTEND_IMAGE_PATH:-jinemi/thermograph/frontend}:${FRONTEND_IMAGE_TAG:-local}
|
||||||
# Its own register() fetches the IndexNow key from backend at boot -- must
|
# Its own register() fetches the IndexNow key from backend at boot -- must
|
||||||
# wait for a real, healthy backend, not just a started container.
|
# wait for a real, healthy backend, not just a started container.
|
||||||
depends_on:
|
depends_on:
|
||||||
|
|
|
||||||
|
|
@ -206,9 +206,16 @@ infra/openbao/verify-parity.sh --env beta # expect 24 k
|
||||||
cd /opt/thermograph-dev && infra/openbao/verify-parity.sh --env dev # expect 12 keys
|
cd /opt/thermograph-dev && infra/openbao/verify-parity.sh --env dev # expect 12 keys
|
||||||
```
|
```
|
||||||
|
|
||||||
Beta needs no special handling: `env-topology.sh` derives `TG_BAO_APPROLE` per
|
Beta needs no special handling: `verify-parity.sh` sources `env-topology.sh` and calls
|
||||||
environment, so beta authenticates as `tg-beta` rather than falling back to prod's
|
`thermograph_topology` per environment, so `TG_BAO_APPROLE` points at
|
||||||
credentials and being denied its own path by `tg-host-prod.hcl`.
|
`/etc/thermograph/openbao-approle-beta` and beta authenticates as `tg-beta` rather than
|
||||||
|
falling back to prod's credentials and being denied its own path by `tg-host-prod.hcl`.
|
||||||
|
|
||||||
|
That sourcing is load-bearing and was missing until 2026-08-01: this document asserted
|
||||||
|
the property while the verifier alone did not have it, so `--env beta` and `--all` both
|
||||||
|
took a 403 and the gate could not check the one environment whose isolation it exists
|
||||||
|
to prove. If you refactor `verify-parity.sh`, the `thermograph_topology` call is not
|
||||||
|
boilerplate.
|
||||||
|
|
||||||
`seed-from-sops.sh` reads every production secret in plaintext. Per `infra/CLAUDE.md`
|
`seed-from-sops.sh` reads every production secret in plaintext. Per `infra/CLAUDE.md`
|
||||||
the equivalent `seed-from-live.sh` is explicitly *not for an agent to run*; this
|
the equivalent `seed-from-live.sh` is explicitly *not for an agent to run*; this
|
||||||
|
|
@ -224,10 +231,27 @@ One PR per hop, following the estate's own promotion model.
|
||||||
+ TG_SECRETS_BACKEND=openbao
|
+ TG_SECRETS_BACKEND=openbao
|
||||||
```
|
```
|
||||||
|
|
||||||
Order: **dev → beta → prod**, with `verify-parity.sh` green throughout. Recommended
|
Order: **dev → beta → prod**, with `verify-parity.sh` green throughout. The gate before
|
||||||
gate before prod: **7 consecutive green parity runs on all three environments**, run
|
prod is **7 consecutive green parity runs on all three environments**. That runs
|
||||||
nightly from `ops-cron.yml` over SSH (which gives continuous evidence without granting
|
nightly from the `secrets-parity` job in `ops-cron.yml` over SSH, which gives
|
||||||
CI any vault access).
|
continuous evidence without granting CI any vault access — the host renders, CI only
|
||||||
|
asks it to. Forgejo's run history for that job is the record; a night the job was
|
||||||
|
skipped, or the runner was down, does not count as green.
|
||||||
|
|
||||||
|
Measured 2026-08-01, immediately after the verifier was fixed:
|
||||||
|
|
||||||
|
| env | keys | result |
|
||||||
|
|---|---|---|
|
||||||
|
| dev | 12 | **PASS** — byte-identical |
|
||||||
|
| beta | 24 | FAIL — 3 values differ |
|
||||||
|
| prod | 32 | FAIL — 3 values differ |
|
||||||
|
|
||||||
|
Identical three keys on both: `THERMOGRAPH_S3_SECRET_KEY`,
|
||||||
|
`THERMOGRAPH_LAKE_S3_SECRET_KEY`, `THERMOGRAPH_VAPID_CONTACT`. All three live in
|
||||||
|
`common.yaml`, which was seeded on 2026-07-31 at 02:30Z — *before* the Contabo
|
||||||
|
rotation landed. dev passes because dev never layers `common`. One
|
||||||
|
`seed-from-sops.sh --env common` fixes beta and prod together; the 7-day clock starts
|
||||||
|
after that, not before.
|
||||||
|
|
||||||
`TG_SECRETS_BACKEND=sops` remains a working two-way door for the whole period. Keep the
|
`TG_SECRETS_BACKEND=sops` remains a working two-way door for the whole period. Keep the
|
||||||
SOPS path for a full release cycle after prod flips — it is the best mitigation
|
SOPS path for a full release cycle after prod flips — it is the best mitigation
|
||||||
|
|
|
||||||
|
|
@ -36,8 +36,13 @@ usage: verify-parity.sh (--all | --env <dev|beta|prod> ...)
|
||||||
Renders each environment through BOTH backends and compares the resulting
|
Renders each environment through BOTH backends and compares the resulting
|
||||||
KEY=value sets. Prints key names and counts only, never values.
|
KEY=value sets. Prints key names and counts only, never values.
|
||||||
|
|
||||||
Exit status: 0 = every requested environment is at parity; 1 = a mismatch.
|
--all means "every environment that lives on THIS host", not all three: dev is
|
||||||
Suitable as a CI / cron gate.
|
on vps1, beta and prod are on vps2, so no single box can check them all. An
|
||||||
|
environment with no checkout here is skipped out loud. Skipping every one of
|
||||||
|
them is an error, not a pass.
|
||||||
|
|
||||||
|
Exit status: 0 = every environment checked here is at parity; 1 = a mismatch,
|
||||||
|
or nothing was checkable. Suitable as a CI / cron gate.
|
||||||
EOF
|
EOF
|
||||||
exit 2
|
exit 2
|
||||||
}
|
}
|
||||||
|
|
@ -52,20 +57,67 @@ while [ $# -gt 0 ]; do
|
||||||
done
|
done
|
||||||
[ ${#TARGETS[@]} -gt 0 ] || usage
|
[ ${#TARGETS[@]} -gt 0 ] || usage
|
||||||
|
|
||||||
|
# env-topology.sh FIRST, and this is not cosmetic. It is the only thing that sets
|
||||||
|
# TG_BAO_APPROLE, and without it render-secrets-openbao.sh:41 falls through to the
|
||||||
|
# bare default — prod's credential — for every environment. On vps2 that made
|
||||||
|
# `--env beta` authenticate as tg-prod and take a 403 from tg-host-prod.hcl on
|
||||||
|
# thermograph/data/env/beta, so the gate could not verify the one environment whose
|
||||||
|
# isolation it exists to prove. `--all` failed the same way, and dev with it.
|
||||||
|
#
|
||||||
|
# deploy.sh and deploy-stack.sh always sourced this; only the verifier did not,
|
||||||
|
# which is the worst place for the omission to be: the tool whose whole job is to
|
||||||
|
# notice a divergence was itself diverging from the path it verifies.
|
||||||
|
#
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
. "$INFRA_DIR/deploy/env-topology.sh"
|
||||||
# shellcheck source=/dev/null
|
# shellcheck source=/dev/null
|
||||||
. "$INFRA_DIR/deploy/render-secrets-openbao.sh"
|
. "$INFRA_DIR/deploy/render-secrets-openbao.sh"
|
||||||
|
|
||||||
overall=0
|
overall=0
|
||||||
|
checked=0
|
||||||
for env_name in "${TARGETS[@]}"; do
|
for env_name in "${TARGETS[@]}"; do
|
||||||
|
# Per-environment topology: TG_BAO_APPROLE (beta's differs, because beta shares a
|
||||||
|
# filesystem with prod), TG_SKIP_COMMON and TG_APP_DIR. Called before the header
|
||||||
|
# line because --all walks three environments and may skip some of them.
|
||||||
|
if ! thermograph_topology "$env_name"; then
|
||||||
|
echo "!! unknown environment: $env_name" >&2
|
||||||
|
overall=1; continue
|
||||||
|
fi
|
||||||
|
|
||||||
|
# NO ENVIRONMENT CAN BE CHECKED FROM A HOST IT DOES NOT LIVE ON, and --all used to
|
||||||
|
# pretend otherwise. dev is on vps1; beta and prod are on vps2. Running --all
|
||||||
|
# anywhere therefore guaranteed a failure on whichever environments are elsewhere,
|
||||||
|
# and the failure was actively misleading: on vps1, prod resolves TG_BAO_APPROLE to
|
||||||
|
# /etc/thermograph/openbao-approle, which on THAT box holds dev's credential, so
|
||||||
|
# prod's check authenticated as tg-dev and took a 403. Fails closed, but reads like
|
||||||
|
# a policy bug rather than "wrong box".
|
||||||
|
#
|
||||||
|
# Identity comes from the ENVIRONMENT'S OWN ADDRESS being bound here, not from its
|
||||||
|
# checkout existing. The first version of this check tested for TG_APP_DIR and was
|
||||||
|
# wrong: vps1 still carries a stale /opt/thermograph from before the estate split,
|
||||||
|
# so prod looked resident there and was checked anyway. A leftover directory is
|
||||||
|
# exactly the kind of thing that outlives the arrangement it belonged to.
|
||||||
|
#
|
||||||
|
# Skipping is announced, never silent. A gate that quietly narrows what it checked
|
||||||
|
# is worse than one that fails, because the run still goes green.
|
||||||
|
if [ -n "${TG_SSH_HOST:-}" ] && command -v ip >/dev/null 2>&1; then
|
||||||
|
if ! ip -4 -o addr show 2>/dev/null | awk '{print $4}' | cut -d/ -f1 \
|
||||||
|
| grep -qxF "$TG_SSH_HOST"; then
|
||||||
|
echo "== parity check: $env_name — SKIPPED, lives on ${TG_HOST:-another host} (${TG_SSH_HOST}), not here"
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
echo "== parity check: $env_name"
|
echo "== parity check: $env_name"
|
||||||
|
checked=$((checked + 1))
|
||||||
|
|
||||||
# dev renders WITHOUT common on both backends. On the SOPS side that is a caller
|
# dev renders WITHOUT common on both backends. On the SOPS side that is a caller
|
||||||
# flag; on the OpenBao side the policy also forbids it. Setting the flag here keeps
|
# flag; on the OpenBao side the policy also forbids it. Taking the flag from the
|
||||||
# the comparison apples-to-apples — and if the flag were wrong, the OpenBao side
|
# topology rather than re-deriving it here keeps the comparison apples-to-apples
|
||||||
# would fail closed with a 403 rather than quietly diverge, which is exactly the
|
# AND keeps this script from drifting from deploy.sh:67, which reads the same
|
||||||
# improvement being verified.
|
# variable. If the flag were wrong, the OpenBao side would fail closed with a 403
|
||||||
skip_common=0
|
# rather than quietly diverge — which is exactly the improvement being verified.
|
||||||
[ "$env_name" = dev ] && skip_common=1
|
skip_common="${TG_SKIP_COMMON:-0}"
|
||||||
|
|
||||||
sops_out=$(mktemp); bao_out=$(mktemp)
|
sops_out=$(mktemp); bao_out=$(mktemp)
|
||||||
# Both temp files hold PLAINTEXT SECRETS. Removed on every exit path below; not a
|
# Both temp files hold PLAINTEXT SECRETS. Removed on every exit path below; not a
|
||||||
|
|
@ -146,11 +198,23 @@ for k in sorted(order):
|
||||||
cleanup
|
cleanup
|
||||||
done
|
done
|
||||||
|
|
||||||
|
# Zero environments checked is a FAILURE, not a pass. Otherwise a --all run on a
|
||||||
|
# host that carries none of them — a new box, a renamed checkout, a typo'd --env —
|
||||||
|
# exits 0 having verified nothing, and a green run is exactly what the 7-day gate
|
||||||
|
# counts. "Nothing was wrong" and "nothing was examined" must not look alike.
|
||||||
|
if [ "$checked" -eq 0 ]; then
|
||||||
|
echo
|
||||||
|
echo "!! nothing was checked: none of [${TARGETS[*]}] has a checkout on this host." >&2
|
||||||
|
echo "!! dev lives on vps1; beta and prod live on vps2. Run this where they are." >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
if [ "$overall" = 0 ]; then
|
if [ "$overall" = 0 ]; then
|
||||||
echo
|
echo
|
||||||
echo "PARITY OK for: ${TARGETS[*]}"
|
echo "PARITY OK — ${checked} environment(s) checked on this host."
|
||||||
echo "Safe to consider flipping TG_SECRETS_BACKEND for these environments."
|
echo "Safe to consider flipping TG_SECRETS_BACKEND for these environments."
|
||||||
echo "Recommended gate before prod: 7 consecutive green runs on all three."
|
echo "Gate before prod: 7 consecutive green runs covering ALL THREE environments."
|
||||||
|
echo "That means both hosts — a green run here says nothing about the other box."
|
||||||
else
|
else
|
||||||
echo
|
echo
|
||||||
echo "PARITY FAILED — do NOT flip TG_SECRETS_BACKEND." >&2
|
echo "PARITY FAILED — do NOT flip TG_SECRETS_BACKEND." >&2
|
||||||
|
|
|
||||||
|
|
@ -50,7 +50,7 @@ new network exposure and no interactive steps.
|
||||||
**As originally written** (pre vps1/vps2 split; kept for history — see the
|
**As originally written** (pre vps1/vps2 split; kept for history — see the
|
||||||
note at the top of this document):
|
note at the top of this document):
|
||||||
|
|
||||||
- Monorepo `emi/thermograph` (domain dirs: `backend/ frontend/ infra/
|
- Monorepo `Jinemi/thermograph` (domain dirs: `backend/ frontend/ infra/
|
||||||
observability/`). Ops tooling lives in `infra/ops/`.
|
observability/`). Ops tooling lives in `infra/ops/`.
|
||||||
- Environments: **dev** = LAN compose stack on the dev machine; **beta** =
|
- Environments: **dev** = LAN compose stack on the dev machine; **beta** =
|
||||||
`75.119.132.91`; **prod** = `169.58.46.181`. SSH as `agent` with
|
`75.119.132.91`; **prod** = `169.58.46.181`. SSH as `agent` with
|
||||||
|
|
|
||||||
|
|
@ -42,7 +42,7 @@ infra repo**, not the app repo — the "branch" column above is which app-repo
|
||||||
tag an environment is meant to track conceptually; the actual pinned versions
|
tag an environment is meant to track conceptually; the actual pinned versions
|
||||||
are `var.hosts[*].environments[*].backend_image_tag` / `frontend_image_tag`
|
are `var.hosts[*].environments[*].backend_image_tag` / `frontend_image_tag`
|
||||||
(e.g. `"sha-<12 hex>"` each — the app is two separately-published images,
|
(e.g. `"sha-<12 hex>"` each — the app is two separately-published images,
|
||||||
`emi/thermograph-backend/app` and `emi/thermograph-frontend/app`), since the
|
`admin_emi/thermograph-backend/app` and `admin_emi/thermograph-frontend/app`), since the
|
||||||
host has no app checkout to derive a tag from. `app_dir` is required with no
|
host has no app checkout to derive a tag from. `app_dir` is required with no
|
||||||
default specifically so vps2's two environments can never collide on the same
|
default specifically so vps2's two environments can never collide on the same
|
||||||
checkout path (prod: `/opt/thermograph`, beta: `/opt/thermograph-beta`). The
|
checkout path (prod: `/opt/thermograph`, beta: `/opt/thermograph-beta`). The
|
||||||
|
|
@ -183,7 +183,8 @@ Terraform brings the stack up — don't let Terraform recreate containers mid-mi
|
||||||
`git.thermograph.org` reaches Forgejo's loopback port on vps1.
|
`git.thermograph.org` reaches Forgejo's loopback port on vps1.
|
||||||
`COOKIE_SECURE` is auto-set to `0` when there's no domain (a Secure
|
`COOKIE_SECURE` is auto-set to `0` when there's no domain (a Secure
|
||||||
cookie is never sent over plain HTTP) and `1` behind Caddy TLS.
|
cookie is never sent over plain HTTP) and `1` behind Caddy TLS.
|
||||||
- The rendered Caddyfile only reverse-proxies the app. The repo's `deploy/Caddyfile`
|
- The rendered Caddyfile only reverse-proxies the app. The repo's
|
||||||
|
`deploy/stack/lb/Caddyfile`
|
||||||
additionally serves the `emigriffith.dev` portfolio and legacy redirects; those are
|
additionally serves the `emigriffith.dev` portfolio and legacy redirects; those are
|
||||||
host-specific and not templated here.
|
host-specific and not templated here.
|
||||||
- Provisioner-based by design: the hosts already exist, so this is not a
|
- Provisioner-based by design: the hosts already exist, so this is not a
|
||||||
|
|
|
||||||
|
|
@ -239,7 +239,7 @@ resource "null_resource" "host" {
|
||||||
# then bring the stack up on the EXPLICIT per-service app image tags — there's no
|
# then bring the stack up on the EXPLICIT per-service app image tags — there's no
|
||||||
# app checkout on this host to derive a tag from (see var.backend_image_tag /
|
# app checkout on this host to derive a tag from (see var.backend_image_tag /
|
||||||
# frontend_image_tag). The compose file reads BACKEND_IMAGE_TAG / FRONTEND_IMAGE_TAG
|
# frontend_image_tag). The compose file reads BACKEND_IMAGE_TAG / FRONTEND_IMAGE_TAG
|
||||||
# (for emi/thermograph-backend/app and emi/thermograph-frontend/app), so we export
|
# (for admin_emi/thermograph-backend/app and admin_emi/thermograph-frontend/app), so we export
|
||||||
# those, not the old single IMAGE_PATH/IMAGE_TAG. docker runs as root (sources both
|
# those, not the old single IMAGE_PATH/IMAGE_TAG. docker runs as root (sources both
|
||||||
# env files in the same shell) so it never depends on the docker group membership
|
# env files in the same shell) so it never depends on the docker group membership
|
||||||
# taking effect in this session.
|
# taking effect in this session.
|
||||||
|
|
@ -257,7 +257,7 @@ resource "null_resource" "host" {
|
||||||
render_thermograph_secrets ${var.app_dir}
|
render_thermograph_secrets ${var.app_dir}
|
||||||
set -a; . /etc/thermograph-topology.env; . /etc/thermograph.env; set +a
|
set -a; . /etc/thermograph-topology.env; . /etc/thermograph.env; set +a
|
||||||
export REGISTRY_HOST="git.thermograph.org" BACKEND_IMAGE_TAG="${var.backend_image_tag}" FRONTEND_IMAGE_TAG="${var.frontend_image_tag}"
|
export REGISTRY_HOST="git.thermograph.org" BACKEND_IMAGE_TAG="${var.backend_image_tag}" FRONTEND_IMAGE_TAG="${var.frontend_image_tag}"
|
||||||
echo "$REGISTRY_TOKEN" | docker login "$REGISTRY_HOST" --username emi --password-stdin
|
echo "$REGISTRY_TOKEN" | docker login "$REGISTRY_HOST" --username admin_emi --password-stdin
|
||||||
docker compose ${local.compose_flags} pull backend frontend
|
docker compose ${local.compose_flags} pull backend frontend
|
||||||
docker compose ${local.compose_flags} up -d --remove-orphans
|
docker compose ${local.compose_flags} up -d --remove-orphans
|
||||||
'
|
'
|
||||||
|
|
|
||||||
|
|
@ -31,12 +31,12 @@ variable "git_branch" {
|
||||||
}
|
}
|
||||||
|
|
||||||
variable "backend_image_tag" {
|
variable "backend_image_tag" {
|
||||||
description = "Backend image tag to pull (emi/thermograph-backend/app), e.g. \"sha-<12 hex>\" (build-push.yml's tag for the backend-repo commit) or a semver tag. The host has no app-repo checkout to derive this from, so it's always explicit."
|
description = "Backend image tag to pull (admin_emi/thermograph-backend/app), e.g. \"sha-<12 hex>\" (build-push.yml's tag for the backend-repo commit) or a semver tag. The host has no app-repo checkout to derive this from, so it's always explicit."
|
||||||
type = string
|
type = string
|
||||||
}
|
}
|
||||||
|
|
||||||
variable "frontend_image_tag" {
|
variable "frontend_image_tag" {
|
||||||
description = "Frontend image tag to pull (emi/thermograph-frontend/app), e.g. \"sha-<12 hex>\" (build-push.yml's tag for the frontend-repo commit) or a semver tag. Always explicit, same as backend_image_tag."
|
description = "Frontend image tag to pull (admin_emi/thermograph-frontend/app), e.g. \"sha-<12 hex>\" (build-push.yml's tag for the frontend-repo commit) or a semver tag. Always explicit, same as backend_image_tag."
|
||||||
type = string
|
type = string
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -151,13 +151,13 @@ hosts = {
|
||||||
# }
|
# }
|
||||||
|
|
||||||
# Optional overrides (shown with their defaults):
|
# Optional overrides (shown with their defaults):
|
||||||
# repo_url = "https://git.thermograph.org/emi/thermograph-infra.git"
|
# repo_url = "https://git.thermograph.org/admin_emi/thermograph-infra.git"
|
||||||
# app_port = 8137
|
# app_port = 8137
|
||||||
# frontend_port = 8080
|
# frontend_port = 8080
|
||||||
#
|
#
|
||||||
# thermograph-infra is a PRIVATE repo, so a host cloning it for the first time
|
# thermograph-infra is a PRIVATE repo, so a host cloning it for the first time
|
||||||
# needs read credentials embedded in repo_url, e.g. a Forgejo deploy token:
|
# needs read credentials embedded in repo_url, e.g. a Forgejo deploy token:
|
||||||
# repo_url = "https://deploy:REPLACE_WITH_TOKEN@git.thermograph.org/emi/thermograph-infra.git"
|
# repo_url = "https://deploy:REPLACE_WITH_TOKEN@git.thermograph.org/admin_emi/thermograph-infra.git"
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------------
|
||||||
# Self-hosted Open-Meteo archive (only used by environments with openmeteo = true)
|
# Self-hosted Open-Meteo archive (only used by environments with openmeteo = true)
|
||||||
|
|
|
||||||
|
|
@ -54,8 +54,8 @@ variable "hosts" {
|
||||||
git_branch = string # this INFRA repo's branch the checkout is reset to
|
git_branch = string # this INFRA repo's branch the checkout is reset to
|
||||||
# Which app images to run, e.g. "sha-<12 hex>" (each matching build-push.yml's tag
|
# Which app images to run, e.g. "sha-<12 hex>" (each matching build-push.yml's tag
|
||||||
# for the commit that app repo built) or a semver tag on a release push. The app is
|
# for the commit that app repo built) or a semver tag on a release push. The app is
|
||||||
# TWO separately-published images now — emi/thermograph-backend/app and
|
# TWO separately-published images now — admin_emi/thermograph-backend/app and
|
||||||
# emi/thermograph-frontend/app — pinned independently. Required, no default: the
|
# admin_emi/thermograph-frontend/app — pinned independently. Required, no default: the
|
||||||
# host's checkout is this infra repo, not the app repos, so there's no "current
|
# host's checkout is this infra repo, not the app repos, so there's no "current
|
||||||
# commit" to derive a tag from; both must be explicit. Bump these (via a normal
|
# commit" to derive a tag from; both must be explicit. Bump these (via a normal
|
||||||
# tfvars edit + apply) whenever an app repo ships a commit you want this
|
# tfvars edit + apply) whenever an app repo ships a commit you want this
|
||||||
|
|
@ -127,9 +127,9 @@ variable "gcp_hosts" {
|
||||||
}
|
}
|
||||||
|
|
||||||
variable "repo_url" {
|
variable "repo_url" {
|
||||||
description = "Git remote to clone from when a host has no checkout yet. Points at THIS repo (thermograph-infra) now, not the app repo -- the app's own source is never checked out on a host; only its published registry images are pulled (see var.hosts[*].backend_image_tag / frontend_image_tag). thermograph-infra is a private repo, so this typically needs embedded read credentials, e.g. a Forgejo deploy token: \"https://<token-name>:<token>@git.thermograph.org/emi/thermograph-infra.git\"."
|
description = "Git remote to clone from when a host has no checkout yet. Points at THIS repo (thermograph-infra) now, not the app repo -- the app's own source is never checked out on a host; only its published registry images are pulled (see var.hosts[*].backend_image_tag / frontend_image_tag). thermograph-infra is a private repo, so this typically needs embedded read credentials, e.g. a Forgejo deploy token: \"https://<token-name>:<token>@git.thermograph.org/admin_emi/thermograph-infra.git\"."
|
||||||
type = string
|
type = string
|
||||||
default = "https://git.thermograph.org/emi/thermograph-infra.git"
|
default = "https://git.thermograph.org/admin_emi/thermograph-infra.git"
|
||||||
sensitive = true
|
sensitive = true
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -62,7 +62,7 @@ persistent, queryable, fleet-wide log store and UI.
|
||||||
> ### ⚠️ Merging is not deploying
|
> ### ⚠️ Merging is not deploying
|
||||||
>
|
>
|
||||||
> `/opt/observability` on vps1 (and vps2) is a checkout of the **archived**
|
> `/opt/observability` on vps1 (and vps2) is a checkout of the **archived**
|
||||||
> `emi/thermograph-observability` repo. It can never `git pull` again — the
|
> `admin_emi/thermograph-observability` repo. It can never `git pull` again — the
|
||||||
> content now lives here, in the mono repo, under `observability/`. A previous
|
> content now lives here, in the mono repo, under `observability/`. A previous
|
||||||
> observability PR merged and had **zero live effect** for exactly this reason.
|
> observability PR merged and had **zero live effect** for exactly this reason.
|
||||||
>
|
>
|
||||||
|
|
@ -147,7 +147,7 @@ LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push \
|
||||||
docker compose -f docker-compose.agent.yml -f docker-compose.agent.beta.yml up -d
|
docker compose -f docker-compose.agent.yml -f docker-compose.agent.beta.yml up -d
|
||||||
```
|
```
|
||||||
|
|
||||||
The mesh must already be up (see the main repo's `deploy/swarm/` / `INFRA.md`);
|
The mesh must already be up (see `infra/deploy/swarm/`);
|
||||||
Swarm is **not** required for the agent itself — it talks plain HTTP over wg0.
|
Swarm is **not** required for the agent itself — it talks plain HTTP over wg0.
|
||||||
|
|
||||||
<!-- TODO(cutover): desktop no longer runs a Thermograph app stack and ships no
|
<!-- TODO(cutover): desktop no longer runs a Thermograph app stack and ships no
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue