Jinemi/thermograph
4
0
Fork 0
Code Issues 2 Pull requests 10 Projects Releases Packages Wiki Activity Actions 1

infra: split the estate into vps1/vps2, moving beta next to prod and dev onto vps1
All checks were successful
PR build (required check) / changes (pull_request) Successful in 6s
Details
secrets-guard / encrypted (pull_request) Successful in 5s
Details
PR build (required check) / build-backend (pull_request) Has been skipped
Details
shell-lint / shellcheck (pull_request) Successful in 7s
Details
PR build (required check) / build-frontend (pull_request) Has been skipped
Details
PR build (required check) / validate-observability (pull_request) Successful in 18s
Details
PR build (required check) / gate (pull_request) Successful in 2s
Details

Browse source 
Environments stop being machines. Until now each box WAS an environment --
"beta" named both a deploy target and a host, "the desktop" named both the
operator's computer and the dev server -- so every path could assume one
environment per host and hardcode /opt/thermograph, /etc/thermograph.env and
ports 8137/8080. That assumption ends here:

  vps1  75.119.132.91  Forgejo, Grafana/Loki, the portfolio site, and DEV
                       (own Postgres, mesh-only on 10.10.0.2:8137)
  vps2  169.58.46.181  PROD and BETA as two Swarm stacks sharing one
                       TimescaleDB instance, plus Centralis, Postfix, backups
  desktop              AI model hosting + flex Swarm capacity, no environment

Nothing live has moved; the ordered cutover is in
infra/deploy/RUNBOOK-vps1-vps2-cutover.md.

deploy/env-topology.sh is the single source of truth: env -> host, checkout,
branch, deploy mode, stack name, env file, LB ports, DB role/database, service
prefix. THERMOGRAPH_ENV is the input, and on vps2 it is the only thing
distinguishing a beta deploy from a prod one -- deploy.sh refuses to run when it
disagrees with the checkout it was invoked from. The host-wide secrets-env and
deploy-mode markers survive only as a fallback for a single-environment box.

Beta gets its own stack file rather than an overlay (a merge cannot REMOVE the
db service, and beta having no database of its own is the point). Its services
are prefixed beta-*: Swarm registers a service's short name as a DNS alias on
every network it joins, so two stacks both calling a service `web` on the shared
data network would let prod's frontend resolve a beta task. Prod's stack, env
and LB config are untouched.

One Postgres, two databases with two roles: deploy/db/provision-env-db.sh
creates thermograph_beta (NOSUPERUSER, owns only its own database, CONNECT
revoked from PUBLIC) plus a <role>_ro for ad-hoc queries, and refuses to run for
the environment that owns the instance -- doing so would demote prod's bootstrap
superuser.

Fixes that co-residency would otherwise have broken silently:

- CI secrets are keyed by host (VPS1_SSH_*, VPS2_SSH_*). SSH_* meant "beta" and
  also "the Forgejo box" because those shared a machine; that conflation is what
  once had the prod backup dumping beta.
- The nightly backup dumps BOTH application databases to separate off-box
  prefixes, and fails loudly on a missing one instead of skipping it.
- Alloy stopped deriving the `host` label from the node -- on vps2 that would
  have filed every beta line as prod, feeding prod's alert rules with beta's
  traffic. It is now derived per source, with a new `node` label for the machine,
  and beta's log volume is mounted via a vps2-only overlay.
- ops/dbq.sh, ops/iceberg.sh and the secrets seed scripts derived their SSH
  target and env-file path from the topology instead of hardcoding beta to
  75.119.132.91 -- which is vps1's address now.
- Dev's overlay binds DEV_BIND_ADDR (loopback by default, the mesh address on
  vps1) instead of 0.0.0.0: correct for a home LAN box, a public exposure of
  unreviewed branches on a VPS.

Terraform's hosts variable is keyed by machine with a nested environments map,
and main.tf flattens (host, environment) pairs so two environments on one box
cannot share a checkout. Caddy's single reference config is split per host.

Docs, onboarding and the runbooks are updated throughout, including the security
rationales that co-residency makes false: separate SSH credentials no longer put
a host boundary between beta and prod, and the boundary that remains is the
database and the filesystem.
This commit is contained in:
Emi Griffith 2026-07-25 15:01:29 -07:00
parent 5f20fba9f5
commit 4e97d8e5dc
85 changed files with 4207 additions and 1482 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

16
.claude/hooks/README.md
View file

@ -13,7 +13,7 @@ They are wired up in `.claude/settings.json`.
## Why prod-guard exists ## Why prod-guard exists
`agent` has passwordless sudo on prod and beta, and the operator's global settings `agent` has passwordless sudo on vps1 and vps2, and the operator's global settings
allow `Bash(ssh prod:*)` outright. Nothing about `ssh prod 'docker service rm …'` allow `Bash(ssh prod:*)` outright. Nothing about `ssh prod 'docker service rm …'`
goes through a pull request, so CI cannot see it — before this hook, any session in goes through a pull request, so CI cannot see it — before this hook, any session in
any directory could delete production with no prompt. any directory could delete production with no prompt.
@ -24,9 +24,17 @@ is wrong by construction — the first destructive verb nobody thought of sails
straight through. Being wrong in the "ask" direction costs a keystroke; being wrong straight through. Being wrong in the "ask" direction costs a keystroke; being wrong
the other way costs production. the other way costs production.
Beta is guarded as strictly as prod: it serves beta.thermograph.org *and* hosts Beta is guarded as strictly as prod — more strictly than the names alone suggest,
Forgejo, so a destructive command there takes out git, CI and the registry at once. now: vps2 (`169.58.46.181`) runs **both** prod and beta as separate Swarm stacks on
The LAN dev box is deliberately unguarded. the same manager, sharing one TimescaleDB instance, so a "beta" command there has
prod's blast radius (see `infra/deploy/env-topology.sh`). vps1 (`75.119.132.91`) is
the other guarded host: Forgejo (git + CI + registry) and Grafana/Loki live there,
plus the `dev` environment, so a destructive command there takes out git, CI and the
registry at once. Mesh IPs did not move in the vps1/vps2 split (vps1 is still
`10.10.0.2`, vps2 is still `10.10.0.1`); what moved is which environment runs where
— `HOST_TOKEN_RE` in `prod-guard.sh` recognises `vps1`/`vps2` alongside the
environment names (`prod`/`beta`) and the raw IPs, so a command naming the host
either way still gets caught. The LAN dev box is deliberately unguarded.
## Changing the classifier ## Changing the classifier

17
.claude/hooks/prod-guard.sh
View file

@ -19,9 +19,15 @@
# That is also what happens if this script errors or jq is missing, so a bug here # That is also what happens if this script errors or jq is missing, so a bug here
# fails toward the prompt rather than toward silent execution. # fails toward the prompt rather than toward silent execution.
# #
# Beta is guarded as strictly as prod: it serves beta.thermograph.org AND hosts # Beta is guarded as strictly as prod -- more strictly than the names alone
# Forgejo, so a destructive command there can take out the git server, CI and the # suggest, now: vps2 (169.58.46.181) runs prod AND beta as two Swarm stacks on
# container registry at once. # the SAME manager, sharing the same TimescaleDB instance, so a "beta" command
# there has PROD's blast radius (shared Docker host, shared Postgres server --
# see infra/deploy/env-topology.sh). vps1 (75.119.132.91) is the other guarded
# host: Forgejo (git + CI + registry), Grafana/Loki, and the dev environment --
# a destructive command there can take out the git server, CI and the container
# registry at once. Mesh IPs did NOT move in the vps1/vps2 split (vps1 is still
# 10.10.0.2, vps2 is still 10.10.0.1); what moved is which environment runs where.
set -uo pipefail set -uo pipefail
payload=$(cat) payload=$(cat)
@ -32,7 +38,10 @@ tool=$(printf '%s' "$payload" | jq -r '.tool_name // ""')
# An ssh target naming a live host, matched as a whole token after any `user@`. # An ssh target naming a live host, matched as a whole token after any `user@`.
# Backslashes are doubled because awk -v processes escape sequences in the value # Backslashes are doubled because awk -v processes escape sequences in the value
# before the regex ever sees it; a single \. arrives as a bare dot (and warns). # before the regex ever sees it; a single \. arrives as a bare dot (and warns).
readonly HOST_TOKEN_RE='^(prod|beta|169\\.58\\.46\\.181|75\\.119\\.132\\.91|10\\.10\\.0\\.[12])$' # vps2 (169.58.46.181) carries BOTH prod and beta; vps1 (75.119.132.91) carries
# Forgejo/Grafana/dev -- "vps1"/"vps2" are recognised alongside the environment
# names and raw IPs so a command naming the host either way still gets caught.
readonly HOST_TOKEN_RE='^(prod|beta|vps1|vps2|169\\.58\\.46\\.181|75\\.119\\.132\\.91|10\\.10\\.0\\.[12])$'
emit() { # $1=allow|ask|deny $2=reason emit() { # $1=allow|ask|deny $2=reason
jq -n --arg d "$1" --arg r "$2" \ jq -n --arg d "$1" --arg r "$2" \

100
.forgejo/workflows/deploy.yml
View file

@ -10,16 +10,32 @@ name: Deploy
# parameterised, so the duplication bought nothing and cost six files to keep in # parameterised, so the duplication bought nothing and cost six files to keep in
# step. # step.
# #
# The two *-deploy-dev.yml workflows are NOT ported. They were documented as # DEV IS A REAL DEPLOY TARGET AGAIN. It previously was not: dev meant a
# inert: they call the monorepo layout at ~/thermograph-dev on the LAN box, # sudo-free compose stack on the operator's desktop, and the two
# which is still a split-era thermograph-infra checkout, so that path does not # *-deploy-dev.yml workflows that drove it were inert. Dev now lives on vps1 at
# exist there. LAN dev is a local `make dev-up` concern, not a CI environment. # /opt/thermograph-dev — a normal fleet host, reached over SSH exactly like beta
# and prod — so it gets a leg here.
#
# SECRETS ARE KEYED BY HOST, NOT BY ENVIRONMENT. `SSH_*` used to mean "beta" and
# `PROD_SSH_*` "prod", which worked only while each environment owned a box. It
# stopped being true in two directions at once: vps2 now runs beta AND prod, and
# the box `SSH_*` pointed at is now vps1, which runs dev, Forgejo and Grafana.
# The old names would have made "the beta secret" and "the Forgejo box secret"
# the same value by accident — the ops-cron file already records one incident
# caused by exactly that conflation. So: VPS1_SSH_* and VPS2_SSH_*, named for
# the machine, and the environment is passed separately as THERMOGRAPH_ENV.
#
# BETA AND PROD DEPLOYING TO ONE BOX DO NOT RACE. They have separate checkouts
# (/opt/thermograph-beta and /opt/thermograph), so the `git reset --hard` in one
# cannot pull the tree out from under the other, and deploy.sh's flock is per
# checkout. The concurrency groups below stay keyed by ref+service, which keeps
# two pushes to the SAME branch serialised — the property that actually matters.
# #
# DELIBERATELY BORING EXPRESSIONS. No dynamic matrix (fromJSON), no # DELIBERATELY BORING EXPRESSIONS. No dynamic matrix (fromJSON), no
# `cond && secrets.A || secrets.B` ternary. Those are GitHub idioms that a # `cond && secrets.A || secrets.B` ternary. Those are GitHub idioms that a
# Forgejo/act runner may evaluate differently, and the failure mode here is # Forgejo/act runner may evaluate differently, and the failure mode here is
# "production does not deploy" or, worse, "deploys with an empty SSH host". The # "production does not deploy" or, worse, "deploys with an empty SSH host". The
# two environments therefore get two explicit, mutually exclusive steps. # three environments therefore get three explicit, mutually exclusive steps.
# #
# What is preserved from the originals, all of it load-bearing: # What is preserved from the originals, all of it load-bearing:
# - fetch-depth: 0, because the image tag is keyed to the LAST COMMIT THAT # - fetch-depth: 0, because the image tag is keyed to the LAST COMMIT THAT
@ -29,14 +45,18 @@ name: Deploy
# - The 12-hex truncation, matching build-push exactly. # - The 12-hex truncation, matching build-push exactly.
# - Per-service, per-environment concurrency with cancel-in-progress: false -- # - Per-service, per-environment concurrency with cancel-in-progress: false --
# a half-finished deploy must never be cancelled by a newer one. # a half-finished deploy must never be cancelled by a newer one.
# - Separate PROD_SSH_* credentials, so a beta credential leak cannot reach # - Separate credentials per host. Note what this does and does not buy now:
# prod. # it keeps vps1 (Forgejo, its CI, and whatever unreviewed branch dev is
# running) away from vps2 entirely. It no longer puts a host boundary
# between beta and prod, because they share vps2 by design — that boundary
# is now at the database (separate roles and databases) and the filesystem
# (separate checkouts and rendered env files).
# - appleboy/ssh-action by full URL; it is not mirrored in Forgejo's default # - appleboy/ssh-action by full URL; it is not mirrored in Forgejo's default
# action registry. # action registry.
on: on:
push: push:
branches: [main, release] branches: [dev, main, release]
paths: paths:
- 'backend/**' - 'backend/**'
- 'frontend/**' - 'frontend/**'
@ -71,10 +91,15 @@ jobs:
run: | run: |
set -euo pipefail set -euo pipefail
# Branch selects the environment. main -> beta, release -> prod. # Branch selects the environment; the environment selects the host and
# the script path. dev -> dev (vps1), main -> beta (vps2),
# release -> prod (vps2). The paths differ because each environment
# has its own checkout — two environments on vps2 must never share
# one — and dev has its own entry point for its secrets policy.
case "${{ github.ref_name }}" in case "${{ github.ref_name }}" in
main) environment=beta ;; dev) environment=dev ; script=/opt/thermograph-dev/infra/deploy/deploy-dev.sh ;;
release) environment=prod ;; main) environment=beta ; script=/opt/thermograph-beta/infra/deploy/deploy.sh ;;
release) environment=prod ; script=/opt/thermograph/infra/deploy/deploy.sh ;;
*) echo "::error::Deploy triggered on unexpected ref '${{ github.ref_name }}'"; exit 1 ;; *) echo "::error::Deploy triggered on unexpected ref '${{ github.ref_name }}'"; exit 1 ;;
esac esac
@ -105,6 +130,7 @@ jobs:
{ {
echo "environment=$environment" echo "environment=$environment"
echo "script=$script"
echo "changed=$changed" echo "changed=$changed"
echo "tag=$tag" echo "tag=$tag"
} >> "$GITHUB_OUTPUT" } >> "$GITHUB_OUTPUT"
@ -126,38 +152,58 @@ jobs:
echo "==> ${{ matrix.service }} -> $environment | changed=$changed | tag=$tag" echo "==> ${{ matrix.service }} -> $environment | changed=$changed | tag=$tag"
- name: Deploy to beta - name: Deploy to dev (vps1)
if: steps.plan.outputs.changed == 'true' && steps.plan.outputs.environment == 'dev'
uses: https://github.com/appleboy/ssh-action@v1.2.0
with:
host: ${{ secrets.VPS1_SSH_HOST }}
username: ${{ secrets.VPS1_SSH_USER }}
key: ${{ secrets.VPS1_SSH_KEY }}
port: ${{ secrets.VPS1_SSH_PORT }}
envs: SERVICE,BACKEND_IMAGE_TAG,FRONTEND_IMAGE_TAG,THERMOGRAPH_ENV
script: ${{ steps.plan.outputs.script }}
env:
SERVICE: ${{ matrix.service }}
THERMOGRAPH_ENV: dev
BACKEND_IMAGE_TAG: ${{ matrix.service == 'backend' && steps.plan.outputs.tag || '' }}
FRONTEND_IMAGE_TAG: ${{ matrix.service == 'frontend' && steps.plan.outputs.tag || '' }}
- name: Deploy to beta (vps2)
if: steps.plan.outputs.changed == 'true' && steps.plan.outputs.environment == 'beta' if: steps.plan.outputs.changed == 'true' && steps.plan.outputs.environment == 'beta'
uses: https://github.com/appleboy/ssh-action@v1.2.0 uses: https://github.com/appleboy/ssh-action@v1.2.0
with: with:
host: ${{ secrets.SSH_HOST }} host: ${{ secrets.VPS2_SSH_HOST }}
username: ${{ secrets.SSH_USER }} username: ${{ secrets.VPS2_SSH_USER }}
key: ${{ secrets.SSH_KEY }} key: ${{ secrets.VPS2_SSH_KEY }}
port: ${{ secrets.SSH_PORT }} port: ${{ secrets.VPS2_SSH_PORT }}
envs: SERVICE,BACKEND_IMAGE_TAG,FRONTEND_IMAGE_TAG envs: SERVICE,BACKEND_IMAGE_TAG,FRONTEND_IMAGE_TAG,THERMOGRAPH_ENV
script: /opt/thermograph/infra/deploy/deploy.sh script: ${{ steps.plan.outputs.script }}
env: env:
SERVICE: ${{ matrix.service }} SERVICE: ${{ matrix.service }}
# THERMOGRAPH_ENV is what makes this a BETA deploy rather than a prod
# one: same host, same credentials, same script — the environment is
# the only thing that differs, and deploy.sh refuses to run if it
# disagrees with the checkout it was invoked from.
THERMOGRAPH_ENV: beta
# Only the matching one is read by deploy.sh for a single-service roll; # Only the matching one is read by deploy.sh for a single-service roll;
# the other stays empty and the persisted .image-tags.env supplies the # the other stays empty and the persisted .image-tags.env supplies the
# sibling's live tag, so this roll never disturbs it. # sibling's live tag, so this roll never disturbs it.
BACKEND_IMAGE_TAG: ${{ matrix.service == 'backend' && steps.plan.outputs.tag || '' }} BACKEND_IMAGE_TAG: ${{ matrix.service == 'backend' && steps.plan.outputs.tag || '' }}
FRONTEND_IMAGE_TAG: ${{ matrix.service == 'frontend' && steps.plan.outputs.tag || '' }} FRONTEND_IMAGE_TAG: ${{ matrix.service == 'frontend' && steps.plan.outputs.tag || '' }}
- name: Deploy to prod - name: Deploy to prod (vps2)
if: steps.plan.outputs.changed == 'true' && steps.plan.outputs.environment == 'prod' if: steps.plan.outputs.changed == 'true' && steps.plan.outputs.environment == 'prod'
uses: https://github.com/appleboy/ssh-action@v1.2.0 uses: https://github.com/appleboy/ssh-action@v1.2.0
with: with:
# A completely separate secret set from beta's, deliberately: a beta host: ${{ secrets.VPS2_SSH_HOST }}
# credential leak must not reach prod. username: ${{ secrets.VPS2_SSH_USER }}
host: ${{ secrets.PROD_SSH_HOST }} key: ${{ secrets.VPS2_SSH_KEY }}
username: ${{ secrets.PROD_SSH_USER }} port: ${{ secrets.VPS2_SSH_PORT }}
key: ${{ secrets.PROD_SSH_KEY }} envs: SERVICE,BACKEND_IMAGE_TAG,FRONTEND_IMAGE_TAG,THERMOGRAPH_ENV
port: ${{ secrets.PROD_SSH_PORT }} script: ${{ steps.plan.outputs.script }}
envs: SERVICE,BACKEND_IMAGE_TAG,FRONTEND_IMAGE_TAG
script: /opt/thermograph/infra/deploy/deploy.sh
env: env:
SERVICE: ${{ matrix.service }} SERVICE: ${{ matrix.service }}
THERMOGRAPH_ENV: prod
BACKEND_IMAGE_TAG: ${{ matrix.service == 'backend' && steps.plan.outputs.tag || '' }} BACKEND_IMAGE_TAG: ${{ matrix.service == 'backend' && steps.plan.outputs.tag || '' }}
FRONTEND_IMAGE_TAG: ${{ matrix.service == 'frontend' && steps.plan.outputs.tag || '' }} FRONTEND_IMAGE_TAG: ${{ matrix.service == 'frontend' && steps.plan.outputs.tag || '' }}

97
.forgejo/workflows/infra-sync.yml
View file

@ -2,65 +2,114 @@ name: Sync infra to hosts
# The infra DOMAIN's own pipeline (new with the monorepo -- the split era had # The infra DOMAIN's own pipeline (new with the monorepo -- the split era had
# no infra deploy workflow at all; infra changes rode along lazily with the # no infra deploy workflow at all; infra changes rode along lazily with the
# next app deploy's `git reset`). On a push to `main` touching infra/**, SSH # next app deploy's `git reset`). On a push touching infra/**, SSH to every
# to beta AND prod, fast-forward the host's /opt/thermograph monorepo checkout # host, fast-forward each ENVIRONMENT's monorepo checkout and re-render that
# and re-render /etc/thermograph.env from the SOPS vault -- so a compose edit # environment's env file from the SOPS vault -- so a compose edit or a secret
# or a secret rotation lands push-button instead of waiting for the next app # rotation lands push-button instead of waiting for the next app deploy.
# deploy.
# #
# Deliberately does NOT roll any service: image tags are the app domains' # Deliberately does NOT roll any service: image tags are the app domains'
# axis, not infra's. A compose change that must recreate containers takes # axis, not infra's. A compose or stack change that must recreate containers
# effect on the next app deploy, or a by-hand # takes effect on the next app deploy, or a by-hand
# `SERVICE=all ... infra/deploy/deploy.sh` run (see that script's header). # `SERVICE=all ... infra/deploy/deploy.sh` run (see that script's header).
# #
# Both hosts track infra via `main` (the split-era model, unchanged): prod's # THREE CHECKOUTS, TWO HOSTS. This is the shape the vps1/vps2 split forces:
# *app images* are staged by the `release` branch, but its checkout follows #
# main -- deploy.sh's own BRANCH default encodes the same thing. # vps1 /opt/thermograph-dev branch dev -> /etc/thermograph.env
# vps2 /opt/thermograph-beta branch main -> /etc/thermograph-beta.env
# vps2 /opt/thermograph branch main -> /etc/thermograph.env
#
# The two checkouts on vps2 are separate directories on purpose: a `git reset
# --hard` for beta must not be able to move prod's tree, and each environment
# renders its own env file from its own vault file. Because the paths differ,
# the beta and prod jobs below can safely run at the same time on one box.
#
# Beta and prod both track `main` (infra is not environment-staged -- only app
# IMAGES are, via tags). Dev tracks `dev`, which is why this workflow now
# triggers on both branches: a dev-branch infra change has to reach the dev
# checkout, and only that one.
#
# Each `render_thermograph_secrets` call passes the environment name and the
# output path EXPLICITLY. It used to rely on the host's
# /etc/thermograph/secrets-env marker, which cannot answer the question on vps2
# -- one marker, two environments. Passing them here means a beta sync can
# never render prod's vault, and vice versa.
on: on:
push: push:
branches: [main] branches: [dev, main]
paths: ['infra/**'] paths: ['infra/**']
workflow_dispatch: {} workflow_dispatch: {}
jobs: jobs:
sync-dev:
# Only the dev branch feeds the dev checkout.
if: github.ref_name == 'dev'
runs-on: docker
concurrency:
group: infra-sync-dev
cancel-in-progress: false
steps:
- name: Sync dev checkout on vps1 + re-render secrets
uses: https://github.com/appleboy/ssh-action@v1.2.0
with:
host: ${{ secrets.VPS1_SSH_HOST }}
username: ${{ secrets.VPS1_SSH_USER }}
key: ${{ secrets.VPS1_SSH_KEY }}
port: ${{ secrets.VPS1_SSH_PORT }}
script: |
set -euo pipefail
cd /opt/thermograph-dev
git fetch --prune origin dev
git reset --hard origin/dev
if [ -f infra/deploy/render-secrets.sh ]; then
. infra/deploy/render-secrets.sh
# SKIP_COMMON is dev's standing rule, not a preference: common.yaml
# is the fleet's shared production credential set, and vps1 also
# runs Forgejo and its CI. See render-secrets.sh.
THERMOGRAPH_SECRETS_SKIP_COMMON=1 \
render_thermograph_secrets /opt/thermograph-dev/infra dev /etc/thermograph.env
fi
echo "synced to $(git log --oneline -1)"
sync-beta: sync-beta:
if: github.ref_name == 'main'
runs-on: docker runs-on: docker
concurrency: concurrency:
group: infra-sync-beta group: infra-sync-beta
cancel-in-progress: false cancel-in-progress: false
steps: steps:
- name: Sync beta checkout + re-render secrets - name: Sync beta checkout on vps2 + re-render secrets
uses: https://github.com/appleboy/ssh-action@v1.2.0 uses: https://github.com/appleboy/ssh-action@v1.2.0
with: with:
host: ${{ secrets.SSH_HOST }} host: ${{ secrets.VPS2_SSH_HOST }}
username: ${{ secrets.SSH_USER }} username: ${{ secrets.VPS2_SSH_USER }}
key: ${{ secrets.SSH_KEY }} key: ${{ secrets.VPS2_SSH_KEY }}
port: ${{ secrets.SSH_PORT }} port: ${{ secrets.VPS2_SSH_PORT }}
script: | script: |
set -euo pipefail set -euo pipefail
cd /opt/thermograph cd /opt/thermograph-beta
git fetch --prune origin main git fetch --prune origin main
git reset --hard origin/main git reset --hard origin/main
if [ -f infra/deploy/render-secrets.sh ]; then if [ -f infra/deploy/render-secrets.sh ]; then
. infra/deploy/render-secrets.sh . infra/deploy/render-secrets.sh
render_thermograph_secrets /opt/thermograph/infra render_thermograph_secrets /opt/thermograph-beta/infra beta /etc/thermograph-beta.env
fi fi
echo "synced to $(git log --oneline -1)" echo "synced to $(git log --oneline -1)"
sync-prod: sync-prod:
if: github.ref_name == 'main'
runs-on: docker runs-on: docker
concurrency: concurrency:
group: infra-sync-prod group: infra-sync-prod
cancel-in-progress: false cancel-in-progress: false
steps: steps:
- name: Sync prod checkout + re-render secrets - name: Sync prod checkout on vps2 + re-render secrets
uses: https://github.com/appleboy/ssh-action@v1.2.0 uses: https://github.com/appleboy/ssh-action@v1.2.0
with: with:
host: ${{ secrets.PROD_SSH_HOST }} host: ${{ secrets.VPS2_SSH_HOST }}
username: ${{ secrets.PROD_SSH_USER }} username: ${{ secrets.VPS2_SSH_USER }}
key: ${{ secrets.PROD_SSH_KEY }} key: ${{ secrets.VPS2_SSH_KEY }}
port: ${{ secrets.PROD_SSH_PORT }} port: ${{ secrets.VPS2_SSH_PORT }}
script: | script: |
set -euo pipefail set -euo pipefail
cd /opt/thermograph cd /opt/thermograph
@ -68,6 +117,6 @@ jobs:
git reset --hard origin/main git reset --hard origin/main
if [ -f infra/deploy/render-secrets.sh ]; then if [ -f infra/deploy/render-secrets.sh ]; then
. infra/deploy/render-secrets.sh . infra/deploy/render-secrets.sh
render_thermograph_secrets /opt/thermograph/infra render_thermograph_secrets /opt/thermograph/infra prod /etc/thermograph.env
fi fi
echo "synced to $(git log --oneline -1)" echo "synced to $(git log --oneline -1)"

3
.forgejo/workflows/observability-validate.yml
View file

@ -1,6 +1,7 @@
name: Validate observability stack name: Validate observability stack
# The observability DOMAIN deploys by hand (`docker compose up -d` on beta + # The observability DOMAIN deploys by hand (`docker compose up -d` on vps1 —
# the monitoring host, not the beta environment, which runs on vps2 now — plus
# the Alloy agent on each node) with no build step, so nothing caught a # the Alloy agent on each node) with no build step, so nothing caught a
# malformed compose file, a broken dashboard JSON, or an unparseable config # malformed compose file, a broken dashboard JSON, or an unparseable config
# until it failed live on the host. This is that missing guard: it parses # until it failed live on the host. This is that missing guard: it parses

147
.forgejo/workflows/ops-cron.yml
View file

@ -6,19 +6,32 @@ name: Ops cron (backup + IndexNow)
# classification table in # classification table in
# thermograph-docs/architecture/repo-topology-and-infrastructure.md §7. # thermograph-docs/architecture/repo-topology-and-infrastructure.md §7.
# #
# Both jobs SSH into the prod host and run inside the already-running compose # The jobs SSH into a host and run inside the already-running stack (docker
# stack (docker compose exec), the same way deploy.sh already runs its own # exec), the same way deploy.sh runs its own post-deploy IndexNow ping -- no new
# post-deploy IndexNow ping -- no new network exposure, no separate dependency # network exposure, no separate dependency install. Runs on the `docker` label
# install. Runs on the `docker` label (the always-on Swarm-hosted runner # (the always-on Swarm-hosted runner deploy/forgejo/ stood up).
# deploy/forgejo/ stood up).
# #
# Targets PROD via the PROD_SSH_* secrets (prod = 169.58.46.181, `agent` user, # WHICH BOX EACH JOB TALKS TO, and why the secret names say so:
# in the docker group so no sudo needed; /etc/thermograph.env is agent-readable). #
# These are the same secrets deploy-prod.yml uses for the release->prod deploy -- # backup, indexnow -> VPS2_SSH_* (vps2 = 169.58.46.181: prod AND beta, the
# NOT the SSH_* secrets, which point at BETA (deploy.yml's `main`->beta path). An # shared TimescaleDB, Postfix, the backups)
# earlier revision reused SSH_* here, so the "prod" backup was silently dumping # forgejo-backup -> VPS1_SSH_* (vps1 = 75.119.132.91: Forgejo, Grafana,
# beta; prod itself had no backup at all. The prod database is the one that must # Loki, and the dev environment)
# be backed up, so both jobs use PROD_SSH_*. #
# The secrets are keyed by HOST rather than by environment because an
# environment no longer implies a machine: vps2 runs two of them. The previous
# names encoded the opposite assumption and had already caused one incident --
# an early revision used SSH_* here, which meant "beta", so the job labelled
# "prod backup" was silently dumping beta while prod had no backup at all. The
# same conflation is why Forgejo's backup used to be described as running "on
# beta": beta and Forgejo happened to share a box, so one secret served both
# meanings. It does not any more.
#
# BOTH APPLICATION DATABASES ARE BACKED UP. prod and beta are separate databases
# (`thermograph` and `thermograph_beta`) on one shared instance. Dumping only
# the prod database would recreate the original failure in a new shape: a job
# that looks like "the backup" while one environment's data is silently
# uncovered. The loop below dumps each, to its own off-box prefix.
# Monorepo port: the compose file now lives under infra/ of the host's # Monorepo port: the compose file now lives under infra/ of the host's
# /opt/thermograph monorepo checkout, so every docker-compose invocation cd's # /opt/thermograph monorepo checkout, so every docker-compose invocation cd's
@ -44,22 +57,22 @@ jobs:
group: ops-backup group: ops-backup
cancel-in-progress: false cancel-in-progress: false
steps: steps:
- name: Dump the prod database over SSH - name: Dump both application databases over SSH
uses: https://github.com/appleboy/ssh-action@v1.2.0 uses: https://github.com/appleboy/ssh-action@v1.2.0
# S3 creds for the encrypted off-box copy to Contabo Object Storage, passed # S3 creds for the encrypted off-box copy to Contabo Object Storage, passed
# into the remote script via `envs:` (the host env file isn't a reliable # into the remote script via `envs:` (a host env file isn't a reliable
# source across boxes -- beta's /etc/thermograph.env isn't readable by the # source across boxes -- on vps1 it isn't readable by the deploy user --
# deploy user -- so the CI secret is the uniform home, like PROD_SSH_*). # so the CI secret is the uniform home, like the VPS*_SSH_* pairs).
env: env:
S3_ENDPOINT: ${{ secrets.S3_ENDPOINT }} S3_ENDPOINT: ${{ secrets.S3_ENDPOINT }}
S3_BUCKET: ${{ secrets.S3_BUCKET }} S3_BUCKET: ${{ secrets.S3_BUCKET }}
S3_ACCESS_KEY: ${{ secrets.S3_ACCESS_KEY }} S3_ACCESS_KEY: ${{ secrets.S3_ACCESS_KEY }}
S3_SECRET_KEY: ${{ secrets.S3_SECRET_KEY }} S3_SECRET_KEY: ${{ secrets.S3_SECRET_KEY }}
with: with:
host: ${{ secrets.PROD_SSH_HOST }} host: ${{ secrets.VPS2_SSH_HOST }}
username: ${{ secrets.PROD_SSH_USER }} username: ${{ secrets.VPS2_SSH_USER }}
key: ${{ secrets.PROD_SSH_KEY }} key: ${{ secrets.VPS2_SSH_KEY }}
port: ${{ secrets.PROD_SSH_PORT }} port: ${{ secrets.VPS2_SSH_PORT }}
envs: S3_ENDPOINT,S3_BUCKET,S3_ACCESS_KEY,S3_SECRET_KEY envs: S3_ENDPOINT,S3_BUCKET,S3_ACCESS_KEY,S3_SECRET_KEY
script: | script: |
set -euo pipefail set -euo pipefail
@ -73,23 +86,45 @@ jobs:
backup_dir="$HOME/thermograph-backups" backup_dir="$HOME/thermograph-backups"
mkdir -p "$backup_dir" mkdir -p "$backup_dir"
stamp="$(date -u +%Y%m%dT%H%M%SZ)" stamp="$(date -u +%Y%m%dT%H%M%SZ)"
out="$backup_dir/thermograph-$stamp.dump" # The db may run under plain compose OR as a Swarm stack task;
# The db may run under plain compose OR as a Swarm stack task # resolve the container either way so the backup survives a
# (prod post-cutover); resolve the container either way so the # deploy-mode switch. One instance now serves both environments.
# backup survives the deploy-mode switch.
dbc=$(docker ps -q --filter "label=com.docker.swarm.service.name=thermograph_db" | head -1) dbc=$(docker ps -q --filter "label=com.docker.swarm.service.name=thermograph_db" | head -1)
[ -z "$dbc" ] && dbc=$(cd /opt/thermograph/infra && docker compose ps -q db 2>/dev/null | head -1) [ -z "$dbc" ] && dbc=$(cd /opt/thermograph/infra && docker compose ps -q db 2>/dev/null | head -1)
[ -n "$dbc" ] || { echo "!! no db container found (compose or stack)"; exit 1; } [ -n "$dbc" ] || { echo "!! no db container found (compose or stack)"; exit 1; }
# Write to a .partial and rename on success so a mid-dump failure can # env:database pairs. Both are dumped every night: they are separate
# never leave a truncated file that looks like a good backup. # databases on one instance, and backing up only one would leave the
docker exec "$dbc" pg_dump -U thermograph -d thermograph \ # other silently uncovered.
--format=custom > "$out.partial" for pair in prod:thermograph beta:thermograph_beta; do
mv "$out.partial" "$out" envname="${pair%%:*}"; dbname="${pair##*:}"
echo "wrote $out ($(du -h "$out" | cut -f1))" # A missing database is a HARD failure, never a quiet skip -- a
# backup job that shrugs off an absent database is exactly how an
# environment ends up with no backups and nobody noticing.
if ! docker exec "$dbc" psql -U thermograph -d postgres -tAc \
"select 1 from pg_database where datname='$dbname'" | grep -q 1; then
echo "!! database '$dbname' ($envname) does not exist on this instance."
if [ "$envname" = prod ]; then
echo "!! That is prod's own database on prod's own instance — this is not a"
echo "!! provisioning gap, something is badly wrong. Do not 'fix' it by creating"
echo "!! an empty database; find out where the real one went."
else
echo "!! provision it first: sudo bash /opt/thermograph-beta/infra/deploy/db/provision-env-db.sh $envname"
fi
exit 1
fi
out="$backup_dir/$dbname-$stamp.dump"
# Write to a .partial and rename on success so a mid-dump failure can
# never leave a truncated file that looks like a good backup.
docker exec "$dbc" pg_dump -U thermograph -d "$dbname" \
--format=custom > "$out.partial"
mv "$out.partial" "$out"
echo "wrote $out ($(du -h "$out" | cut -f1))"
dumps="${dumps:-} $envname:$out"
done
# The dumps are the disaster-recovery copy, not a versioned # The dumps are the disaster-recovery copy, not a versioned
# archive -- keep the last 14 days and let the rest age out. # archive -- keep the last 14 days and let the rest age out.
find "$backup_dir" -name 'thermograph-*.dump' -mtime +14 -delete find "$backup_dir" -name '*.dump' -mtime +14 -delete
find "$backup_dir" -name 'thermograph-*.dump.partial' -mtime +1 -delete find "$backup_dir" -name '*.dump.partial' -mtime +1 -delete
# --- off-box copy to Contabo Object Storage (age-encrypted) --- # --- off-box copy to Contabo Object Storage (age-encrypted) ---
# Streams the just-written dump through age (to the vault's age # Streams the just-written dump through age (to the vault's age
# recipient -- the same key each host renders secrets with, so # recipient -- the same key each host renders secrets with, so
@ -105,23 +140,33 @@ jobs:
RCLONE_CONFIG_ARCHIVE_SECRET_ACCESS_KEY="$S3_SECRET_KEY" \ RCLONE_CONFIG_ARCHIVE_SECRET_ACCESS_KEY="$S3_SECRET_KEY" \
RCLONE_CONFIG_ARCHIVE_REGION=default RCLONE_CONFIG_ARCHIVE_FORCE_PATH_STYLE=true RCLONE_CONFIG_ARCHIVE_REGION=default RCLONE_CONFIG_ARCHIVE_FORCE_PATH_STYLE=true
recip=age1xx4dzs0dxlwvkv9sjuqzsphl7lfrxannkfken374yu2qvvcte9sqzktqt2 recip=age1xx4dzs0dxlwvkv9sjuqzsphl7lfrxannkfken374yu2qvvcte9sqzktqt2
s3base="$S3_BUCKET/backups/db/prod" # One prefix per environment, so a restore never has to guess which
age -r "$recip" < "$out" | rclone rcat "archive:$s3base/$(basename "$out").age" # database a dump came from: backups/db/prod/ and backups/db/beta/.
echo "off-box: uploaded $(basename "$out").age to $s3base" for entry in $dumps; do
rclone delete --min-age 30d "archive:$s3base/" 2>/dev/null || true envname="${entry%%:*}"; f="${entry##*:}"
s3base="$S3_BUCKET/backups/db/$envname"
age -r "$recip" < "$f" | rclone rcat "archive:$s3base/$(basename "$f").age"
echo "off-box: uploaded $(basename "$f").age to $s3base"
rclone delete --min-age 30d "archive:$s3base/" 2>/dev/null || true
done
else else
echo "!! S3_* secrets unset -- skipped off-box push (add them as repo secrets)" echo "!! S3_* secrets unset -- skipped off-box push (add them as repo secrets)"
fi fi
forgejo-backup: forgejo-backup:
name: Forgejo backup (beta) -> S3 name: Forgejo backup (vps1) -> S3
runs-on: docker runs-on: docker
# Forgejo (the git host + all CI history) lives on beta and had NO backup at # Forgejo (the git host + all CI history) lives on vps1 and had NO backup at
# all. Dumps its Postgres db (consistent) + tars its data volume (repos, LFS, # all. Dumps its Postgres db (consistent) + tars its data volume (repos, LFS,
# config, avatars), age-encrypts both, pushes off-box, keeps 30 days. Runs as # config, avatars), age-encrypts both, pushes off-box, keeps 30 days. Runs as
# the beta SSH user (SSH_*), which is in the docker group -- so docker exec/run # vps1's SSH user (VPS1_SSH_*), which is in the docker group -- so docker
# need no host sudo, and the data tar runs inside a throwaway container (the # exec/run need no host sudo, and the data tar runs inside a throwaway
# volume dir is root-owned on the host). rclone + age must be present on beta. # container (the volume dir is root-owned on the host). rclone + age must be
# present on vps1.
#
# This job follows FORGEJO, not beta. It used to use the same secret as the
# beta deploy purely because Forgejo and beta shared a box; beta has since
# moved to vps2 and Forgejo has not moved at all.
concurrency: concurrency:
group: ops-forgejo-backup group: ops-forgejo-backup
cancel-in-progress: false cancel-in-progress: false
@ -134,16 +179,16 @@ jobs:
S3_ACCESS_KEY: ${{ secrets.S3_ACCESS_KEY }} S3_ACCESS_KEY: ${{ secrets.S3_ACCESS_KEY }}
S3_SECRET_KEY: ${{ secrets.S3_SECRET_KEY }} S3_SECRET_KEY: ${{ secrets.S3_SECRET_KEY }}
with: with:
host: ${{ secrets.SSH_HOST }} host: ${{ secrets.VPS1_SSH_HOST }}
username: ${{ secrets.SSH_USER }} username: ${{ secrets.VPS1_SSH_USER }}
key: ${{ secrets.SSH_KEY }} key: ${{ secrets.VPS1_SSH_KEY }}
port: ${{ secrets.SSH_PORT }} port: ${{ secrets.VPS1_SSH_PORT }}
envs: S3_ENDPOINT,S3_BUCKET,S3_ACCESS_KEY,S3_SECRET_KEY envs: S3_ENDPOINT,S3_BUCKET,S3_ACCESS_KEY,S3_SECRET_KEY
script: | script: |
set -euo pipefail set -euo pipefail
[ -n "${S3_ACCESS_KEY:-}" ] || { echo "!! S3_* secrets unset"; exit 1; } [ -n "${S3_ACCESS_KEY:-}" ] || { echo "!! S3_* secrets unset"; exit 1; }
command -v rclone >/dev/null 2>&1 || { echo "!! rclone missing on beta"; exit 1; } command -v rclone >/dev/null 2>&1 || { echo "!! rclone missing on vps1"; exit 1; }
command -v age >/dev/null 2>&1 || { echo "!! age missing on beta"; exit 1; } command -v age >/dev/null 2>&1 || { echo "!! age missing on vps1"; exit 1; }
export RCLONE_CONFIG_ARCHIVE_TYPE=s3 RCLONE_CONFIG_ARCHIVE_PROVIDER=Other \ export RCLONE_CONFIG_ARCHIVE_TYPE=s3 RCLONE_CONFIG_ARCHIVE_PROVIDER=Other \
RCLONE_CONFIG_ARCHIVE_ENDPOINT="$S3_ENDPOINT" \ RCLONE_CONFIG_ARCHIVE_ENDPOINT="$S3_ENDPOINT" \
RCLONE_CONFIG_ARCHIVE_ACCESS_KEY_ID="$S3_ACCESS_KEY" \ RCLONE_CONFIG_ARCHIVE_ACCESS_KEY_ID="$S3_ACCESS_KEY" \
@ -177,10 +222,10 @@ jobs:
- name: Ping IndexNow if the URL set changed - name: Ping IndexNow if the URL set changed
uses: https://github.com/appleboy/ssh-action@v1.2.0 uses: https://github.com/appleboy/ssh-action@v1.2.0
with: with:
host: ${{ secrets.PROD_SSH_HOST }} host: ${{ secrets.VPS2_SSH_HOST }}
username: ${{ secrets.PROD_SSH_USER }} username: ${{ secrets.VPS2_SSH_USER }}
key: ${{ secrets.PROD_SSH_KEY }} key: ${{ secrets.VPS2_SSH_KEY }}
port: ${{ secrets.PROD_SSH_PORT }} port: ${{ secrets.VPS2_SSH_PORT }}
script: | script: |
set -euo pipefail set -euo pipefail
cd /opt/thermograph/infra cd /opt/thermograph/infra

60
CLAUDE.md
View file

@ -15,24 +15,30 @@ every change, so a stale one is a correctness bug, not a documentation bug.
| Branch | Deploys to | Workflow | | Branch | Deploys to | Workflow |
|---|---|---| |---|---|---|
| feature branch | nothing | PR into `dev` | | feature branch | nothing | PR into `dev` |
| `dev` | nothing | integration branch only — see below | | `dev` | dev (vps1, mesh-only, own Postgres) | `deploy.yml` |
| `main` | beta (beta.thermograph.org) | `deploy.yml` | | `main` | beta (beta.thermograph.org, vps2) | `deploy.yml` |
| `release` | prod (thermograph.org) | `deploy.yml` | | `release` | prod (thermograph.org, vps2) | `deploy.yml` |
`dev`, `main` and `release` are protected: **everything is a PR**, for humans and `dev`, `main` and `release` are protected: **everything is a PR**, for humans and
agents alike. Promotion is one PR per hop, `dev` → `main` → `release`. agents alike. Promotion is one PR per hop, `dev` → `main` → `release`.
**`dev` deploys nowhere.** It is purely the integration branch that feature PRs **`dev` is a first-class hosted environment, not just an integration branch.**
land on before promotion to `main`. The two LAN-dev deploy workflows were deleted It runs on vps1 — the same box as Forgejo and the monitoring stack — with its
rather than kept: they called a monorepo path that does not exist on the LAN box own Postgres container, reachable only on the WireGuard mesh
(`~/thermograph-dev` is still a split-era `thermograph-infra` checkout), so they (`10.10.0.2:8137`): no public DNS record, no Caddy site, no TLS. It is deployed
had been inert since cutover. Run LAN dev locally with `infra/`'s `make dev-up`. by CI like beta and prod. The desktop hosts no Thermograph environment at all;
`make dev-up` there is a laptop convenience for running the stack locally, not
a deployment target.
**One workflow deploys everything.** `deploy.yml` handles both services and both **One workflow deploys everything.** `deploy.yml` handles both services and all
environments: the branch selects the environment (`main` → beta, `release` → three environments: the branch selects the environment (`dev` → vps1, `main` →
prod), a matrix covers backend and frontend, and each leg checks whether this beta on vps2, `release` → prod on vps2), a matrix covers backend and frontend,
push actually touched its domain before rolling. `build-push.yml` is the same and each leg checks whether this push actually touched its domain before
shape for images. They replaced six and two near-identical files respectively. rolling. `build-push.yml` is the same shape for images. `THERMOGRAPH_ENV` is the
input that tells a leg which environment it's deploying — load-bearing on vps2,
which runs beta and prod side by side and has no other way to tell them apart.
See `infra/deploy/env-topology.sh` for the single source of truth on where each
environment's checkout, branch, stack and ports live.
## The deploy contract ## The deploy contract
@ -44,15 +50,25 @@ SERVICE=backend|frontend|all BACKEND_IMAGE_TAG=sha-<12hex> FRONTEND_IMAGE_TAG=
``` ```
`deploy.sh` resets the host checkout, renders secrets from the SOPS vault, then `deploy.sh` resets the host checkout, renders secrets from the SOPS vault, then
either rolls compose services or — if `/etc/thermograph/deploy-mode` contains either rolls compose services or — if `infra/deploy/env-topology.sh` says the
`stack` — execs `infra/deploy/stack/deploy-stack.sh`. environment's deploy mode is `stack` — execs `infra/deploy/stack/deploy-stack.sh`.
The old host-wide `/etc/thermograph/deploy-mode` marker survives only as a
fallback for a by-hand run with no environment resolvable any other way; it
cannot describe vps2 alone, since vps2 runs beta and prod side by side.
- **prod runs Swarm.** Its stack is `infra/deploy/stack/thermograph-stack.yml` - **prod and beta both run Swarm, as two separate stacks co-resident on vps2.**
(db, web, worker, lake, daemon, frontend, autoscaler, autoscaler-lake). Prod's is `infra/deploy/stack/thermograph-stack.yml` (db, web, worker, lake,
`deploy-stack.sh` also offers `STACK_TEST=1`: a full parallel rehearsal on daemon, frontend, autoscaler, autoscaler-lake). Beta's is the separate
throwaway volumes and ports that cannot touch live data. `infra/deploy/stack/thermograph-beta-stack.yml` — its services are prefixed
- **beta and LAN dev run compose**, from `infra/docker-compose.yml` (`beta-web`, `beta-worker`, …) because Swarm registers a service's short name
(db, backend, lake, daemon, frontend). as a DNS alias on every network it joins, and beta shares prod's `data`
network to reach the database. Beta has no `db` service of its own: one
TimescaleDB instance serves both, on separate databases and separate
NOSUPERUSER roles. `deploy-stack.sh` also offers `STACK_TEST=1`: a full
parallel rehearsal on throwaway volumes and ports that cannot touch live data.
- **dev runs compose**, from `infra/docker-compose.yml` (db, backend, lake,
daemon, frontend), on its own host (vps1) with its own Postgres container —
the one environment not sharing a database with anything else.
- Each service's live tag is persisted host-side, so a single-service roll never - Each service's live tag is persisted host-side, so a single-service roll never
disturbs the sibling's running tag. disturbs the sibling's running tag.
@ -76,7 +92,7 @@ reunification.
- Anything named "prefetch" must never spend the Open-Meteo quota. - Anything named "prefetch" must never spend the Open-Meteo quota.
Nominatim ≤ 1 req/s. Nominatim ≤ 1 req/s.
- The compose project name is pinned (`name: thermograph` in - The compose project name is pinned (`name: thermograph` in
`infra/docker-compose.yml`); LAN dev overrides with `infra/docker-compose.yml`); dev (and a local `make dev-up`) overrides with
`COMPOSE_PROJECT_NAME=thermograph-dev`. Don't remove either half — the pinned `COMPOSE_PROJECT_NAME=thermograph-dev`. Don't remove either half — the pinned
name is what makes the Swarm stack's external volume names line up. name is what makes the Swarm stack's external volume names line up.
- `CUTOVER-NOTES.md` is the source of truth for what is and isn't live yet. - `CUTOVER-NOTES.md` is the source of truth for what is and isn't live yet.

35
CUTOVER-NOTES.md
View file

@ -153,3 +153,38 @@ Checked every split-repo branch by dry-run merge; migrated everything real:
historical. historical.
- Infra `Makefile` local-image targets still assume sibling checkouts — update - Infra `Makefile` local-image targets still assume sibling checkouts — update
when first needed locally. when first needed locally.
## vps1/vps2 host-topology re-architecture (landed 2026-07-25, NOT YET EXECUTED)
**This section documents a second re-architecture, layered on top of the
monorepo cutover above. It is landed IN THE REPO — code, stack files, deploy
scripts, docs — but has NOT been run against the live estate.** The boxes
described elsewhere in this file (prod at `169.58.46.181`, beta at
`75.119.132.91`, a desktop LAN dev box) are still what is actually running
today. Do not treat anything below as live until the cutover runbook has been
executed and this note updated.
What changes, in one line: hosts are renamed by **role** instead of by
environment, and beta moves onto the same box as prod.
- **vps1** (mesh `10.10.0.2`, public `75.119.132.91`) — unchanged box, new job.
Keeps Forgejo, Grafana/Loki/Alloy and the `emigriffith.dev` portfolio; gains
the **dev** environment (its own Postgres, plain compose, mesh-only —
`10.10.0.2:8137`, no DNS record, no Caddy site, no TLS). This is the box that
used to be called "beta".
- **vps2** (mesh `10.10.0.1`, public `169.58.46.181`) — unchanged box, new job.
Keeps prod; gains **beta** as a second, co-resident Docker Swarm stack
(prefixed services, own checkout, own env file, own loopback LB ports). One
TimescaleDB instance now serves both prod and beta, on separate databases and
separate roles. This is the box that used to be called "prod".
- **desktop** (mesh `10.10.0.3`) — loses the LAN dev stack and the Forgejo
Actions runner entirely; becomes AI-model hosting plus flex Swarm capacity.
`make dev-up` still works there as a laptop convenience, but it is no longer
an environment.
The mesh IPs did not move — only which environment lives on which box did. See
`infra/deploy/env-topology.sh` for the single source of truth on per-environment
checkout paths, branches, stack names, ports and DB roles, and
**[`infra/deploy/RUNBOOK-vps1-vps2-cutover.md`](infra/deploy/RUNBOOK-vps1-vps2-cutover.md)**
for the execution runbook. Until that runbook has been run, `CLAUDE.md`,
`README.md` and `docs/onboarding/` describe the target state, not today's.

7
README.md
View file

@ -10,7 +10,7 @@ for: **per-domain images, per-domain deploys, and an async FE/BE contract**.
|---|---|---| |---|---|---|
| `backend/` | FastAPI graded-climate API, accounts, notifications (Discord bot, push, mail), data pipeline | `build-push` → image `emi/thermograph/backend`; `deploy` | | `backend/` | FastAPI graded-climate API, accounts, notifications (Discord bot, push, mail), data pipeline | `build-push` → image `emi/thermograph/backend`; `deploy` |
| `frontend/` | Public client: static JS/CSS + SSR pages | same `build-push` / `deploy` workflows, matrixed by domain; image `emi/thermograph/frontend` | | `frontend/` | Public client: static JS/CSS + SSR pages | same `build-push` / `deploy` workflows, matrixed by domain; image `emi/thermograph/frontend` |
| `infra/` | Compose (beta, LAN dev) + the Swarm stack (prod), deploy scripts, terraform, SOPS secrets vault, ops cron | `infra-sync` (host checkout + secrets render), `secrets-guard`, `ops-cron` | | `infra/` | Compose (dev, on vps1) + two Swarm stacks co-resident on vps2 (beta, prod), deploy scripts, terraform, SOPS secrets vault, ops cron | `infra-sync` (host checkout + secrets render), `secrets-guard`, `ops-cron` |
| `observability/` | Loki + Grafana + Alloy stack | `observability-validate` | | `observability/` | Loki + Grafana + Alloy stack | `observability-validate` |
`thermograph-docs` deliberately **stays its own repo** (ADRs + runbooks, no `thermograph-docs` deliberately **stays its own repo** (ADRs + runbooks, no
@ -36,6 +36,9 @@ The one intentionally *coupled* piece is `pr-build.yml`: a single always-running
path-filtered required check would deadlock auto-merge). path-filtered required check would deadlock auto-merge).
Branch model (unchanged from the split era): PRs → `dev`, `main` → beta, Branch model (unchanged from the split era): PRs → `dev`, `main` → beta,
`release` → prod; infra tracked via `main` on all hosts. `release` → prod. Infra isn't environment-staged the same way app images are:
beta's and prod's checkouts (both on vps2) track `main`; dev's checkout (on
vps1) tracks `dev` itself, since it's the one environment that isn't a
rehearsal for something downstream.
**Before pointing anything live at this repo, read `CUTOVER-NOTES.md`.** **Before pointing anything live at this repo, read `CUTOVER-NOTES.md`.**

51
docs/onboarding/01-orientation.md
View file

@ -55,40 +55,59 @@ and `PAYLOAD_VER` is the lever that moves it.** See
## The estate ## The estate
Four machines on a WireGuard mesh (`10.10.0.0/24`): Hosts are named by **role**, not by environment, plus the operator's desktop —
all on a WireGuard mesh (`10.10.0.0/24`):
| Host | Mesh IP | Public | Runs | | Host | Mesh IP | Public | Runs |
|---|---|---|---| |---|---|---|---|
| **prod** | `10.10.0.1` | `169.58.46.181`, thermograph.org | Docker **Swarm** app stack, TimescaleDB, Postfix, backups, Centralis | | **vps1** | `10.10.0.2` | `75.119.132.91`, git.thermograph.org, dashboard.thermograph.org | Forgejo (git + CI + registry), Grafana + Loki + Alloy, the `emigriffith.dev` portfolio, and **dev** — its own Postgres, plain compose, **mesh-only** (no public DNS, no Caddy site, no TLS) |
| **beta** | `10.10.0.2` | `75.119.132.91`, beta.thermograph.org | **compose** app stack, Forgejo (git + CI + registry), Grafana + Loki | | **vps2** | `10.10.0.1` | `169.58.46.181`, thermograph.org, beta.thermograph.org | **prod** and **beta** as two separate Docker **Swarm** stacks, Centralis, Postfix, backups |
| **desktop** | `10.10.0.3` | — | LAN dev stack, the Forgejo Actions runner, your editor | | **desktop** | `10.10.0.3` | — | AI-model hosting (voice-to-text, an upcoming-feature LLM) + flex Swarm capacity. Hosts **no** Thermograph environment — `make dev-up` there is a laptop convenience only |
| phone | — | — | alerts | | phone | — | — | alerts |
vps2 runs prod and beta **co-resident on one box**: one TimescaleDB instance
serves both, on separate databases and separate roles (`thermograph` /
`thermograph_beta`, the latter `NOSUPERUSER`/`NOCREATEDB`, `CONNECT` revoked
from `PUBLIC`); separate checkouts, env files and loopback LB ports; beta's
Swarm service names are prefixed (`beta-web`, not `web`) because Swarm
registers a service's short name as a DNS alias on every network it joins, and
beta shares prod's network to reach the database. Beta rehearses prod's actual
orchestrator this way — that's the point of putting it next to prod rather
than next to dev.
The mesh IPs did not move when this topology landed — vps1 is the box that used
to be called "beta", vps2 the one that used to be called "prod". What moved is
which environment lives where. `infra/deploy/env-topology.sh` is the single
source of truth for every per-environment path, port and role; `THERMOGRAPH_ENV`
(`dev`/`beta`/`prod`) is the input a deploy leg passes, since vps2 can no longer
tell beta and prod apart by which host it's running on.
Two consequences you will hit within the first week: Two consequences you will hit within the first week:
- **Forgejo is mesh-only.** `git.thermograph.org` resolves publicly to beta's - **Forgejo is mesh-only.** `git.thermograph.org` resolves publicly to vps1's
IP, but beta's Caddy rejects `/v2/*` (the registry API) from outside the mesh. IP, but vps1's Caddy rejects `/v2/*` (the registry API) from outside the
Any host that pulls images needs `10.10.0.2 git.thermograph.org` in mesh. Any host that pulls images needs `10.10.0.2 git.thermograph.org` in
`/etc/hosts`. `/etc/hosts`.
- **Prod and beta run different orchestrators.** Prod is Swarm - **Beta and prod run the same orchestrator now — Swarm, as two stacks on one
(`infra/deploy/stack/`), beta and LAN dev are compose box.** Dev is the only environment on compose (`infra/docker-compose.yml`),
(`infra/docker-compose.yml`). `deploy.sh` routes between them by reading and it lives alone on vps1. Most of the time you don't care which stack
`/etc/thermograph/deploy-mode` on the box. Most of the time you don't care — you're reading — but when you're reading logs or naming containers, you do
but when you're reading logs or naming containers, you do (see [observability](09-observability.md)), and on vps2 you additionally have
(see [observability](09-observability.md)). to say *which* environment: `deploy.sh` there routes on the explicit
`THERMOGRAPH_ENV` a caller passes, not on which host it's running on.
## How a change travels ## How a change travels
``` ```
PR ──(required check: `gate`)──▶ dev ──▶ LAN dev box PR ──(required check: `gate`)──▶ dev ──▶ dev (vps1, mesh-only)
│ │
promotion PR promotion PR
▼ ▼
main ──▶ beta.thermograph.org main ──▶ beta.thermograph.org (vps2)
│ │
promotion PR ← the owner's call promotion PR ← the owner's call
▼ ▼
release ──▶ thermograph.org release ──▶ thermograph.org (vps2)
``` ```
`dev`, `main` and `release` are **protected** — no direct pushes, for humans or `dev`, `main` and `release` are **protected** — no direct pushes, for humans or

13
docs/onboarding/02-setup.md
View file

@ -172,11 +172,18 @@ ends fail closed. With Discord unconfigured it logs once and runs cron-only.
```bash ```bash
cd infra cd infra
make dev-up # docker-compose.yml + docker-compose.dev.yml overlay: make dev-up # docker-compose.yml + docker-compose.dev.yml overlay:
# uncapped CPU, backend published on 0.0.0.0:8137 for the LAN # uncapped CPU, backend published on 0.0.0.0:8137 for your LAN
make dev-down make dev-down
``` ```
The dev overlay exports `COMPOSE_PROJECT_NAME=thermograph-dev` so the LAN stack This is a **laptop convenience**, not the hosted dev environment: the actual
`dev` environment now runs on vps1 with its own Postgres, deployed by CI, and
bound only to the WireGuard mesh (`10.10.0.2:8137`) — never `0.0.0.0`, since
vps1 is a public box. See [Infra and secrets](08-infra-secrets.md). A local
`make dev-up` is fine on `0.0.0.0` because it's your own machine's LAN, not the
public internet.
The dev overlay exports `COMPOSE_PROJECT_NAME=thermograph-dev` so this stack
keeps volumes separate from anything else. **Do not remove either half of the keeps volumes separate from anything else. **Do not remove either half of the
project-name pinning** — `infra/docker-compose.yml` pins `name: thermograph`, project-name pinning** — `infra/docker-compose.yml` pins `name: thermograph`,
and without it running compose from `infra/` derives the project name `infra`, and without it running compose from `infra/` derives the project name `infra`,
@ -216,7 +223,7 @@ it will be fresher than this page.
| Hook | When | What | | Hook | When | What |
|---|---|---| |---|---|---|
| `prod-guard.sh` | before Bash / live-host MCP calls | Classifies by **allowlist**: only positively-recognised read-only commands pass; everything else asks. Beta is guarded as strictly as prod — it hosts Forgejo, Grafana *and* beta.thermograph.org. | | `prod-guard.sh` | before Bash / live-host MCP calls | Classifies by **allowlist**: only positively-recognised read-only commands pass; everything else asks. vps1 is guarded as strictly as vps2 — it hosts Forgejo, Grafana *and* the mesh-only `dev` environment, so a destructive command there takes out git, CI and the registry at once, not just a dev sandbox. |
| `secrets-guard.sh` | before Write/Edit | Denies any direct write to `infra/deploy/secrets/*.yaml`. Use `sops edit`. | | `secrets-guard.sh` | before Write/Edit | Denies any direct write to `infra/deploy/secrets/*.yaml`. Use `sops edit`. |
| `lint-after-edit.sh` | after Write/Edit | shellchecks an edited `*.sh` and feeds findings straight back. Exits quietly if shellcheck is missing. | | `lint-after-edit.sh` | after Write/Edit | shellchecks an edited `*.sh` and feeds findings straight back. Exits quietly if shellcheck is missing. |

46
docs/onboarding/03-repo-map.md
View file

@ -114,24 +114,36 @@ Deep dive: [05-frontend.md](05-frontend.md).
## `infra/` — how and where images run ## `infra/` — how and where images run
``` ```
docker-compose.yml the compose stack (beta + LAN dev): docker-compose.yml the compose stack — dev's, on vps1 (and a local
db, backend, lake, daemon, frontend. `make dev-up`): db, backend, lake, daemon, frontend.
`name: thermograph` is PINNED — see below. `name: thermograph` is PINNED — see below.
docker-compose.dev.yml LAN overlay: uncapped, backend on 0.0.0.0:8137 docker-compose.dev.yml dev overlay: uncapped, backend bound to
`${DEV_BIND_ADDR:-127.0.0.1}` — vps1's mesh IP
(10.10.0.2) at deploy time, never 0.0.0.0
docker-compose.openmeteo.yml self-hosted Open-Meteo overlay (prod-only) docker-compose.openmeteo.yml self-hosted Open-Meteo overlay (prod-only)
Makefile compose orchestration only Makefile compose orchestration only
deploy/ deploy/
deploy.sh ★ the single entry point for beta and prod env-topology.sh ★ single source of truth: env → host, checkout,
render-secrets.sh renders /etc/thermograph.env from the SOPS vault branch, deploy mode, stack name, env file, LB
ports, DB role/name, service prefix
deploy.sh ★ the single entry point for dev, beta and prod
deploy-dev.sh thin wrapper: routes deploy.sh at dev's compose
overlay and secrets policy
render-secrets.sh renders the env file for a given environment from
the SOPS vault
secrets/ the vault: common.yaml, prod/beta/dev.yaml, secrets/ the vault: common.yaml, prod/beta/dev.yaml,
centralis.prod.yaml, example.yaml centralis.prod.yaml, example.yaml
stack/ the SWARM path (live on prod): stack/ the SWARM path — both live on vps2:
thermograph-stack.yml, deploy-stack.sh, thermograph-stack.yml (prod),
autoscale.sh, the LB thermograph-beta-stack.yml (beta, prefixed
services, no `db` of its own), deploy-stack.sh,
autoscale.sh, lb/ (per-environment Caddyfiles)
db/provision-env-db.sh creates each environment's database + role on the
shared TimescaleDB instance
provision-dev.sh one-time bootstrap of the dev environment on vps1
swarm/, forgejo/ mesh + Forgejo/CI-runner provisioning swarm/, forgejo/ mesh + Forgejo/CI-runner provisioning
Caddyfile the reverse proxy (path-splits FE vs BE) Caddyfile the reverse proxy (path-splits FE vs BE)
provision-*.sh host bootstrap: agent access, dev LAN, mail, secrets provision-*.sh host bootstrap: agent access, mail, secrets
db/init/ TimescaleDB init + tuning
migrations/ hand-run SQL migrations migrations/ hand-run SQL migrations
ops/ ops/
dbq.sh read-only psql into any environment's db container dbq.sh read-only psql into any environment's db container
@ -141,26 +153,26 @@ ACCESS.md, DEPLOY.md, DEPLOY-DEV.md (partly stale — see traps)
``` ```
**Why the project name is pinned.** Compose creates volumes **Why the project name is pinned.** Compose creates volumes
`thermograph_pgdata` / `_appdata` / `_applogs`, and the Swarm stack declares `thermograph_pgdata` / `_appdata` / `_applogs`, and the Swarm stacks declare
those exact names as `external: true` at the same mount paths. Running compose those exact names as `external: true` at the same mount paths. Running compose
from `infra/` without `name: thermograph` derives project `infra`, which makes a from `infra/` without `name: thermograph` derives project `infra`, which makes a
whole new stack with empty volumes next to the running one. `deploy-dev.sh` whole new stack with empty volumes next to the running one. `deploy-dev.sh`
exports `COMPOSE_PROJECT_NAME=thermograph-dev` (env wins over the file key) to exports `COMPOSE_PROJECT_NAME=thermograph-dev` (env wins over the file key) to
keep LAN dev separate on purpose. Keep both halves. keep dev separate on purpose. Keep both halves.
Deep dive: [08-infra-secrets.md](08-infra-secrets.md). Deep dive: [08-infra-secrets.md](08-infra-secrets.md).
## `observability/` — the logging stack ## `observability/` — the logging stack
``` ```
docker-compose.yml Loki + Grafana (runs on beta) docker-compose.yml Loki + Grafana (runs on vps1)
loki/config.yml mesh-only, filesystem storage, 30-day retention loki/config.yml mesh-only, filesystem storage, 30-day retention
grafana/provisioning/ datasource + dashboard provider (auto-loaded) grafana/provisioning/ datasource + dashboard provider (auto-loaded)
grafana/provisioning/alerting/ rules, the Discord contact point, the policy grafana/provisioning/alerting/ rules, the Discord contact point, the policy
grafana/dashboards/*.json the fleet-logs dashboard grafana/dashboards/*.json the fleet-logs dashboard
alloy/config.alloy the per-node shipper alloy/config.alloy the per-node shipper
alloy/docker-compose.agent.yml runs Alloy on a node (ALLOY_NODE per host) alloy/docker-compose.agent.yml runs Alloy on a node (ALLOY_NODE per host)
caddy-grafana.conf beta's Grafana vhost, kept for reference caddy-grafana.conf vps1's Grafana vhost, kept for reference
``` ```
No build, no deploy automation — it ships by hand. **The repo is the only No build, no deploy automation — it ships by hand. **The repo is the only
@ -177,10 +189,10 @@ Deep dive: [09-observability.md](09-observability.md).
| `pr-build.yml` | PR → `dev`/`main` | The `gate` required check. Diffs the PR, builds only touched domains. Deliberately **not** path-filtered. | | `pr-build.yml` | PR → `dev`/`main` | The `gate` required check. Diffs the PR, builds only touched domains. Deliberately **not** path-filtered. |
| `build.yml` | `workflow_call` | Build one domain's image; run the backend suite **inside** the built image. | | `build.yml` | `workflow_call` | Build one domain's image; run the backend suite **inside** the built image. |
| `build-push.yml` | push to `dev`/`main`/`release` touching an app domain, or a `v*.*.*` tag | Builds + pushes `sha-<12hex>` images. A version tag builds **both**. | | `build-push.yml` | push to `dev`/`main`/`release` touching an app domain, or a `v*.*.*` tag | Builds + pushes `sha-<12hex>` images. A version tag builds **both**. |
| `deploy.yml` | push to `main`/`release` touching an app domain | Branch selects environment; matrix covers services; SSHes and runs `deploy.sh`. | | `deploy.yml` | push to `dev`/`main`/`release` touching an app domain | Branch selects environment — three legs: `dev`→vps1, `main`→beta on vps2, `release`→prod on vps2; matrix covers services; SSHes and runs `deploy.sh` with `THERMOGRAPH_ENV` set. |
| `infra-sync.yml` | push to `main` touching `infra/**` | Fast-forwards each host's checkout and re-renders secrets. Rolls **no** service. | | `infra-sync.yml` | push to `main` touching `infra/**` | Fast-forwards beta's and prod's checkouts (both on vps2) and re-renders their secrets. Rolls **no** service. |
| `observability-validate.yml` | push touching `observability/**`, or `workflow_call` | Parses every artifact; validates Alloy config with the pinned binary; strict alerting checks. | | `observability-validate.yml` | push touching `observability/**`, or `workflow_call` | Parses every artifact; validates Alloy config with the pinned binary; strict alerting checks. |
| `ops-cron.yml` | daily 03:00 UTC | **THE prod backup** (pg_dump) + IndexNow. Uses `PROD_SSH_*`. | | `ops-cron.yml` | daily 03:00 UTC | **THE prod backup** (pg_dump) + IndexNow, against prod on vps2. Uses `VPS2_SSH_*`. |
| `secrets-guard.yml` | every PR and push | Fails if any `infra/deploy/secrets/*.yaml` isn't SOPS-encrypted. | | `secrets-guard.yml` | every PR and push | Fails if any `infra/deploy/secrets/*.yaml` isn't SOPS-encrypted. |
| `shell-lint.yml` | every PR and push | shellcheck (pinned v0.11.0 + sha256) over every `*.sh`. | | `shell-lint.yml` | every PR and push | shellcheck (pinned v0.11.0 + sha256) over every `*.sh`. |

7
docs/onboarding/04-backend.md
View file

@ -17,7 +17,7 @@ them is impossible.
| **web** | `THERMOGRAPH_ROLE=web` | Serves requests. Never starts the notifier, even if it would win the election — so the web tier scales to N replicas without also scaling background sweeps. | | **web** | `THERMOGRAPH_ROLE=web` | Serves requests. Never starts the notifier, even if it would win the election — so the web tier scales to N replicas without also scaling background sweeps. |
| **worker** | `THERMOGRAPH_ROLE=worker` | Owns the subscription notifier. Still serves requests today. | | **worker** | `THERMOGRAPH_ROLE=worker` | Owns the subscription notifier. Still serves requests today. |
| **all** | default | Both — the single-process dev default. | | **all** | default | Both — the single-process dev default. |
| **lake** | `THERMOGRAPH_ROLE=lake` | `deploy/entrypoint.sh` execs `uvicorn lake_app:app` on port 8141 instead. No database, no migrations. Prod only. | | **lake** | `THERMOGRAPH_ROLE=lake` | `deploy/entrypoint.sh` execs `uvicorn lake_app:app` on port 8141 instead. No database, no migrations. Runs in every environment — its own Swarm service on prod and beta (`lake` / `beta-lake`), a compose service on dev. |
| **daemon** | compose/Swarm sets the command to `/usr/local/bin/thermograph-daemon` | The Go binary: Discord gateway + cron timers. | | **daemon** | compose/Swarm sets the command to `/usr/local/bin/thermograph-daemon` | The Go binary: Discord gateway + cron timers. |
`deploy/entrypoint.sh` runs `alembic upgrade head` (retried — a fresh Postgres `deploy/entrypoint.sh` runs `alembic upgrade head` (retried — a fresh Postgres
@ -169,7 +169,7 @@ warming, also warm-only.
**The catch-all proxy** — the frontend owns every page and asset. In prod and **The catch-all proxy** — the frontend owns every page and asset. In prod and
beta, Caddy path-splits directly, so this proxy is never exercised. It exists as beta, Caddy path-splits directly, so this proxy is never exercised. It exists as
the fallback for environments with no Caddy in front (LAN dev, bare-metal). It the fallback for environments with no Caddy in front (dev, bare-metal). It
forwards `X-Forwarded-Host`/`-Proto` so the frontend can build correct absolute forwards `X-Forwarded-Host`/`-Proto` so the frontend can build correct absolute
URLs instead of resolving to the internal hop's own address. Because it's URLs instead of resolving to the internal hop's own address. Because it's
registered dead last, every real backend route wins first — registered dead last, every real backend route wins first —
@ -236,7 +236,8 @@ never overlaps a job with itself, and drops ticks that fire mid-run.
## The lake (`lake_app.py` + `data/era5lake.py`) ## The lake (`lake_app.py` + `data/era5lake.py`)
Prod-only Swarm service, same image, `THERMOGRAPH_ROLE=lake`. Runs as a Swarm service on both prod and beta (`lake` / `beta-lake`, same
image, `THERMOGRAPH_ROLE=lake`), and as a compose service on dev.
The lake is parquet in an S3-compatible bucket (Contabo, `era5-thermograph`), The lake is parquet in an S3-compatible bucket (Contabo, `era5-thermograph`),
extracted once from the public Earthmover ERA5 Icechunk archive by extracted once from the public Earthmover ERA5 Icechunk archive by

2
docs/onboarding/05-frontend.md
View file

@ -80,7 +80,7 @@ The static mount is registered last so the explicit routes win.
The shell HTML files in `static/` carry an `__ORIGIN__` placeholder for the The shell HTML files in `static/` carry an `__ORIGIN__` placeholder for the
link-preview/Open Graph tags — preview crawlers need absolute URLs and the host link-preview/Open Graph tags — preview crawlers need absolute URLs and the host
differs between LAN and prod. The substituted HTML and its ETag are **memoized differs between dev and prod. The substituted HTML and its ETag are **memoized
per origin**, not recomputed per request. per origin**, not recomputed per request.
Origin resolution prefers `X-Forwarded-Host` over `Host`. That matters when the Origin resolution prefers `X-Forwarded-Host` over `Host`. That matters when the

85
docs/onboarding/07-ci-and-release.md
View file

@ -3,15 +3,15 @@
## The branch model ## The branch model
``` ```
PR ──(required check: `gate`)──▶ dev ──▶ LAN dev box PR ──(required check: `gate`)──▶ dev ──▶ dev (vps1, mesh-only)
│ │
promotion PR promotion PR
▼ ▼
main ──▶ beta.thermograph.org main ──▶ beta.thermograph.org (vps2)
│ │
promotion PR ← the owner's call, always promotion PR ← the owner's call, always
▼ ▼
release ──▶ thermograph.org release ──▶ thermograph.org (vps2)
``` ```
`dev`, `main` and `release` are protected — direct pushes are blocked for `dev`, `main` and `release` are protected — direct pushes are blocked for
@ -40,9 +40,10 @@ for the separate hotfix path.
**Commit counts are not deliverable content.** Before promoting, compare the two **Commit counts are not deliverable content.** Before promoting, compare the two
tips' *trees*. Identical trees mean the same changes are already on the target tips' *trees*. Identical trees mean the same changes are already on the target
under different SHAs — promoting there merges nothing while still firing the under different SHAs — promoting there merges nothing while still firing the
target's deploy workflows, including `infra-sync` re-rendering target's deploy workflows, including `infra-sync` re-rendering prod's and
`/etc/thermograph.env` on prod and beta. The ahead/behind numbers will look like beta's env files (`/etc/thermograph.env` and `/etc/thermograph-beta.env`, both
real work. `promote` refuses this outright. on vps2). The ahead/behind numbers will look like real work. `promote` refuses
this outright.
**Expect `dev`, `main` and `release` to be mutually divergent, and expect **Expect `dev`, `main` and `release` to be mutually divergent, and expect
`--ff-only` to fail.** Every promotion merges the source into the target, `--ff-only` to fail.** Every promotion merges the source into the target,
@ -111,16 +112,17 @@ Triggers on pushes to `dev`/`main`/`release` touching `backend/**` or
- The registry is mesh-only, so the runner host needs - The registry is mesh-only, so the runner host needs
`10.10.0.2 git.thermograph.org` in `/etc/hosts`. `10.10.0.2 git.thermograph.org` in `/etc/hosts`.
### `deploy.yml` — one file, two environments, two services ### `deploy.yml` — one file, three environments, two services
Formerly six near-identical files. Branch selects the environment Formerly six near-identical files. Branch selects the environment
(`main → beta`, `release → prod`); a matrix covers the services. (`dev → dev`, `main → beta`, `release → prod`) — **three legs now, not two** —
and a matrix covers the services.
The style is **deliberately boring**: no dynamic `fromJSON` matrix, no The style is **deliberately boring**: no dynamic `fromJSON` matrix, no
`cond && secrets.A || secrets.B` ternary. Those are GitHub idioms a Forgejo/act `cond && secrets.A || secrets.B` ternary. Those are GitHub idioms a Forgejo/act
runner may evaluate differently, and the failure mode here is "production does runner may evaluate differently, and the failure mode here is "production does
not deploy" or, worse, "deploys with an empty SSH host". So the two environments not deploy" or, worse, "deploys with an empty SSH host". So each environment
get two explicit, mutually exclusive steps and the tag is computed in shell. gets its own explicit, mutually exclusive step and the tag is computed in shell.
Preserved from the originals, all load-bearing: Preserved from the originals, all load-bearing:
@ -129,11 +131,34 @@ Preserved from the originals, all load-bearing:
- 12-hex truncation matching `build-push.yml` exactly. - 12-hex truncation matching `build-push.yml` exactly.
- Per-service, per-environment concurrency with **`cancel-in-progress: false`** — - Per-service, per-environment concurrency with **`cancel-in-progress: false`** —
a half-finished deploy must never be cancelled by a newer one. a half-finished deploy must never be cancelled by a newer one.
- **Separate `PROD_SSH_*` credentials**, so a beta credential leak cannot reach
prod.
- `appleboy/ssh-action` referenced by full URL; it isn't mirrored in Forgejo's - `appleboy/ssh-action` referenced by full URL; it isn't mirrored in Forgejo's
default action registry. default action registry.
**SSH credentials are keyed by HOST, not by environment**: `VPS1_SSH_*` and
`VPS2_SSH_*`, replacing the old `SSH_*` (which actually meant beta) and
`PROD_SSH_*` (which actually meant prod). That old naming used to read as a
security boundary — "separate `PROD_SSH_*` credentials, so a beta credential
leak cannot reach prod" — and on the old topology it happened to be true,
because beta and prod were also on separate hosts. They no longer are: beta and
prod are now two co-resident Swarm stacks on vps2, so a `VPS2_SSH_*` credential
reaches both by construction, and there is no host-level SSH boundary between
them to preserve. The boundary that replaces it lives at two other levels
instead: the **database** (beta connects as its own `NOSUPERUSER` role to its
own database, `CONNECT` revoked from `PUBLIC` on prod's) and the **filesystem**
(separate checkouts, separate rendered env files, separate deploy locks —
`env-topology.sh` derives all of it so nothing is shared by accident).
The credential boundary that *is* still real, and still matters, is
**vps1-vs-vps2**: vps1 runs Forgejo, its own CI, and whatever's currently on
`dev` — unreviewed by definition, since `dev` is exactly the branch PRs land on
before review gates them into `main`. That is precisely why the `dev`
environment renders `dev.yaml` alone and never layers `common.yaml` (the
fleet's shared production credentials, including a read-write S3 keypair and
the VAPID push-signing key) — see [Infra and secrets](08-infra-secrets.md). A
`VPS1_SSH_*` leak should never be able to reach a production credential, which
is a property `common.yaml`'s dev exclusion buys independently of which SSH key
was used to get there.
There's also a per-leg refinement: the workflow-level `paths` filter only says There's also a per-leg refinement: the workflow-level `paths` filter only says
*backend or frontend moved*, so each leg re-checks whether **this** push touched *backend or frontend moved*, so each leg re-checks whether **this** push touched
**this** domain. Without it a backend-only push would also roll the frontend, **this** domain. Without it a backend-only push would also roll the frontend,
@ -144,17 +169,20 @@ skipping silently strands a change.
### `infra-sync.yml` — infra's own pipeline ### `infra-sync.yml` — infra's own pipeline
Push to `main` touching `infra/**` → SSH to beta **and** prod, fast-forward Push to `main` touching `infra/**` → SSH to vps2, fast-forward **both**
`/opt/thermograph`, re-render `/etc/thermograph.env` from the vault. checkouts there (`/opt/thermograph` for prod, `/opt/thermograph-beta` for
beta), re-render each one's own env file from the vault.
**Rolls no service.** Image tags are the app domains' axis, not infra's. A **Rolls no service.** Image tags are the app domains' axis, not infra's. A
compose change that must recreate containers takes effect on the next app compose or stack change that must recreate containers takes effect on the next
deploy, or on a by-hand `SERVICE=all … infra/deploy/deploy.sh`. app deploy, or on a by-hand `SERVICE=all … infra/deploy/deploy.sh` (or
`deploy-dev.sh` for dev).
Note the asymmetry: app code *is* environment-staged (`dev`→`main`→`release` Note the asymmetry: app code *is* environment-staged (`dev`→`main`→`release`
maps to LAN→beta→prod via image tags); **infra is not** — both hosts track infra maps to dev(vps1)→beta→prod(vps2) via image tags); **infra is not** — beta's and
via `main`. Prod's *app images* are staged by `release`, but its checkout prod's checkouts both track infra via `main`, while dev's checkout tracks `dev`
follows `main`. itself. Prod's *app images* are staged by `release`, but its checkout follows
`main`.
### `observability-validate.yml` ### `observability-validate.yml`
@ -171,13 +199,17 @@ silent no-op is exactly what this check exists to prevent.
### `ops-cron.yml` — the prod backup ### `ops-cron.yml` — the prod backup
Daily at 03:00 UTC: `pg_dump` on prod, plus an IndexNow ping. Both SSH into prod Daily at 03:00 UTC: `pg_dump` on prod, plus an IndexNow ping. Both SSH into
and run inside the already-running stack. vps2 and run inside prod's already-running stack — specifically prod's, not
beta's, even though both now live on the same box.
**It uses `PROD_SSH_*`, not `SSH_*`.** An earlier revision reused `SSH_*` — so **It uses `VPS2_SSH_*`, not a beta-flavoured secret.** An earlier revision (on
the "prod" backup was silently dumping *beta*, and prod had no backup at all. If the old topology) reused the credential meant for beta's host — so the "prod"
you touch this file, verify a dump actually lands in backup was silently dumping *beta*, and prod had no backup at all. On vps2's
`agent@prod:~/thermograph-backups/`. co-residency, the host-keyed secret alone doesn't disambiguate prod from beta
the way it used to; the script also needs to target prod's checkout and `db`
specifically. If you touch this file, verify a dump actually lands in
`agent@vps2:~/thermograph-backups/` and that it's prod's data, not beta's.
### `secrets-guard.yml` and `shell-lint.yml` — the backstops ### `secrets-guard.yml` and `shell-lint.yml` — the backstops
@ -199,7 +231,8 @@ guard.
## Runner facts ## Runner facts
- Jobs run on the `docker` label — the LAN box's Forgejo Actions runner. - Jobs run on the `docker` label — vps1's Forgejo Actions runner (vps1 is where
Forgejo itself lives; the desktop no longer runs a runner).
- Job containers get the **host's docker.sock** automounted - Job containers get the **host's docker.sock** automounted
(Docker-outside-of-Docker, no privileged mode). The job image (Docker-outside-of-Docker, no privileged mode). The job image
(`node:20-bookworm`) ships no docker CLI, which is why several workflows (`node:20-bookworm`) ships no docker CLI, which is why several workflows

209
docs/onboarding/08-infra-secrets.md
View file

@ -7,23 +7,36 @@ Read [`infra/CLAUDE.md`](../../infra/CLAUDE.md) alongside this.
## The machines ## The machines
| Host | Public | Mesh | Orchestrator | Login | Two VPS boxes named by **role**, not by environment, plus the operator's
desktop:
| Host | Public | Mesh | Runs | Login |
|---|---|---|---|---| |---|---|---|---|---|
| **prod** | `169.58.46.181` / thermograph.org | `10.10.0.1` | Docker **Swarm** | `agent`, passwordless sudo | | **vps1** | `75.119.132.91` / git.thermograph.org, dashboard.thermograph.org | `10.10.0.2` | Forgejo, Grafana + Loki + Alloy, `emigriffith.dev`, and **dev** (own Postgres, compose, mesh-only) | `agent`, passwordless sudo |
| **beta** | `75.119.132.91` / beta.thermograph.org | `10.10.0.2` | **compose** | `agent`, passwordless sudo | | **vps2** | `169.58.46.181` / thermograph.org, beta.thermograph.org | `10.10.0.1` | **prod** and **beta**, as two co-resident Docker **Swarm** stacks; Centralis; Postfix; backups | `agent`, passwordless sudo |
| **desktop** | — | `10.10.0.3` | compose (LAN dev) | it's your box | | **desktop** | — | `10.10.0.3` | AI-model hosting + flex Swarm capacity. No Thermograph environment; `make dev-up` is a laptop convenience there | it's your box |
```bash ```bash
ssh -i ~/.ssh/thermograph_agent_ed25519 agent@169.58.46.181 # prod ssh -i ~/.ssh/thermograph_agent_ed25519 agent@169.58.46.181 # vps2 (prod + beta)
ssh -i ~/.ssh/thermograph_agent_ed25519 agent@75.119.132.91 # beta ssh -i ~/.ssh/thermograph_agent_ed25519 agent@75.119.132.91 # vps1 (Forgejo + Grafana + dev)
``` ```
Beta also hosts **Forgejo** (git + CI + registry) and **Grafana + Loki**, which The mesh IPs did not move when this topology landed — vps1 is the box that used
is why beta is guarded as strictly as prod: a destructive command there takes to be called "beta" (still mesh `10.10.0.2`), vps2 the one that used to be
out git, CI and the registry at once. The LAN dev box is deliberately called "prod" (still mesh `10.10.0.1`). What moved is which environment runs on
unguarded. which box: beta moved from vps1 onto vps2 to sit next to prod, and dev moved
onto vps1.
Every root-effective command on the VPSes is logged by `auditd` **vps1 is guarded as strictly as vps2**, not more loosely because it "used to
be beta" or because dev now lives there. It hosts Forgejo, its CI runner and
the registry, plus Grafana/Loki — a destructive command there takes out git, CI
and the registry at once. Dev being mesh-only and a public VPS's tenant, rather
than someone's desktop, is exactly why it gets the same guard as everything
else on that box: it shares a kernel with the estate's source of truth for code
and the monitoring stack, sudo there is passwordless, and it runs whatever
branch is currently in flight.
Every root-effective command on both VPSes is logged by `auditd`
(`ausearch -k agentcmd`). That's a feature — fixes are attributable. (`ausearch -k agentcmd`). That's a feature — fixes are attributable.
Prefer Centralis for routine work (`run_on_host`, `fleet_status`, Prefer Centralis for routine work (`run_on_host`, `fleet_status`,
@ -31,20 +44,36 @@ Prefer Centralis for routine work (`run_on_host`, `fleet_status`,
## The two orchestrators ## The two orchestrators
Which path a host takes is decided by **`/etc/thermograph/deploy-mode`**: the Which path an environment takes is decided by **`infra/deploy/env-topology.sh`**
string `stack` makes `deploy.sh` exec `deploy/stack/deploy-stack.sh`; anything (`thermograph_topology <env>`, keyed on `THERMOGRAPH_ENV=dev|beta|prod`): it
else is compose. The workflows never need to know which mode a host runs. derives the checkout, branch, deploy mode, stack/compose name, env file, LB
ports and DB role for that environment, and `deploy.sh` execs
`deploy/stack/deploy-stack.sh` when the mode is `stack`, or rolls compose
otherwise. The old host-wide `/etc/thermograph/deploy-mode` marker still exists
as a **fallback** for a by-hand run on a box that predates this file, but it
cannot describe vps2 on its own — vps2 runs two environments, so "which mode
does this host use" is no longer a well-formed question there.
| | compose (beta, LAN dev) | Swarm (prod) | | | compose (dev) | Swarm (prod, beta) |
|---|---|---| |---|---|---|
| File | `infra/docker-compose.yml` | `infra/deploy/stack/thermograph-stack.yml` | | Host | vps1 | vps2, both stacks co-resident |
| Services | `db`, `backend`, `lake`, `daemon`, `frontend` | `db`, `web`, `worker`, `lake`, `daemon`, `frontend`, `autoscaler`, `autoscaler-lake` | | File | `infra/docker-compose.yml` | `infra/deploy/stack/thermograph-stack.yml` (prod), `thermograph-beta-stack.yml` (beta) |
| Services | `db`, `backend`, `lake`, `daemon`, `frontend` | prod: `db`, `web`, `worker`, `lake`, `daemon`, `frontend`, `autoscaler`, `autoscaler-lake`. beta: `beta-web`, `beta-worker`, `beta-lake`, `beta-daemon`, `beta-frontend` — prefixed, no `db`, no autoscalers |
| Rolling | `up -d --no-deps <targets>` | start-first, health-gated, auto-rollback | | Rolling | `up -d --no-deps <targets>` | start-first, health-gated, auto-rollback |
| Tag file | `deploy/.image-tags.env` | `deploy/.stack-image-tags.env` | | Tag file | `deploy/.image-tags.env` | `deploy/.stack-image-tags.env` |
Note prod splits `backend` into **`web`** and **`worker`** (the Note prod splits `backend` into **`web`** and **`worker`** (the
`THERMOGRAPH_ROLE` split from [04](04-backend.md)); compose runs one `backend` `THERMOGRAPH_ROLE` split from [04](04-backend.md)); dev's compose stack runs one
service in role `all`. `backend` service in role `all`. Beta's Swarm services are prefixed
(`beta-web`, not `web`) because Swarm registers a service's short name as a DNS
alias on every network it joins — two stacks both naming a service `web` on
the shared `data` network would make `web` ambiguous, and beta joins that
network (declared `external` in its stack file) purely to reach prod's `db`.
Beta keeps its own `internal` overlay for beta-to-beta traffic and has no `db`
service of its own: **one TimescaleDB instance serves both environments**, on
separate databases (`thermograph` / `thermograph_beta`) and separate roles
(beta's is `NOSUPERUSER`/`NOCREATEDB`/`NOCREATEROLE`, `CONNECT` revoked from
`PUBLIC` on prod's database).
`STACK_TEST=1` rehearses the entire Swarm deploy under stack name `STACK_TEST=1` rehearses the entire Swarm deploy under stack name
`thermograph-test` on throwaway volumes and ports `18137`/`18080`. Leftover `thermograph-test` on throwaway volumes and ports `18137`/`18080`. Leftover
@ -58,11 +87,15 @@ coupling is why `name: thermograph` is pinned in the compose file — running
compose from `infra/` without it derives project `infra`, silently creating a compose from `infra/` without it derives project `infra`, silently creating a
whole new stack with empty volumes beside the running one. `deploy-dev.sh` whole new stack with empty volumes beside the running one. `deploy-dev.sh`
exports `COMPOSE_PROJECT_NAME=thermograph-dev` (env wins over the file key) to exports `COMPOSE_PROJECT_NAME=thermograph-dev` (env wins over the file key) to
keep LAN dev separate on purpose. **Keep both halves.** keep dev separate on purpose. **Keep both halves.**
## `deploy/deploy.sh` — read this before you touch a deploy ## `deploy/deploy.sh` — read this before you touch a deploy
Single entry point for beta and prod. Contract: Single entry point for dev, beta and prod — the same script for all three;
`infra/deploy/env-topology.sh` is what tells it which checkout, branch, stack
and ports apply. Contract, run from that environment's own checkout
(`/opt/thermograph` for prod, `/opt/thermograph-beta` for beta, both on vps2;
`/opt/thermograph-dev` on vps1 via the `deploy-dev.sh` wrapper):
```bash ```bash
SERVICE=backend BACKEND_IMAGE_TAG=sha-<12hex> /opt/thermograph/infra/deploy/deploy.sh SERVICE=backend BACKEND_IMAGE_TAG=sha-<12hex> /opt/thermograph/infra/deploy/deploy.sh
@ -76,13 +109,17 @@ What it does, in order, and why each step is the way it is:
the whole run. Backend and frontend deploys can fire for the same push the whole run. Backend and frontend deploys can fire for the same push
seconds apart and both SSH into one checkout; concurrent runs race the git seconds apart and both SSH into one checkout; concurrent runs race the git
reset, the compose project and the tag file. `-w 600` bounds the wait. reset, the compose project and the tag file. `-w 600` bounds the wait.
2. **Render secrets** from the SOPS vault into `/etc/thermograph.env`, then 2. **Render secrets** from the SOPS vault into that environment's own env file
source it, so a by-hand run interpolates the same as the systemd unit does. (`/etc/thermograph.env` for prod, `/etc/thermograph-beta.env` for beta,
Guarded on the helper's existence so the very deploy that *introduces* likewise `/etc/thermograph.env` on dev's separate host), then source it, so
`render-secrets.sh` is safe. a by-hand run interpolates the same as CI's does. Guarded on the helper's
existence so the very deploy that *introduces* `render-secrets.sh` is safe.
3. **`git reset --hard origin/$BRANCH`** on the checkout root (`BRANCH` defaults 3. **`git reset --hard origin/$BRANCH`** on the checkout root (`BRANCH` defaults
to `main`). ⚠️ **Uncommitted edits in `/opt/thermograph` evaporate here.** to `main` for beta/prod, `dev` for dev). ⚠️ **Uncommitted edits in that
The one exception is `deploy/.image-tags.env`, untracked on purpose. checkout evaporate here** — on vps2 that's `/opt/thermograph` for a prod
deploy or `/opt/thermograph-beta` for a beta one; each reset only ever
touches its own checkout, never the sibling's. The one exception is
`deploy/.image-tags.env`, untracked on purpose.
4. **Read `.image-tags.env`** so a single-service roll re-renders compose with 4. **Read `.image-tags.env`** so a single-service roll re-renders compose with
*both* services' real tags and never accidentally recreates or downgrades the *both* services' real tags and never accidentally recreates or downgrades the
sibling. sibling.
@ -115,12 +152,15 @@ What it does, in order, and why each step is the way it is:
Rollback is redeploying the previous image tag. But **do not treat image Rollback is redeploying the previous image tag. But **do not treat image
retention as a rollback guarantee**: both `deploy.sh` and `deploy-stack.sh` retention as a rollback guarantee**: both `deploy.sh` and `deploy-stack.sh`
*end* by deleting images outside the running pair. On beta that succeeds, so *end* by deleting images outside the running pair. Beta now runs Swarm exactly
beta typically holds **no local rollback target at all**; on prod it usually like prod (it used to be compose, on the old topology, where this pruning
fails only because Swarm's stopped task containers still hold references, which reliably left no rollback target at all); on both Swarm stacks it usually
is why prod keeps a handful. Verify the target tag is actually present before fails only because Swarm's stopped task containers still hold references,
promising a rollback — Centralis's `rollback_to(dry_run: true)` checks both host which is why prod and beta both tend to keep a handful. Dev, the one remaining
and registry. compose environment, is the one where this prune still reliably succeeds and
leaves nothing to roll back to. Verify the target tag is actually present
before promising a rollback — Centralis's `rollback_to(dry_run: true)` checks
both host and registry.
`docker system prune -a` on a live box is forbidden for the same reason. `docker system prune -a` on a live box is forbidden for the same reason.
`docker image prune -f` (dangling only) is the safe form. `docker image prune -f` (dangling only) is the safe form.
@ -129,55 +169,81 @@ and registry.
**The single source of truth is `infra/deploy/secrets/*.yaml`**, committed **The single source of truth is `infra/deploy/secrets/*.yaml`**, committed
encrypted (values only — keys stay readable so diffs mean something) and encrypted (values only — keys stay readable so diffs mean something) and
rendered into `/etc/thermograph.env` at deploy time by rendered at deploy time by `deploy/render-secrets.sh` into each environment's
`deploy/render-secrets.sh`. own env file (`/etc/thermograph.env` for prod, `/etc/thermograph-beta.env` for
beta, `/etc/thermograph.env` again on dev's separate host).
**Cycling a key is: edit → commit → deploy.** No SSH, no hand-edited root file, **Cycling a key is: edit → commit → deploy.** No SSH, no hand-edited root file,
no per-host duplication. no per-host duplication.
| File | Holds | | File | Holds |
|---|---| |---|---|
| `common.yaml` | the 16 values identical on prod **and** beta — VAPID keypair, metrics token, IndexNow key, `REGISTRY_TOKEN`, S3 endpoint/bucket and both S3 keypairs, plus shared non-secret config | | `common.yaml` | the 16 values identical on prod **and** beta — VAPID keypair, metrics token, IndexNow key, `REGISTRY_TOKEN`, S3 endpoint/bucket and both S3 keypairs, plus shared non-secret config. **Never layered under `dev`** — see below |
| `prod.yaml` | prod's own — the three held-back credentials, sizing, `THERMOGRAPH_BASE_URL`, Discord + mail credentials that exist nowhere else | | `prod.yaml` | prod's own — the three held-back credentials, sizing, `THERMOGRAPH_BASE_URL`, Discord + mail credentials that exist nowhere else |
| `beta.yaml` | beta's own — the three held-back credentials, sizing, base URL | | `beta.yaml` | beta's own — the three held-back credentials, sizing, base URL |
| `dev.yaml` | the LAN box's own, **self-contained**, 12 values, no production credential | | `dev.yaml` | dev's own, **self-contained**, 12 values, no production credential |
| `centralis.prod.yaml` | Centralis's nine variables → `/etc/centralis.env`, its own renderer | | `centralis.prod.yaml` | Centralis's nine variables → `/etc/centralis.env`, its own renderer |
| `example.yaml` | format reference / CI fixture (fake values) | | `example.yaml` | format reference / CI fixture (fake values) |
| `../../.sops.yaml` | which age recipients files encrypt to (plaintext config) | | `../../.sops.yaml` | which age recipients files encrypt to (plaintext config) |
The renderer concatenates `common.yaml` then `<env>.yaml`, so a **host value The renderer concatenates `common.yaml` then `<env>.yaml`, so a **host value
wins** (last occurrence of a duplicate key). `<env>` comes from wins** (last occurrence of a duplicate key). `<env>` is passed explicitly as
`/etc/thermograph/secrets-env` on the box. `THERMOGRAPH_ENV` by the deploy caller — required now that vps2 alone hosts two
environments and a host-wide marker can't tell them apart; the marker
(`/etc/thermograph/secrets-env`) survives only as the fallback for a by-hand
run on a single-environment box.
### Two design decisions worth understanding ### Two design decisions worth understanding — re-derived for vps1/vps2
**Three credentials are deliberately *not* in `common.yaml`** — **Three credentials are deliberately *not* in `common.yaml`** —
`POSTGRES_PASSWORD`, `THERMOGRAPH_AUTH_SECRET`, `THERMOGRAPH_DATABASE_URL`. `POSTGRES_PASSWORD`, `THERMOGRAPH_AUTH_SECRET`, `THERMOGRAPH_DATABASE_URL`.
They hold identical values on prod and beta today, so by the mechanical rule They hold identical *values* on prod and beta today (beta was seeded from
they belong there. They're kept per-host anyway because these are the prod), so by the mechanical rule they'd belong in `common.yaml`. They're kept
credentials that let one environment *act as* another: with them, a foothold on per-host anyway, and the reason used to be phrased as a host-isolation
beta (the more exposed box — public Forgejo and Grafana) is a foothold on prod's argument — "a foothold on beta is a foothold on prod's database, and beta is
database and prod's session signing. They match because beta was seeded from the more exposed box." **That framing no longer holds**: beta and prod are now
prod, not because the two are meant to be one system. Keeping them per-host two Swarm stacks co-resident on vps2, so a foothold on the host reaches both
costs one extra line and buys the ability to diverge. regardless of which file a credential lives in. What these three credentials
still buy, correctly stated, is a **database-level** and **file-level**
boundary, not a host-level one: beta's `THERMOGRAPH_DATABASE_URL` names its own
`NOSUPERUSER` role (`thermograph_beta`) against its own database on the shared
TimescaleDB instance, with `CONNECT` revoked from `PUBLIC` on prod's database,
and each environment's `POSTGRES_PASSWORD`/`THERMOGRAPH_AUTH_SECRET` live in a
separate file so rotating prod's stops implying "and beta's too." Keeping them
per-host costs one extra line and buys the ability to diverge — that part is
unchanged. What changed is what "diverge" is defending against: not a beta
compromise reaching a separate prod host, but a beta compromise reaching
prod's role and session-signing key on the host they now already share.
**`dev` renders `dev.yaml` alone** — `deploy-dev.sh` exports **The credential boundary that actually maps to hosts now is vps1-vs-vps2, not
`THERMOGRAPH_SECRETS_SKIP_COMMON=1`. Eleven of `common.yaml`'s sixteen values beta-vs-prod** — which is exactly why `dev` renders `dev.yaml` **alone**,
are live production credentials, including a **read-write** object-storage never layering `common.yaml` (the fleet's shared production credentials):
keypair on the bucket that also holds prod's database backups, and the VAPID `deploy-dev.sh` exports `THERMOGRAPH_SECRETS_SKIP_COMMON=1`. Eleven of
private key that signs push to real subscribers. The render is plaintext `common.yaml`'s sixteen values are live production credentials, including a
concatenation, so layering it would put all of those into every container in the **read-write** object-storage keypair on the bucket that also holds prod's
dev stack — on the operator's desktop, which is also the CI runner whose database backups, and the VAPID private key that signs push to real
`docker`-labelled jobs get the host socket, running unreviewed `dev` code. An subscribers. The render is plaintext concatenation, so layering it would put
all of those into every container in the dev stack — on vps1, the box that
also runs Forgejo and its CI runner, whose `docker`-labelled jobs get the host
socket, running whatever branch is currently `dev` (unreviewed by definition:
that's the branch PRs land on before the gate promotes them to `main`). An
override in `dev.yaml` wouldn't help: last-wins governs *consumers*, but the override in `dev.yaml` wouldn't help: last-wins governs *consumers*, but the
production value is still physically a line in the file. production value is still physically a line in the file. This is the load-
bearing boundary in the vault today, not a per-environment credential split
that co-residency has already dissolved at the host level.
Dev works fine without them: no S3 keys means the `lake` service answers 503 and Dev works fine without `common.yaml`'s values: no S3 keys means the `lake`
history falls through to the Open-Meteo archive (an accelerator, never a service answers 503 and history falls through to the Open-Meteo archive (an
dependency); VAPID and IndexNow both self-generate and persist to the `appdata` accelerator, never a dependency); VAPID and IndexNow both self-generate and
volume; with no metrics token `/api/v2/metrics` is direct-loopback-only, which persist to the `appdata` volume; with no metrics token `/api/v2/metrics` is
is right on a LAN box. One `common.yaml` value is simply *wrong* for dev — direct-loopback-only, which is right for a mesh-only box. One `common.yaml`
`THERMOGRAPH_COOKIE_SECURE=1` silently breaks login over plain HTTP. value is simply *wrong* for dev — `THERMOGRAPH_COOKIE_SECURE=1` silently breaks
login over dev's plain-HTTP mesh URL.
None of this makes dev unimportant or unguarded. It's a public VPS's tenant
now, sharing a box with Forgejo and the monitoring stack — not somebody's
desktop — which is precisely why it gets its own exclusion from `common.yaml`
rather than being treated as beneath the vault's concern.
### Working with the vault ### Working with the vault
@ -223,12 +289,14 @@ infra/ops/iceberg.sh prod -c "select count(*) from era5_daily"
``` ```
Both exec **into the container**. No database is exposed over TCP — each listens Both exec **into the container**. No database is exposed over TCP — each listens
only on its private docker network, and prod's is a Swarm **overlay the prod only on its private docker network, and prod's and beta's are both Swarm
host itself cannot route to**, so `ssh -L` works for beta and is *impossible* **overlays the vps2 host itself cannot route to**, so `ssh -L` is *impossible*
for prod. Exec works identically everywhere with no ports, no tunnels, no infra for either. (Dev's, on vps1, is plain compose, so `ssh -L` would actually work
changes. The prod container name is a Swarm task name that changes on every there — but exec is used uniformly across environments anyway, so it doesn't
redeploy, so it's resolved at call time via `docker ps --filter name=`, never matter which one happens to allow the shortcut.) Exec works identically
hardcoded. everywhere with no ports, no tunnels, no infra changes. The prod and beta
container names are Swarm task names that change on every redeploy, so they're
resolved at call time via `docker ps --filter name=`, never hardcoded.
Queries connect as **`thermograph_ro`** — `NOSUPERUSER`, granted only Queries connect as **`thermograph_ro`** — `NOSUPERUSER`, granted only
`pg_read_all_data`. Read-only is enforced by Postgres, not by convention: `pg_read_all_data`. Read-only is enforced by Postgres, not by convention:
@ -244,8 +312,9 @@ confirmation — on prod, know what you are doing.
## Backups ## Backups
The nightly `ops-cron.yml` `pg_dump` into `agent@prod:~/thermograph-backups/` The nightly `ops-cron.yml` `pg_dump` into `agent@vps2:~/thermograph-backups/`
is it. Known gaps, stated plainly: **a single copy on the same box as the (prod's data specifically — see [CI and release](07-ci-and-release.md)) is it.
Known gaps, stated plainly: **a single copy on the same box as the
database, no offsite yet**, and restores must handle TimescaleDB's database, no offsite yet**, and restores must handle TimescaleDB's
`continuous_agg` circular-FK warning (`--disable-triggers`). The `backups/` `continuous_agg` circular-FK warning (`--disable-triggers`). The `backups/`
prefix in the object-storage bucket belongs to those jobs — don't write outside prefix in the object-storage bucket belongs to those jobs — don't write outside

71
docs/onboarding/09-observability.md
View file

@ -1,26 +1,34 @@
# 9. Observability # 9. Observability
Fleet-wide log aggregation: **Loki + Grafana on beta**, fed by a **Grafana Fleet-wide log aggregation: **Loki + Grafana on vps1**, fed by a **Grafana
Alloy** agent on every node, all over the WireGuard mesh. Alloy** agent on every node, all over the WireGuard mesh. vps2 runs two
environments (beta, prod) behind that one Alloy agent; vps1 runs Alloy
alongside the things it's shipping logs *from* (Forgejo, Grafana itself, dev).
``` ```
prod (10.10.0.1) beta (10.10.0.2) desktop / LAN dev (10.10.0.3) vps2 (10.10.0.1) vps1 (10.10.0.2) desktop (10.10.0.3)
prod + beta Loki, Grafana, dev
┌────────────┐ ┌──────────────────┐ ┌────────────┐ ┌────────────┐ ┌──────────────────┐ ┌────────────┐
│ Alloy agent│──wg0──┐ │ Loki ◀── Alloy │ ┌───│ Alloy agent│ │ Alloy agent│──wg0──┐ │ Loki ◀── Alloy │ ┌───│ Alloy agent│
└────────────┘ └──▶│ Grafana (Caddy) │◀─┘ └────────────┘ └────────────┘ └──▶│ Grafana (Caddy) │◀─┘ └────────────┘
docker+caddy+app └──────────────────┘ docker+caddy+app docker+caddy+app └──────────────────┘ docker (AI models)
``` ```
- **Loki** — filesystem storage, **30-day retention**, listening on the mesh IP - **Loki** — filesystem storage, **30-day retention**, listening on the mesh IP
`10.10.0.2:3100`. Never public. `10.10.0.2:3100`. Never public.
- **Grafana** — the UI, fronted by beta's Caddy at - **Grafana** — the UI, fronted by vps1's Caddy at
**`dashboard.thermograph.org`** (Google SSO, pre-provisioned accounts only, **`dashboard.thermograph.org`** (Google SSO, pre-provisioned accounts only,
`allow_sign_up=false`, plus a break-glass local admin). Use that hostname `allow_sign_up=false`, plus a break-glass local admin). Use that hostname
everywhere; **never** `grafana.thermograph.org`. everywhere; **never** `grafana.thermograph.org`.
- **Alloy** — on each node, shipping three sources: every Docker container's - **Alloy** — one agent per *host*, not per environment: vps2's single agent
stdout/stderr, Caddy's host access logs, and the app's structured JSON logs ships logs for both beta and prod, vps1's ships Forgejo/Grafana/dev, and the
(`errors`/`access`/`audit` `*.jsonl`, parsed so `level`/`tag`/`phase` become desktop's ships whatever runs there now. Each ships three sources: every
labels). Every line is tagged `host = prod|beta|dev`. Docker container's stdout/stderr, Caddy's host access logs, and the app's
structured JSON logs (`errors`/`access`/`audit` `*.jsonl`, parsed so
`level`/`tag`/`phase` become labels). Every app log line is tagged
`host = prod|beta|dev` — that label is the *environment*, not the box, which
matters on vps2 where one Alloy agent ships lines carrying two different
`host` values.
There is **no build and no deploy automation** for this domain — it ships by There is **no build and no deploy automation** for this domain — it ships by
hand. But `observability-validate.yml` parses every artifact that ships to a hand. But `observability-validate.yml` parses every artifact that ships to a
@ -42,20 +50,24 @@ Start wide, then narrow:
2. `logs_overview since="1h"` — where the volume is, and what has gone quiet. 2. `logs_overview since="1h"` — where the volume is, and what has gone quiet.
3. `logs_query service=… host=…` — the actual lines. 3. `logs_query service=… host=…` — the actual lines.
### ⚠ The prod trap ### ⚠ The prod (and now beta) trap
Prod runs a Swarm stack, and its `job="docker"` streams carry **no `service` Prod and beta both run Swarm stacks — co-resident on vps2 — and their
label**. A raw `{host="prod", service="backend"}` query matches nothing and `job="docker"` streams carry **no `service` label**. A raw
reads as *"no logs"* when it actually means *"wrong selector"*. `{host="prod", service="backend"}` query matches nothing and reads as *"no
logs"* when it actually means *"wrong selector"*. This used to be a prod-only
trap when beta ran compose; it now applies to both environments on vps2, and
dev (compose, on vps1) is the one environment this does *not* affect.
Use Centralis's `logs_query` `service` argument, which rewrites it to a Use Centralis's `logs_query` `service` argument, which rewrites it to a
container name-regex with the churning task suffix wildcarded. Prod service container name-regex with the churning task suffix wildcarded. Prod service
names are `thermograph_web`, `_worker`, `_frontend`, `_db`, `_autoscaler`, names are `thermograph_web`, `_worker`, `_frontend`, `_db`, `_autoscaler`,
`_lake`. `_lake`; beta's are the same names prefixed `beta-` (`beta-web`, `beta-worker`,
…) with no `_db` or autoscaler equivalents.
Related: **prod container names change on every redeploy** (the Swarm task Related: **prod and beta container names both change on every redeploy** (the
suffix). Never hardcode one; resolve via `docker ps --filter name=` at call Swarm task suffix). Never hardcode one; resolve via `docker ps --filter name=`
time. at call time.
## The dashboard ## The dashboard
@ -81,16 +93,16 @@ Three reasons, all learned the hard way:
- Grafana's factory default pointed at the literal string - Grafana's factory default pointed at the literal string
`<example@email.com>` — a paging path that had never delivered a message to `<example@email.com>` — a paging path that had never delivered a message to
anyone. anyone.
- Beta's Grafana relays SMTP through **prod's** Postfix. Email alerts therefore - vps1's Grafana relays SMTP through **vps2's** Postfix (prod's mail service).
travel through the box most likely to be on fire, and vanish exactly when they Email alerts therefore travel through the box most likely to be on fire, and
matter. vanish exactly when they matter.
- Discord is off-estate, works when prod is dead, and reaches a phone. - Discord is off-estate, works when prod is dead, and reaches a phone.
`#ops-alerts` is dedicated. `#dev` / `#uat` / `#prod` are the *product's* alert `#ops-alerts` is dedicated. `#dev` / `#uat` / `#prod` are the *product's* alert
subscription output — real deliveries to real subscribers — so routing ops noise subscription output — real deliveries to real subscribers — so routing ops noise
there would corrupt the evidence. there would corrupt the evidence.
The webhook URL is a secret and lives only in beta's gitignored `.env` as The webhook URL is a secret and lives only in vps1's gitignored `.env` as
`DISCORD_ALERT_WEBHOOK_URL`. Grafana refuses to start if it's unset. A literal `DISCORD_ALERT_WEBHOOK_URL`. Grafana refuses to start if it's unset. A literal
webhook URL committed to the repo is a **hard CI failure**. webhook URL committed to the repo is a **hard CI failure**.
@ -135,7 +147,7 @@ pages nobody. **The only proof is a message actually arriving in
## Deploying an observability change ## Deploying an observability change
> ⚠️ **Merging is not deploying — and this one has already bitten.** > ⚠️ **Merging is not deploying — and this one has already bitten.**
> `/opt/observability` on beta and prod is a checkout of the **archived** > `/opt/observability` on vps1 and vps2 is a checkout of the **archived**
> `emi/thermograph-observability` repo. It can never `git pull` again; the > `emi/thermograph-observability` repo. It can never `git pull` again; the
> content now lives here under `observability/`. A previous observability PR > content now lives here under `observability/`. A previous observability PR
> merged and had **zero live effect** for exactly this reason. > merged and had **zero live effect** for exactly this reason.
@ -146,13 +158,14 @@ hand: back up the live file with a UTC-timestamped suffix, `scp` the new one up,
Two gotchas in that sequence: Two gotchas in that sequence:
- Use `sudo docker restart <container>` on **prod** — not `docker compose`, - Use `sudo docker restart <container>` on **vps2** — not `docker compose`,
which there demands a `GF_SECURITY_ADMIN_PASSWORD` it has no `.env` to which there demands a `GF_SECURITY_ADMIN_PASSWORD` it has no `.env` to
interpolate from. Beta's stack *does* have an `.env`, so compose works there. interpolate from. vps1's stack *does* have an `.env`, so compose works there.
- **`docker restart` will not pick up a new environment variable.** Any change - **`docker restart` will not pick up a new environment variable.** Any change
that introduces one needs `docker compose up -d grafana` on beta. that introduces one needs `docker compose up -d grafana` on vps1.
Alerting lives on **beta only** — prod and dev run Alloy agents, not Grafana. Alerting lives on **vps1 only** — vps2 (prod and beta alike) and dev run Alloy
agents, not Grafana.
Full step-by-step, including rollback, is in Full step-by-step, including rollback, is in
[`observability/README.md`](../../observability/README.md). [`observability/README.md`](../../observability/README.md).
@ -177,9 +190,9 @@ Full step-by-step, including rollback, is in
- **Grafana cannot alert on its own death.** `AlertingWatchdog` is a *manual* - **Grafana cannot alert on its own death.** `AlertingWatchdog` is a *manual*
dead-man's switch — it relies on somebody noticing the daily message stopped. dead-man's switch — it relies on somebody noticing the daily message stopped.
An external uptime pinger from off-estate would close this properly. An external uptime pinger from off-estate would close this properly.
- **Beta runs a `docker-compose.override.yml` that is not in this repo**, wiring - **vps1 runs a `docker-compose.override.yml` that is not in this repo**, wiring
Grafana's SMTP to prod's Postfix. It should be mirrored into the tracked Grafana's SMTP to vps2's Postfix (prod's). It should be mirrored into the
compose file or deleted; alerting no longer depends on it. tracked compose file or deleted; alerting no longer depends on it.
## When something breaks ## When something breaks

19
docs/onboarding/10-recipes.md
View file

@ -79,14 +79,17 @@ git commit && push && open a PR
``` ```
Then it reaches the hosts by either route: an `infra/**` push to `main` fires Then it reaches the hosts by either route: an `infra/**` push to `main` fires
`infra-sync.yml` (re-renders `/etc/thermograph.env` on beta and prod, rolls `infra-sync.yml` (re-renders prod's and beta's env files — `/etc/thermograph.env`
nothing), or the next app deploy renders it as step 2 of `deploy.sh`. and `/etc/thermograph-beta.env`, both on vps2 — rolls nothing), or the next app
deploy renders it as step 2 of `deploy.sh`. Dev's vault (`dev.yaml`) is never
part of that sync — it's rendered only by dev's own deploy on vps1, and it
never layers `common.yaml` regardless.
Decide **which file** first: Decide **which file** first:
- identical on prod *and* beta, and not a "lets one environment act as another" - identical on prod *and* beta, and not a "lets one environment act as another"
credential → `common.yaml`; credential → `common.yaml` (never reaches dev, on vps1, at all);
- per-host, or one of `POSTGRES_PASSWORD` / `THERMOGRAPH_AUTH_SECRET` / - per-environment, or one of `POSTGRES_PASSWORD` / `THERMOGRAPH_AUTH_SECRET` /
`THERMOGRAPH_DATABASE_URL` → `prod.yaml` / `beta.yaml`; `THERMOGRAPH_DATABASE_URL` → `prod.yaml` / `beta.yaml`;
- dev → `dev.yaml`, and give it **its own** value. Never a copy of prod's. - dev → `dev.yaml`, and give it **its own** value. Never a copy of prod's.
@ -209,9 +212,11 @@ SERVICE=backend BACKEND_IMAGE_TAG=sha-<previous12hex> \
``` ```
**Check the tag actually exists first.** Both deploy scripts *end* by deleting **Check the tag actually exists first.** Both deploy scripts *end* by deleting
images outside the running pair, so beta typically holds no local rollback images outside the running pair. Beta now runs Swarm exactly like prod
target at all. `rollback_to(dry_run: true)` checks host and registry and will (co-resident on vps2), so both usually keep a handful of stopped-task
tell you when it's in neither. references; dev, the one remaining compose environment (on vps1), is the one
that typically holds no local rollback target at all. `rollback_to(dry_run:
true)` checks host and registry and will tell you when it's in neither.
--- ---

61
docs/onboarding/11-traps.md
View file

@ -48,19 +48,31 @@ walkthrough as superseded — but the drift goes further than that note admits.
The **key-management** sections (Part 1) and the `/etc/hosts` registry note are The **key-management** sections (Part 1) and the `/etc/hosts` registry note are
still accurate and useful. still accurate and useful.
### `infra/DEPLOY-DEV.md` and `infra/ACCESS.md` reference deleted workflows ### `infra/DEPLOY-DEV.md` and `infra/ACCESS.md` reference deleted workflows — and predate dev becoming a real environment
Both talk about `.forgejo/workflows/deploy-dev.yml`. It does not exist. The two Both talk about `.forgejo/workflows/deploy-dev.yml`. It does not exist under
LAN-dev deploy workflows were deleted rather than ported at the CI that name. The two original LAN-dev deploy workflows were deleted rather than
consolidation — they were documented as inert (they targeted a monorepo layout ported at the CI consolidation — at the time they were genuinely inert (they
at `~/thermograph-dev` on a box still holding a split-era checkout). **LAN dev targeted a monorepo layout at `~/thermograph-dev` on a box still holding a
is a local `make dev-up` concern, not a CI environment.** split-era checkout), which is where the old claim **"LAN dev is a local `make
dev-up` concern, not a CI environment"** came from.
**That claim is now false, and worth flagging precisely because it used to be
true.** `dev` is a first-class hosted environment on vps1 — its own Postgres
container, mesh-only exposure, deployed by CI like beta and prod, as the third
leg of `deploy.yml` (see [07](07-ci-and-release.md) and
[08](08-infra-secrets.md)). The desktop's `make dev-up` is still a real,
useful thing — a laptop-local rehearsal — but it is no longer the *only* way
`dev` runs, and neither doc has been updated to say so.
`ACCESS.md` §3c also lists `{build,pr-build,deploy-dev,deploy}.yml` as the `ACCESS.md` §3c also lists `{build,pr-build,deploy-dev,deploy}.yml` as the
workflow set; there are now nine files and the deploy/build ones were collapsed. workflow set; there are now nine files and the deploy/build ones were collapsed.
It also still describes the host table from before the vps1/vps2 rename — see
`CUTOVER-NOTES.md`'s vps1/vps2 section for the current one.
Everything else in `ACCESS.md` — the host table, the agent-access model, the key Everything else in `ACCESS.md` — the agent-access model, the key rotation
rotation procedure, the Swarm/Forgejo tracks — is current and worth reading. procedure, the Swarm/Forgejo tracks — is current and worth reading; just
mentally translate host names.
### `backend/README.md` is written for the split-repo era ### `backend/README.md` is written for the split-repo era
@ -93,7 +105,7 @@ it — that's the standing instruction in the root `CLAUDE.md`.
### `observability/README.md`: merging is not deploying ### `observability/README.md`: merging is not deploying
`/opt/observability` on beta and prod is a checkout of the **archived** `/opt/observability` on vps1 and vps2 is a checkout of the **archived**
`emi/thermograph-observability` repo. It can never `git pull` again. A previous `emi/thermograph-observability` repo. It can never `git pull` again. A previous
observability PR merged and had **zero live effect** for exactly this reason. observability PR merged and had **zero live effect** for exactly this reason.
The README documents the hand-ship procedure; until the checkouts are re-pointed The README documents the hand-ship procedure; until the checkouts are re-pointed
@ -153,7 +165,7 @@ test run.
`infra/docker-compose.yml` pins `name: thermograph`. Without it, running compose `infra/docker-compose.yml` pins `name: thermograph`. Without it, running compose
from `infra/` derives project `infra` — a silently **new** stack with empty from `infra/` derives project `infra` — a silently **new** stack with empty
volumes beside the running one. `deploy-dev.sh` exports volumes beside the running one. `deploy-dev.sh` exports
`COMPOSE_PROJECT_NAME=thermograph-dev` to keep LAN dev separate on purpose. `COMPOSE_PROJECT_NAME=thermograph-dev` to keep dev separate on purpose.
Keep both halves. Keep both halves.
### `deploy.sh` hard-resets the host checkout ### `deploy.sh` hard-resets the host checkout
@ -167,11 +179,13 @@ roll the sibling onto `local`.
### The deploy scripts eat your rollback image ### The deploy scripts eat your rollback image
Both `deploy.sh` and `deploy-stack.sh` **end** by deleting images outside the Both `deploy.sh` and `deploy-stack.sh` **end** by deleting images outside the
running pair. On beta that succeeds, so beta typically holds **no local rollback running pair. Beta now runs Swarm co-resident with prod on vps2, so both
target at all**; on prod it usually fails only because Swarm's stopped task usually fail to fully prune only because Swarm's stopped task containers still
containers still hold references. `docker system prune -a` is forbidden on a hold references — dev, the one remaining compose environment (on vps1), is the
live box for the same reason — but the deploy script is the thing that actually one that typically holds **no local rollback target at all**. `docker system
eats the image, not prune. Verify a tag exists before promising a rollback. prune -a` is forbidden on a live box for the same reason — but the deploy
script is the thing that actually eats the image, not prune. Verify a tag
exists before promising a rollback.
### `daemon` and `lake` are never deploy targets on their own ### `daemon` and `lake` are never deploy targets on their own
@ -189,13 +203,14 @@ line, in file order. A `| head -1` therefore grabbed `db`'s image, and
and dropped the daemon from *every* backend deploy. The script now builds the and dropped the daemon from *every* backend deploy. The script now builds the
image reference from the same variables compose interpolates. image reference from the same variables compose interpolates.
### Prod's Loki streams carry no `service` label ### Prod's (and beta's) Loki streams carry no `service` label
Prod runs Swarm; its `job="docker"` streams have no `service` label. A raw Prod and beta both run Swarm, co-resident on vps2; their `job="docker"` streams
`{host="prod", service="backend"}` query matches nothing and reads as "no logs" have no `service` label. A raw `{host="prod", service="backend"}` query matches
when it means "wrong selector". Use `logs_query`'s `service` argument. Prod nothing and reads as "no logs" when it means "wrong selector". Use
container names also change on every redeploy (the task suffix) — resolve via `logs_query`'s `service` argument. Prod and beta container names also change on
`docker ps --filter name=` at call time, never hardcode. every redeploy (the task suffix) — resolve via `docker ps --filter name=` at
call time, never hardcode. Dev, on compose, doesn't have this problem.
### A Grafana alert rule can provision cleanly and never fire ### A Grafana alert rule can provision cleanly and never fire
@ -212,9 +227,9 @@ It's executable documentation until state is bootstrapped.
### The registry is mesh-only ### The registry is mesh-only
`git.thermograph.org` resolves publicly to beta's IP, but beta's Caddy rejects `git.thermograph.org` resolves publicly to vps1's IP, but vps1's Caddy rejects
`/v2/*` from outside the WireGuard mesh. Any host that pulls needs `/v2/*` from outside the WireGuard mesh. Any host that pulls needs
`10.10.0.2 git.thermograph.org` in `/etc/hosts`. Beta itself doesn't — it *is* `10.10.0.2 git.thermograph.org` in `/etc/hosts`. vps1 itself doesn't — it *is*
the box. the box.
### Forgejo runners only have the labels they registered with ### Forgejo runners only have the labels they registered with

11
docs/onboarding/README.md
View file

@ -42,12 +42,13 @@ readings.
the superseded original — see [traps](11-traps.md).) the superseded original — see [traps](11-traps.md).)
- **`infra/`** — compose and Swarm files, deploy scripts, the SOPS secrets - **`infra/`** — compose and Swarm files, deploy scripts, the SOPS secrets
vault, Terraform, ops query tooling. vault, Terraform, ops query tooling.
- **`observability/`** — Loki + Grafana on beta, an Alloy agent per node. - **`observability/`** — Loki + Grafana on vps1, an Alloy agent per node.
Branches stage environments: PR → `dev` (LAN dev) → `main` (beta) → `release` Branches stage environments: PR → `dev` (vps1, mesh-only) → `main` (beta, vps2)
(prod). The two app domains build and deploy **independently** — that → `release` (prod, vps2). The two app domains build and deploy
independence is the whole reason the split-then-reunify history exists, and **independently** — that independence is the whole reason the split-then-reunify
[contracts](06-contracts.md) is the list of things that keep it safe. history exists, and [contracts](06-contracts.md) is the list of things that
keep it safe.
## Your first day ## Your first day

25
infra/.claude/skills/key-gaps/SKILL.md
View file

@ -28,12 +28,24 @@ python3 .claude/skills/key-gaps/key_gaps.py \
## Audit the LIVE boxes (read-only, key names only) ## Audit the LIVE boxes (read-only, key names only)
Gather the key names over SSH (never the values), then audit. Hosts/keys per INFRA.md: Gather the key names over SSH (never the values), then audit. Hosts per the
root `CLAUDE.md` topology:
- **vps2** (`169.58.46.181`) hosts BOTH prod and beta as separate Swarm
stacks — TWO live env files on the same box: `/etc/thermograph.env` (prod)
and `/etc/thermograph-beta.env` (beta). "The live env file per environment"
is no longer one path per host; it's one path per environment, and both
environments' files live on this one host. Get both in the same SSH round
trip, or two separate commands against the same host — never assume one
host means one file here.
- **vps1** (`75.119.132.91`) hosts dev, at the same `/etc/thermograph.env`
path (it's a different host, so no collision with prod's file of the same
name).
```sh ```sh
K=~/.ssh/thermograph_agent_ed25519 K=~/.ssh/thermograph_agent_ed25519
# sudo: /etc/thermograph.env is root-owned (0640). On prod `agent` can read it # sudo: the env files are root-owned (0640). On these boxes `agent` can often
# directly too, but sudo works uniformly on both boxes. # read them directly too, but sudo works uniformly regardless.
# #
# The character class must allow DIGITS. This was `^[A-Z_]+=` until 2026-07-24, # The character class must allow DIGITS. This was `^[A-Z_]+=` until 2026-07-24,
# which silently skipped every key whose name contains a digit — in this estate # which silently skipped every key whose name contains a digit — in this estate
@ -44,9 +56,10 @@ K=~/.ssh/thermograph_agent_ed25519
# no audit: the invented finding sends someone to provision a credential that # no audit: the invented finding sends someone to provision a credential that
# already exists, and it was briefly recorded as a root cause of a real bug. # already exists, and it was briefly recorded as a root cause of a real bug.
# The match still stops at the `=`, so no value is ever read. # The match still stops at the `=`, so no value is ever read.
ssh -i $K agent@169.58.46.181 'sudo grep -oE "^[A-Z][A-Z0-9_]*=" /etc/thermograph.env' > /tmp/prod.keys # prod ssh -i $K agent@169.58.46.181 'sudo grep -oE "^[A-Z][A-Z0-9_]*=" /etc/thermograph.env' > /tmp/prod.keys # vps2: prod
ssh -i $K agent@75.119.132.91 'sudo grep -oE "^[A-Z][A-Z0-9_]*=" /etc/thermograph.env' > /tmp/beta.keys # beta ssh -i $K agent@169.58.46.181 'sudo grep -oE "^[A-Z][A-Z0-9_]*=" /etc/thermograph-beta.env' > /tmp/beta.keys # vps2: beta
python3 .claude/skills/key-gaps/key_gaps.py prod=/tmp/prod.keys beta=/tmp/beta.keys ssh -i $K agent@75.119.132.91 'sudo grep -oE "^[A-Z][A-Z0-9_]*=" /etc/thermograph.env' > /tmp/dev.keys # vps1: dev
python3 .claude/skills/key-gaps/key_gaps.py prod=/tmp/prod.keys beta=/tmp/beta.keys dev=/tmp/dev.keys
``` ```
## Verify a SOPS cutover matches live ## Verify a SOPS cutover matches live

17
infra/.env.example
View file

@ -1,9 +1,11 @@
# Local docker-compose overrides. Copy to .env (gitignored) for `docker compose # Local docker-compose overrides. Copy to .env (gitignored) for `docker compose
# up` / `make up` on a dev box: cp .env.example .env # up` / `make up` on a laptop: cp .env.example .env
# #
# Compose auto-reads a repo-root .env for ${VAR} interpolation in # Compose auto-reads a repo-root .env for ${VAR} interpolation in
# docker-compose.yml. In production these live in /etc/thermograph.env instead # docker-compose.yml. On the fleet (dev/beta/prod), these values live in each
# (loaded by the systemd unit), so this file is only for local runs. # environment's own rendered env file instead (/etc/thermograph.env,
# /etc/thermograph-beta.env — see deploy/secrets/README.md), so this file is
# only for a local, unmanaged run.
# Database password. Compose uses it to initialize the postgres container AND to # Database password. Compose uses it to initialize the postgres container AND to
# build the app's THERMOGRAPH_DATABASE_URL. Change it before first `up`. # build the app's THERMOGRAPH_DATABASE_URL. Change it before first `up`.
@ -33,8 +35,13 @@ THERMOGRAPH_INTERNAL_TOKEN=
# replicate with another -- see docker-compose.yml's db service comment. # replicate with another -- see docker-compose.yml's db service comment.
# TIMESCALEDB_TAG=latest-pg18 # TIMESCALEDB_TAG=latest-pg18
# Postgres sizing. Terraform sets these per host in prod/beta (prod DB_MEMORY # Postgres sizing. Only meaningful for an environment that runs its OWN db
# 16g); local/beta default to 8g / 2 CPUs. # service — dev's compose stack (this file) and prod's Swarm stack, which
# sizes the ONE shared TimescaleDB instance on vps2 (see
# deploy/secrets/prod.yaml, currently DB_MEMORY=16g). Beta shares that same
# instance rather than running a second one, so a DB_MEMORY/DB_CPUS value in
# beta's own vault file no longer sizes anything — don't be misled by its
# presence there. Local/dev default to 8g / 2 CPUs.
# DB_MEMORY=8g # DB_MEMORY=8g
# DB_CPUS=2 # DB_CPUS=2

158
infra/ACCESS.md
View file

@ -5,28 +5,55 @@ provisioning / Postgres / Terraform work described in `terraform/README.md`
and (for the historical Track A/Track B plan this whole effort grew from, and (for the historical Track A/Track B plan this whole effort grew from,
including the decision to split this repo out of the app monorepo) including the decision to split this repo out of the app monorepo)
`thermograph-docs/runbooks/implementation-handoff.md` and `thermograph-docs/runbooks/implementation-handoff.md` and
`thermograph-docs/architecture/repo-topology-and-infrastructure.md`. Terraform `thermograph-docs/architecture/repo-topology-and-infrastructure.md`.
provisions **prod** (the new 48 GB / 12-core box, `thermograph.org`) and **Note:** `terraform/README.md`'s own host table still describes the
**beta** (the old VPS, `75.119.132.91`) — see `terraform/README.md` / pre-cutover shape (`prod` and `beta` as two single-purpose boxes) and has not
`terraform.tfvars.example`. **None of this touches the app repo** (its yet been updated to the vps1/vps2 split below — that file is out of this
backend/frontend source, `Dockerfile`, or CI) — this repo owns only how and pass's scope; treat this document as the current source of truth for the
where the already-built app image runs; the app repo owns building it. physical topology in the meantime.
``` ```
Track 1: deploy/provision-agent-access.sh — a dedicated full-root login for me Track 1: deploy/provision-agent-access.sh — a dedicated full-root login for me
Track 2: deploy/swarm/ — Swarm cluster spanning THREE nodes Track 2: deploy/swarm/ — Swarm mesh spanning THREE nodes
Track 3: deploy/forgejo/ + .forgejo/ — Forgejo + Forgejo Actions, replacing GitHub Track 3: deploy/forgejo/ + .forgejo/ — Forgejo + Forgejo Actions, replacing GitHub
``` ```
**Three nodes, not two**: prod, beta, and **the desktop** (the LAN dev ## The estate (renamed by role, not by environment)
machine — same box that already runs the pre-Forgejo GitHub self-hosted
runner). This matches the canonical Track B design in the handoff doc; an The same two VPS boxes as always — **neither public IP nor WireGuard mesh
earlier revision of this whole effort covered just prod+beta and has been address moved** — plus the operator's desktop, now named for what they *do*
realigned. rather than which environment happens to live there:
| Host | Public IP | Mesh IP | Role |
|------|-----------|---------|------|
| **vps1** | `75.119.132.91` | `10.10.0.2` | "Operational programs": Forgejo (git + CI + registry, `git.thermograph.org`), Grafana/Loki/Alloy (`dashboard.thermograph.org`), the `emigriffith.dev` portfolio, and the **dev** environment (mesh-only, `10.10.0.2:8137`, no public DNS/Caddy/TLS) |
| **vps2** | `169.58.46.181` | `10.10.0.1` | "The deployed environment": **prod** and **beta**, as two separate Docker Swarm stacks, plus Centralis, Postfix, the backups, and the one shared TimescaleDB instance both stacks use |
| **desktop** | — | `10.10.0.3` | AI-model hosting (voice-to-text, an upcoming-feature LLM) plus flex Swarm-worker capacity. Hosts **no** Thermograph environment — `make dev-up` still works there, but only as a laptop-local rehearsal |
This is a **rename**, not a re-provisioning: vps1 is the box that used to be
called "beta" (it already ran Forgejo and Grafana before this split), and vps2
is the box that used to be called "prod". What moved is the **beta
environment** — off vps1 and onto vps2, so a beta green light is evidence
about prod (same orchestrator, same Postgres build, same Caddy, same mesh
position) — and the **dev environment**, off the desktop and onto vps1, so the
desktop can retire from the Thermograph estate entirely. See
`deploy/env-topology.sh`'s header comment for the full rationale.
**Three nodes on the Swarm mesh, not two**: vps2 (manager), vps1 (worker), and
the desktop (worker, flex capacity + AI model hosting). One manager, not more:
Raft needs 3 nodes for real quorum-based HA, and this cluster only has 3 nodes
total, so a second manager would still fall short of real HA while adding
split-brain risk. If the manager (vps2) goes down, the workers keep running
whatever was already scheduled on them (Forgejo, pinned to vps1) but the
cluster can't reschedule anything until vps2 is back — acceptable for a small
cluster whose only Swarm-scheduled workload today is Forgejo (prod and beta's
app stacks are separate Swarm stacks that also happen to run on vps2, the
manager, because their volumes are local to that node — see
`deploy/stack/thermograph-stack.yml`'s header).
## Track 1 — Agent access ## Track 1 — Agent access
Run `sudo bash deploy/provision-agent-access.sh` on **prod and beta** (not Run `sudo bash deploy/provision-agent-access.sh` on **vps1 and vps2** (not
the desktop — that's wherever you're already working from, no separate the desktop — that's wherever you're already working from, no separate
access-provisioning step needed there). Creates a dedicated `agent` user (not access-provisioning step needed there). Creates a dedicated `agent` user (not
raw root login — a distinct name gives a clean audit trail) with passwordless raw root login — a distinct name gives a clean audit trail) with passwordless
@ -42,20 +69,27 @@ Both VPS boxes are provisioned and reachable as of this writing:
| Host | Role | Public IP | Login | | Host | Role | Public IP | Login |
|------|------|-----------|-------| |------|------|-----------|-------|
| prod | Swarm manager; Thermograph's live prod home (`release`, thermograph.org) | `169.58.46.181` | `agent` | | vps2 | Swarm manager; prod AND beta live here (`thermograph.org`, `beta.thermograph.org`) | `169.58.46.181` | `agent` |
| beta | Swarm worker; `main`/beta.thermograph.org + hosts Forgejo | `75.119.132.91` | `agent` | | vps1 | Swarm worker; Forgejo + Grafana/Loki + dev (`git.thermograph.org`, `dashboard.thermograph.org`) | `75.119.132.91` | `agent` |
| desktop | Swarm worker, LAN dev machine + Forgejo Actions runner | (local) | (already have access) | | desktop | Swarm worker, AI-model hosting | (local) | (already have access) |
``` ```
ssh -i ~/.ssh/thermograph_agent_ed25519 agent@169.58.46.181 # prod ssh -i ~/.ssh/thermograph_agent_ed25519 agent@169.58.46.181 # vps2 (prod + beta)
ssh -i ~/.ssh/thermograph_agent_ed25519 agent@75.119.132.91 # beta ssh -i ~/.ssh/thermograph_agent_ed25519 agent@75.119.132.91 # vps1 (dev + Forgejo + Grafana)
``` ```
**On vps2, always say which environment you mean.** A shell on vps2 can act on
either `/opt/thermograph` (prod) or `/opt/thermograph-beta` (beta) — there is
no "the app" on that box any more. `deploy.sh` refuses to run if
`THERMOGRAPH_ENV` disagrees with the checkout it was invoked from (see
`deploy/env-topology.sh`), which catches the one genuinely dangerous mistake
here: running prod's checkout with `THERMOGRAPH_ENV=beta`, or the reverse.
The private half of the agent's dedicated keypair lives at The private half of the agent's dedicated keypair lives at
`~/.ssh/thermograph_agent_ed25519` on the operator's own machine (never in `~/.ssh/thermograph_agent_ed25519` on the operator's own machine (never in
this repo, never in a CI secret) — same convention `DEPLOY.md` already uses this repo, never in a CI secret) — same convention `DEPLOY.md` already uses
for the separate CI deploy key (`~/.ssh/thermograph_ci`). If you're setting for the separate CI deploy keys. If you're setting this up fresh elsewhere,
this up fresh elsewhere, generate a new pair the same way generate a new pair the same way
(`ssh-keygen -t ed25519 -a 100 -C "claude-agent@thermograph-infra" -f (`ssh-keygen -t ed25519 -a 100 -C "claude-agent@thermograph-infra" -f
~/.ssh/thermograph_agent_ed25519 -N ""`), embed the new public half in ~/.ssh/thermograph_agent_ed25519 -N ""`), embed the new public half in
`AGENT_PUBKEY` at the top of `provision-agent-access.sh`, and re-run it on `AGENT_PUBKEY` at the top of `provision-agent-access.sh`, and re-run it on
@ -67,31 +101,21 @@ directory that gets cleaned up, not anywhere-in-repo. Losing it just means
regenerating and re-running the provisioning script; it does not lock either regenerating and re-running the provisioning script; it does not lock either
box, since your own login is untouched. box, since your own login is untouched.
**Live as of 2026-07-21** (was: "no Docker / Terraform not applied / Swarm
inactive" — all now done): Docker is installed on all three nodes; the
WireGuard mesh and Docker Swarm are up with all three nodes `Ready` (prod
manager, beta + desktop workers); Terraform has been applied to both VPS
boxes (prod stood up fresh on `release`, beta rebuilt on `main`); Forgejo is
serving at `git.thermograph.org`; and the desktop runs the Forgejo Actions
runner. See the verification checklist at the end.
## Track 2 — Swarm ## Track 2 — Swarm
See `deploy/swarm/README.md` for the exact order of operations across all See `deploy/swarm/README.md` for the exact order of operations across all
**three** nodes (WireGuard mesh first, then swarm init/join, then lock the **three** nodes (WireGuard mesh first, then swarm init/join, then lock the
Swarm ports down to the tunnel interface, then label beta for Forgejo Swarm ports down to the tunnel interface, then label vps1 for Forgejo
placement). This cluster's only job is hosting Forgejo — it does not placement). This cluster's Swarm-scheduled workload is Forgejo; prod and
orchestrate the Terraform-managed app deploys. beta's app stacks are separate `docker stack deploy`s that happen to live on
the same manager node (vps2) because their volumes are local to it today.
## Track 3 — Forgejo, replacing GitHub ## Track 3 — Forgejo, replacing GitHub
### 3a. Stand up Forgejo ### 3a. Stand up Forgejo
See `deploy/forgejo/README.md` — deploys the stack (pinned to beta), then See `deploy/forgejo/README.md` — deploys the stack (pinned to **vps1**), then
walks through registering the Actions runner **on the desktop** as a plain walks through registering the Actions runner as a plain systemd service
systemd service (`register-lan-runner.sh`), not as a Swarm-scheduled (`register-lan-runner.sh`), not as a Swarm-scheduled container.
container. Only one Swarm secret to mint now (`forgejo_db_password`) — the
runner token is no longer a Swarm secret, since the runner isn't a Swarm
service anymore.
### 3b. Migrate the repo (mirror first, verify, then cut over) ### 3b. Migrate the repo (mirror first, verify, then cut over)
1. In Forgejo: **+ New Migration → GitHub**. Point it at 1. In Forgejo: **+ New Migration → GitHub**. Point it at
@ -103,13 +127,12 @@ service anymore.
3. **Don't retarget any secret or disable a GitHub workflow yet** — see 3d. 3. **Don't retarget any secret or disable a GitHub workflow yet** — see 3d.
### 3c. Workflows ### 3c. Workflows
`.forgejo/workflows/{build,pr-build,deploy-dev,deploy}.yml` mirror `.forgejo/workflows/{build,pr-build,deploy}.yml` (a single `Deploy` workflow
now covers all three environments — see its own header comment) mirror
`.github/workflows/*.yml`, copied with the mechanical changes Forgejo needs: `.github/workflows/*.yml`, copied with the mechanical changes Forgejo needs:
- `runs-on: ubuntu-latest` → `runs-on: docker` (one of the two labels the - `runs-on: ubuntu-latest` → `runs-on: docker` (the runner's `docker` label —
desktop's runner registers under — see 3a; general CI/deploy jobs get a containerized jobs get a fresh container via the docker-socket-automounted
fresh container, the LAN-deploy job in `deploy-dev.yml` runs bare/host-native runner).
under the `thermograph-lan` label instead, since it needs real filesystem
access).
- `appleboy/ssh-action` referenced by full GitHub URL (not mirrored in - `appleboy/ssh-action` referenced by full GitHub URL (not mirrored in
Forgejo's default action registry, unlike `actions/checkout` / Forgejo's default action registry, unlike `actions/checkout` /
`actions/setup-python`, which resolve unchanged). `actions/setup-python`, which resolve unchanged).
@ -119,44 +142,51 @@ service anymore.
`build` status check required (Settings → Branches). Forgejo does **not** `build` status check required (Settings → Branches). Forgejo does **not**
auto-merge on green by itself — a ready, green, mergeable PR is merged auto-merge on green by itself — a ready, green, mergeable PR is merged
**explicitly** (`POST .../pulls/{n}/merge`, `{"Do": "squash"}`), per **explicitly** (`POST .../pulls/{n}/merge`, `{"Do": "squash"}`), per
`CLAUDE.md`. That merge is an ordinary push, so `deploy-dev.yml`'s push `CLAUDE.md`. That merge is an ordinary push, so `deploy.yml`'s push
trigger fires naturally afterward. trigger fires naturally afterward.
- **Branch/deploy mapping (settled):** `.forgejo/workflows/deploy.yml` - **Branch/deploy mapping (settled):** `.forgejo/workflows/deploy.yml`
("Deploy to beta VPS") triggers on `main` and SSHes to **beta** triggers on `dev`, `main` and `release`, and SSHes to the environment each
(`75.119.132.91`) running `deploy/deploy.sh`. **Prod is not deployed by a branch maps to (`dev` → dev on **vps1**, `main` → beta on **vps2**,
push-triggered workflow** — it's deployed with `terraform apply` `release` → prod on **vps2**), running `deploy/deploy-dev.sh` or
(`terraform/README.md`) on the `release` branch. So the promotion chain is `deploy/deploy.sh` per `deploy/env-topology.sh`. Credentials are keyed by
`dev` → `main` (this workflow deploys beta) → `release` (Terraform deploys **host** (`VPS1_SSH_*`, `VPS2_SSH_*`), not by environment, since vps2 alone
prod). There is deliberately no `release`-triggered workflow. now answers to two of them — see the workflow's own header comment for why
that rename matters.
A second, independent set of Forgejo workflows A second, independent set of Forgejo workflows
(`.forgejo/workflows/{ci,build-push,ops-cron}.yml`) was added by a parallel (`.forgejo/workflows/{build-push,ops-cron}.yml`) was added by a parallel
effort implementing `thermograph-docs/runbooks/implementation-handoff.md` Track A chunk effort implementing `thermograph-docs/runbooks/implementation-handoff.md` Track A chunk
7 — see that doc and its own reconciliation PR (which already fixed its 7 — see that doc and its own reconciliation PR. `build-push.yml` (image →
runner-label and ssh-action guesses to match the real infrastructure here). Forgejo's built-in registry) and `ops-cron.yml` (backup + IndexNow) are novel
`ci.yml` duplicated this PR's `build.yml` exactly and was dropped in that and still present; the latter's secrets are also keyed by host now (see its
reconciliation; `build-push.yml` (image → Forgejo's built-in registry) and own header comment).
`ops-cron.yml` (backup + IndexNow) are novel and still present.
### 3d. Cut over — DONE (2026-07-21) ### 3d. Cut over — DONE (2026-07-21)
The GitHub → Forgejo cutover is complete; this records what was done: The GitHub → Forgejo cutover is complete; this records what was done:
1. Deploy secrets (`SSH_HOST`, `SSH_USER`, `SSH_KEY`, `SSH_PORT`) live in 1. Deploy secrets live in Forgejo's repo Secrets, keyed by host
Forgejo's repo Secrets, pointing at beta with a dedicated CI deploy key. (`VPS1_SSH_*`, `VPS2_SSH_*`) since the vps1/vps2 split, pointing at each
2. The LAN dev runner was re-pointed to Forgejo via box with a dedicated CI deploy key.
`deploy/forgejo/register-lan-runner.sh` (systemd --user, on the desktop). 2. The Actions runner is registered against Forgejo via
`deploy/forgejo/register-lan-runner.sh`.
3. The repo was migrated into Forgejo; `dev`/`main`/`release` all promoted 3. The repo was migrated into Forgejo; `dev`/`main`/`release` all promoted
through Forgejo and deploys verified live. through Forgejo and deploys verified live.
4. GitHub is retired as git host and CI (PR "Retire GitHub as the git host and 4. GitHub is retired as git host and CI (PR "Retire GitHub as the git host and
CI platform") — `origin` remains only as a read-only mirror of history. CI platform") — `origin` remains only as a read-only mirror of history.
## Verification checklist — all met (2026-07-21) ## Verification checklist — all met (2026-07-21, topology as of 2026-07-25)
- [x] `ssh agent@<prod>` and `ssh agent@<beta>` both work; `sudo whoami` → `root` - [x] `ssh agent@<vps1>` and `ssh agent@<vps2>` both work; `sudo whoami` → `root`
- [x] Password SSH auth confirmed dead on both boxes (tested with an actual - [x] Password SSH auth confirmed dead on both boxes (tested with an actual
failed login attempt, not just config inspection) failed login attempt, not just config inspection)
- [x] `docker node ls` (from prod) shows all three nodes `Ready` - [x] `docker node ls` (from vps2, the manager) shows all three nodes `Ready`
- [x] Swarm ports (2377/7946/4789) unreachable from outside the WireGuard tunnel - [x] Swarm ports (2377/7946/4789) unreachable from outside the WireGuard tunnel
- [x] `https://git.thermograph.org` serves Forgejo over TLS; `/v2/` registry - [x] `https://git.thermograph.org` serves Forgejo over TLS; `/v2/` registry
API is 403 from off-mesh (firewalled to the WireGuard CIDR) API is 403 from off-mesh (firewalled to the WireGuard CIDR)
- [x] A test PR flow verified: required `build` check runs → explicit squash - [x] A test PR flow verified: required `build` check runs → explicit squash
merge → LAN dev deploy lands merge → deploy lands on the branch's mapped environment
- [x] GitHub retired as git host + CI; kept only as a read-only history mirror - [x] GitHub retired as git host + CI; kept only as a read-only history mirror
TODO(cutover): re-verify the checklist above against the vps1/vps2 rename
specifically (e.g. confirm the Forgejo Swarm label now reads `vps1`'s node
name, not a leftover `beta` node name) — this document was updated to match
the new topology brief, but a fresh live check wasn't run as part of this
pass.

96
infra/CLAUDE.md
View file

@ -8,33 +8,67 @@ orchestrator.
## The machines ## The machines
Per `ACCESS.md`: **dev** (operator's box — LAN dev server and CI runner), Two VPS boxes plus the operator's desktop, on one WireGuard mesh, named by
**prod** (`169.58.46.181`, thermograph.org, `agent` user, passwordless sudo), ROLE rather than by environment:
**beta** (`75.119.132.91`, beta.thermograph.org + Forgejo + Grafana/Loki,
`agent` user). All four are on a WireGuard mesh. Hosts' `/opt/thermograph` is a - **vps1** — `75.119.132.91`, mesh `10.10.0.2`. "Operational programs": Forgejo
checkout of **this monorepo**, not an infra-only repo. (git + CI + registry, `git.thermograph.org`), Grafana/Loki/Alloy
(`dashboard.thermograph.org`), the `emigriffith.dev` portfolio site, and the
**dev** environment (`/opt/thermograph-dev`, tracks `dev`), with its own
Postgres container. Dev is mesh-only here — published on `10.10.0.2:8137`,
no public DNS, no Caddy site, no TLS.
- **vps2** — `169.58.46.181`, mesh `10.10.0.1`. "The deployed environment" —
everything an external user can reach: **prod** (`/opt/thermograph`, tracks
`main`, Swarm stack `thermograph`) AND **beta** (`/opt/thermograph-beta`,
tracks `main`, Swarm stack `thermograph-beta`), Centralis, Postfix, and the
backups. One shared TimescaleDB instance serves both, on separate
databases/roles (`thermograph` / `thermograph_beta`).
- **desktop** — mesh `10.10.0.3`. Hosts no Thermograph environment at all: AI
model hosting plus flex Swarm-worker capacity. `make dev-up` still works
there as a laptop-local rehearsal, but that is not an environment.
`agent` user, passwordless sudo, on vps1 and vps2. A host's `/opt/thermograph*`
is a checkout of **this monorepo**, not an infra-only repo — note vps2 carries
**two** such checkouts side by side (`/opt/thermograph` and
`/opt/thermograph-beta`), which is the one thing most deploy logic here has to
get right that a single-environment host never had to.
## Deploy paths ## Deploy paths
- **`deploy/deploy.sh`** — the single entry point for beta and prod. Resets the - **`deploy/env-topology.sh`** — sourced by every path below. `THERMOGRAPH_ENV`
host checkout to `BRANCH` (default `main`), renders secrets, then routes: (`dev`/`beta`/`prod`) is the input; host, checkout, branch, deploy mode,
if `/etc/thermograph/deploy-mode` says `stack` it execs stack/compose name, env file, LB ports, DB role/database and service-name
`deploy/stack/deploy-stack.sh`, otherwise it rolls compose services. Per-service prefix are all derived from it. This exists because vps2 alone runs two
tags persist in untracked `deploy/.image-tags.env` (compose) and environments — a host-wide marker can no longer answer "which environment is
this", so the caller (the deploy workflow) says so explicitly.
- **`deploy/deploy.sh`** — the single entry point for dev, beta and prod.
Resets the host checkout to `BRANCH`, renders secrets, then routes: beta and
prod (`TG_DEPLOY_MODE=stack`) exec `deploy/stack/deploy-stack.sh`; dev
(`compose`) rolls compose services. The old host-wide
`/etc/thermograph/deploy-mode` marker is honoured only when
`env-topology.sh` can't resolve an environment at all (a checkout that
predates the split, or a by-hand run with nothing set). Per-service tags
persist in untracked `deploy/.image-tags.env` (compose) and
`deploy/.stack-image-tags.env` (stack) so the two never mix. `deploy/.stack-image-tags.env` (stack) so the two never mix.
- **`deploy/stack/deploy-stack.sh`** — the Swarm path, live on prod. `backend` - **`deploy/stack/deploy-stack.sh`** — the Swarm path, live on vps2 for both
rolls web **and** worker; `frontend` rolls frontend; `all` runs a full prod and beta, picking the stack file/ports/DB role/service prefix out of
`docker stack deploy`. Start-first, health-gated, auto-rollback. `env-topology.sh`. `backend` rolls web **and** worker; `frontend` rolls
`STACK_TEST=1` rehearses the whole thing under stack name `thermograph-test` frontend; `all` runs a full `docker stack deploy`. Start-first, health-gated,
auto-rollback. `STACK_TEST=1` rehearses under stack name `thermograph-test`
on throwaway volumes and ports `18137`/`18080`. on throwaway volumes and ports `18137`/`18080`.
- **`deploy/deploy-dev.sh`** — thin LAN-dev wrapper (dev compose overlay, - **`deploy/deploy-dev.sh`** — thin wrapper around `deploy.sh` for dev on vps1
`~/thermograph-dev`). Its CI trigger is inert; see root `CLAUDE.md`. (dev compose overlay, `/opt/thermograph-dev`, `THERMOGRAPH_SECRETS_SKIP_COMMON=1`).
Deployed like beta/prod now — a push to `dev` triggers the same `Deploy`
workflow over SSH, no LAN-specific runner involved.
- **`Makefile`** — compose orchestration only: `up`, `down`, `db-up`, `dev-up`, - **`Makefile`** — compose orchestration only: `up`, `down`, `db-up`, `dev-up`,
`om-up`, `om-backfill`. `om-up`, `om-backfill`.
Volumes are the reason the compose project name is pinned: compose creates Volumes are the reason the compose project name is pinned: compose creates
`thermograph_pgdata`/`_appdata`/`_applogs`, and the Swarm stack declares those `thermograph_pgdata`/`_appdata`/`_applogs`, and the Swarm stack declares those
same names as `external: true` at the same mount paths. same names as `external: true` at the same mount paths. Beta's stack has no
`db` of its own — it reaches prod's `db` service over prod's `thermograph_internal`
overlay (declared `external: true` in beta's stack file) and keeps a separate
`internal` overlay for its own beta-to-beta traffic.
## Rules ## Rules
@ -42,13 +76,29 @@ same names as `external: true` at the same mount paths.
an apply would attempt full re-provisioning of live hosts. Terraform here is an apply would attempt full re-provisioning of live hosts. Terraform here is
executable documentation until state is bootstrapped. executable documentation until state is bootstrapped.
- **Secrets: SOPS vault only** (`deploy/secrets/*.yaml`, `sops edit` → commit → - **Secrets: SOPS vault only** (`deploy/secrets/*.yaml`, `sops edit` → commit →
deploy). Never hand-edit `/etc/thermograph.env` — it is a rendered artifact. deploy). Never hand-edit `/etc/thermograph.env` / `/etc/thermograph-beta.env`
`secrets-guard` CI rejects plaintext. `deploy/secrets/seed-from-live.sh` reads — they are rendered artifacts. `secrets-guard` CI rejects plaintext.
production secrets and is **not** for an agent to run. `deploy/secrets/seed-from-live.sh` reads production secrets and is **not**
for an agent to run.
- **vps1 must never hold prod credentials.** It's the box that runs Forgejo,
its CI runner, and dev's unreviewed branch — the opposite of an isolation
boundary. This is why dev renders `dev.yaml` **alone** and never layers
`common.yaml` (the fleet's shared production credential set). See
`deploy/secrets/README.md`.
- **A foothold on vps2 is a foothold on both beta and prod's host** — they are
co-resident by design now. What still separates them is the database
(separate roles/databases, `CONNECT` revoked from `PUBLIC`) and the
filesystem (separate checkouts, separate rendered env files). Don't describe
beta and prod as isolated at the host or SSH-credential level; they aren't
anymore.
- **The ops cron (`.forgejo/workflows/ops-cron.yml`, at the repo root) is THE - **The ops cron (`.forgejo/workflows/ops-cron.yml`, at the repo root) is THE
prod backup.** It uses `PROD_SSH_*`, not `SSH_*` — an earlier revision reused backup for both application databases and for Forgejo.** Secrets are keyed
`SSH_*` and silently dumped beta while prod had no backup at all. If you touch by host, not environment: `VPS2_SSH_*` for the `backup`/`indexnow` jobs
it, verify a dump actually lands in `agent@prod:~/thermograph-backups/`. (prod and beta's databases, both on vps2) and `VPS1_SSH_*` for
`forgejo-backup` (Forgejo now lives on vps1, not co-located with beta). If
you touch it, verify a dump actually lands for **both** the prod and beta
databases, not just one — an earlier revision backed up only prod and
silently left beta uncovered.
- Shell here runs as root over SSH against live hosts with no test suite in - Shell here runs as root over SSH against live hosts with no test suite in
front of it. `shell-lint` CI (pinned shellcheck) is the only guard — keep the front of it. `shell-lint` CI (pinned shellcheck) is the only guard — keep the
tree at zero findings. tree at zero findings.

233
infra/DEPLOY-DEV.md
View file

@ -1,33 +1,45 @@
# Dev CI/CD → LAN server on this machine # Dev CI/CD → the dev environment on vps1
Parallel to the prod pipeline (`main` → VPS, see `DEPLOY.md`), the **`dev`** Parallel to beta/prod's pipeline (`main`→beta, `release`→prod, see `DEPLOY.md`),
branch continuously deploys to a **LAN server running on this computer**. Git the **`dev`** branch continuously deploys to the **dev environment on vps1**
hosting and CI are self-hosted **Forgejo** (`git.thermograph.org`, reachable (`75.119.132.91`, mesh `10.10.0.2`) — `/opt/thermograph-dev`, a normal fleet
at `http://10.10.0.2:3080` over the WireGuard mesh) — GitHub is retired. checkout reached over SSH exactly like beta and prod, **not** a stack on the
operator's desktop or LAN any more. Git hosting and CI are self-hosted
**Forgejo** (`git.thermograph.org`, also on vps1, reachable at
`http://10.10.0.2:3080` over the WireGuard mesh) — GitHub is retired.
``` ```
open PR ──▶ CI (build + boot/health, on the LAN Forgejo runner) open PR ──▶ CI (build + boot/health, on the `docker`-labeled Forgejo runner)
│ green │ green
▼ ▼
merge into dev (explicit — Forgejo does not auto-merge on its own; merge into dev (explicit — Forgejo does not auto-merge on its own;
see "Landing a PR" below) see "Landing a PR" below)
│ │
▼ ▼
LAN runner (thermograph-lan label) runs deploy/deploy-dev.sh the Deploy workflow SSHes into vps1 (VPS1_SSH_*, THERMOGRAPH_ENV=dev)
and runs deploy/deploy-dev.sh
│ │
▼ ▼
~/thermograph-dev updated ─▶ docker compose stack (backend + frontend + /opt/thermograph-dev updated ─▶ docker compose stack (backend + frontend +
Postgres 18) brought up, backend on 0.0.0.0:8137 Postgres) brought up, backend published
MESH-ONLY on 10.10.0.2:8137
``` ```
Reachable at `http://<lan-ip>:8137/` from any device on your Wi-Fi. Reachable at `http://10.10.0.2:8137/` from anything already on the WireGuard
mesh, and from nowhere else — no public DNS record, no Caddy site, no TLS.
vps1 is a public VPS running whatever branch is currently in flight (including
unreviewed code), so unlike the old LAN box it must **never** publish on
`0.0.0.0`.
The dev server runs the **same containerized stack as prod** (`docker-compose.yml`), The dev stack runs the **same containerized stack as prod** (`docker-compose.yml`),
overlaid with `docker-compose.dev.yml`: **uncapped** (no CPU limits, unlike prod's overlaid with `docker-compose.dev.yml`: **uncapped** (no CPU limits, unlike
backend=4 / frontend=2 / db=2) and backend published on `0.0.0.0:8137` for the prod's Swarm-managed replicas) and backend published on the WireGuard address
LAN (prod binds loopback behind Caddy; frontend has no published port here (`10.10.0.2:8137` — set via `DEV_BIND_ADDR` in `deploy/env-topology.sh`; prod
either way, no Caddy to reach it directly, so it's only reached through and beta bind loopback behind their own Caddy LBs instead). Frontend has no
backend's own reverse-proxy fallback). Bring it up by hand with `make dev-up`. published port either way — reached only through backend's own
reverse-proxy fallback. Bring dev up by hand with `make dev-up`, which is a
**laptop-local convenience** (uncapped, loopback-published) — not the dev
environment itself, which only ever runs on vps1.
## Why it's built this way ## Why it's built this way
@ -39,26 +51,40 @@ via the API (`POST .../pulls/{n}/merge`, `{"Do": "squash"}`). Nothing merges
a ready PR for you automatically just because checks pass, unlike GitHub's a ready PR for you automatically just because checks pass, unlike GitHub's
old in-workflow `ci-cd.yml` (retired along with the rest of `.github/`). old in-workflow `ci-cd.yml` (retired along with the rest of `.github/`).
**Dev moved off the desktop and onto vps1 so the desktop could retire from the
Thermograph estate entirely.** The desktop now hosts AI models and offers flex
Swarm-worker capacity — it is no longer "the LAN dev server" and no longer the
primary CI runner story. What dev keeps that beta and prod do not:
- It renders `dev.yaml` **alone**, never layering `common.yaml` (the fleet's
shared production credentials) — vps1 also runs Forgejo and its CI, and dev
runs whatever branch is in flight. See the long note in
`deploy/render-secrets.sh` and `deploy/secrets/README.md`.
- It is **mesh-only** — see above. A public VPS running unreviewed branches
must never be reachable from the open internet the way the old LAN box
(behind a home router, reachable only on the Wi-Fi) safely could be.
## Moving parts ## Moving parts
| File | Purpose | | File | Purpose |
|------|---------| |------|---------|
| `.forgejo/workflows/pr-build.yml` | required status check for PRs into `dev` (calls `build.yml`) | | `.forgejo/workflows/pr-build.yml` | required status check for PRs into `dev` (calls `build.yml`) |
| `.forgejo/workflows/deploy-dev.yml` | build + deploy on pushes to `dev` (direct, or a PR merge) | | `.forgejo/workflows/deploy.yml` | the single Deploy workflow — `dev` branch pushes SSH into vps1 and run `deploy-dev.sh` |
| `.forgejo/workflows/build.yml` | shared build gate: deps, backend tests, JS syntax check, boot/health check | | `.forgejo/workflows/build.yml` | shared build gate: deps, backend tests, JS syntax check, boot/health check |
| `deploy/deploy-dev.sh` | pull `dev` into `~/thermograph-dev`, `docker compose up` the uncapped LAN stack, health check | | `deploy/env-topology.sh` | source of truth: dev = vps1, `/opt/thermograph-dev`, branch `dev`, compose mode, mesh-only bind address |
| `deploy/deploy-dev.sh` | thin wrapper around `deploy.sh`: dev's compose overlay + secrets policy, then delegates to the shared pull/roll/health-check logic |
| `deploy/secrets/dev.yaml` | dev's SOPS vault — its own secrets and config, rendered to `/etc/thermograph.env` at deploy time (see "Config and secrets") | | `deploy/secrets/dev.yaml` | dev's SOPS vault — its own secrets and config, rendered to `/etc/thermograph.env` at deploy time (see "Config and secrets") |
| `docker-compose.dev.yml` | dev overlay: no CPU limits, backend published on `0.0.0.0:8137` (frontend unpublished) | | `docker-compose.dev.yml` | dev overlay: no CPU limits, backend published on the mesh address (frontend unpublished) |
| `deploy/provision-dev-lan.sh` | one-time sudo-free bootstrap of the checkout + runner | | `deploy/provision-dev.sh` | one-time bootstrap of the checkout + vault marker on vps1 |
| `deploy/forgejo/register-lan-runner.sh` | registers the Forgejo Actions runner on this machine | | `deploy/forgejo/register-lan-runner.sh` | registers the Forgejo Actions runner (name predates the vps1/vps2 split — see the script's own header) |
`deploy-dev.yml`'s `deploy` job runs on the `thermograph-lan` label — **not** `deploy.yml`'s dev leg SSHes in with `VPS1_SSH_*` and runs
`[self-hosted, thermograph-lan]` (the array form is what the original GitHub `/opt/thermograph-dev/infra/deploy/deploy-dev.sh` directly over SSH — the same
version used; GitHub implicitly tags every self-hosted runner `self-hosted`, shape as the beta and prod legs, just a different host and script. There is no
but Forgejo's runner has only the labels it was explicitly registered with, longer a `thermograph-lan`-labeled runner job doing host-native
so the array form is permanently unschedulable there — a real bug found and `systemctl`/`docker-compose` calls from inside a CI job; the deploy happens the
fixed once, worth knowing about if a "deploy" job ever silently stops same way for all three environments now (a `docker`-labeled runner opens an SSH
appearing in the Actions history again). session to the target host and runs that environment's deploy script there).
`deploy-dev.sh`'s git auth (`GH_TOKEN`/`GITHUB_TOKEN`, Forgejo's per-job `deploy-dev.sh`'s git auth (`GH_TOKEN`/`GITHUB_TOKEN`, Forgejo's per-job
token exposed under that name for compatibility) is scoped to `REPO_URL`'s token exposed under that name for compatibility) is scoped to `REPO_URL`'s
@ -68,15 +94,25 @@ hardcoded host would make git silently skip the header (no error) and fall
through to whatever ambient credentials happen to be available, not fail through to whatever ambient credentials happen to be available, not fail
loudly. loudly.
## The self-hosted runner ## The Forgejo Actions runner
A Forgejo Actions **runner** (`forgejo-runner`) runs on this machine, labels TODO(cutover): `deploy/forgejo/register-lan-runner.sh`'s own header comment
still describes "the desktop" as the canonical placement for this runner, and
its `docker`-labeled jobs are what actually execute `deploy.yml`'s SSH-based
dev/beta/prod deploy steps today. Whether the runner has actually moved to
vps1 (alongside Forgejo) or still runs on the desktop under the new topology
is not settled by anything read for this pass — that script is out of this
document's edit scope. Confirm where it actually lives before relying on the
description below.
A Forgejo Actions **runner** (`forgejo-runner`) serves this repo, labels
`docker` (containerized jobs, node:20-bookworm, with the host's docker.sock `docker` (containerized jobs, node:20-bookworm, with the host's docker.sock
automounted in — `container.docker_host: automount` in automounted in — `container.docker_host: automount` in
`~/forgejo-runner/config.yaml` — so `docker build`/`push` work without `~/forgejo-runner/config.yaml` — so `docker build`/`push` work without
privileged Docker-in-Docker) and `thermograph-lan` (host-native jobs, for privileged Docker-in-Docker) and `thermograph-lan` (a legacy label from when a
`deploy-dev.sh`'s systemctl/docker-compose calls). Installed under deploy job ran host-native on the LAN box; nothing in the current `deploy.yml`
`~/forgejo-runner`, runs as the `forgejo-runner` systemd `--user` service. schedules against it, since every environment now deploys over SSH from the
`docker`-labeled job instead).
```bash ```bash
# status / logs # status / logs
@ -87,62 +123,66 @@ journalctl --user -u forgejo-runner -f
bash deploy/forgejo/register-lan-runner.sh <forgejo_url> <registration_token> bash deploy/forgejo/register-lan-runner.sh <forgejo_url> <registration_token>
``` ```
`git.thermograph.org`'s public DNS resolves to beta's **public** IP, which Registry pushes (`build-push.yml`) run against the runner host's own
the registry's mesh-only ACL (see `deploy/forgejo/README.md`) rejects for automounted Docker daemon, so it's that **host's** own resolver that needs to
`/v2/` paths — `docker build-push.yml` pushes run against the *host's* route `git.thermograph.org` over the mesh, not anything settable from a job
automounted daemon, so it's this **host's** own resolver that needs to route container. If registry pushes ever start failing with an HTTP/TLS mismatch or
git.thermograph.org over the mesh, not anything settable from a job a 403 on `/v2/`, check `getent hosts git.thermograph.org` on the runner host —
container. If registry pushes ever start failing with an HTTP/TLS mismatch it needs to resolve to vps1's WireGuard IP (`10.10.0.2`), typically via a
or a 403 on `/v2/`, check `getent hosts git.thermograph.org` here — it needs `/etc/hosts` line, not the public one (a non-issue if the runner is co-located
to resolve to beta's WireGuard IP (`10.10.0.2`), typically via a `/etc/hosts` with Forgejo on vps1 itself, since `git.thermograph.org` then resolves to
line, not the public one. itself either way).
## The LAN dev service (Docker Compose) ## The dev environment (Docker Compose, on vps1)
Runs as a Docker Compose stack, not a bare systemd unit — `docker ps --filter Runs as a Docker Compose stack, not a bare systemd unit — `docker ps --filter
name=thermograph-dev` shows `thermograph-dev-backend-1`, `thermograph-dev-frontend-1` name=thermograph-dev` on vps1 shows `thermograph-dev-backend-1`,
(repo-split Stage 4 split the single "app" container in two — frontend has no `thermograph-dev-frontend-1` (frontend has no published port, reached only
published port, reached only through backend's own reverse-proxy fallback) and through backend's own reverse-proxy fallback) and `thermograph-dev-db-1`
`thermograph-dev-db-1` (TimescaleDB). A legacy pre-container (TimescaleDB, dev's **own** container — not the shared instance on vps2).
`thermograph-dev.service` systemd --user unit may still exist from before this
cutover; it's stale and `deploy-dev.sh` stops/disables it on every run — don't
trust `systemctl --user status thermograph-dev` as a liveness check, use `docker
ps` instead.
```bash ```bash
docker ps --filter name=thermograph-dev # is it up? docker ps --filter name=thermograph-dev # is it up?
docker compose -f docker-compose.yml -f docker-compose.dev.yml logs -f backend frontend # app logs (from ~/thermograph-dev) docker compose -f docker-compose.yml -f docker-compose.dev.yml logs -f backend frontend # app logs (from /opt/thermograph-dev/infra)
``` ```
Bootstrap (or rebuild) it by hand: Bootstrap (or rebuild) it:
```bash ```bash
bash deploy/provision-dev-lan.sh sudo bash deploy/provision-dev.sh
``` ```
The dedicated checkout at `~/thermograph-dev` is separate from your working tree, replacing the retired `deploy/provision-dev-lan.sh`, which bootstrapped dev on
so a deploy never touches uncommitted edits in `~/Code/Thermograph`. the operator's desktop as a sudo-free `systemd --user` stack. Dev is
provisioned like any other fleet environment now — a checkout at
`/opt/thermograph-dev`, root-owned, deployed over SSH by CI like beta and
prod. The dedicated checkout is separate from your own working tree, so a
deploy never touches uncommitted edits there.
## Config and secrets ## Config and secrets
Dev's configuration lives in the same SOPS vault as prod's and beta's — Dev's configuration lives in the same SOPS vault as prod's and beta's —
`deploy/secrets/dev.yaml`, encrypted to the same age recipient, rendered to `deploy/secrets/dev.yaml`, encrypted to the same age recipient, rendered to
`/etc/thermograph.env` by the same `render-secrets.sh` at deploy time. Changing a `/etc/thermograph.env` by the same `render-secrets.sh` at deploy time. Changing a
dev value is `sops deploy/secrets/dev.yaml` → commit → deploy, exactly as for the dev value is `sops deploy/secrets/dev.yaml` → commit → deploy, exactly as for
VPS boxes. **`deploy/secrets/README.md` is the reference**; what follows is only beta and prod. **`deploy/secrets/README.md` is the reference**; what follows
what is specific to this machine. is only what is specific to this environment.
Dev renders `dev.yaml` **alone**. It does *not* layer `common.yaml`, which is the Dev renders `dev.yaml` **alone**. It does *not* layer `common.yaml`, which is the
fleet's shared production credential set — registry token, both S3 keypairs (one fleet's shared production credential set — registry token, both S3 keypairs (one
read-write pair, on the bucket that also holds prod's backups), the VAPID private read-write pair, on the bucket that also holds prod's backups), the VAPID private
key that signs push to real subscribers, the IndexNow key, the metrics token. This key that signs push to real subscribers, the IndexNow key, the metrics token. This
box is the CI runner and runs unreviewed `dev`-branch code; a plaintext render of box also runs Forgejo and its CI, and executes unreviewed `dev`-branch code; a
that set into `/etc/thermograph.env` would put all of it in every dev container's plaintext render of that set into `/etc/thermograph.env` would put all of it in
environment. `deploy-dev.sh` exports `THERMOGRAPH_SECRETS_SKIP_COMMON=1` to prevent every dev container's environment. `deploy-dev.sh` exports
it, and **aborts the deploy** if the renderer in the checkout cannot honour that. `THERMOGRAPH_SECRETS_SKIP_COMMON=1` to prevent it, and **aborts the deploy** if
the renderer in the checkout cannot honour that. This is a property of the
`dev` **environment** (encoded in `deploy/env-topology.sh`'s `TG_SKIP_COMMON`),
not just of `deploy-dev.sh` — so it holds regardless of which entry point is
used to deploy it.
So dev holds its own `THERMOGRAPH_AUTH_SECRET` (its own, not prod's — the `daemon` So dev holds its own `THERMOGRAPH_AUTH_SECRET` (its own, not prod's — the `daemon`
service refuses to start without one), its own `POSTGRES_PASSWORD`, a LAN service refuses to start without one), its own `POSTGRES_PASSWORD`, a mesh-only
`THERMOGRAPH_BASE_URL`, `THERMOGRAPH_COOKIE_SECURE=0` (plain HTTP — the shared `THERMOGRAPH_BASE_URL`, `THERMOGRAPH_COOKIE_SECURE=0` (plain HTTP — the shared
value is `1` and would silently break login here), and the non-secret config it value is `1` and would silently break login here), and the non-secret config it
would otherwise have inherited. It holds **no** S3 keys, no VAPID keypair, no would otherwise have inherited. It holds **no** S3 keys, no VAPID keypair, no
@ -150,20 +190,26 @@ IndexNow key, no Discord or mail credentials: the lake falls through to the
Open-Meteo archive without bucket creds, and the app generates and persists its own Open-Meteo archive without bucket creds, and the app generates and persists its own
VAPID and IndexNow keys into the `appdata` volume. VAPID and IndexNow keys into the `appdata` volume.
Two dev-only mechanics, both because this box has **no passwordless sudo** and the Two dev-only mechanics carried over from the old LAN setup, both worth keeping
runner is a `systemd --user` service with no tty: even though dev is no longer sudo-free (vps1's `agent` user has passwordless
sudo like any other fleet host) — the render path just doesn't strictly need
them any more:
- the age key is read from `~/.config/sops/age/keys.txt`, not - the age key can be read from `~/.config/sops/age/keys.txt` as a fallback
`/etc/thermograph/age.key` — a root-owned `0400` key would send the renderer down when `/etc/thermograph/age.key` isn't present or readable — see
its `sudo cat` path, where it would hang on a prompt no CI job can answer; `deploy-dev.sh`'s own comment on this;
- `/etc/thermograph.env` is **pre-created** owned by the deploying user, so the - `/etc/thermograph.env` being pre-created and owned by the deploying user
renderer takes its in-place write path and no deploy needs `sudo` at all. lets the renderer take its in-place write path without needing `sudo` at
all, which still matters under the Forgejo runner's `systemd --user`
context (no tty to answer a `sudo` prompt).
Setting this up on a fresh dev machine is four typed commands — see **"Setting up a Setting this up on a fresh dev host is a few typed commands — see **"Setting up a
new dev machine"** in `deploy/secrets/README.md`. Until the new dev machine"** in `deploy/secrets/README.md` (written for the old desktop
`/etc/thermograph/secrets-env` marker exists the box renders nothing and falls back setup; the mechanics are the same on vps1, just root-owned rather than
to `deploy-dev.sh`'s built-in `POSTGRES_PASSWORD` default, which is the safe state sudo-free). Until the `/etc/thermograph/secrets-env` marker exists the box
and the one to return to (delete the marker) if the vault ever gets in the way. renders nothing and falls back to `deploy-dev.sh`'s built-in
`POSTGRES_PASSWORD` default, which is the safe state and the one to return to
(delete the marker) if the vault ever gets in the way.
`REGISTRY_TOKEN` is deliberately not in the vault: dev pulls with the persistent `REGISTRY_TOKEN` is deliberately not in the vault: dev pulls with the persistent
`docker login` credential already in this host's docker config, like the prod/beta `docker login` credential already in this host's docker config, like the prod/beta
@ -196,25 +242,28 @@ with `{"Do": "squash", "delete_branch_after_merge": true}`.
## Day-to-day ## Day-to-day
- **Ship:** open a PR into `dev`, confirm CI is green, merge it explicitly — - **Ship:** open a PR into `dev`, confirm CI is green, merge it explicitly —
the merge triggers the deploy here. the merge triggers a deploy to vps1.
- **Manual redeploy:** trigger `deploy-dev.yml` via `workflow_dispatch` - **Manual redeploy:** trigger the `Deploy` workflow via `workflow_dispatch`
(`POST /api/v1/repos/{owner}/{repo}/actions/workflows/deploy-dev.yml/dispatches`, (`POST /api/v1/repos/{owner}/{repo}/actions/workflows/deploy.yml/dispatches`,
`{"ref": "dev"}`), or locally `APP_DIR=~/thermograph-dev bash deploy/deploy-dev.sh`. `{"ref": "dev"}`), or directly on vps1:
- **Promote to prod:** merge/fast-forward `dev` into `main` and push — that `APP_DIR=/opt/thermograph-dev bash deploy/deploy-dev.sh`.
triggers the VPS pipeline in `DEPLOY.md`. - **Promote to beta:** merge/fast-forward `dev` into `main` and push — that
triggers the beta leg of the same `Deploy` workflow, per `DEPLOY.md`.
## Monitoring ## Monitoring
Logs and dashboards moved to a **separate project** — a central Grafana + Loki Logs and dashboards live in a **separate project** — a central Grafana + Loki
stack fed by a Grafana Alloy agent on every node (including this LAN dev box), stack, hosted on **vps1 itself**, fed by a Grafana Alloy agent on every node
over the WireGuard mesh (`thermograph-observability` on Forgejo; UI at (prod and beta on vps2, dev and the desktop's own agent, all over the
`https://dashboard.thermograph.org`). The dev node's Alloy agent ships its WireGuard mesh; `thermograph-observability` on Forgejo; UI at
container/app logs under `host="dev"`, so LAN dev shows up alongside prod and `https://dashboard.thermograph.org`). Dev's Alloy agent ships its
beta in the same dashboards. This replaced the old `scripts/dashboard.py`. container/app logs under its own `host`/service labels, so dev shows up
alongside prod and beta in the same dashboards. This replaced the old
`scripts/dashboard.py`.
For a quick terminal check without Grafana, the app still serves raw counters at For a quick terminal check without Grafana, the app still serves raw counters at
the gated `GET /api/v2/metrics` route. Dev sets no `THERMOGRAPH_METRICS_TOKEN`, so the gated `GET /api/v2/metrics` route. Dev sets no `THERMOGRAPH_METRICS_TOKEN`, so
the route is direct-loopback-only — `curl localhost:8137/api/v2/metrics` from this the route is direct-loopback-only — `curl localhost:8137/api/v2/metrics` from vps1
machine, which is where you already are. (A token exists on the VPS boxes to allow itself. (A token exists on prod/beta to allow reads through an SSH tunnel; dev has
reads through an SSH tunnel; dev has no reason to carry a copy, and the estate's no reason to carry a copy, and the estate's copy must not land here — see
copy must not land here — see `deploy/secrets/README.md`.) `deploy/secrets/README.md`.)

188
infra/DEPLOY.md
View file

@ -1,62 +1,71 @@
# Deploying Thermograph to a prod VPS # Deploying Thermograph to prod (and beta) on vps2
> **Prod and beta are now provisioned by Terraform** (`terraform/README.md`) and > **Prod and beta both run as Docker Swarm stacks on vps2** (`169.58.46.181`,
> run as a **`docker compose` stack** (backend + frontend + Postgres/TimescaleDB; > mesh `10.10.0.1`) — two separate stacks, two separate checkouts
> repo-split Stage 4 split the single "app" service in two), not the > (`/opt/thermograph` and `/opt/thermograph-beta`), sharing one host because a
> manual venv + systemd model this document originally described. Current shape: > beta green light is meant to be evidence about prod (same orchestrator, same
> prod = new box `169.58.46.181`, branch `release`, deployed with `terraform > Postgres build, same Caddy, same mesh position). `deploy/env-topology.sh` is
> apply`; beta = old box `75.119.132.91`, branch `main`, deployed when `main` is > the single source of truth for which environment lives where; every path
> pushed (`.forgejo/workflows/deploy.yml` SSHes in and runs `deploy/deploy.sh`, > below sources it rather than hardcoding a path or a port. The **manual
> which `docker compose pull`s the image `build-push.yml` already pushed for > walkthrough below (`provision.sh`, `thermograph.service`, the venv) is
> that commit and `up`s it -- repo-split Stage 6's registry-pull cutover; > legacy** — kept as a from-scratch, no-orchestrator reference; prod and beta's
> `deploy.sh` no longer builds in place). The **manual walkthrough below > actual process model is a Swarm stack (`deploy/stack/`), and dev's is plain
> (`provision.sh`, `thermograph.service`, the venv) is legacy** — kept as a > compose on vps1 (see `DEPLOY-DEV.md`), so the systemd/venv specifics below no
> from-scratch, no-Terraform reference; the process model is compose, so the > longer match how any live environment actually runs.
> systemd/venv specifics no longer match how prod/beta actually run.
Pipeline (beta, on `main`): **push to `main` on Forgejo → `build-push.yml` builds Pipeline, both environments: **push to `main` (beta) or `release` (prod) on
+ pushes the image, tagged by SHA → Forgejo Actions SSHes to beta → Forgejo → `build-push.yml` builds + pushes the image, tagged by SHA → the
`deploy/deploy.sh` (`git reset` + `docker login` + `docker compose pull && up`, single `Deploy` workflow (`.forgejo/workflows/deploy.yml`) SSHes into **vps2**
schema migration runs in the app entrypoint) → Caddy fronts it with Let's Encrypt with `THERMOGRAPH_ENV=beta` or `THERMOGRAPH_ENV=prod` → `deploy/deploy.sh`
TLS.** Prod (on `release`) is deployed with `terraform apply`, not a push resolves the right checkout/branch/stack from `env-topology.sh`, `git reset`s
trigger — its `remote-exec` provisioner does the same pull-instead-of-build. it, renders that environment's secrets, and execs `deploy/stack/deploy-stack.sh`
(GitHub is retired/archived — Forgejo, self-hosted at `git.thermograph.org` / (schema migration runs as a one-shot task before the roll) → each environment's
`10.10.0.2:3080` over the WireGuard mesh, is now the sole git host and CI.) own loopback Caddy LB fronts it with TLS.** Credentials are keyed by **host**
(`VPS2_SSH_*`), not by environment — vps2 answers to both prod and beta, so the
environment is passed as data (`THERMOGRAPH_ENV`), not baked into which secret
is used. (GitHub is retired/archived — Forgejo, self-hosted at
`git.thermograph.org` on **vps1** over the WireGuard mesh, is now the sole git
host and CI.)
Files that make this work: Files that make this work:
| File | Where it lives in prod | Purpose | | File | Where it lives on vps2 | Purpose |
|------|------------------------|---------| |------|------------------------|---------|
| `.forgejo/workflows/deploy.yml` | Forgejo | CI job that SSHes in and runs the deploy script (beta, on `main`) | | `.forgejo/workflows/deploy.yml` | Forgejo | CI job that SSHes in and runs the deploy script, for all three environments |
| `.forgejo/workflows/build-push.yml` | Forgejo | builds + pushes the SHA-tagged image `deploy.sh`/Terraform pull | | `.forgejo/workflows/build-push.yml` | Forgejo | builds + pushes the SHA-tagged image `deploy.sh` pulls |
| `deploy/deploy.sh` | `/opt/thermograph/deploy/deploy.sh` | `git reset` + `docker login` + `docker compose pull && up` + health check + warm | | `deploy/env-topology.sh` | `/opt/thermograph{,-beta}/infra/deploy/env-topology.sh` | resolves `THERMOGRAPH_ENV` to a checkout, branch, stack name, ports, DB role |
| `docker-compose.yml` | `/opt/thermograph/docker-compose.yml` | app + Postgres/TimescaleDB services (the process model) | | `deploy/deploy.sh` | `/opt/thermograph{,-beta}/infra/deploy/deploy.sh` | `git reset` + render secrets + routes to the Swarm-stack deploy |
| `deploy/thermograph.env.example` | `/etc/thermograph.env` | Postgres password, VAPID/auth secrets, `WORKERS`, sizing, base path | | `deploy/stack/thermograph-stack.yml` / `thermograph-beta-stack.yml` | same | prod's / beta's Swarm stack (db, web, worker, lake, daemon, frontend; beta has no `db` of its own) |
| `deploy/Caddyfile` | `/etc/caddy/Caddyfile` | reverse proxy + automatic HTTPS | | `deploy/thermograph.env.example` | `/etc/thermograph.env` (prod) / `/etc/thermograph-beta.env` (beta) | Postgres password, VAPID/auth secrets, `WORKERS`, sizing, base path |
| `terraform/` | run from your machine | provisions the box + brings the stack up (replaces `provision.sh`) | | `deploy/stack/lb/Caddyfile` / `Caddyfile.beta` | each stack's loopback LB container | reverse proxy from the host's real Caddy into the Swarm overlay |
| `deploy/provision.sh`, `deploy/thermograph.service` | *(legacy)* | pre-compose venv+systemd bootstrap — superseded by Terraform | | `terraform/` | run from your machine | provisions the box (host-level only; does not know about the two-environment split — see `ACCESS.md`) |
| `deploy/provision.sh`, `deploy/thermograph.service` | *(legacy)* | pre-compose venv+systemd bootstrap — superseded by the Swarm stack |
The app serves on **loopback only**; Caddy is the only thing exposed to the Each app serves on **loopback only** (prod `127.0.0.1:8137`/`:8080`, beta
internet (ports 80/443). The parquet cache in `data/cache/` lives inside the `127.0.0.1:8237`/`:8180`); the host's real Caddy is the only thing exposed to
`/opt/thermograph` checkout and is gitignored, so `git pull` never touches it. the internet (ports 80/443), reverse-proxying into whichever loopback LB
matches the domain requested. Each environment's own checkout keeps its own
gitignored cache under `data/cache/`, so a `git pull` in one never touches the
other's.
**Any deploy host needs a `/etc/hosts` entry for the registry.** **Any deploy host needs a `/etc/hosts` entry for the registry.**
`git.thermograph.org`'s public DNS resolves to beta's public IP, and beta's `git.thermograph.org`'s public DNS resolves to **vps1's** public IP, and
own Caddy rejects `/v2/*` (the registry API) from anything outside the vps1's own Caddy rejects `/v2/*` (the registry API) from anything outside the
WireGuard mesh (10.10.0.0/24) — confirmed live: `docker login`/`pull` from WireGuard mesh (10.10.0.0/24) — confirmed live: `docker login`/`pull` from
prod failed with a 403 until adding `10.10.0.2 git.thermograph.org` to vps2 failed with a 403 until adding `10.10.0.2 git.thermograph.org` to
`/etc/hosts` (10.10.0.2 is beta's mesh IP; every deploy host is already on `/etc/hosts` (10.10.0.2 is vps1's mesh IP; every deploy host is already on
the mesh, this is purely a DNS-routing gap, not a connectivity one). Beta the mesh, this is purely a DNS-routing gap, not a connectivity one). vps1
itself doesn't need this — it's the box Forgejo runs on, so `git.thermograph.org` itself doesn't need this — it's the box Forgejo runs on, so `git.thermograph.org`
just resolves to itself either way. `build-push.yml`'s own header comment just resolves to itself either way. `build-push.yml`'s own header comment
documents the same requirement for the CI runner host. documents the same requirement for the CI runner host.
**Current production layout** (`deploy/Caddyfile`): the app owns **Current production layout**: prod's Caddy config owns **`thermograph.org`**
**`thermograph.org`** at its root (`THERMOGRAPH_BASE=/`, so uvicorn serves `/`, at its root (`THERMOGRAPH_BASE=/`, so uvicorn serves `/`, `/calendar`,
`/calendar`, `/api/v2/…` with no prefix), and **`emigriffith.dev`** serves a `/api/v2/…` with no prefix) and beta's owns **`beta.thermograph.org`** the same
static portfolio at its root with `emigriffith.dev/thermograph*` permanently way. **`emigriffith.dev`** is a separate static portfolio site, served from
redirecting to `thermograph.org` (prefix stripped). Both domains' `A` records **vps1**, not vps2 — it shares no host, checkout or Caddy config with either
point at the same VPS; Caddy provisions a separate cert for each. app environment. `emigriffith.dev/thermograph*` permanently redirects to
`thermograph.org` (prefix stripped).
> **Note:** `deploy.sh` (the CI deploy) only pulls code, reinstalls deps, and > **Note:** `deploy.sh` (the CI deploy) only pulls code, reinstalls deps, and
> restarts the app service — it does **not** touch Caddy or `/etc/thermograph.env`. > restarts the app service — it does **not** touch Caddy or `/etc/thermograph.env`.
@ -202,24 +211,32 @@ If the repo is **public**, skip this — `git pull` needs no auth.
## Accounts & notifications ## Accounts & notifications
Accounts, subscriptions, and notifications are **authoritative and not Accounts, subscriptions, and notifications are **authoritative and not
regenerable** — back them up. On **prod/beta they live in Postgres/TimescaleDB** regenerable** — back them up. **Prod, beta and dev all keep them in
(the compose `db` service, on the `pgdata` volume) alongside the climate record; Postgres/TimescaleDB** — prod and beta on the one shared instance on vps2 (as
on **LAN dev** they fall back to SQLite (`data/accounts.sqlite`) when the `thermograph` and `thermograph_beta` databases, separate roles, `CONNECT`
`THERMOGRAPH_DATABASE_URL` isn't a Postgres URL. revoked from `PUBLIC` — see `deploy/db/provision-env-db.sh`), dev in its own
`db` container on vps1. SQLite (`data/accounts.sqlite`) is only a fallback for
running the app **outside** any of these stacks (a bare venv, the test suite)
when `THERMOGRAPH_DATABASE_URL` isn't a Postgres URL — it's not what any live
environment actually uses.
- **Back up the Postgres volume** (e.g. `pg_dump` the `thermograph` database) as - **Back up the Postgres data** (e.g. `pg_dump` the relevant database) as part
part of the VPS backup routine — losing it wipes all accounts and alerts. (On of each environment's backup routine — losing it wipes all accounts and
a SQLite LAN dev box, back up `data/accounts.sqlite` and its `-wal`/`-shm` alerts for that environment. (Only a bare-venv/offline run backs up
sidecars instead.) `data/accounts.sqlite` and its `-wal`/`-shm` sidecars instead.)
- **Multiple workers, one notifier (leader election).** Prod runs several uvicorn - **Multiple workers, one notifier (leader election).** Each environment runs
workers (`WORKERS`, currently 8 on prod / 4 on beta). The subscription several uvicorn workers (`WORKERS`, sized per environment — see
evaluator and the recurring scheduler must run in exactly one, so the workers `deploy/secrets/{prod,beta,dev}.yaml` and each stack file's replica/worker
elect a single leader — a host-local `flock` (`THERMOGRAPH_SINGLETON_LOCK`, counts). The subscription evaluator and the recurring scheduler must run in
set by the compose app service to `/app/data/notifier.lock`), or a cluster-wide exactly one, so the workers elect a single leader — a host-local `flock`
Postgres advisory lock (`THERMOGRAPH_SINGLETON_PG=1`) if the notifier must be (`THERMOGRAPH_SINGLETON_LOCK`, set by the compose/stack app service to
single across *hosts*. See `backend/core/singleton.py`. `THERMOGRAPH_ROLE` `/app/data/notifier.lock`), or a cluster-wide Postgres advisory lock
(`web`/`worker`/`all`, default `all`) further restricts which processes may own (`THERMOGRAPH_SINGLETON_PG=1`) if the notifier must be single across
it, so a stateless web tier can scale without scaling notifiers. *replicas on multiple hosts* — this is what prod's Swarm stack uses, since
`web` scales to N replicas under the autoscaler. See
`backend/core/singleton.py`. `THERMOGRAPH_ROLE` (`web`/`worker`/`all`,
default `all`) further restricts which processes may own it, so a stateless
web tier can scale without scaling notifiers.
- **Env vars** (all optional, sensible defaults): - **Env vars** (all optional, sensible defaults):
- `THERMOGRAPH_COOKIE_SECURE` — set to `1` when serving over HTTPS (the VPS/TLS - `THERMOGRAPH_COOKIE_SECURE` — set to `1` when serving over HTTPS (the VPS/TLS
deploy) so the session cookie is `Secure`. Leave unset for plain-HTTP LAN, where deploy) so the session cookie is `Secure`. Leave unset for plain-HTTP LAN, where
@ -296,9 +313,11 @@ Why route through a local MTA rather than a provider's API:
`ReadWritePaths=…`) needs no change. `ReadWritePaths=…`) needs no change.
**Default is safe:** `THERMOGRAPH_MAIL_BACKEND` defaults to `console` — it logs **Default is safe:** `THERMOGRAPH_MAIL_BACKEND` defaults to `console` — it logs
the message and sends nothing — so LAN dev and the test suite exercise the whole the message and sends nothing — so dev and the test suite exercise the whole
signup path with no mail server and no risk of mailing a real person. Production signup path with no mail server and no risk of mailing a real person. Prod
opts in with `=smtp`. opts in with `=smtp`; beta stays on `console` too (see
`deploy/stack/thermograph-beta-stack.yml`'s header for why beta must never
hold live mail or Discord credentials).
### Postfix unit topology — and the check that lies ### Postfix unit topology — and the check that lies
@ -431,8 +450,8 @@ Check placement with <https://www.mail-tester.com>, which scores all four at onc
confirmation or password-reset link is mailed. It currently defaults to a confirmation or password-reset link is mailed. It currently defaults to a
per-boot random value, so every outstanding link would break on restart. per-boot random value, so every outstanding link would break on restart.
- **Digest signups** land in the `pending_digest` table in the accounts DB - **Digest signups** land in the `pending_digest` table in the accounts DB
(Postgres on prod/beta; authoritative, back it up). The form ships ahead of (Postgres on every environment; authoritative, back it up). The form ships
delivery on purpose: ahead of delivery on purpose:
addresses are collected now and confirmed once SMTP is live. addresses are collected now and confirmed once SMTP is live.
- **Logs:** `journalctl -u postfix -f`; queue with `mailq`. - **Logs:** `journalctl -u postfix -f`; queue with `mailq`.
@ -464,26 +483,33 @@ regenerated with `python gen_cities.py [n_global] [n_english] [n_extended]`.
proved unreliable (wrong matches), so they're edited by hand; a city not in the list proved unreliable (wrong matches), so they're edited by hand; a city not in the list
simply shows its flavor blurb. Add entries freely — a test checks the slugs are valid. simply shows its flavor blurb. Add entries freely — a test checks the slugs are valid.
- **Archive warming runs automatically on every deploy.** `deploy.sh` (prod) and - **Archive warming runs automatically on every deploy of prod and dev — not
`deploy-dev.sh` (dev) launch `warm_cities.py` detached in the background after the beta.** `deploy/stack/deploy-stack.sh` (prod) and `deploy.sh`'s compose path
health check, so the ~750 city pages serve from cache and a crawl can't burst the (dev, via `deploy-dev.sh`) launch `warm_cities.py` detached in the background
archive API quota. It's idempotent (skips already-cached cells), so only the first after the health check, so the ~750 city pages serve from cache and a crawl
deploy does the full ~25-min warm; later deploys just top up new cities. A page can't burst the archive API quota. Beta deliberately skips this
also self-heals (fetches its archive once) if hit before warming finishes. To run (`TG_POST_DEPLOY=0` in `env-topology.sh`) — it would spend the shared
it by hand: `cd backend && python warm_cities.py --pace 2`. Logs: prod upstream archive quota filling a cache only a rehearsal environment reads.
`logs/warm-cities.log`; dev `journalctl --user -u thermograph-warm-cities`. Warming is idempotent (skips already-cached cells), so only the first deploy
does the full ~25-min warm; later deploys just top up new cities. A page
also self-heals (fetches its archive once) if hit before warming finishes. To
run it by hand: `cd backend && python warm_cities.py --pace 2`. Logs land at
`logs/warm-cities.log` inside each environment's own backend container (dev
is a normal container stack on vps1 now, not a bare systemd unit; there's no
`journalctl --user` path any more).
- **Submit the sitemap** in Google Search Console: `https://thermograph.org/sitemap.xml`. - **Submit the sitemap** in Google Search Console: `https://thermograph.org/sitemap.xml`.
- `jinja2` is a new dependency (already in `requirements.txt`). - `jinja2` is a new dependency (already in `requirements.txt`).
## Monitoring ## Monitoring
Fleet-wide logs and dashboards live in a **separate project** — a central Fleet-wide logs and dashboards live in a **separate project** — a central
Grafana + Loki stack fed by a Grafana Alloy agent on every node, over the Grafana + Loki stack, hosted on **vps1** alongside Forgejo, fed by a Grafana
WireGuard mesh (`thermograph-observability` on Forgejo; UI at Alloy agent on every node over the WireGuard mesh (`thermograph-observability`
**`https://dashboard.thermograph.org`**, Google SSO). It ingests every on Forgejo; UI at **`https://dashboard.thermograph.org`**, Google SSO). It
container's stdout, Caddy's access logs, and the app's structured JSON logs ingests every container's stdout, Caddy's access logs, and the app's
(`logs/{errors,access,audit}/*.jsonl`). This replaced the old SSH-tailed structured JSON logs (`logs/{errors,access,audit}/*.jsonl`) — from prod and
`scripts/dashboard.py`. beta on vps2 as well as dev on vps1, each distinguishable by its `host`/service
labels. This replaced the old SSH-tailed `scripts/dashboard.py`.
The app still exposes raw counters at the gated `GET /api/v2/metrics` route The app still exposes raw counters at the gated `GET /api/v2/metrics` route
(loopback-only; refuses any request carrying a proxy `X-Forwarded-*` header, so (loopback-only; refuses any request carrying a proxy `X-Forwarded-*` header, so

10
infra/Makefile
View file

@ -30,8 +30,14 @@ db-up:
db-down: db-down:
docker compose stop db docker compose stop db
# The LAN dev stack: uncapped (no CPU limits) and published on 0.0.0.0:8137 so # The dev overlay: uncapped (no CPU limits). Default publish is 0.0.0.0:8137,
# phones on the Wi-Fi can reach it (the base file is prod: capped + loopback). # right for a laptop-local rehearsal (e.g. phones on the same Wi-Fi) -- this is
# also the *fleet* dev environment's own overlay, but that environment now runs
# on vps1 (a public VPS, not a LAN box) and deploys via deploy/deploy-dev.sh,
# which overrides the bind address to the WireGuard mesh IP only
# (DEV_BIND_ADDR, see deploy/env-topology.sh) -- never invoke `make dev-up`
# directly on a fleet host, or it publishes on every interface. (The base file
# is prod/beta's shape: capped + loopback, fronted by their own Caddy LB.)
DEV_COMPOSE = docker compose -f docker-compose.yml -f docker-compose.dev.yml DEV_COMPOSE = docker compose -f docker-compose.yml -f docker-compose.dev.yml
dev-up: dev-up:
POSTGRES_PASSWORD=$${POSTGRES_PASSWORD:-thermograph-dev} $(DEV_COMPOSE) up -d --build POSTGRES_PASSWORD=$${POSTGRES_PASSWORD:-thermograph-dev} $(DEV_COMPOSE) up -d --build

87
infra/README.md
View file

@ -3,8 +3,9 @@
Infrastructure for [Thermograph](https://thermograph.org): Terraform host Infrastructure for [Thermograph](https://thermograph.org): Terraform host
provisioning, the SOPS+age secrets vault, WireGuard/Swarm networking, Forgejo, provisioning, the SOPS+age secrets vault, WireGuard/Swarm networking, Forgejo,
Caddy, mail, and the deploy scripts that run the already-built app images on each Caddy, mail, and the deploy scripts that run the already-built app images on each
host. This is a domain of the `emi/thermograph` monorepo — hosts' `/opt/thermograph` host. This is a domain of the `emi/thermograph` monorepo — a host's checkout
is a checkout of the whole monorepo, and `infra/` never builds app source; it only (`/opt/thermograph`, `/opt/thermograph-beta`, or `/opt/thermograph-dev`) is a
checkout of the whole monorepo, and `infra/` never builds app source; it only
runs published images. runs published images.
- **`terraform/`** — provisions/configures hosts (SSH-driven by default; an - **`terraform/`** — provisions/configures hosts (SSH-driven by default; an
@ -13,37 +14,61 @@ runs published images.
executable documentation, not a routine operation. executable documentation, not a routine operation.
- **`deploy/secrets/`** — the git-native SOPS+age secrets vault (every app secret, - **`deploy/secrets/`** — the git-native SOPS+age secrets vault (every app secret,
encrypted at rest, rendered at deploy time). See `deploy/secrets/README.md`. encrypted at rest, rendered at deploy time). See `deploy/secrets/README.md`.
- **`deploy/swarm/`, `deploy/forgejo/`** — the WireGuard/Swarm cluster hosting - **`deploy/swarm/`, `deploy/forgejo/`** — the WireGuard/Swarm mesh spanning
Forgejo (git + CI + registry). See `ACCESS.md`. vps1, vps2 and the desktop, and Forgejo (git + CI + registry), pinned to
- **`deploy/deploy.sh`** — the single deploy entry point for beta and prod. vps1. See `ACCESS.md`.
Takes `SERVICE=backend|frontend|all` plus `BACKEND_IMAGE_TAG`/`FRONTEND_IMAGE_TAG`, - **`deploy/env-topology.sh`** — the single source of truth for where each
resets the host checkout, renders secrets, and routes to the right orchestrator. environment (`dev`/`beta`/`prod`) lives: host, checkout path, branch, deploy
- **`deploy/stack/`** — the **Swarm** path, live on **prod**: mode, stack/compose name, env file, LB ports, DB role/database, service-name
`thermograph-stack.yml` (db, web, worker, lake, daemon, frontend, autoscaler, prefix. Every deploy path sources it and derives its behavior from it rather
autoscaler-lake), `deploy-stack.sh`, `autoscale.sh`, and the LB. Rolling updates than guessing from the host it happens to be running on — necessary since
are start-first, health-gated, with auto-rollback. `STACK_TEST=1` rehearses the vps2 alone now runs two environments.
whole stack on throwaway volumes and ports. - **`deploy/deploy.sh`** — the single deploy entry point for dev, beta and prod.
- **`docker-compose*.yml`** — the **compose** path, live on **beta** and LAN dev Takes `SERVICE=backend|frontend|all` plus `BACKEND_IMAGE_TAG`/`FRONTEND_IMAGE_TAG`
(db, backend, lake, daemon, frontend). `docker-compose.dev.yml` is the LAN (and, on vps2, `THERMOGRAPH_ENV=beta|prod` to say which of the two checkouts
overlay; `docker-compose.openmeteo.yml` is the self-hosted Open-Meteo overlay. it's acting on), resets the host checkout, renders secrets, and routes to the
right orchestrator per `env-topology.sh`.
- **`deploy/deploy-dev.sh`** — a thin dev-specific wrapper around `deploy.sh`
(dev compose overlay, dev's secrets policy). See `DEPLOY-DEV.md`.
- **`deploy/stack/`** — the **Swarm** path, live on **vps2** for both prod and
beta: `thermograph-stack.yml` (prod: db, web, worker, lake, daemon, frontend,
autoscaler, autoscaler-lake) and `thermograph-beta-stack.yml` (beta: the same
service shape minus `db` and the autoscalers, every service name prefixed
`beta-`). `deploy-stack.sh`, `autoscale.sh` and `lb/` are shared by both.
Rolling updates are start-first, health-gated, with auto-rollback.
`STACK_TEST=1` rehearses the whole stack on throwaway volumes and ports.
- **`docker-compose*.yml`** — the **compose** path, live only on **dev** (vps1)
(db, backend, lake, daemon, frontend). `docker-compose.dev.yml` is dev's
mesh-only overlay; `docker-compose.openmeteo.yml` is the self-hosted
Open-Meteo overlay (prod only). `make dev-up` also runs this path locally as
a laptop convenience — that is not an "environment", just a local rehearsal.
Which path a host takes is decided by `/etc/thermograph/deploy-mode`: the string Which path an environment takes is decided by `deploy/env-topology.sh`
`stack` makes `deploy.sh` exec `deploy/stack/deploy-stack.sh`; anything else is (`TG_DEPLOY_MODE`, keyed by `dev`/`beta`/`prod`): dev is `compose`, beta and prod
compose. The workflows never need to know which mode a host runs. are both `stack`. The old host-wide marker `/etc/thermograph/deploy-mode` still
exists as a fallback for a by-hand run with no explicit environment, but it
cannot describe vps2, which runs two environments in two different checkouts —
so it is no longer the thing that decides where files go. The workflows never
need to know which mode an environment runs; they only pass `THERMOGRAPH_ENV`.
## Branches & how changes reach each environment ## Branches & how changes reach each environment
- **`main`** — what **prod and beta** run. `infra-sync.yml` fires on a push to - **`dev`** — deploys to **dev on vps1** (`/opt/thermograph-dev`).
`main` touching `infra/**`, fast-forwards each host's `/opt/thermograph` checkout - **`main`** — deploys to **beta on vps2** (`/opt/thermograph-beta`).
and re-renders `/etc/thermograph.env` from the vault. It deliberately does - **`release`** — deploys to **prod on vps2** (`/opt/thermograph`).
**not** roll any service: image tags are the app domains' axis, not infra's. A
compose or stack change that must recreate containers takes effect on the next
app deploy, or a by-hand `SERVICE=all … deploy/deploy.sh`.
- **`dev`** — what LAN dev would run via `deploy/deploy-dev.sh`. The CI trigger
for this is currently inert (the LAN box still holds a split-era checkout); use
`make dev-up` locally.
- **`release`** — consumed by app deploys only. Both hosts track infra via `main`;
prod's *app images* are staged by `release`, but its checkout follows `main`.
Note the asymmetry with the app domains: app code IS environment-staged App code IS environment-staged this way (`dev`→`main`→`release` maps to
(`dev`→`main`→`release` maps to LAN→beta→prod via image tags); infra is not. vps1/dev → vps2/beta → vps2/prod via image tags, one `Deploy` workflow keyed by
branch). **Infra itself is not environment-staged** the same way: `infra-sync.yml`
fires on a push touching `infra/**` — on `dev` it fast-forwards vps1's dev
checkout, on `main` it fast-forwards **both** of vps2's checkouts (beta and
prod) — re-rendering each environment's own env file from the vault. It
deliberately does **not** roll any service: image tags are the app domains'
axis, not infra's. A compose or stack change that must recreate containers
takes effect on the next app deploy, or a by-hand `SERVICE=all …
deploy/deploy.sh` (or `deploy-dev.sh`) run.
Note the asymmetry this leaves: dev's infra checkout tracks `dev`, the same
branch its app images are staged by. Beta's and prod's infra checkouts both
track `main` — prod's *app images* are staged by `release`, but prod's *infra
checkout* follows `main`, same as beta's.

120
infra/deploy/Caddyfile
View file

@ -1,120 +0,0 @@
# /etc/caddy/Caddyfile on the VPS.
# Each domain's A/AAAA record must already point at this VPS — Caddy provisions a
# Let's Encrypt cert on first request and auto-renews. Nothing else to do for TLS
# (just make sure ports 80 and 443 are open).
#
# Layout:
# thermograph.org/* -> the Thermograph app (path-split across
# backend/frontend -- see below)
# emigriffith.dev/ -> static portfolio site (served straight from disk)
# emigriffith.dev/thermograph* -> permanent redirect to thermograph.org (the app moved)
#
# Thermograph now owns thermograph.org's root, so both services run with
# THERMOGRAPH_BASE=/ (see /etc/thermograph.env) — pages, assets and API all sit
# at "/" with no sub-path prefix. Repo-split Stage 4: backend and frontend are
# two containers now (docker-compose.yml), each on its own loopback port --
# Caddy path-splits directly to whichever owns a given path, so the browser
# still sees one apparent origin. Repo-split Stage 7a flipped which side owns
# the enumerated list: frontend now owns everything (content pages, the
# interactive tool's SPA shells, every static asset, the dynamic IndexNow key
# file) except the short, stable set below, which mirrors backend/web/app.py's
# own routing exactly (a single catch-all proxy to frontend for everything
# else) -- unlike frontend's paths, backend's don't grow every time a new
# static asset filename is added. A gap in this list still just degrades to
# "one extra hop" through backend's own proxy fallback, never a 404.
thermograph.org {
encode zstd gzip
@backend_paths path /api/* /digest /discord/interactions
# Active health check on the same cheap /healthz route each container's own
# HEALTHCHECK uses (Dockerfile) — so a deploy that's still restarting/booting
# never gets proxied into (a reload alone has no gate, hop-1 runbook hazard #10).
# 15s (was 5s): plenty responsive for a process that only restarts on a deploy,
# and a quarter of the polling load.
handle @backend_paths {
reverse_proxy 127.0.0.1:8137 {
health_uri /healthz
health_interval 15s
health_timeout 3s
health_status 2xx
}
}
handle {
reverse_proxy 127.0.0.1:8080 {
health_uri /healthz
health_interval 15s
health_timeout 3s
health_status 2xx
}
}
# Access-log hygiene: the default JSON encoder serializes full request headers,
# the TLS block, and response headers on every line (measured ~1,133B/line) --
# strip those with the `filter` format encoder. Also strip the query string from
# the logged URI: Caddy's default logger records request.uri *including* the
# query string, so every `?q=<search text>` a visitor typed sat in Loki next to
# their client IP for the full 30-day retention -- a real privacy leak, not just
# noise. Bot/crawler skipping stays out of here: `log_skip` needs Caddy >= 2.7
# and an upgrade is out of scope, so that's handled downstream in Alloy's
# loki.process "caddy" stage instead (see observability/alloy/config.alloy).
log {
output file /var/log/caddy/thermograph.log {
roll_size 20MiB
roll_keep 5
}
format filter {
wrap json
fields {
request>headers delete
request>tls delete
resp_headers delete
request>uri regexp \?.* ""
}
}
}
}
emigriffith.dev {
encode zstd gzip
# Thermograph moved to its own domain. Send the old sub-path there with a
# permanent redirect, stripping the /thermograph prefix so deep links map
# straight across (…/thermograph/calendar -> thermograph.org/calendar). The
# bare /thermograph (no trailing slash) goes to the new root.
handle_path /thermograph/* {
redir https://thermograph.org{uri} permanent
}
handle /thermograph {
redir https://thermograph.org/ permanent
}
# Portfolio at the root. Point `root` at the built static site (for the Astro
# portfolio that's its `dist/` output). file_server serves index.html for
# directories and returns a real 404 for missing paths.
handle {
root * /var/www/emigriffith
file_server
}
log {
output file /var/log/caddy/emigriffith.log
}
}
# Old bookmarks to the raw IP (the pre-domain URL) would otherwise get bounced to
# HTTPS-on-the-IP, which has no cert and fails. Redirect them to the portfolio domain.
http://75.119.132.91 {
redir https://emigriffith.dev{uri} permanent
}
# Optional: redirect www -> apex for either domain. Add the www CNAME/A record
# first, then uncomment the matching block.
# www.emigriffith.dev {
# redir https://emigriffith.dev{uri} permanent
# }
# www.thermograph.org {
# redir https://thermograph.org{uri} permanent
# }

61
infra/deploy/Caddyfile.vps1 Normal file
View file

@ -0,0 +1,61 @@
# /etc/caddy/Caddyfile on **vps1** (75.119.132.91) — the operational-programs box.
#
# vps1 serves:
# emigriffith.dev -> static portfolio site (from disk)
# git.thermograph.org -> Forgejo (see deploy/forgejo/caddy-git.conf)
# dashboard.thermograph.org -> Grafana (see observability/caddy-grafana.conf)
#
# and hosts the **dev** environment, which is deliberately ABSENT from this
# file. Dev has no site block, no DNS record and no certificate: it publishes on
# the WireGuard address only (http://10.10.0.2:8137) and is reachable from the
# mesh and nowhere else. Dev runs whatever branch is in flight, on the same box
# as Forgejo and its CI — putting it behind a public hostname would be handing
# the internet an unreviewed build. If you ever need it reachable from a phone,
# join the phone to the mesh; do not add a site block here.
#
# beta.thermograph.org is NOT here either — beta moved to vps2, next to prod.
# That is the one edit in this file most likely to be made by mistake: a "beta"
# reference elsewhere in the repo that still points at 75.119.132.91 means this
# box, which no longer runs beta at all.
#
# Each domain's A/AAAA record must already point here — Caddy provisions a
# Let's Encrypt cert on first request and auto-renews.
emigriffith.dev {
encode zstd gzip
# Thermograph moved to its own domain. Send the old sub-path there with a
# permanent redirect, stripping the /thermograph prefix so deep links map
# straight across (…/thermograph/calendar -> thermograph.org/calendar). The
# bare /thermograph (no trailing slash) goes to the new root.
handle_path /thermograph/* {
redir https://thermograph.org{uri} permanent
}
handle /thermograph {
redir https://thermograph.org/ permanent
}
# Portfolio at the root. Point `root` at the built static site (for the Astro
# portfolio that's its `dist/` output). file_server serves index.html for
# directories and returns a real 404 for missing paths.
handle {
root * /var/www/emigriffith
file_server
}
log {
output file /var/log/caddy/emigriffith.log
}
}
# Old bookmarks to the raw IP (the pre-domain URL) would otherwise get bounced to
# HTTPS-on-the-IP, which has no cert and fails. Redirect them to the portfolio domain.
http://75.119.132.91 {
redir https://emigriffith.dev{uri} permanent
}
# Optional: redirect www -> apex. Add the www CNAME/A record first, then
# uncomment.
# www.emigriffith.dev {
# redir https://emigriffith.dev{uri} permanent
# }

145
infra/deploy/Caddyfile.vps2 Normal file
View file

@ -0,0 +1,145 @@
# /etc/caddy/Caddyfile on **vps2** (169.58.46.181) — the public-facing box.
#
# vps2 serves BOTH environments an external user can reach:
# thermograph.org -> prod (loopback LB on 127.0.0.1:8137 / :8080)
# beta.thermograph.org -> beta (loopback LB on 127.0.0.1:8237 / :8180)
#
# and Centralis at mcp.thermograph.org, which is provisioned separately.
#
# Each domain's A/AAAA record must already point here — Caddy provisions a
# Let's Encrypt cert on first request and auto-renews. Nothing else to do for
# TLS (just make sure ports 80 and 443 are open).
#
# This file was split out of a single deploy/Caddyfile that described both
# boxes' sites at once. That was workable while each box served an unrelated
# set; it stopped being workable when the two ports 8137/8080 started meaning
# "prod on vps2" here and nothing at all on the other box. See Caddyfile.vps1
# for git/dashboard/portfolio.
#
# Thermograph owns thermograph.org's root, so both services run with
# THERMOGRAPH_BASE=/ — pages, assets and API all sit at "/" with no sub-path
# prefix. Backend and frontend are separate containers, each on its own loopback
# port; Caddy path-splits directly to whichever owns a given path, so the
# browser still sees one apparent origin. The enumerated backend list below
# mirrors backend/web/app.py's own routing exactly (a single catch-all proxy to
# frontend for everything else) -- unlike frontend's paths, backend's don't grow
# every time a new static asset filename is added. A gap in this list still just
# degrades to "one extra hop" through backend's own proxy fallback, never a 404.
thermograph.org {
encode zstd gzip
@backend_paths path /api/* /digest /discord/interactions
# Active health check on the same cheap /healthz route each container's own
# HEALTHCHECK uses (Dockerfile) — so a deploy that's still restarting/booting
# never gets proxied into (a reload alone has no gate, hop-1 runbook hazard #10).
# 15s (was 5s): plenty responsive for a process that only restarts on a deploy,
# and a quarter of the polling load.
handle @backend_paths {
reverse_proxy 127.0.0.1:8137 {
health_uri /healthz
health_interval 15s
health_timeout 3s
health_status 2xx
}
}
handle {
reverse_proxy 127.0.0.1:8080 {
health_uri /healthz
health_interval 15s
health_timeout 3s
health_status 2xx
}
}
# Access-log hygiene: the default JSON encoder serializes full request headers,
# the TLS block, and response headers on every line (measured ~1,133B/line) --
# strip those with the `filter` format encoder. Also strip the query string from
# the logged URI: Caddy's default logger records request.uri *including* the
# query string, so every `?q=<search text>` a visitor typed sat in Loki next to
# their client IP for the full 30-day retention -- a real privacy leak, not just
# noise. Bot/crawler skipping stays out of here: `log_skip` needs Caddy >= 2.7
# and an upgrade is out of scope, so that's handled downstream in Alloy's
# loki.process "caddy" stage instead (see observability/alloy/config.alloy).
#
# The FILENAME is load-bearing now: Alloy attributes each access log to an
# environment by filename (thermograph.log -> host="prod", beta.log ->
# host="beta"). One Caddy fronts two environments here, so a single log file
# would file every beta request under prod. Renaming either file means
# updating loki.process "caddy" in observability/alloy/config.alloy.
log {
output file /var/log/caddy/thermograph.log {
roll_size 20MiB
roll_keep 5
}
format filter {
wrap json
fields {
request>headers delete
request>tls delete
resp_headers delete
request>uri regexp \?.* ""
}
}
}
}
# Beta — same shape as prod, pointed at beta's loopback LB. It moved here from
# its own box; the DNS A record for beta.thermograph.org must point at vps2.
#
# Deliberately NOT indexed: beta serves the same content as prod at a different
# hostname, which is a duplicate-content problem for search engines and an
# invitation for users to land on a rehearsal environment from a search result.
# The app itself never pings IndexNow from beta (see env-topology.sh's
# TG_POST_DEPLOY), and this header closes the other half.
beta.thermograph.org {
encode zstd gzip
header {
X-Robots-Tag "noindex, nofollow"
}
@backend_paths path /api/* /digest /discord/interactions
handle @backend_paths {
reverse_proxy 127.0.0.1:8237 {
health_uri /healthz
health_interval 15s
health_timeout 3s
health_status 2xx
}
}
handle {
reverse_proxy 127.0.0.1:8180 {
health_uri /healthz
health_interval 15s
health_timeout 3s
health_status 2xx
}
}
log {
output file /var/log/caddy/beta.log {
roll_size 20MiB
roll_keep 5
}
format filter {
wrap json
fields {
request>headers delete
request>tls delete
resp_headers delete
request>uri regexp \?.* ""
}
}
}
}
# Optional: redirect www -> apex. Add the www CNAME/A record first, then
# uncomment.
# www.thermograph.org {
# redir https://thermograph.org{uri} permanent
# }

409
infra/deploy/RUNBOOK-vps1-vps2-cutover.md Normal file
View file

@ -0,0 +1,409 @@
# Runbook — moving beta to vps2 and dev to vps1
**Status: not executed.** Everything this runbook describes is landed in the
repo and live nowhere. The estate still runs the old shape until someone works
through the steps below.
## What changes
| | before | after |
|---|---|---|
| `75.119.132.91` (**vps1**) | beta + Forgejo + Grafana/Loki | Forgejo + Grafana/Loki + **dev** |
| `169.58.46.181` (**vps2**) | prod + Centralis + Postfix + backups | **prod + beta** + Centralis + Postfix + backups |
| desktop | LAN dev server + CI runner | AI models + flex Swarm worker, **no environment** |
| databases | one Postgres per environment | one instance on vps2, `thermograph` + `thermograph_beta` |
Mesh addresses do **not** move: vps1 stays `10.10.0.2`, vps2 stays `10.10.0.1`.
Every "beta = 75.119.132.91" reference anywhere is wrong after this — that
address is vps1, and sending a beta-intended command there is the most likely
way to do damage during this cutover.
## Order, and why it is this order
Beta moves first and completely, while dev stays where it is. Then dev moves.
The two halves are independent, so a problem in one never forces a rollback of
the other — and beta is the half that carries a public hostname and a TLS
certificate, so it gets done while you have the most attention.
Prod is touched twice, and both touches are additive: a new database + role on
its Postgres instance, and a new site block in its Caddyfile. Prod's stack file,
its services, its volumes and its env file are not modified by this work at all.
**Expected downtime:** beta, roughly the length of a dump/restore plus a stack
bring-up (~10–20 min). Dev, as long as you like. **Prod: none** — unless step 3
or 8 is done wrong, which is what the verification lines are for.
---
## 0. Before you start
- [ ] **Lower the DNS TTL** on `beta.thermograph.org` to 300s, at least an hour
before step 7. Do this first; it is the only step with a lead time.
- [ ] **Create the new Forgejo Actions secrets** (Settings → Secrets). The
workflows on this branch reference them and will fail without them:
VPS1_SSH_HOST=75.119.132.91 VPS1_SSH_USER=agent
VPS1_SSH_KEY=<the CI deploy key> VPS1_SSH_PORT=22
VPS2_SSH_HOST=169.58.46.181 VPS2_SSH_USER=agent
VPS2_SSH_KEY=<the CI deploy key> VPS2_SSH_PORT=22
Keep the old `SSH_*` / `PROD_SSH_*` secrets until step 11 — they are the
rollback path, and deleting them early strands you.
- [ ] **Take a fresh prod backup and confirm it landed**, rather than trusting
the schedule:
```
# from a machine with repo access
# Actions -> "Ops cron (backup + IndexNow)" -> Run workflow
ssh agent@169.58.46.181 'ls -lh ~/thermograph-backups | tail -3'
```
- [ ] **Confirm you can decrypt the vault** (`sops -d infra/deploy/secrets/beta.yaml`
from `infra/`). Steps 3 and 5 need the rendered beta password.
- [ ] Have the PR merged to `dev` and promoted to `main`. Hosts pull `main`, so
nothing below works from an unmerged branch.
---
## 1. Give beta a checkout on vps2
```
ssh agent@169.58.46.181
sudo git clone --branch main http://10.10.0.2:3080/emi/thermograph.git /opt/thermograph-beta
sudo chown -R agent:agent /opt/thermograph-beta
```
Prod's checkout at `/opt/thermograph` is untouched. Two checkouts side by side
is the whole point: a `git reset --hard` for beta must never be able to move
prod's tree.
**Verify:** `git -C /opt/thermograph-beta log --oneline -1` matches
`git -C /opt/thermograph log --oneline -1`.
---
## 2. Render beta's env file on vps2
```
ssh agent@169.58.46.181
cd /opt/thermograph-beta/infra
. deploy/render-secrets.sh
render_thermograph_secrets /opt/thermograph-beta/infra beta /etc/thermograph-beta.env
```
The age key is already at `/etc/thermograph/age.key` on this box (prod uses it);
beta needs no second copy.
**Verify** — beta's file must name beta's role and database, and prod's must be
untouched:
```
sudo grep -c . /etc/thermograph-beta.env # non-zero
sudo grep -o 'thermograph_beta' /etc/thermograph-beta.env | head -1
sudo grep -o '@db:5432/thermograph$' /etc/thermograph.env # prod: still 'thermograph'
```
---
## 3. Create beta's role and database on the shared instance
This is the first step that touches prod's Postgres. It only ever adds; it drops
nothing.
```
ssh agent@169.58.46.181
sudo bash /opt/thermograph-beta/infra/deploy/db/provision-env-db.sh beta
```
**Verify the isolation is real** — this is the whole justification for one
instance serving two environments, so actually run it:
```
DBC=$(docker ps -q --filter label=com.docker.swarm.service.name=thermograph_db | head -1)
# beta's role must NOT be able to reach prod's database:
docker exec "$DBC" psql -U thermograph_beta -d thermograph -c 'select 1' # must FAIL
# beta's read-only role exists and really is read-only:
docker exec "$DBC" psql -U thermograph_beta_ro -d thermograph_beta -c 'select 1' # must SUCCEED
docker exec "$DBC" psql -U thermograph_beta_ro -d thermograph_beta \
-c 'create table _x(i int)' # must FAIL
# prod's own role is UNTOUCHED — still a superuser:
docker exec "$DBC" psql -U thermograph -d postgres -tAc \
"select rolsuper from pg_roles where rolname='thermograph'" # must be t
# and prod is still healthy:
curl -fsS -o /dev/null -w '%{http_code}\n' https://thermograph.org/api/health
```
The script **refuses to run for prod** — prod's `thermograph` role is the
instance's bootstrap superuser, not a guest, and provisioning it would demote
it. That guard is why the superuser check above should never fail; run it
anyway.
If the first command SUCCEEDS, stop. `CONNECT` was not revoked and beta would be
able to read production data; re-run the provisioning script and re-check before
going further.
**Rollback:** `DROP DATABASE thermograph_beta; DROP ROLE thermograph_beta;` —
prod is unaffected either way.
---
## 4. Move beta's data (optional)
Beta's existing database still lives on vps1. Decide honestly whether you want
it: beta's value is a rehearsal of prod, and a fresh schema with a handful of
test accounts is often *better* than carrying old beta state across. If you do
want it:
```
# on vps1 — dump the old beta database
ssh agent@75.119.132.91 \
'docker exec thermograph-db-1 pg_dump -U thermograph -d thermograph --format=custom' \
> /tmp/beta-premove.dump
# on vps2 — restore into beta's database as beta's role
scp /tmp/beta-premove.dump agent@169.58.46.181:/tmp/
ssh agent@169.58.46.181 '
DBC=$(docker ps -q --filter label=com.docker.swarm.service.name=thermograph_db | head -1)
docker cp /tmp/beta-premove.dump "$DBC":/tmp/
docker exec "$DBC" pg_restore -U thermograph -d thermograph_beta --no-owner --role=thermograph_beta /tmp/beta-premove.dump
'
```
`--no-owner --role=thermograph_beta` matters: the dump's objects are owned by the
old `thermograph` role, and restoring them verbatim would leave beta's tables
owned by a role beta does not connect as.
**Do not** restore a *prod* dump into beta. Beta's daemon runs the notification
timers, and beta's subscriber table would then be full of real people. Beta's
vault sets `THERMOGRAPH_MAIL_BACKEND=console` so nothing would actually send,
but that is one `sops edit` away from not being true.
---
## 5. Bring beta's stack up on vps2
```
ssh agent@169.58.46.181
cd /opt/thermograph-beta
THERMOGRAPH_ENV=beta SERVICE=all \
BACKEND_IMAGE_TAG=<current beta backend tag> \
FRONTEND_IMAGE_TAG=<current beta frontend tag> \
infra/deploy/deploy.sh
```
Get the current tags from the old beta host first
(`cat /opt/thermograph/infra/deploy/.image-tags.env` on vps1), so beta comes up
on exactly the code it was already running.
**Verify** — beta answers on its own loopback ports, and prod's are unmoved:
```
curl -fsS -o /dev/null -w 'beta %{http_code}\n' http://127.0.0.1:8237/healthz
curl -fsS -o /dev/null -w 'prod %{http_code}\n' http://127.0.0.1:8137/healthz
docker stack services thermograph-beta # 5 services, all 1/1
docker stack services thermograph # prod: unchanged, still 1/1
```
**Verify the DNS-collision fix actually held** — this is the failure mode that
would be subtle and awful in production:
```
# prod's frontend must resolve prod's web, not beta's
docker exec $(docker ps -q -f label=com.docker.swarm.service.name=thermograph_frontend | head -1) \
getent hosts web
docker exec $(docker ps -q -f label=com.docker.swarm.service.name=thermograph_frontend | head -1) \
getent hosts beta-web # should NOT resolve on prod's network
```
**Rollback:** `docker stack rm thermograph-beta` and remove the LB container
(`docker rm -f thermograph-beta-lb`). Beta on vps1 is still running and still
serving; nothing has moved yet.
---
## 6. Add beta's site block to vps2's Caddy
`infra/deploy/Caddyfile.vps2` is the reference copy. Append its
`beta.thermograph.org` block to the live `/etc/caddy/Caddyfile` on vps2.
```
ssh agent@169.58.46.181
sudo caddy validate --config /etc/caddy/Caddyfile # BEFORE reloading
sudo systemctl reload caddy
```
Validate before reload, always: a malformed Caddyfile takes `thermograph.org`
down with it, and prod is on this box now.
Caddy cannot issue the certificate until DNS moves (step 7), so expect the beta
hostname to fail TLS until then. That is fine and expected.
---
## 7. Move DNS
Point `beta.thermograph.org`'s A record at **169.58.46.181**.
Then wait for propagation and the certificate:
```
dig +short beta.thermograph.org # 169.58.46.181
curl -fsS -o /dev/null -w '%{http_code}\n' https://beta.thermograph.org/api/health
ssh agent@169.58.46.181 'sudo journalctl -u caddy -n 30 --no-pager | grep -i certificate'
```
**Rollback:** point the A record back at 75.119.132.91. Beta on vps1 is still
up (you have not stopped it yet — that is step 9), so this is a clean revert
for as long as you leave it running.
---
## 8. Stand dev up on vps1
Independent of everything above; do it whenever.
```
ssh agent@75.119.132.91
sudo bash /opt/thermograph-dev/infra/deploy/provision-dev.sh # clones if absent
```
The script writes `/etc/thermograph/secrets-env` = `dev` and removes any
`deploy-mode` marker. Then deploy:
```
cd /opt/thermograph-dev
SERVICE=all BACKEND_IMAGE_TAG=<dev tag> FRONTEND_IMAGE_TAG=<dev tag> \
infra/deploy/deploy-dev.sh
```
**Verify it is mesh-only.** This is the security-relevant check of the whole
cutover — vps1 is a public box and dev runs unreviewed branches:
```
ss -ltnp | grep 8137 # MUST show 10.10.0.2:8137, never 0.0.0.0:8137
curl -fsS -o /dev/null -w '%{http_code}\n' http://10.10.0.2:8137/healthz # from the mesh
curl --max-time 5 http://75.119.132.91:8137/healthz # MUST fail/refuse
```
Also confirm dev did **not** get the fleet's shared production credentials —
`dev.yaml` is rendered alone, and vps1 is exactly the box that must not hold
them:
```
sudo grep -c 'THERMOGRAPH_S3_SECRET_KEY\|REGISTRY_TOKEN\|VAPID_PRIVATE' /etc/thermograph.env # expect 0
```
---
## 9. Retire beta from vps1
Only after beta has been serving from vps2 through step 7 for long enough that
you believe it.
```
ssh agent@75.119.132.91
cd /opt/thermograph/infra
docker compose down # stops the old beta app stack
# remove beta's site block from /etc/caddy/Caddyfile (see Caddyfile.vps1)
sudo caddy validate --config /etc/caddy/Caddyfile && sudo systemctl reload caddy
```
**Keep the old `thermograph_pgdata` volume on vps1** until you are certain beta
on vps2 is healthy and backed up. It is the only copy of pre-move beta data.
Delete it deliberately, later, not as part of this runbook.
Forgejo, Grafana, Loki and the portfolio site all keep running on this box
untouched — do not stop anything else here.
---
## 10. Repoint the Alloy agents
Both nodes need the new variables; the label model changed (see
`observability/alloy/config.alloy`).
```
# vps2 — two environments, so two log volumes and the beta overlay
ssh agent@169.58.46.181
cd /opt/thermograph/observability/alloy
ALLOY_NODE=vps2 ALLOY_ENV=prod \
APPLOGS_VOLUME=thermograph_applogs BETA_APPLOGS_VOLUME=thermograph-beta_applogs \
LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push \
docker compose -f docker-compose.agent.yml -f docker-compose.agent.beta.yml up -d
# vps1
ssh agent@75.119.132.91
cd /opt/thermograph/observability/alloy
ALLOY_NODE=vps1 ALLOY_ENV=dev APPLOGS_VOLUME=thermograph-dev_applogs \
LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push \
docker compose -f docker-compose.agent.yml up -d
```
**Verify each environment is labelled as itself** — the point of the change is
that beta's logs on vps2 are not filed as prod:
```
# in Grafana Explore, or via the Loki API:
# {host="beta"} |= "" -> should show beta traffic, from node="vps2"
# {host="prod"} |= "" -> prod only
# {host="dev"} |= "" -> from node="vps1"
# count by (host) (count_over_time({job="docker"}[5m])) -> three values, not one
```
If `{host="beta"}` is empty while beta is clearly serving, the container-name
relabel did not match — check `docker ps --format '{{.Names}}'` on vps2 against
the `/thermograph-beta_.*` regex.
---
## 11. Clean up
- [ ] Delete the old `SSH_*` and `PROD_SSH_*` Forgejo secrets (only now — they
were the rollback path).
- [ ] Run the ops cron by hand and confirm **both** databases are dumped:
`ssh agent@169.58.46.181 'ls ~/thermograph-backups | tail -4'` should show
both a `thermograph-*.dump` and a `thermograph_beta-*.dump`.
- [ ] Confirm the Forgejo backup job still works — it now uses `VPS1_SSH_*`,
and it follows Forgejo (vps1), not beta.
- [ ] Restore the DNS TTL on `beta.thermograph.org`.
- [ ] Desktop: stop the old LAN dev stack and free the box for the AI models.
Leave it joined to the Swarm as a worker; nothing schedules onto it today
(every app service is pinned `node.role == manager`), which is what makes
it safe to also run models there.
- [ ] Update Centralis — see below.
---
## Centralis (outside this repo)
Centralis' own text still describes the old estate, and it is not in this
repository. These need a separate change in the Centralis repo:
- The `onboarding` tool's estate paragraph ("Four machines on a WireGuard
mesh... beta (beta.thermograph.org, Forgejo, Grafana + Loki), the operator's
desktop (LAN dev, the main CI runner)").
- The MCP server's own protocol-level `instructions` string, which carries a
second, separately-worded copy of the same paragraph.
- The `thermograph-orientation` skill's estate table, and `thermograph-ops`'
release-flow diagram ("LAN dev box"), its "ssh -L works for beta and is
impossible for prod" claim (beta is a Swarm overlay now, so it behaves like
prod), and its prod-only Swarm service-name list (beta has
`beta-`-prefixed ones now).
- Tool descriptions that hardcode the old shape: `fleet_status` ("Swarm services
on prod, compose containers on beta/dev"), `estate_status` and `rollback_to`
("Centralis cannot SSH to the dev desktop"), `logs_query`'s per-environment
service lists (Forgejo/Grafana/Loki are listed under `beta`; they belong under
`dev`/vps1 now), and `secrets_render`'s assumption that a host's own
`secrets-env` marker identifies one environment — vps2 has two.
---
## If it goes wrong
| symptom | most likely cause | action |
|---|---|---|
| prod 502s after step 5 | beta's stack collided with prod's service DNS | `docker stack rm thermograph-beta`; prod recovers on its own. Check beta's services are `beta-`-prefixed |
| beta 502s, prod fine | beta's LB or stack not up | `docker ps \| grep beta-lb`; `docker stack ps thermograph-beta --no-trunc` |
| beta can't reach the database | role/database missing, or the wrong network | re-run `provision-env-db.sh beta`; confirm beta's tasks are on `thermograph_internal` |
| beta serves prod's data | beta's env file has prod's URL | check `/etc/thermograph-beta.env` names `thermograph_beta` twice; re-render (step 2). **Stop beta until fixed** |
| prod's nightly backup fails | `thermograph_beta` doesn't exist yet | either finish step 3 or revert the ops-cron change; the job fails loudly by design rather than skipping silently |
| dev reachable from the internet | `DEV_BIND_ADDR` not applied | `ss -ltnp \| grep 8137`; redeploy via `deploy-dev.sh`, which sets it from `env-topology.sh` |

41
infra/deploy/backup/README.md
View file

@ -6,11 +6,21 @@ single VPS. Driven by `.forgejo/workflows/ops-cron.yml` (03:00 UTC daily +
## What is backed up ## What is backed up
| Prefix in bucket `era5-thermograph` | Source | Job | Retention off-box | | Prefix in bucket `era5-thermograph` | Source | Job | Host / secrets | Retention off-box |
|---|---|---|---| |---|---|---|---|---|
| `backups/db/prod/` | prod Postgres/TimescaleDB (`pg_dump --format=custom`) | `backup` (prod, `PROD_SSH_*`) | 30 days | | `backups/db/prod/` | prod's `thermograph` database (`pg_dump --format=custom`) | `backup` | vps2, `VPS2_SSH_*` | 30 days |
| `backups/forgejo/db/` | Forgejo Postgres (`forgejo_db`) | `forgejo-backup` (beta, `SSH_*`) | 30 days | | `backups/db/beta/` | beta's `thermograph_beta` database, same shared instance | `backup` | vps2, `VPS2_SSH_*` | 30 days |
| `backups/forgejo/data/` | Forgejo data volume (repos/LFS/config) | `forgejo-backup` | 30 days | | `backups/forgejo/db/` | Forgejo Postgres (`forgejo_db`) | `forgejo-backup` | vps1, `VPS1_SSH_*` | 30 days |
| `backups/forgejo/data/` | Forgejo data volume (repos/LFS/config) | `forgejo-backup` | vps1, `VPS1_SSH_*` | 30 days |
Both application databases live on the ONE shared TimescaleDB instance on
vps2 now (separate roles/databases, not separate boxes) — the `backup` job
dumps each by name so beta isn't silently left uncovered. Forgejo moved to
**vps1** with the vps1/vps2 rename (it used to be co-located with beta on the
box now called vps2's predecessor); its backup targets vps1 accordingly. See
`ops-cron.yml`'s own header comment for the history: an earlier revision keyed
these secrets by environment name rather than host, which is what let "the
prod backup" silently dump the wrong box for a while.
Everything is streamed through **`age`** encryption before upload — encrypted to the Everything is streamed through **`age`** encryption before upload — encrypted to the
vault recipient `age1xx4dz…`, so the private key each host already has at vault recipient `age1xx4dz…`, so the private key each host already has at
@ -23,9 +33,10 @@ decrypts it. Nothing plaintext leaves the box.
(also mirrored into the SOPS vault `deploy/secrets/{prod,beta}.yaml` for host-side (also mirrored into the SOPS vault `deploy/secrets/{prod,beta}.yaml` for host-side
use). Contabo needs **path-style** addressing (`force_path_style=true`, use). Contabo needs **path-style** addressing (`force_path_style=true`,
`provider=Other`, `region=default`). `provider=Other`, `region=default`).
- `rclone` + `age` must be installed on prod and beta (one-time: `apt-get install -y - `rclone` + `age` must be installed on vps1 and vps2 (one-time: `apt-get install -y
rclone age`). The prod job apt-installs them if missing; the beta user has no sudo, rclone age`). The prod/beta job on vps2 apt-installs them if missing; check
so they are pre-installed there. whether the Forgejo-backup job's user on vps1 has the sudo to do the same, or
whether they need pre-installing there.
## Restore (DR runbook) ## Restore (DR runbook)
@ -33,17 +44,27 @@ Prereq on the restoring host: `rclone` + `age` + the age key at
`/etc/thermograph/age.key` (or the operator key), and rclone configured for the `/etc/thermograph/age.key` (or the operator key), and rclone configured for the
`archive:` remote (env vars `RCLONE_CONFIG_ARCHIVE_*` or `/etc/rclone/rclone.conf`). `archive:` remote (env vars `RCLONE_CONFIG_ARCHIVE_*` or `/etc/rclone/rclone.conf`).
### Prod database ### Prod / beta database
Both restore into the SAME shared instance on vps2 (`<db-container>` is that
one instance's container/task, resolved the same way `dbq.sh` does) — only
the bucket prefix and the target database/role differ:
```sh ```sh
# list available encrypted dumps # list available encrypted dumps
rclone ls archive:era5-thermograph/backups/db/prod/ rclone ls archive:era5-thermograph/backups/db/prod/
rclone ls archive:era5-thermograph/backups/db/beta/
# download newest, decrypt, restore into a fresh db # download newest, decrypt, restore into a fresh db
OBJ=archive:era5-thermograph/backups/db/prod/<name>.dump.age OBJ=archive:era5-thermograph/backups/db/prod/<name>.dump.age
rclone cat "$OBJ" | sudo age -d -i /etc/thermograph/age.key \ rclone cat "$OBJ" | sudo age -d -i /etc/thermograph/age.key \
| docker exec -i <db-container> pg_restore -U thermograph -d thermograph --clean --if-exists | docker exec -i <db-container> pg_restore -U thermograph -d thermograph --clean --if-exists
# beta: same instance, its own role/database
OBJ=archive:era5-thermograph/backups/db/beta/<name>.dump.age
rclone cat "$OBJ" | sudo age -d -i /etc/thermograph/age.key \
| docker exec -i <db-container> pg_restore -U thermograph -d thermograph_beta --clean --if-exists
``` ```
### Forgejo ### Forgejo (restore on vps1)
```sh ```sh
# database # database
rclone cat archive:era5-thermograph/backups/forgejo/db/<name>.dump.age \ rclone cat archive:era5-thermograph/backups/forgejo/db/<name>.dump.age \

7
infra/deploy/db/init/20-tuning.sh
View file

@ -1,7 +1,10 @@
#!/bin/bash #!/bin/bash
# Postgres memory / performance tuning, scaled to the container's DB_MEMORY budget so # Postgres memory / performance tuning, scaled to the container's DB_MEMORY budget so
# the same init serves every host (beta 8g; prod 16g on the 48 GB box) with no # the same init works for any host with no hardcoding. There is ONE shared TimescaleDB
# hardcoding. Runs once on a fresh data volume from /docker-entrypoint-initdb.d, after # instance now (prod and beta are separate databases/roles on it, not separate
# containers), sized from prod's vault values (16g on the 48 GB box) -- beta no longer
# gets a second database or a second tuning pass. Dev keeps its own, separate container
# (8g). Runs once on a fresh data volume from /docker-entrypoint-initdb.d, after
# 10-timescaledb.sql enables timescaledb. Settings are written via ALTER SYSTEM # 10-timescaledb.sql enables timescaledb. Settings are written via ALTER SYSTEM
# (persisted to postgresql.auto.conf, which the timescaledb image's own # (persisted to postgresql.auto.conf, which the timescaledb image's own
# timescaledb-tune postgresql.conf defers to); the container's post-init restart brings # timescaledb-tune postgresql.conf defers to); the container's post-init restart brings

161
infra/deploy/db/provision-env-db.sh Executable file
View file

@ -0,0 +1,161 @@
#!/usr/bin/env bash
# Give an environment its own role + database on the SHARED Postgres instance.
#
# sudo bash infra/deploy/db/provision-env-db.sh beta # run on vps2
#
# Since beta moved onto vps2 there is ONE TimescaleDB instance serving two
# environments. That is a capacity decision — one Postgres to tune, back up and
# keep on one extension build — and it is emphatically NOT a decision to let the
# two environments see each other's data. This script is what makes the second
# half true:
#
# * a role per environment (thermograph_beta), with its own password taken
# from that environment's own vault render,
# * a database per environment (thermograph_beta) OWNED by that role,
# * CONNECT revoked from PUBLIC on it, so the split is enforced by Postgres
# rather than by everyone remembering to use the right URL,
# * and NO superuser, NO CREATEDB, NO CREATEROLE on that role.
#
# The prod role keeps its own database and cannot be reached with beta's
# credentials; beta's role cannot connect to prod's database at all.
#
# Idempotent by construction — safe to re-run after a password rotation (it
# re-applies the password) or on a fresh instance. It never drops anything.
#
# WHY THIS IS NOT WIRED INTO deploy.sh: creating roles and databases is a
# privileged, once-per-environment act, and a deploy that can mint database
# roles is a deploy that can mint them wrongly at 3am. Run it by hand from the
# runbook when an environment is first stood up, and after a password rotation.
set -euo pipefail
ENV_NAME="${1:?usage: provision-env-db.sh <env> (beta|prod)}"
SELF_DIR=$(cd "$(dirname "$0")" && pwd)
# shellcheck source=infra/deploy/env-topology.sh
. "$SELF_DIR/../env-topology.sh"
thermograph_topology "$ENV_NAME"
if [ "$TG_DEPLOY_MODE" != stack ]; then
echo "!! $ENV_NAME does not use the shared instance (dev keeps its own db container)" >&2
exit 2
fi
# REFUSE to run for the environment that OWNS the instance.
#
# This script provisions a GUEST environment onto someone else's Postgres. Run
# for prod, it would target prod's `thermograph` role — which is the instance's
# bootstrap superuser, not a guest — and the `ALTER ROLE ... NOSUPERUSER` below
# would strip superuser from the role the whole instance is administered with.
# Prod's roles predate this script and are created by the container's own
# initdb; there is nothing here for prod to need.
if [ "$TG_DB_SERVICE" = "${TG_STACK_NAME}_db" ]; then
echo "!! '$ENV_NAME' OWNS this Postgres instance (${TG_DB_SERVICE} is its own stack's db)." >&2
echo "!! This script provisions a guest environment onto a shared instance." >&2
echo "!! Running it here would demote ${TG_DB_USER} from superuser. Refusing." >&2
exit 2
fi
# The database SERVER is prod's, wherever we are provisioning FOR.
DB_CID=$(docker ps -q --filter "label=com.docker.swarm.service.name=${TG_DB_SERVICE}" | head -1)
if [ -z "$DB_CID" ]; then
echo "!! no running task for ${TG_DB_SERVICE} on this host" >&2
echo "!! run this on vps2, with prod's stack up." >&2
exit 1
fi
# The new role's password is whatever that environment's vault says it is, so
# the database and the app can never disagree about it. Read from the rendered
# env file (a deploy of that environment writes it) rather than from sops here:
# this script should not need the age key.
if [ ! -r "$TG_ENV_FILE" ]; then
PW=$(sudo grep -m1 '^POSTGRES_PASSWORD=' "$TG_ENV_FILE" 2>/dev/null | cut -d= -f2- || true)
else
PW=$(grep -m1 '^POSTGRES_PASSWORD=' "$TG_ENV_FILE" | cut -d= -f2- || true)
fi
if [ -z "${PW:-}" ]; then
echo "!! no POSTGRES_PASSWORD in $TG_ENV_FILE" >&2
echo "!! deploy $ENV_NAME once first so the vault renders it, then re-run." >&2
exit 1
fi
echo "==> Provisioning role/database '${TG_DB_USER}'/'${TG_DB_NAME}' on ${TG_DB_SERVICE}"
# The bootstrap superuser of the instance is prod's POSTGRES_USER (`thermograph`),
# and psql runs inside the container over its local socket — the database is
# never exposed on a TCP port anyone outside the overlay can reach.
#
# The password reaches psql as an environment variable rather than a -v
# argument, so it is not in psql's argv inside the container. It IS briefly in
# the `docker exec` argv on the host; that is visible only to root on vps2, who
# can read the vault render anyway. It is never interpolated into SQL text:
# :'pw' is psql's quote-and-escape form, which is also what makes a password
# containing a quote safe.
#
# Note both statements below are generated and run via \gexec rather than a
# DO block. psql does NOT substitute :variables inside dollar-quoted strings,
# so a DO $$ ... :'role' ... $$ body would be sent to the server literally.
docker exec -i -e ENV_DB_PASSWORD="$PW" "$DB_CID" \
psql -v ON_ERROR_STOP=1 -U thermograph -d postgres \
-v role="$TG_DB_USER" -v dbname="$TG_DB_NAME" <<'SQL'
\set pw `echo "$ENV_DB_PASSWORD"`
-- Role: create if absent.
SELECT format('CREATE ROLE %I LOGIN', :'role')
WHERE NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = :'role')
\gexec
-- Always (re)apply the password and the negative privileges, so a vault
-- rotation is just "rotate, redeploy, re-run this" — and so a role that was
-- created by hand with more rights than it should have gets corrected.
ALTER ROLE :"role" WITH LOGIN PASSWORD :'pw' NOSUPERUSER NOCREATEDB NOCREATEROLE;
-- CREATE DATABASE cannot run inside a transaction block, hence \gexec here too.
SELECT format('CREATE DATABASE %I OWNER %I', :'dbname', :'role')
WHERE NOT EXISTS (SELECT 1 FROM pg_database WHERE datname = :'dbname')
\gexec
SQL
# Extension + privileges have to run INSIDE the new database, hence a second
# connection. CREATE EXTENSION needs superuser, which is why it is done here
# rather than left to the app's own migration to attempt as the env role.
docker exec -i "$DB_CID" \
psql -v ON_ERROR_STOP=1 -U thermograph -d "$TG_DB_NAME" \
-v role="$TG_DB_USER" -v dbname="$TG_DB_NAME" -v rorole="${TG_DB_USER}_ro" <<'SQL'
CREATE EXTENSION IF NOT EXISTS timescaledb;
-- Nobody but this environment's roles connects to this database. Without this,
-- PUBLIC retains CONNECT and any role on the instance could read it.
REVOKE CONNECT ON DATABASE :"dbname" FROM PUBLIC;
GRANT CONNECT ON DATABASE :"dbname" TO :"role";
-- The role owns the database but not necessarily the public schema, which is
-- owned by the bootstrap superuser on a fresh database; Alembic needs to create
-- tables in it.
ALTER SCHEMA public OWNER TO :"role";
-- The read-only role that ad-hoc queries use (ops/dbq.sh). Every environment
-- gets one, named <role>_ro, so a human or an agent poking at data cannot
-- write. Without it, beta's queries would have had to connect as the OWNER —
-- read-write on its own database — losing a guarantee prod and dev already had
-- purely because beta was newer.
SELECT format('CREATE ROLE %I LOGIN', :'rorole')
WHERE NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = :'rorole')
\gexec
ALTER ROLE :"rorole" WITH LOGIN NOSUPERUSER NOCREATEDB NOCREATEROLE;
GRANT CONNECT ON DATABASE :"dbname" TO :"rorole";
GRANT USAGE ON SCHEMA public TO :"rorole";
GRANT SELECT ON ALL TABLES IN SCHEMA public TO :"rorole";
GRANT SELECT ON ALL SEQUENCES IN SCHEMA public TO :"rorole";
-- Future tables too: without this, every new migration would create a table the
-- read-only role cannot see, and the omission would only surface as a confusing
-- "permission denied" months later.
ALTER DEFAULT PRIVILEGES FOR ROLE :"role" IN SCHEMA public
GRANT SELECT ON TABLES TO :"rorole";
ALTER DEFAULT PRIVILEGES FOR ROLE :"role" IN SCHEMA public
GRANT SELECT ON SEQUENCES TO :"rorole";
SQL
echo "==> OK: ${TG_DB_USER} owns ${TG_DB_NAME} (timescaledb enabled, PUBLIC revoked)"
echo " Verify isolation:"
echo " docker exec $DB_CID psql -U ${TG_DB_USER} -d thermograph -c 'select 1' # must FAIL"

99
infra/deploy/deploy-dev.sh
View file

@ -1,41 +1,54 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# LAN-dev deploy: roll the per-service registry-pull stack (see deploy.sh) onto # Dev deploy: roll the per-service registry-pull stack (see deploy.sh) onto the
# the ~/thermograph-dev overlay instead of prod/beta's loopback-only stack. # dev compose overlay instead of beta/prod's Swarm stacks.
# #
# This script assumes $APP_DIR is a checkout of the monorepo (deploy assets under infra/), # Dev lives on vps1 now, at /opt/thermograph-dev — not on the operator's desktop.
# the same way /opt/thermograph is on prod/beta. That is the live state: the LAN # The desktop hosts no Thermograph environment any more; it runs the AI models
# box's ~/thermograph-dev was reprovisioned as an infra checkout during the # and offers flex Swarm capacity. What moved is only WHERE and HOW dev is
# 2026-07-22 cutover (the app monorepo is archived), provision-dev-lan.sh's # reached: it is a normal fleet host now (SSH deploy from CI, SOPS render at
# REPO_URL defaults to this repo, and the app repos' deploy-dev.yml workflows # /etc/thermograph.env, mesh-only exposure on 10.10.0.2:8137), rather than a
# invoke this script on the thermograph-lan runner. This IS the live LAN-dev # sudo-free systemd --user stack on someone's Wi-Fi.
# deploy path. #
# $APP_DIR is a checkout of the monorepo (deploy assets under infra/), the same
# way /opt/thermograph is for prod. A laptop can still point APP_DIR at a
# checkout under $HOME and get the old local behaviour.
# #
# Design: a thin wrapper around deploy.sh, not a duplicate. deploy.sh already owns # Design: a thin wrapper around deploy.sh, not a duplicate. deploy.sh already owns
# the entire per-service registry-pull mechanism -- secrets sourcing, docker login, # the entire per-service registry-pull mechanism -- secrets sourcing, docker login,
# the retry-pull loop, --no-deps single-service rolls vs. --remove-orphans `all`, # the retry-pull loop, --no-deps single-service rolls vs. --remove-orphans `all`,
# .image-tags.env persistence, and the 8137/8080 health checks. None of that is # .image-tags.env persistence, and the 8137/8080 health checks. None of that is
# dev-specific; the only things LAN dev actually changes are WHERE it deploys # dev-specific; the only things dev actually changes are WHERE it deploys (a
# (a separate checkout + branch) and WHICH compose files are in play (the base # separate checkout + branch), WHICH compose files are in play (the base file
# file plus docker-compose.dev.yml's uncapped/LAN-published overrides). Both are # plus docker-compose.dev.yml's uncapped/mesh-published overrides), and its
# expressible as environment (APP_DIR/BRANCH that deploy.sh already reads, and # secrets policy (below). All are expressible as environment (APP_DIR/BRANCH
# docker compose's own COMPOSE_FILE variable), so re-exec'ing deploy.sh with that # that deploy.sh already reads, and docker compose's own COMPOSE_FILE
# environment set covers it with no forked copy of the pull/roll/health logic to # variable), so re-exec'ing deploy.sh with that environment set covers it with
# drift out of sync. If LAN dev ever needs deploy logic that genuinely diverges # no forked copy of the pull/roll/health logic to drift out of sync. If dev ever
# from prod/beta (not just "different files/directory"), promote this to a # needs deploy logic that genuinely diverges from beta/prod (not just "different
# standalone script at that point rather than growing special cases into deploy.sh. # files/directory"), promote this to a standalone script at that point rather
# than growing special cases into deploy.sh.
# #
# Usage, mirrors deploy.sh directly: # Usage, mirrors deploy.sh directly:
# # roll just the backend onto a dev-tagged image: # # roll just the backend onto a dev-tagged image:
# ssh dev-box 'SERVICE=backend BACKEND_IMAGE_TAG=sha-<12hex> deploy/deploy-dev.sh' # ssh vps1 'SERVICE=backend BACKEND_IMAGE_TAG=sha-<12hex> /opt/thermograph-dev/infra/deploy/deploy-dev.sh'
# # bring the whole dev stack up (both tags required, same as deploy.sh): # # bring the whole dev stack up (both tags required, same as deploy.sh):
# ssh dev-box 'SERVICE=all BACKEND_IMAGE_TAG=sha-<a> FRONTEND_IMAGE_TAG=sha-<b> deploy/deploy-dev.sh' # ssh vps1 'SERVICE=all BACKEND_IMAGE_TAG=sha-<a> FRONTEND_IMAGE_TAG=sha-<b> /opt/thermograph-dev/infra/deploy/deploy-dev.sh'
set -euo pipefail set -euo pipefail
# Dev context: a separate checkout + branch from prod/beta's /opt/thermograph # Dev context: its own checkout and its own branch, on its own host (vps1).
# (main), so a dev deploy never touches or is touched by the prod/beta one. # Both come from deploy/env-topology.sh so there is one place that says where
APP_DIR="${APP_DIR:-$HOME/thermograph-dev}" # dev lives. $APP_DIR is still honoured when set explicitly — that is how a
BRANCH="${BRANCH:-dev}" # laptop points this at a checkout under $HOME.
_dev_self="$(cd "$(dirname "$0")" && pwd)"
# shellcheck source=infra/deploy/env-topology.sh
. "$_dev_self/env-topology.sh"
thermograph_topology dev
APP_DIR="${APP_DIR:-$TG_APP_DIR}"
BRANCH="${BRANCH:-$TG_BRANCH}"
export APP_DIR BRANCH export APP_DIR BRANCH
# deploy.sh resolves the environment itself; say so explicitly rather than
# letting it fall back to a host marker that, on vps1, names dev anyway.
export THERMOGRAPH_ENV=dev
# Point every `docker compose` invocation inside deploy.sh at the LAN-dev overlay # Point every `docker compose` invocation inside deploy.sh at the LAN-dev overlay
# (uncapped CPU, backend published on 0.0.0.0:8137, frontend unpublished -- see # (uncapped CPU, backend published on 0.0.0.0:8137, frontend unpublished -- see
@ -72,23 +85,33 @@ export COMPOSE_PROJECT_NAME="${COMPOSE_PROJECT_NAME:-thermograph-dev}"
# genuinely needs, and nothing else. # genuinely needs, and nothing else.
export THERMOGRAPH_SECRETS_SKIP_COMMON=1 export THERMOGRAPH_SECRETS_SKIP_COMMON=1
# (1b) Docker on this box is the snap package, confined to $HOME (plus a short # (1b) Snap-packaged Docker is confined to $HOME (plus a short allowlist) and
# allowlist) -- it silently can't see /etc/thermograph.env at all, so # silently can't see /etc/thermograph.env at all, so env_file: /etc/thermograph.env
# env_file: /etc/thermograph.env resolves to nothing for every container no # resolves to nothing for every container no matter how correctly it's rendered.
# matter how correctly it's rendered. Mirror the render to a path under # Where that is the case, mirror the render to a path under $APP_DIR;
# $APP_DIR too; docker-compose.dev.yml points env_file at this copy instead of # docker-compose.dev.yml carries a matching optional env_file entry.
# /etc/thermograph.env. Untracked (see infra/.gitignore); re-rendered on every #
# deploy, never committed. # CONDITIONAL, not unconditional: this writes a plaintext copy of the render
export THERMOGRAPH_SECRETS_ENV_FILE_MIRROR="$APP_DIR/infra/deploy/dev-secrets.env" # into the checkout, and on vps1 (ordinary apt Docker, which reads /etc fine)
# that copy would be pure liability. Detect the confinement rather than add a
# knob someone has to remember — the docker binary resolving under /snap is
# exactly the condition that breaks the /etc path.
_docker_bin="$(command -v docker || true)"
if [ -n "$_docker_bin" ] && case "$(readlink -f "$_docker_bin")" in /snap/*) true ;; *) false ;; esac; then
export THERMOGRAPH_SECRETS_ENV_FILE_MIRROR="$APP_DIR/infra/deploy/dev-secrets.env"
fi
# (2) The age key is read from wherever it is READABLE, not necessarily # (2) The age key is read from wherever it is READABLE, not necessarily
# /etc/thermograph/age.key. render-secrets.sh falls back to `sudo cat` for a # /etc/thermograph/age.key. render-secrets.sh falls back to `sudo cat` for a
# root-owned 0400 key, and this box has no passwordless sudo -- under the Forgejo # root-owned 0400 key, which needs passwordless sudo.
# runner (a systemd --user service, no tty) that sudo cannot prompt, so the #
# fleet-standard placement would make every dev deploy fail at the render. Prefer # On vps1 that is the normal case: the `agent` user has passwordless sudo and
# the standard path when it is readable; otherwise the operator's own keyring, which # the key sits at the fleet-standard path, so this block does nothing. The
# is where dev's key already lives (this box is the machine `sops` is run on), so # fallback exists for the laptop case — a checkout where the key lives only in
# provisioning dev needs no second copy of the estate's recovery root. # the operator's own keyring (that machine is where `sops` is run), and where a
# non-interactive shell could not prompt for sudo even if the key were in /etc.
# Preferring the standard path when readable keeps vps1 on the fleet convention
# and means provisioning dev needs no second copy of the estate's recovery root.
if [ -z "${THERMOGRAPH_AGE_KEY:-}" ] && [ ! -r /etc/thermograph/age.key ] \ if [ -z "${THERMOGRAPH_AGE_KEY:-}" ] && [ ! -r /etc/thermograph/age.key ] \
&& [ -r "$HOME/.config/sops/age/keys.txt" ]; then && [ -r "$HOME/.config/sops/age/keys.txt" ]; then
export THERMOGRAPH_AGE_KEY="$HOME/.config/sops/age/keys.txt" export THERMOGRAPH_AGE_KEY="$HOME/.config/sops/age/keys.txt"

96
infra/deploy/deploy.sh
View file

@ -26,11 +26,59 @@
# two axes are independent and rarely change together. # two axes are independent and rarely change together.
set -euo pipefail set -euo pipefail
APP_DIR="${APP_DIR:-/opt/thermograph}" # --- which environment is this? ------------------------------------------------
# vps2 runs beta AND prod, so "the host" no longer answers this — the caller does,
# via THERMOGRAPH_ENV (the deploy workflow always passes it). A by-hand run on a
# single-environment box still falls back to the host marker, and a box with
# neither still defaults to prod's historical paths, so nothing about an existing
# single-env host changes.
#
# Resolved from the SCRIPT'S OWN LOCATION, not a hardcoded /opt/thermograph:
# invoking /opt/thermograph-beta/infra/deploy/deploy.sh must act on the beta
# checkout even if something in the environment says otherwise. That is also the
# check that catches the one genuinely dangerous mistake on a two-environment
# host — running prod's checkout with THERMOGRAPH_ENV=beta, or the reverse.
SELF_DIR=$(cd "$(dirname "$0")" && pwd)
SELF_APP_DIR=$(cd "$SELF_DIR/../.." && pwd)
# Guarded exactly like render-secrets.sh below: the deploy that INTRODUCES this
# file runs with a checkout that predates it (it arrives with the git reset
# further down, after which deploy.sh re-execs). Missing => keep the pre-split
# behaviour rather than fail.
if [ -f "$SELF_DIR/env-topology.sh" ]; then
# shellcheck source=infra/deploy/env-topology.sh
. "$SELF_DIR/env-topology.sh"
ENV_NAME=$(thermograph_env_name)
# Nothing to go on anywhere: this is a pre-split host whose paths are prod's.
[ -n "$ENV_NAME" ] || ENV_NAME=prod
thermograph_topology "$ENV_NAME"
if [ "$SELF_APP_DIR" != "$TG_APP_DIR" ] && [ -z "${APP_DIR:-}" ]; then
echo "!! environment/checkout mismatch: THERMOGRAPH_ENV=$ENV_NAME expects" >&2
echo "!! $TG_APP_DIR but this script lives in $SELF_APP_DIR." >&2
echo "!! On vps2 that means beta and prod have been crossed. Refusing to deploy." >&2
echo "!! If this checkout really is $ENV_NAME (a rehearsal copy, a relocated" >&2
echo "!! checkout), say so explicitly: APP_DIR=$SELF_APP_DIR ..." >&2
exit 2
fi
APP_DIR="${APP_DIR:-$TG_APP_DIR}"
BRANCH="${BRANCH:-$TG_BRANCH}"
ENV_FILE="$TG_ENV_FILE"
DEPLOY_MODE="$TG_DEPLOY_MODE"
[ "$TG_SKIP_COMMON" = 1 ] && export THERMOGRAPH_SECRETS_SKIP_COMMON=1
# The second pass (after the re-exec) must resolve the SAME environment, and
# the stack path needs it too.
export THERMOGRAPH_ENV="$ENV_NAME"
else
ENV_NAME=""
APP_DIR="${APP_DIR:-/opt/thermograph}"
BRANCH="${BRANCH:-main}"
ENV_FILE=/etc/thermograph.env
DEPLOY_MODE=""
fi
# Monorepo layout: git operations act on the checkout root ($APP_DIR); all # Monorepo layout: git operations act on the checkout root ($APP_DIR); all
# compose files and deploy assets live under infra/. # compose files and deploy assets live under infra/.
INFRA_DIR="$APP_DIR/infra" INFRA_DIR="$APP_DIR/infra"
BRANCH="${BRANCH:-main}"
# Which service this deploy rolls: backend | frontend | all. Defaults to `all` # Which service this deploy rolls: backend | frontend | all. Defaults to `all`
# (a full-stack bring-up) so a by-hand run with both tags still works; the # (a full-stack bring-up) so a by-hand run with both tags still works; the
# per-repo deploy.yml workflows always pass an explicit single service. # per-repo deploy.yml workflows always pass an explicit single service.
@ -69,16 +117,20 @@ fi
# after which deploy.sh re-execs), so a missing helper simply falls back to the # after which deploy.sh re-execs), so a missing helper simply falls back to the
# existing /etc/thermograph.env. Then source it so a by-hand run interpolates the # existing /etc/thermograph.env. Then source it so a by-hand run interpolates the
# same as the systemd unit does. See deploy/render-secrets.sh + deploy/secrets/. # same as the systemd unit does. See deploy/render-secrets.sh + deploy/secrets/.
#
# $ENV_FILE, not a hardcoded /etc/thermograph.env: on vps2 prod renders
# prod.yaml there while beta renders beta.yaml to /etc/thermograph-beta.env.
# One file per environment, never shared.
if [ -f "$INFRA_DIR/deploy/render-secrets.sh" ]; then if [ -f "$INFRA_DIR/deploy/render-secrets.sh" ]; then
# shellcheck source=infra/deploy/render-secrets.sh # shellcheck source=infra/deploy/render-secrets.sh
. "$INFRA_DIR/deploy/render-secrets.sh" . "$INFRA_DIR/deploy/render-secrets.sh"
render_thermograph_secrets "$INFRA_DIR" render_thermograph_secrets "$INFRA_DIR" "$ENV_NAME" "$ENV_FILE"
fi fi
# /etc/thermograph.env is rendered at deploy time from the SOPS vault — it # $ENV_FILE is rendered at deploy time from the SOPS vault — it cannot exist at
# cannot exist at lint time, so don't ask shellcheck to follow it. # lint time, so don't ask shellcheck to follow it.
set -a set -a
# shellcheck source=/dev/null # shellcheck source=/dev/null
. /etc/thermograph.env 2>/dev/null || true . "$ENV_FILE" 2>/dev/null || true
set +a set +a
# Pre-warm the ~750 city-page archives so /climate pages serve from cache and a # Pre-warm the ~750 city-page archives so /climate pages serve from cache and a
@ -122,12 +174,18 @@ if [ -z "${DEPLOY_SH_REEXECED:-}" ]; then
exec "$0" "$@" exec "$0" "$@"
fi fi
# Stack-mode routing: a host whose /etc/thermograph/deploy-mode says "stack" # Stack-mode routing: prod and beta are both Swarm stacks, dev is compose. The
# (prod, after the Swarm cutover) deploys via the Swarm stack path instead of # mode is a property of the ENVIRONMENT (env-topology.sh), not of the host --
# compose. Checked AFTER the reset+re-exec so the stack script is always the # vps2 runs two stacks, and a host-wide /etc/thermograph/deploy-mode marker
# freshly-pulled one, and the SERVICE/tag contract passes through unchanged -- # cannot describe a host that runs more than one environment. The marker is
# the app repos' workflows never need to know which mode a host runs. # still honoured when the topology file is absent (a checkout that predates it).
if [ "$(cat /etc/thermograph/deploy-mode 2>/dev/null || true)" = "stack" ]; then # Checked AFTER the reset+re-exec so the stack script is always the freshly
# pulled one, and the SERVICE/tag contract passes through unchanged -- the
# deploy workflow never needs to know which mode an environment runs.
if [ -z "$DEPLOY_MODE" ]; then
DEPLOY_MODE=$(cat /etc/thermograph/deploy-mode 2>/dev/null || true)
fi
if [ "$DEPLOY_MODE" = "stack" ]; then
exec bash "$INFRA_DIR/deploy/stack/deploy-stack.sh" exec bash "$INFRA_DIR/deploy/stack/deploy-stack.sh"
fi fi
@ -137,6 +195,20 @@ fi
# rename the project and recreate the stack under a second name. # rename the project and recreate the stack under a second name.
cd "$INFRA_DIR" cd "$INFRA_DIR"
# Compose-mode environments (dev) carry their project name, file list and bind
# address in the topology table. Set only when not already in the environment,
# so deploy-dev.sh's own exports and a by-hand override both still win.
if [ -n "${TG_COMPOSE_PROJECT:-}" ]; then
export COMPOSE_PROJECT_NAME="${COMPOSE_PROJECT_NAME:-$TG_COMPOSE_PROJECT}"
export COMPOSE_FILE="${COMPOSE_FILE:-$TG_COMPOSE_FILE}"
fi
# Which address the dev overlay publishes on. Defaults to loopback in the
# compose file; dev on vps1 sets the mesh address here, because a public VPS
# must never publish an unreviewed branch's stack on 0.0.0.0.
if [ -n "${TG_BIND_ADDR:-}" ]; then
export DEV_BIND_ADDR="${DEV_BIND_ADDR:-$TG_BIND_ADDR}"
fi
# Registry-pull cutover: pull the image each app repo's build-push.yml already # Registry-pull cutover: pull the image each app repo's build-push.yml already
# built and pushed, instead of building in place. This checkout is # built and pushed, instead of building in place. This checkout is
# thermograph-infra, not an app repo, so there's no "current commit" to derive # thermograph-infra, not an app repo, so there's no "current commit" to derive

237
infra/deploy/env-topology.sh Normal file
View file

@ -0,0 +1,237 @@
#!/usr/bin/env bash
# WHERE EACH ENVIRONMENT LIVES — the single source of truth, sourced by every
# deploy path. Not executable on its own; `. env-topology.sh` then call
# `thermograph_topology <env>`.
#
# Why this file exists at all. Until the vps1/vps2 split, "which environment is
# this?" was answerable from the machine you were standing on: one host ran one
# environment, so a host-wide marker (/etc/thermograph/secrets-env) and a
# host-wide deploy mode (/etc/thermograph/deploy-mode) were enough, and every
# path could hardcode /opt/thermograph, /etc/thermograph.env, ports 8137/8080.
# vps2 now runs BOTH beta and prod. Every one of those assumptions becomes a
# collision: two checkouts, two rendered env files, two stacks, two loopback
# port pairs, two image-tag files, two deploy locks — on one box. So the
# environment is now an INPUT (THERMOGRAPH_ENV, passed by the deploy caller),
# and everything else is derived here rather than re-guessed per script.
#
# The host markers survive as the fallback for a by-hand run with no explicit
# env (`deploy.sh` on prod still means prod), but they are no longer the thing
# that decides where files go.
#
# THE ESTATE
#
# vps1 75.119.132.91 10.10.0.2 "operational programs"
# Forgejo (git + CI + registry), Grafana/Loki/Alloy, emigriffith.dev,
# and the DEV environment with its own Postgres. Nothing here is
# reachable by an external user except git/dashboard/the portfolio site;
# dev itself is mesh-only, deliberately (see DEV_BIND_ADDR below).
#
# vps2 169.58.46.181 10.10.0.1 "the deployed environment"
# Everything an external user can touch: prod (thermograph.org) AND beta
# (beta.thermograph.org), Centralis, Postfix, the backups, and the ONE
# TimescaleDB instance that serves both environments on separate
# databases with separate roles.
#
# desktop 10.10.0.3 operator's box
# AI model hosting (voice-to-text, the LLM behind upcoming features) and
# flex capacity as a Swarm worker. It hosts NO Thermograph environment —
# that is the whole point of the vps1/vps2 split. A laptop-style
# `make dev-up` still works locally; it is just not the dev server.
#
# WHY BETA SITS NEXT TO PROD RATHER THAN NEXT TO DEV
#
# Beta is a pre-production rehearsal, so what it needs to rehearse is prod's
# environment: the same orchestrator (Swarm, not compose), the same Postgres
# major and extension build, the same Caddy in front, the same mail path, the
# same mesh position. Co-locating beta with dev bought resemblance to the thing
# it is NOT trying to predict. Co-locating it with prod means a beta green light
# is evidence about prod. The cost — a bad beta deploy shares a machine with
# prod — is bounded by the per-environment isolation this file defines: separate
# stacks, separate volumes, separate DB roles, separate CPU limits.
# thermograph_topology <env>
#
# Exports the TG_* variables describing that environment. Every value is a
# derived fact about the estate, not a preference: change one here and the
# deploy scripts, the stack files and the runbooks all follow.
thermograph_topology() {
local env_name="${1:?thermograph_topology needs an environment: dev|beta|prod}"
# Reset first: this function is sourced into long-lived shells (deploy.sh
# sources it before and after its self re-exec), and a stale TG_STACK_NAME
# from a previous call would silently target the wrong stack.
TG_ENV=""; TG_HOST=""; TG_APP_DIR=""; TG_BRANCH=""; TG_DEPLOY_MODE=""
TG_STACK_NAME=""; TG_STACK_FILE=""; TG_LB_CONFIG=""
TG_COMPOSE_PROJECT=""; TG_COMPOSE_FILE=""
TG_ENV_FILE=""; TG_STACK_ENV_FILE=""; TG_LB_NAME=""
TG_LB_HTTP_PORT=""; TG_LB_FE_PORT=""; TG_DB_NAME=""; TG_DB_USER=""
TG_TAGS_FILE=""; TG_LOCK_FILE=""; TG_BIND_ADDR=""; TG_SKIP_COMMON=0
TG_SVC_PREFIX=""; TG_DATA_NETWORK=""; TG_DB_SERVICE=""; TG_POST_DEPLOY=0
TG_SSH_HOST=""; TG_SSH_TARGET=""
case "$env_name" in
prod)
TG_ENV=prod
TG_HOST=vps2
# Unchanged from before the split — prod keeps every path it already has,
# so nothing about the live prod host moves during the cutover.
TG_SSH_HOST=169.58.46.181
TG_SSH_TARGET=agent@169.58.46.181
TG_APP_DIR=/opt/thermograph
TG_BRANCH=main
TG_DEPLOY_MODE=stack
TG_STACK_NAME=thermograph
TG_STACK_FILE=deploy/stack/thermograph-stack.yml
TG_ENV_FILE=/etc/thermograph.env
TG_STACK_ENV_FILE=/etc/thermograph/stack.env
TG_LB_NAME=thermograph-lb
TG_LB_CONFIG=deploy/stack/lb/Caddyfile
TG_LB_HTTP_PORT=8137
TG_LB_FE_PORT=8080
TG_DB_NAME=thermograph
TG_DB_USER=thermograph
# Prod's services keep their bare names (web, worker, frontend, ...): the
# cutover must not touch prod's stack file, its env or its LB config.
TG_SVC_PREFIX=""
# Where the database lives. Prod's own overlay, which is `attachable` and
# which beta joins as an external network.
TG_DATA_NETWORK=thermograph_internal
TG_DB_SERVICE=thermograph_db
# Post-deploy city warm + IndexNow ping. Prod only — see the beta note.
TG_POST_DEPLOY=1
;;
beta)
TG_ENV=beta
TG_HOST=vps2
# Same box, same public IP as prod — beta and prod are two stacks on one
# Swarm manager now, not two separate hosts. A beta SSH target is
# therefore prod's blast radius too (see .claude/hooks/prod-guard.sh).
TG_SSH_HOST=169.58.46.181
TG_SSH_TARGET=agent@169.58.46.181
# A SECOND checkout on the same box. Separate from prod's so the two can
# sit on different commits of this repo, and so `git reset --hard` in one
# deploy can never yank the tree out from under the other's running
# deploy. Everything below is likewise a distinct name/port/path from
# prod's, on purpose: co-location is only safe if nothing is shared by
# accident. The one deliberate exception is the database SERVER (see
# TG_DB_* — separate role and database on a shared instance).
TG_APP_DIR=/opt/thermograph-beta
TG_BRANCH=main
TG_DEPLOY_MODE=stack
TG_STACK_NAME=thermograph-beta
TG_STACK_FILE=deploy/stack/thermograph-beta-stack.yml
TG_ENV_FILE=/etc/thermograph-beta.env
TG_STACK_ENV_FILE=/etc/thermograph/beta-stack.env
TG_LB_NAME=thermograph-beta-lb
TG_LB_CONFIG=deploy/stack/lb/Caddyfile.beta
# NOT 8137/8080: prod's loopback LB already owns those on this host.
TG_LB_HTTP_PORT=8237
TG_LB_FE_PORT=8180
TG_DB_NAME=thermograph_beta
# Its own role, not prod's `thermograph` superuser-of-its-own-database.
# One instance is a capacity decision; it is not a decision to let a beta
# deploy running an unmerged branch read or write the production database.
TG_DB_USER=thermograph_beta
# Beta's Swarm services are named beta-web, beta-worker, ... Swarm
# registers a service's SHORT name as a DNS alias on every network it
# joins, so two stacks that both call a service `web` on one shared
# network make `web` ambiguous — prod's frontend could resolve a beta
# task. Prefixing beta's names removes the collision without editing a
# single line of prod's stack.
TG_SVC_PREFIX="beta-"
# Beta has no db service of its own: it joins prod's overlay (declared
# `external` in its stack file) purely to reach `db`, and keeps its own
# `internal` overlay for beta-to-beta traffic.
TG_DATA_NETWORK=thermograph_internal
TG_DB_SERVICE=thermograph_db
# No warm, no IndexNow on beta. IndexNow announces URLs to Bing/DDG/
# Yandex — from beta that means asking search engines to index
# beta.thermograph.org, which is the opposite of what beta is for. The
# city warm is worse than useless here: it spends the shared upstream
# archive quota to fill a cache that only a rehearsal environment reads.
# (Both ran on beta under compose; that was a side effect of beta and
# prod sharing one script, not a decision.)
TG_POST_DEPLOY=0
;;
dev)
TG_ENV=dev
TG_HOST=vps1
# vps1 — Forgejo/Grafana/dev, mesh-only for dev itself (see TG_BIND_ADDR).
TG_SSH_HOST=75.119.132.91
TG_SSH_TARGET=agent@75.119.132.91
TG_APP_DIR=/opt/thermograph-dev
# The only environment that tracks `dev`; beta and prod both run this
# repo's `main` (infra is not environment-staged — app code is, via image
# tags). See the branch model in infra/README.md.
TG_BRANCH=dev
# Compose, not Swarm: dev's value is a fast, legible, single-box stack
# you can `docker compose logs` at. It is the one environment not
# pretending to be prod.
TG_DEPLOY_MODE=compose
TG_COMPOSE_PROJECT=thermograph-dev
TG_COMPOSE_FILE="docker-compose.yml:docker-compose.dev.yml"
TG_ENV_FILE=/etc/thermograph.env
# Dev keeps its OWN Postgres container (a `db` service in its compose
# project). It is not on the shared instance and must not be: the shared
# instance lives on vps2 and holds real user data.
TG_DB_NAME=thermograph
TG_DB_USER=thermograph
# Mesh-only. vps1 is a public VPS, so the dev overlay must not publish on
# 0.0.0.0 the way it did on the operator's LAN box — dev runs whatever
# branch is in flight, including unreviewed ones. Reachable at
# http://10.10.0.2:8137 from anything on the WireGuard mesh, and from
# nowhere else. There is deliberately no dev DNS record and no Caddy site.
TG_BIND_ADDR=10.10.0.2
# Dev NEVER layers common.yaml — that file is the internet-facing fleet's
# shared production credential set (both S3 keypairs, the VAPID signing
# key, REGISTRY_TOKEN, the metrics token), and dev runs whatever branch is
# in flight on the same box as Forgejo and its CI runner. This lives here
# rather than only in deploy-dev.sh so the protection is a property of the
# ENVIRONMENT, not of which entry point happened to be used. See the long
# note in render-secrets.sh.
TG_SKIP_COMMON=1
TG_DB_SERVICE=db
# Dev warms nothing and pings nothing: it must never spend the upstream
# archive quota, and it must never tell a search engine it exists.
TG_POST_DEPLOY=0
;;
*)
echo "!! unknown environment '$env_name' (expected dev|beta|prod)" >&2
return 2
;;
esac
# Derived, never hand-set: keeping these next to the definitions above is what
# stops a second environment on one host from sharing a lock or a tag file.
TG_LOCK_FILE="$TG_APP_DIR/infra/deploy/.deploy.lock"
if [ "$TG_DEPLOY_MODE" = stack ]; then
TG_TAGS_FILE="$TG_APP_DIR/infra/deploy/.stack-image-tags.env"
else
TG_TAGS_FILE="$TG_APP_DIR/infra/deploy/.image-tags.env"
fi
export TG_ENV TG_HOST TG_APP_DIR TG_BRANCH TG_DEPLOY_MODE
export TG_STACK_NAME TG_STACK_FILE TG_LB_CONFIG TG_COMPOSE_PROJECT TG_COMPOSE_FILE
export TG_ENV_FILE TG_STACK_ENV_FILE TG_LB_NAME
export TG_LB_HTTP_PORT TG_LB_FE_PORT TG_DB_NAME TG_DB_USER
export TG_TAGS_FILE TG_LOCK_FILE TG_BIND_ADDR TG_SKIP_COMMON
export TG_SVC_PREFIX TG_DATA_NETWORK TG_DB_SERVICE TG_POST_DEPLOY
export TG_SSH_HOST TG_SSH_TARGET
}
# thermograph_env_name
#
# Which environment is this run for? Explicit input first (the deploy workflows
# pass THERMOGRAPH_ENV, and on vps2 it is the ONLY thing distinguishing a beta
# deploy from a prod one), then the host marker for a by-hand run on a
# single-environment box. Empty output means "cannot tell" and the caller must
# fail rather than guess — guessing wrong on vps2 means deploying beta's image
# tag onto prod.
thermograph_env_name() {
local marker="${THERMOGRAPH_SECRETS_ENV_FILE:-/etc/thermograph/secrets-env}"
if [ -n "${THERMOGRAPH_ENV:-}" ]; then
printf '%s\n' "$THERMOGRAPH_ENV"
return 0
fi
cat "$marker" 2>/dev/null || true
}

96
infra/deploy/forgejo/README.md
View file

@ -1,22 +1,24 @@
# Forgejo on the Swarm cluster # Forgejo on the Swarm cluster
Runs as `deploy/forgejo/docker-stack.yml` — the only Swarm-scheduled workload Runs as `deploy/forgejo/docker-stack.yml` — the only Swarm-scheduled workload
this cluster carries (the Thermograph app itself stays on the this cluster carries (prod and beta's app stacks are separate `docker stack
Terraform-managed `docker compose` deploys; see `terraform/README.md`). deploy`s that happen to run on the manager node, vps2 — see
Pinned to the **beta** node (old VPS) via the `role=forge` label from `deploy/stack/README` context in `DEPLOY.md`). Pinned to the **vps1** node
(`75.119.132.91`) via the `role=forge` label from
`deploy/swarm/label-forge-node.sh`. `deploy/swarm/label-forge-node.sh`.
The Actions **runner** is deliberately *not* part of this stack — it runs on The Actions **runner** is deliberately *not* part of this stack — see
the **desktop** as a plain systemd service (`register-lan-runner.sh` below), `DEPLOY-DEV.md` and the note in `register-lan-runner.sh`'s own header for
per `thermograph-docs/runbooks/implementation-handoff.md` Track B step 5. That's the where it runs today. That's the canonical placement per
canonical placement; an earlier revision of this stack ran the runner as a `thermograph-docs/runbooks/implementation-handoff.md` Track B step 5; an
Swarm-scheduled Docker-in-Docker sidecar pinned to beta, which is gone now. earlier revision of this stack ran the runner as a Swarm-scheduled
Docker-in-Docker sidecar pinned to the Forgejo node, which is gone now.
## Prerequisites ## Prerequisites
1. All three nodes have joined the swarm (`deploy/swarm/`) and beta is 1. All three nodes have joined the swarm (`deploy/swarm/`) and vps1 is
labeled `role=forge`. labeled `role=forge`.
2. `docker node ls` (from the manager) shows all three `Ready`. 2. `docker node ls` (from the manager, vps2) shows all three `Ready`.
## One-time setup: Swarm secret ## One-time setup: Swarm secret
@ -43,27 +45,27 @@ its first health check just fails harmlessly until it is.
This stack has **no auto-deploy trigger** — nothing in `.forgejo/workflows/` This stack has **no auto-deploy trigger** — nothing in `.forgejo/workflows/`
redeploys it on push. A change to `docker-stack.yml` only takes effect once redeploys it on push. A change to `docker-stack.yml` only takes effect once
someone re-runs `docker stack deploy` by hand on the manager (prod). someone re-runs `docker stack deploy` by hand on the manager (vps2).
`db`/`forgejo` both carry `resources.limits` (defaults: db 1 CPU/1g, forgejo 2 `db`/`forgejo` both carry `resources.limits` (defaults: db 1 CPU/1g, forgejo 2
CPU/2g — several times observed steady-state usage), overridable with CPU/2g — several times observed steady-state usage), overridable with
`FORGEJO_DB_CPUS`/`FORGEJO_DB_MEMORY`/`FORGEJO_CPUS`/`FORGEJO_MEMORY` env vars `FORGEJO_DB_CPUS`/`FORGEJO_DB_MEMORY`/`FORGEJO_CPUS`/`FORGEJO_MEMORY` env vars
before `docker stack deploy`, same convention as the app stack. before `docker stack deploy`, same convention as the app stack.
## DNS + TLS: reusing beta's existing Caddy, not a second reverse proxy ## DNS + TLS: reusing vps1's existing Caddy, not a second reverse proxy
Forgejo is pinned to beta (`role=forge`) — but beta is also **today's live Forgejo is pinned to vps1 (`role=forge`) — the same box that also runs
thermograph.org host**, and its Caddy already owns ports 80/443 Grafana/Loki/Alloy and the `emigriffith.dev` portfolio site, each fronted by
(`/etc/caddy/Caddyfile` on that box). A second ingress (Traefik) trying to that host's own Caddy. A second ingress (Traefik) trying to bind the same
bind the same ports would collide with it. So there's no Traefik in this ports would collide with it. So there's no Traefik in this stack: `forgejo`'s
stack: `forgejo`'s web port publishes to `127.0.0.1:3080` only (host-local), web port publishes to `127.0.0.1:3080` only (host-local), and vps1's
and beta's *existing* Caddy gets one more site block reverse-proxying to it — *existing* Caddy gets one more site block reverse-proxying to it — same
same pattern as its `thermograph.org` block, same automatic-HTTPS. pattern as its other site blocks, same automatic-HTTPS.
1. Point the Forgejo domain (default `git.thermograph.org`; override with 1. Point the Forgejo domain (default `git.thermograph.org`; override with
`FORGEJO_DOMAIN=...` before `docker stack deploy`) at **beta's** public IP `FORGEJO_DOMAIN=...` before `docker stack deploy`) at **vps1's** public IP
— that's where the task actually runs, not prod's or the desktop's. — that's where the task actually runs, not vps2's or the desktop's.
2. Append `deploy/forgejo/caddy-git.conf` to beta's `/etc/caddy/Caddyfile`, 2. Append `deploy/forgejo/caddy-git.conf` to vps1's `/etc/caddy/Caddyfile`,
adjusting the domain if you didn't use the default, then `systemctl reload adjusting the domain if you didn't use the default, then `systemctl reload
caddy`. caddy`.
3. That file also resolves the registry-exposure hazard (#15 in 3. That file also resolves the registry-exposure hazard (#15 in
@ -75,28 +77,28 @@ same pattern as its `thermograph.org` block, same automatic-HTTPS.
## Registry access from mesh clients ## Registry access from mesh clients
Any node that needs `docker login`/push/pull against the registry (the Any node that needs `docker login`/push/pull against the registry (the CI
desktop's CI runner building/pushing images, later any Swarm node pulling runner building/pushing images, any Swarm node pulling them, prod or beta on
them) must reach `git.thermograph.org` **over the WireGuard tunnel**, not vps2 pulling app images) must reach `git.thermograph.org` **over the
beta's public IP — otherwise Caddy's `/v2/*` block above refuses the WireGuard tunnel**, not vps1's public IP — otherwise Caddy's `/v2/*` block
connection. Public DNS resolves the domain to beta's public IP, so add a above refuses the connection. Public DNS resolves the domain to vps1's public
`/etc/hosts` override on each such node pinning it to beta's WireGuard IP, so add a `/etc/hosts` override on each such node pinning it to vps1's
address instead: WireGuard address instead:
``` ```
echo "10.10.0.2 git.thermograph.org" | sudo tee -a /etc/hosts echo "10.10.0.2 git.thermograph.org" | sudo tee -a /etc/hosts
``` ```
(`10.10.0.2` is beta's WG address per `deploy/swarm/README.md`'s peer (`10.10.0.2` is vps1's WG address per `deploy/swarm/README.md`'s peer
numbering — adjust if you assigned it differently.) The git/web UI keeps numbering — adjust if you assigned it differently.) The git/web UI keeps
working normally for everyone else since only `/v2/*` is restricted. working normally for everyone else since only `/v2/*` is restricted.
## Register the Actions runner (on the desktop, not through Swarm) ## Register the Actions runner
Once Forgejo answers at its domain: Once Forgejo answers at its domain:
```bash ```bash
# On the desktop: # On the runner host (see DEPLOY-DEV.md for where that is today):
# Forgejo web UI -> Site Administration -> Actions -> Runners -> Create new Runner # Forgejo web UI -> Site Administration -> Actions -> Runners -> Create new Runner
# (or, for a repo-scoped runner: <repo> -> Settings -> Actions -> Runners) # (or, for a repo-scoped runner: <repo> -> Settings -> Actions -> Runners)
# copy the registration token, then: # copy the registration token, then:
@ -104,8 +106,8 @@ bash deploy/forgejo/register-lan-runner.sh https://<forgejo-domain> <token>
``` ```
See that script's header for exactly what it replaces (the pre-Forgejo GitHub See that script's header for exactly what it replaces (the pre-Forgejo GitHub
self-hosted runner on this same machine) and why it registers with two self-hosted runner) and why it registers with two labels where there used to
labels where there used to be two separate runners. be two separate runners.
`config.yaml`'s `runner.capacity` is raised from the tool's default of 1 to 8 `config.yaml`'s `runner.capacity` is raised from the tool's default of 1 to 8
(override with `CAPACITY=`) — a single PR push fires `pr-build`, (override with `CAPACITY=`) — a single PR push fires `pr-build`,
@ -113,18 +115,16 @@ labels where there used to be two separate runners.
no `needs:` between them), so capacity 1 serializes work that could run in no `needs:` between them), so capacity 1 serializes work that could run in
parallel, and even capacity 3 (an earlier, undocumented hand-tune) leaves parallel, and even capacity 3 (an earlier, undocumented hand-tune) leaves
`build-backend`/`build-frontend`/`validate-observability` queued behind `build-backend`/`build-frontend`/`validate-observability` queued behind
those three before they get a slot. The desktop has 16 cores / 34GB free those three before they get a slot.
today; 8 concurrent jobs is comfortable headroom without starving LAN dev's
own compose stack.
**Adding more runner capacity should mean raising this number, or adding a **Adding more runner capacity should mean raising this number, or adding a
second runner on the desktop itself — not putting a runner on prod or second runner alongside it — not putting a runner on prod or beta (vps2).**
beta.** `container.docker_host: automount` gives job containers the *host's* `container.docker_host: automount` gives job containers the *host's* Docker
Docker socket; on prod or beta that would mean any CI job has root-equivalent socket; on vps2 that would mean any CI job has root-equivalent access to both
access to whatever else is running there (the live app stack, or Forgejo the prod and beta stacks running there. An earlier revision of this stack
itself). An earlier revision of this stack actually did run the runner as a actually ran the runner as a Swarm-hosted container on the box Forgejo was
Swarm-hosted container on beta and was deliberately reverted to the desktop pinned to and was deliberately reverted for this same class of reason — see
for this reason — see the note at the top of `docker-stack.yml`. the note at the top of `docker-stack.yml`.
## Custom CI job image (`ci-runner/`) ## Custom CI job image (`ci-runner/`)
@ -155,8 +155,8 @@ docker push git.thermograph.org/emi/thermograph/ci-runner:vN
`register-lan-runner.sh`'s `LABELS` default points at the current tag, so `register-lan-runner.sh`'s `LABELS` default points at the current tag, so
fresh registrations pick it up automatically. The live runner is cut over by fresh registrations pick it up automatically. The live runner is cut over by
editing the `labels` array in `~/forgejo-runner/.runner` on the desktop (same editing the `labels` array in `~/forgejo-runner/.runner` on the runner host
runner id/token, no re-registration needed) and restarting the service — (same runner id/token, no re-registration needed) and restarting the service —
**verify a real job runs green under the new image before relying on it**, **verify a real job runs green under the new image before relying on it**,
same way v1's break was caught. Only after that verification should the same way v1's break was caught. Only after that verification should the
now-redundant `apt-get install docker.io` / `python3-yaml` steps be removed now-redundant `apt-get install docker.io` / `python3-yaml` steps be removed
@ -165,7 +165,7 @@ still running on the stock `node:20-bookworm` image.
## Why Postgres here and not the Thermograph app's TimescaleDB ## Why Postgres here and not the Thermograph app's TimescaleDB
Separate instance, separate network (`forgejo_net`, not the app's compose Separate instance, separate network (`forgejo_net`, not the app's overlay
network), separate volume. Forgejo is a distinct product with its own schema network), separate volume. Forgejo is a distinct product with its own schema
and its own backup/restore lifecycle — sharing a database with the app would and its own backup/restore lifecycle — sharing a database with the app would
couple two things that should be able to fail, migrate, and restore couple two things that should be able to fail, migrate, and restore
@ -177,7 +177,7 @@ independently.
docker service ls # forgejo_db, forgejo_forgejo both Running, 1/1 docker service ls # forgejo_db, forgejo_forgejo both Running, 1/1
curl -I https://git.thermograph.org/ # 200, valid cert (Caddy's, not a new one) curl -I https://git.thermograph.org/ # 200, valid cert (Caddy's, not a new one)
curl -I https://git.thermograph.org/v2/ # 403 from anywhere off the WireGuard mesh curl -I https://git.thermograph.org/v2/ # 403 from anywhere off the WireGuard mesh
# On the desktop, after registering the runner: # On the runner host, after registering the runner:
systemctl --user status forgejo-runner # active, both labels registered systemctl --user status forgejo-runner # active, both labels registered
``` ```

11
infra/deploy/forgejo/caddy-git.conf
View file

@ -3,10 +3,11 @@
# ends up as a confusing orphaned comment in production config with no # ends up as a confusing orphaned comment in production config with no
# surrounding context (docker-stack.yml isn't visible from there). # surrounding context (docker-stack.yml isn't visible from there).
# #
# Target: /etc/caddy/Caddyfile on beta (the box `forgejo` is pinned to — see # Target: /etc/caddy/Caddyfile on vps1 (the box `forgejo` is pinned to — see
# docker-stack.yml). Reuses beta's existing Caddy instead of a second reverse # docker-stack.yml; this is the box also called "vps1", 75.119.132.91).
# proxy/ACME flow: Forgejo publishes its web UI to 127.0.0.1:3080 only # Reuses vps1's existing Caddy instead of a second reverse proxy/ACME flow:
# (host-local), and this block is the only thing that ever talks to it. # Forgejo publishes its web UI to 127.0.0.1:3080 only (host-local), and this
# block is the only thing that ever talks to it.
# #
# Registry exposure (hazard #15, see docker-stack.yml's header comment): # Registry exposure (hazard #15, see docker-stack.yml's header comment):
# git.thermograph.org/v2/* is the built-in OCI registry API. It's blocked # git.thermograph.org/v2/* is the built-in OCI registry API. It's blocked
@ -15,7 +16,7 @@
# else (web UI, git-over-HTTP, PR pages) stays public like the rest of the # else (web UI, git-over-HTTP, PR pages) stays public like the rest of the
# site. # site.
# #
# Update the domain below to match whatever you actually pointed at beta's # Update the domain below to match whatever you actually pointed at vps1's
# IP, if not git.thermograph.org. # IP, if not git.thermograph.org.
# --- copy from here down into /etc/caddy/Caddyfile --- # --- copy from here down into /etc/caddy/Caddyfile ---

31
infra/deploy/forgejo/docker-stack.yml
View file

@ -9,18 +9,18 @@
# (deploy/forgejo/register-lan-runner.sh) — the same place the pre-Forgejo # (deploy/forgejo/register-lan-runner.sh) — the same place the pre-Forgejo
# GitHub self-hosted runner already lived, not a Swarm-scheduled container. # GitHub self-hosted runner already lived, not a Swarm-scheduled container.
# #
# No Traefik here. Forgejo is pinned to beta (node.labels.role == forge) # No Traefik here. Forgejo is pinned to vps1 (node.labels.role == forge)
# because that's what was chosen, but beta is ALSO today's live thermograph.org # because that's what was chosen, and vps1 is ALSO the emigriffith.dev portfolio
# host — Caddy already owns its ports 80/443 (see /etc/caddy/Caddyfile on that # host — Caddy already owns its ports 80/443 (see /etc/caddy/Caddyfile on that
# box). A second reverse proxy binding those same ports would either fail to # box). A second reverse proxy binding those same ports would either fail to
# start or fight Caddy. Instead: forgejo's web port publishes to # start or fight Caddy. Instead: forgejo's web port publishes to
# 127.0.0.1:3080 only (host-local, mode: host), and beta's existing Caddy gets # 127.0.0.1:3080 only (host-local, mode: host), and vps1's existing Caddy gets
# a new site block reverse-proxying git.thermograph.org -> 127.0.0.1:3080, # a new site block reverse-proxying git.thermograph.org -> 127.0.0.1:3080,
# same pattern as its thermograph.org block. TLS is Caddy's existing # same pattern as its other blocks. TLS is Caddy's existing automatic-HTTPS
# automatic-HTTPS (HTTP-01), not a second ACME flow. # (HTTP-01), not a second ACME flow.
# #
# Deploy from the manager node (prod), after all three nodes (prod, beta, # Deploy from the manager node (vps2, which runs prod), after all three nodes
# desktop) have joined the swarm and beta is labeled role=forge: # (vps1, vps2, desktop) have joined the swarm and vps1 is labeled role=forge:
# #
# docker stack deploy -c deploy/forgejo/docker-stack.yml forgejo # docker stack deploy -c deploy/forgejo/docker-stack.yml forgejo
# #
@ -32,7 +32,7 @@
# firewall the /v2/ registry API paths so only WireGuard-mesh clients can # firewall the /v2/ registry API paths so only WireGuard-mesh clients can
# reach them. That's implemented in the Caddy site block (see # reach them. That's implemented in the Caddy site block (see
# deploy/forgejo/README.md), not here — nothing in this stack file is # deploy/forgejo/README.md), not here — nothing in this stack file is
# registry-specific, the restriction lives entirely in Caddy's config on beta. # registry-specific, the restriction lives entirely in Caddy's config on vps1.
services: services:
db: db:
@ -96,9 +96,10 @@ services:
FORGEJO__service__ENABLE_PASSWORD_SIGNIN_FORM: "false" FORGEJO__service__ENABLE_PASSWORD_SIGNIN_FORM: "false"
FORGEJO__service__REGISTER_MANUAL_CONFIRM: "true" FORGEJO__service__REGISTER_MANUAL_CONFIRM: "true"
FORGEJO__service__DEFAULT_USER_IS_RESTRICTED: "true" FORGEJO__service__DEFAULT_USER_IS_RESTRICTED: "true"
# Outbound mail via prod's Postfix null client over the WireGuard mesh # Outbound mail via vps2's Postfix null client (prod's box) over the
# (10.10.0.1:25 — mynetworks permits beta); self-signed cert on :25, so # WireGuard mesh (10.10.0.1:25 — mynetworks permits vps1, 10.10.0.2,
# trust it. Send-only; used for admin/approval and notification mail. # where this Forgejo runs); self-signed cert on :25, so trust it.
# Send-only; used for admin/approval and notification mail.
FORGEJO__mailer__ENABLED: "true" FORGEJO__mailer__ENABLED: "true"
FORGEJO__mailer__PROTOCOL: "smtp" FORGEJO__mailer__PROTOCOL: "smtp"
FORGEJO__mailer__SMTP_ADDR: "10.10.0.1" FORGEJO__mailer__SMTP_ADDR: "10.10.0.1"
@ -117,17 +118,17 @@ services:
networks: [forgejo_net] networks: [forgejo_net]
ports: ports:
# SSH for git@ clones — published on whichever node the task lands on # SSH for git@ clones — published on whichever node the task lands on
# (pinned to beta by the placement constraint below, so effectively # (pinned to vps1 by the placement constraint below, so effectively
# always beta's public IP, port 2222). # always vps1's public IP, port 2222).
- target: 2222 - target: 2222
published: 2222 published: 2222
protocol: tcp protocol: tcp
mode: host mode: host
# Web UI/API. Swarm's port schema has no host_ip scoping, so this binds # Web UI/API. Swarm's port schema has no host_ip scoping, so this binds
# 0.0.0.0:3080 on beta — NOT actually localhost-only by itself. A # 0.0.0.0:3080 on vps1 — NOT actually localhost-only by itself. A
# DOCKER-USER iptables rule (see deploy/forgejo/README.md) is what # DOCKER-USER iptables rule (see deploy/forgejo/README.md) is what
# actually restricts it, since Docker's own iptables rules bypass ufw # actually restricts it, since Docker's own iptables rules bypass ufw
# for published ports. beta's Caddy reverse-proxies to 127.0.0.1:3080. # for published ports. vps1's Caddy reverse-proxies to 127.0.0.1:3080.
- target: 3000 - target: 3000
published: 3080 published: 3080
protocol: tcp protocol: tcp

27
infra/deploy/forgejo/register-lan-runner.sh
View file

@ -4,17 +4,26 @@
# runner (see DEPLOY-DEV.md) — this replaces that runner, it doesn't add a # runner (see DEPLOY-DEV.md) — this replaces that runner, it doesn't add a
# second one. Sudo-free, systemd --user, same pattern as the app service. # second one. Sudo-free, systemd --user, same pattern as the app service.
# #
# This is THE runner (docs/runbooks/implementation-handoff.md Track B step 5 # It registers with BOTH labels the workflows historically needed: general
# — canonical placement is the desktop, not a Swarm-hosted container), so it # CI/build/deploy jobs (`docker`, containerized via this machine's own
# registers with BOTH labels the workflows need, where one runner used to be # already-installed Docker — no Docker-in-Docker sidecar needed, since a real
# two: general CI/build/deploy.yml jobs (`docker`, containerized via this # host with a real Docker install needs no such indirection) and the LAN-deploy
# machine's own already-installed Docker — no Docker-in-Docker sidecar needed, # job (`thermograph-lan`, bare/host-native — it wrote to ~/thermograph-dev and
# unlike the Swarm-hosted design this replaces, since a real host with a real # restarted a systemd --user service, which only worked running directly on the
# Docker install needs no such indirection) and the LAN-deploy job
# (`thermograph-lan`, bare/host-native — it writes to ~/thermograph-dev and
# restarts a systemd --user service, which only works running directly on the
# host, not inside a container). # host, not inside a container).
# #
# `thermograph-lan` IS NOW OBSOLETE. Dev moved off this machine to vps1 and is
# deployed over SSH by the same `Deploy` workflow that ships beta and prod;
# there is no host-native LAN deploy job left for that label to serve. Nothing
# breaks by keeping it registered — no workflow requests it — but do not build
# anything new on it.
#
# TODO(cutover): whether this machine keeps serving the `docker` label at all is
# an open call. It is no longer the only runner (the Swarm-hosted one is
# always-on), and the box's new job is AI model hosting plus flex Swarm-worker
# capacity. Keeping it is fine — it is real CI capacity — but the estate no
# longer depends on it, so decide deliberately rather than by inertia.
#
# bash deploy/forgejo/register-lan-runner.sh <forgejo_url> <registration_token> # bash deploy/forgejo/register-lan-runner.sh <forgejo_url> <registration_token>
# #
# Get <registration_token> from the Forgejo web UI: # Get <registration_token> from the Forgejo web UI:

10
infra/deploy/openmeteo/README.md
View file

@ -184,6 +184,10 @@ credit; keep it in place.
## 9. Not on dev/beta ## 9. Not on dev/beta
This overlay runs only on the self-hosting host (prod). Dev and beta leave This overlay runs only for prod specifically — not for every environment on
`THERMOGRAPH_ARCHIVE_URL` unset and use the public Open-Meteo archive API — do prod's host. Beta now shares vps2 with prod, but that doesn't extend the
not bring the overlay up there. self-hosted archive to it: beta is its own Swarm stack
(`thermograph-beta-stack.yml`), which never sets `THERMOGRAPH_ARCHIVE_URL`, so
it reaches the public Open-Meteo archive API like dev does. Dev leaves it
unset for the same reason on vps1. Do not bring this overlay up for beta or
dev, and don't assume co-residency with prod on vps2 changes that.

30
infra/deploy/provision-dev-lan.sh
View file

@ -1,30 +0,0 @@
#!/usr/bin/env bash
# One-time bootstrap for the Thermograph LAN dev server on THIS machine.
#
# Sudo-free: the app runs as a Docker Compose stack (see deploy-dev.sh), and
# `linger` keeps the runner/services running across logout/reboot. Re-runnable
# (idempotent).
#
# bash deploy/provision-dev-lan.sh
set -euo pipefail
APP_DIR="${APP_DIR:-$HOME/thermograph-dev}"
REPO_URL="${REPO_URL:-http://10.10.0.2:3080/emi/thermograph-infra.git}"
BRANCH="${BRANCH:-dev}"
here="$(cd "$(dirname "$0")" && pwd)"
echo "==> Enabling linger so the service survives logout/reboot"
loginctl enable-linger "$USER" \
|| echo " (couldn't enable linger; the service still runs while you're logged in)"
echo "==> Cloning/refreshing $APP_DIR and starting the service"
APP_DIR="$APP_DIR" REPO_URL="$REPO_URL" BRANCH="$BRANCH" bash "$here/deploy-dev.sh"
cat <<EOF
Done. The dev server runs as the 'thermograph-dev' systemd --user service.
status: systemctl --user status thermograph-dev
logs: journalctl --user -u thermograph-dev -f
restart: systemctl --user restart thermograph-dev
stop: systemctl --user stop thermograph-dev
EOF

79
infra/deploy/provision-dev.sh Executable file
View file

@ -0,0 +1,79 @@
#!/usr/bin/env bash
# One-time bootstrap of the DEV environment on vps1.
#
# sudo bash infra/deploy/provision-dev.sh
#
# Replaces provision-dev-lan.sh, which bootstrapped dev on the operator's
# desktop as a sudo-free systemd --user stack with `linger`. Dev is a normal
# fleet environment now: a checkout at /opt/thermograph-dev, secrets rendered
# from the SOPS vault at deploy time, deployed over SSH by CI like beta and
# prod. The desktop hosts no Thermograph environment at all.
#
# What dev keeps that beta and prod do not:
# - It renders dev.yaml ALONE, never layering common.yaml (the fleet's shared
# production credentials). vps1 also runs Forgejo and its CI, and dev runs
# whatever branch is in flight — see the long note in render-secrets.sh.
# - It is MESH-ONLY: published on 10.10.0.2:8137 (wg0), never 0.0.0.0. No DNS
# record, no Caddy site, no TLS. Anything that can reach it is already on
# the WireGuard mesh.
#
# Idempotent; re-run it after changing the branch or repointing the remote.
set -euo pipefail
SELF_DIR=$(cd "$(dirname "$0")" && pwd)
# shellcheck source=infra/deploy/env-topology.sh
. "$SELF_DIR/env-topology.sh"
thermograph_topology dev
REPO_URL="${REPO_URL:-http://10.10.0.2:3080/emi/thermograph.git}"
APP_DIR="${APP_DIR:-$TG_APP_DIR}"
BRANCH="${BRANCH:-$TG_BRANCH}"
if [ "$(id -u)" -ne 0 ]; then
echo "!! run this as root (it writes /opt and /etc/thermograph)" >&2
exit 1
fi
echo "==> Checkout: $APP_DIR on $BRANCH"
if [ -d "$APP_DIR/.git" ]; then
git -C "$APP_DIR" remote set-url origin "$REPO_URL"
git -C "$APP_DIR" fetch --prune origin "$BRANCH"
git -C "$APP_DIR" checkout -B "$BRANCH" "origin/$BRANCH"
else
git clone --branch "$BRANCH" "$REPO_URL" "$APP_DIR"
fi
echo "==> Marking this checkout's environment"
mkdir -p /etc/thermograph
# The host marker still exists for by-hand runs. vps1 runs exactly one
# environment, so a marker is sufficient here — unlike vps2, where beta and prod
# share a box and THERMOGRAPH_ENV must be passed explicitly.
printf 'dev\n' > /etc/thermograph/secrets-env
# Dev is compose, not Swarm. env-topology.sh is what actually decides this; the
# marker is only the fallback for a checkout that predates it.
rm -f /etc/thermograph/deploy-mode
if [ ! -f /etc/thermograph/age.key ]; then
cat >&2 <<'EOF'
!! No age key at /etc/thermograph/age.key.
!! Dev cannot render its vault without it, and deploy-dev.sh will fall back to
!! the un-vaulted defaults. Copy the key over (0400 root:root), then re-run:
!! install -m 0400 -o root -g root age.key /etc/thermograph/age.key
EOF
fi
cat <<EOF
Done. Dev is checked out at $APP_DIR on $BRANCH.
Deploy it:
SERVICE=all BACKEND_IMAGE_TAG=sha-<a> FRONTEND_IMAGE_TAG=sha-<b> \\
$APP_DIR/infra/deploy/deploy-dev.sh
Reach it (mesh only):
http://${TG_BIND_ADDR}:8137
Confirm it is NOT publicly exposed — this must show ${TG_BIND_ADDR}:8137 and
never 0.0.0.0:8137:
ss -ltnp | grep 8137
EOF

62
infra/deploy/provision-mail.sh
View file

@ -35,6 +35,12 @@
# sudo MAIL_DOMAIN=thermograph.org bash deploy/provision-mail.sh # sudo MAIL_DOMAIN=thermograph.org bash deploy/provision-mail.sh
# sudo MAIL_DOMAIN=thermograph.org RELAYHOST='[smtp.provider.com]:587' \ # sudo MAIL_DOMAIN=thermograph.org RELAYHOST='[smtp.provider.com]:587' \
# RELAY_USER=apikey RELAY_PASSWORD=secret bash deploy/provision-mail.sh # RELAY_USER=apikey RELAY_PASSWORD=secret bash deploy/provision-mail.sh
#
# MAIL_ENV picks which environment's deploy mode (env-topology.sh) sizes the
# Docker mail gateway below -- see the "WHICH gateway" comment. Defaults to
# prod; only matters if this box ever runs an environment in compose mode
# (it doesn't today -- prod and beta are both Swarm on this box; compose-mode
# dev lives on the other box entirely and doesn't run Postfix).
set -euo pipefail set -euo pipefail
MAIL_DOMAIN="${MAIL_DOMAIN:-thermograph.org}" MAIL_DOMAIN="${MAIL_DOMAIN:-thermograph.org}"
@ -43,6 +49,12 @@ RELAYHOST="${RELAYHOST:-}"
RELAY_USER="${RELAY_USER:-}" RELAY_USER="${RELAY_USER:-}"
RELAY_PASSWORD="${RELAY_PASSWORD:-}" RELAY_PASSWORD="${RELAY_PASSWORD:-}"
MAIL_ENV="${MAIL_ENV:-prod}"
SELF_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
# shellcheck source=infra/deploy/env-topology.sh
. "$SELF_DIR/env-topology.sh"
thermograph_topology "$MAIL_ENV"
if [[ $EUID -ne 0 ]]; then if [[ $EUID -ne 0 ]]; then
echo "run as root (sudo)" >&2 echo "run as root (sudo)" >&2
exit 1 exit 1
@ -68,17 +80,21 @@ postconf -e "myorigin = ${MAIL_DOMAIN}"
# docker-compose.yml). Set DOCKER_MAIL_GATEWAY="" for a pure loopback-only null # docker-compose.yml). Set DOCKER_MAIL_GATEWAY="" for a pure loopback-only null
# client (app running natively on the host, not in a container). # client (app running natively on the host, not in a container).
# #
# WHICH gateway depends on the host's deploy mode: plain compose (beta, LAN) # WHICH gateway to listen on comes from MAIL_ENV's deploy mode (env-topology.sh,
# uses the pinned compose bridge (172.19.0.1/172.19.0.0/16, the defaults); # TG_DEPLOY_MODE) -- not a hardcoded beta/prod split. Beta moved onto this SAME
# Swarm-stack mode (prod) uses the docker_gwbridge gateway instead -- # box as prod and is Swarm too now, so "compose" no longer means "beta"; it
# overlay tasks have no compose-bridge gateway -- so prod is provisioned with # means dev, which lives on the other box entirely and never runs this script.
# DOCKER_MAIL_GATEWAY=172.18.0.1 DOCKER_MAIL_SUBNET=172.18.0.0/16 (plus its # Compose mode uses the pinned compose bridge (172.19.0.1/172.19.0.0/16, the
# MESH_MAIL_* listener below). Do NOT list an address that doesn't exist on # defaults); stack mode uses the docker_gwbridge gateway instead -- overlay
# the host: Postfix's master fails to bind and takes ALL listeners down -- # tasks have no compose-bridge gateway -- so a stack-mode MAIL_ENV (prod or
# exactly what happened when the compose bridge (172.19.0.1) vanished at the # beta; same box, same gateway) gets DOCKER_MAIL_GATEWAY=172.18.0.1
# stack cutover while still listed in inet_interfaces. Also note: postfix on # DOCKER_MAIL_SUBNET=172.18.0.0/16 (plus the MESH_MAIL_* listener below). Do
# this distro is an umbrella unit; restart `postfix@-`, not `postfix`, for # NOT list an address that doesn't exist on the host: Postfix's master fails
# inet_interfaces changes to take effect. # to bind and takes ALL listeners down -- exactly what happened when the
# compose bridge (172.19.0.1) vanished at the stack cutover while still listed
# in inet_interfaces. Also note: postfix on this distro is an umbrella unit;
# restart `postfix@-`, not `postfix`, for inet_interfaces changes to take
# effect.
# #
# THE SAME TRAP FIRES AT BOOT, NOT JUST ON RENUMBERING (prod outage 2026-07-24). # THE SAME TRAP FIRES AT BOOT, NOT JUST ON RENUMBERING (prod outage 2026-07-24).
# A Docker bridge address does not exist until dockerd creates it, and the stock # A Docker bridge address does not exist until dockerd creates it, and the stock
@ -92,15 +108,23 @@ postconf -e "myorigin = ${MAIL_DOMAIN}"
# it orders postfix@ after docker.service and wg-quick@wg0.service and retries # it orders postfix@ after docker.service and wg-quick@wg0.service and retries
# on failure. If you add an address here that some other daemon creates, add # on failure. If you add an address here that some other daemon creates, add
# that daemon to the drop-in too. # that daemon to the drop-in too.
DOCKER_MAIL_GATEWAY="${DOCKER_MAIL_GATEWAY-172.19.0.1}" if [ "$TG_DEPLOY_MODE" = stack ]; then
DOCKER_MAIL_SUBNET="${DOCKER_MAIL_SUBNET-172.19.0.0/16}" DOCKER_MAIL_GATEWAY="${DOCKER_MAIL_GATEWAY-172.18.0.1}"
# Optional WireGuard-mesh listener: other mesh nodes (e.g. beta's Forgejo, whose DOCKER_MAIL_SUBNET="${DOCKER_MAIL_SUBNET-172.18.0.0/16}"
else
DOCKER_MAIL_GATEWAY="${DOCKER_MAIL_GATEWAY-172.19.0.1}"
DOCKER_MAIL_SUBNET="${DOCKER_MAIL_SUBNET-172.19.0.0/16}"
fi
# Optional WireGuard-mesh listener: other mesh nodes (e.g. vps1's Forgejo, whose
# mailer posts to 10.10.0.1:25 — see deploy/forgejo/docker-stack.yml) can relay # mailer posts to 10.10.0.1:25 — see deploy/forgejo/docker-stack.yml) can relay
# through this box. Prod runs with MESH_MAIL_LISTEN=10.10.0.1 and # through this box. This host (vps2 -- prod AND beta) runs with
# MESH_MAIL_PEERS=10.10.0.2/32; both default OFF so a plain run stays a strict # MESH_MAIL_LISTEN=10.10.0.1 and MESH_MAIL_PEERS=10.10.0.2/32; both default OFF
# null client. Without these, re-running this script on prod would silently drop # so a plain run stays a strict null client. 10.10.0.2 is vps1 (Forgejo,
# the mesh listener and break Forgejo's outbound mail — the live config was # Grafana, dev) -- mesh IPs did not move in the vps1/vps2 split, only which
# originally hand-applied and this script is the source of truth for it now. # environment runs where, so this is NOT "beta's" address. Without these,
# re-running this script on vps2 would silently drop the mesh listener and
# break Forgejo's outbound mail — the live config was originally hand-applied
# and this script is the source of truth for it now.
MESH_MAIL_LISTEN="${MESH_MAIL_LISTEN-}" MESH_MAIL_LISTEN="${MESH_MAIL_LISTEN-}"
MESH_MAIL_PEERS="${MESH_MAIL_PEERS-}" MESH_MAIL_PEERS="${MESH_MAIL_PEERS-}"
postconf -e "inet_protocols = ipv4" postconf -e "inet_protocols = ipv4"

46
infra/deploy/provision-secrets.sh
View file

@ -1,16 +1,37 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# Provision a host to render secrets from the SOPS vault at deploy time. Idempotent; # Provision a NAMED ENVIRONMENT to render secrets from the SOPS vault at deploy
# run once per box (prod/beta) as a sudo-capable user. # time. Idempotent; run once per environment (not once per box) as a
# sudo-capable user — vps2 carries two (prod AND beta), so it needs this run
# TWICE, each pointing SECRETS_ENV_FILE at that environment's own marker.
# #
# SECRETS_ENV=prod AGE_KEY_SRC=/path/to/age.key bash deploy/provision-secrets.sh # SECRETS_ENV=prod AGE_KEY_SRC=/path/to/age.key bash deploy/provision-secrets.sh
# SECRETS_ENV=beta AGE_KEY_SRC=/path/to/age.key \
# SECRETS_ENV_FILE=/etc/thermograph/beta-secrets-env bash deploy/provision-secrets.sh
# #
# Installs `sops` + `age`, then installs the age PRIVATE key at /etc/thermograph/age.key # Installs `sops` + `age` (idempotent, and shared across every environment on the
# (0400) and writes /etc/thermograph/secrets-env. After this, deploy.sh renders # box — one age key decrypts every deploy/secrets/<env>.yaml, since they're all
# /etc/thermograph.env from deploy/secrets/*.yaml. See deploy/secrets/README.md. # encrypted to the same recipient), then installs the age PRIVATE key at
# /etc/thermograph/age.key (0400) and writes SECRETS_ENV_FILE (default
# /etc/thermograph/secrets-env — right for a single-environment host, dev
# included). After this, deploy.sh renders that environment's own
# /etc/thermograph*.env from deploy/secrets/*.yaml. See deploy/secrets/README.md.
#
# SECRETS_ENV_FILE only matters for a by-hand run with THERMOGRAPH_ENV unset:
# env-topology.sh's thermograph_env_name() falls back to reading it to decide
# which environment a bare `deploy.sh` means, and a host running two
# environments (vps2) cannot answer that from one shared marker — a by-hand
# deploy of beta there must export THERMOGRAPH_SECRETS_ENV_FILE to match
# whatever path this script wrote. CI-driven deploys pass THERMOGRAPH_ENV
# explicitly and never consult this file at all.
set -euo pipefail set -euo pipefail
SECRETS_ENV="${SECRETS_ENV:?set SECRETS_ENV=prod|beta}" SECRETS_ENV="${SECRETS_ENV:?set SECRETS_ENV=prod|beta|dev}"
AGE_KEY_SRC="${AGE_KEY_SRC:?set AGE_KEY_SRC=/path/to/the/age/private/key}" AGE_KEY_SRC="${AGE_KEY_SRC:?set AGE_KEY_SRC=/path/to/the/age/private/key}"
# Where to write the env marker. Default matches the original, single-environment
# behavior; a SECOND environment on the same box (vps2's beta, alongside prod's
# default marker) must pass a distinct path so provisioning one can never
# overwrite the other's marker.
SECRETS_ENV_FILE="${SECRETS_ENV_FILE:-/etc/thermograph/secrets-env}"
SOPS_VERSION="${SOPS_VERSION:-v3.13.2}" SOPS_VERSION="${SOPS_VERSION:-v3.13.2}"
AGE_VERSION="${AGE_VERSION:-v1.3.1}" AGE_VERSION="${AGE_VERSION:-v1.3.1}"
BIN="${BIN:-/usr/local/bin}" BIN="${BIN:-/usr/local/bin}"
@ -35,11 +56,16 @@ install_age() {
install_sops install_sops
install_age install_age
echo "==> Installing age key -> /etc/thermograph/age.key (0400) and marker (${SECRETS_ENV})" echo "==> Installing age key -> /etc/thermograph/age.key (0400) and marker ${SECRETS_ENV_FILE} (${SECRETS_ENV})"
sudo mkdir -p /etc/thermograph sudo mkdir -p /etc/thermograph "$(dirname "$SECRETS_ENV_FILE")"
sudo install -m 0400 "$AGE_KEY_SRC" /etc/thermograph/age.key sudo install -m 0400 "$AGE_KEY_SRC" /etc/thermograph/age.key
printf '%s\n' "$SECRETS_ENV" | sudo tee /etc/thermograph/secrets-env >/dev/null printf '%s\n' "$SECRETS_ENV" | sudo tee "$SECRETS_ENV_FILE" >/dev/null
sudo chmod 0644 /etc/thermograph/secrets-env sudo chmod 0644 "$SECRETS_ENV_FILE"
echo "==> Done. Verify: sops --version && ls -l /etc/thermograph/" echo "==> Done. Verify: sops --version && ls -l /etc/thermograph/"
echo " Next: seed deploy/secrets/*.yaml, dry-run render, then deploy (see README)." echo " Next: seed deploy/secrets/*.yaml, dry-run render, then deploy (see README)."
if [ "$SECRETS_ENV_FILE" != /etc/thermograph/secrets-env ]; then
echo " NOTE: non-default marker -- a by-hand deploy of ${SECRETS_ENV} on this box"
echo " must export THERMOGRAPH_SECRETS_ENV_FILE=${SECRETS_ENV_FILE} (CI-driven"
echo " deploys pass THERMOGRAPH_ENV explicitly and never consult this file)."
fi

27
infra/deploy/render-secrets.sh
View file

@ -15,12 +15,21 @@
# (prod|beta|dev). A host without them keeps whatever /etc/thermograph.env it already # (prod|beta|dev). A host without them keeps whatever /etc/thermograph.env it already
# has — so merging this is a safe no-op until a host is deliberately migrated. See # has — so merging this is a safe no-op until a host is deliberately migrated. See
# deploy/secrets/README.md. # deploy/secrets/README.md.
# render_thermograph_secrets <repo> [env_name] [out_file]
#
# env_name and out_file are explicit since vps2 began running two environments.
# A host-wide marker file cannot name two environments, and one output path
# cannot hold two renders: on vps2 prod renders prod.yaml to /etc/thermograph.env
# while beta renders beta.yaml to /etc/thermograph-beta.env. Both arguments keep
# their pre-split defaults (marker file, /etc/thermograph.env), so a
# single-environment host and a by-hand run behave exactly as before.
render_thermograph_secrets() { render_thermograph_secrets() {
local repo="${1:-.}" # repo root holding deploy/secrets local repo="${1:-.}" # repo root holding deploy/secrets
local key="${THERMOGRAPH_AGE_KEY:-/etc/thermograph/age.key}" local key="${THERMOGRAPH_AGE_KEY:-/etc/thermograph/age.key}"
local marker="${THERMOGRAPH_SECRETS_ENV_FILE:-/etc/thermograph/secrets-env}" local marker="${THERMOGRAPH_SECRETS_ENV_FILE:-/etc/thermograph/secrets-env}"
local env_name local env_name="${2:-}"
env_name=$(cat "$marker" 2>/dev/null || true) local out="${3:-/etc/thermograph.env}"
[ -n "$env_name" ] || env_name=$(cat "$marker" 2>/dev/null || true)
# The per-host file is required; common.yaml is optional so the initial cutover can # The per-host file is required; common.yaml is optional so the initial cutover can
# seed each host as an exact copy of its live env (byte-identical render, no value # seed each host as an exact copy of its live env (byte-identical render, no value
@ -31,7 +40,7 @@ render_thermograph_secrets() {
# (a) Not configured for SOPS at all — no env marker, or no age key. The LAN # (a) Not configured for SOPS at all — no env marker, or no age key. The LAN
# dev box is legitimately in this state. Saying so and succeeding is right. # dev box is legitimately in this state. Saying so and succeeding is right.
if [ -z "$env_name" ] || [ ! -f "$key" ]; then if [ -z "$env_name" ] || [ ! -f "$key" ]; then
echo "==> SOPS secrets not configured here; using existing /etc/thermograph.env" echo "==> SOPS secrets not configured here; using existing $out"
return 0 return 0
fi fi
@ -57,7 +66,7 @@ render_thermograph_secrets() {
return 1 return 1
fi fi
echo "==> Rendering /etc/thermograph.env from deploy/secrets (common + ${env_name})" echo "==> Rendering $out from deploy/secrets (common + ${env_name})"
# The age private key is root-owned (0400). Read it directly if we can, else via # The age private key is root-owned (0400). Read it directly if we can, else via
# sudo into SOPS_AGE_KEY — so the key never has to be readable by the deploy user. # sudo into SOPS_AGE_KEY — so the key never has to be readable by the deploy user.
# (The render needs sudo to write /etc/thermograph.env below anyway.) # (The render needs sudo to write /etc/thermograph.env below anyway.)
@ -119,12 +128,12 @@ render_thermograph_secrets() {
# failed in-place `cat` write to status 0. Capture the status instead, so a # failed in-place `cat` write to status 0. Capture the status instead, so a
# half-written /etc/thermograph.env fails loudly rather than deploying stale. # half-written /etc/thermograph.env fails loudly rather than deploying stale.
local rc=0 local rc=0
if [ -f /etc/thermograph.env ] && [ -w /etc/thermograph.env ]; then if [ -f "$out" ] && [ -w "$out" ]; then
cat "$tmp" > /etc/thermograph.env || rc=1 cat "$tmp" > "$out" || rc=1
elif install -m 0640 "$tmp" /etc/thermograph.env 2>/dev/null; then : elif install -m 0640 "$tmp" "$out" 2>/dev/null; then :
elif sudo install -m 0640 -o "$(id -un)" -g "$(id -gn)" "$tmp" /etc/thermograph.env 2>/dev/null; then : elif sudo install -m 0640 -o "$(id -un)" -g "$(id -gn)" "$tmp" "$out" 2>/dev/null; then :
else else
echo "!! cannot write /etc/thermograph.env (need file write access or passwordless sudo)" >&2 echo "!! cannot write $out (need file write access or passwordless sudo)" >&2
rc=1 rc=1
fi fi

142
infra/deploy/secrets/README.md
View file

@ -16,8 +16,8 @@ no per-host duplication.
|------|-------|------------| |------|-------|------------|
| `common.yaml` | The 16 values identical on **prod and beta** — VAPID keypair, metrics token, IndexNow key, `REGISTRY_TOKEN`, S3 endpoint/bucket and both S3 keypairs, plus shared non-secret config. **Not layered under `dev`** — see below | ✅ | | `common.yaml` | The 16 values identical on **prod and beta** — VAPID keypair, metrics token, IndexNow key, `REGISTRY_TOKEN`, S3 endpoint/bucket and both S3 keypairs, plus shared non-secret config. **Not layered under `dev`** — see below | ✅ |
| `prod.yaml` | Prod's own — the three held-back credentials below, sizing (`APP_CPUS`, `DB_CPUS`, `DB_MEMORY`, `WORKERS`), `THERMOGRAPH_BASE_URL`, and the Discord + mail credentials that exist nowhere else | ✅ | | `prod.yaml` | Prod's own — the three held-back credentials below, sizing (`APP_CPUS`, `DB_CPUS`, `DB_MEMORY`, `WORKERS`), `THERMOGRAPH_BASE_URL`, and the Discord + mail credentials that exist nowhere else | ✅ |
| `beta.yaml` | Beta's own — the three held-back credentials, sizing, `THERMOGRAPH_BASE_URL` | ✅ | | `beta.yaml` | Beta's own — the three held-back credentials, `THERMOGRAPH_BASE_URL`, `APP_CPUS`/`WORKERS`, plus `THERMOGRAPH_MAIL_BACKEND=console` and `THERMOGRAPH_DISCORD_BOT=0`. Its `POSTGRES_PASSWORD` and `THERMOGRAPH_DATABASE_URL` name beta's **own role and database** (`thermograph_beta`) on the instance it shares with prod. `DB_CPUS`/`DB_MEMORY` were **removed**: beta no longer runs a database of its own, and leaving them would have implied it sized one — the shared instance is sized by `prod.yaml`'s values (see `deploy/stack/thermograph-stack.yml`'s `db` service) | ✅ |
| `dev.yaml` | The LAN dev box's own — **self-contained**, 12 values, no production credential among them | ✅ | | `dev.yaml` | Dev's own (vps1) — **self-contained**, 12 values, no production credential among them | ✅ |
| `centralis.prod.yaml` | **Centralis's** nine variables, rendered to `/etc/centralis.env` on prod. A different service, a different output file, a different renderer — see below | ✅ | | `centralis.prod.yaml` | **Centralis's** nine variables, rendered to `/etc/centralis.env` on prod. A different service, a different output file, a different renderer — see below | ✅ |
| `example.yaml` | Format reference / CI fixture (fake values) | ✅ | | `example.yaml` | Format reference / CI fixture (fake values) | ✅ |
| `../../.sops.yaml` | Which age recipient files are encrypted to (plaintext config) | — | | `../../.sops.yaml` | Which age recipient files are encrypted to (plaintext config) | — |
@ -25,28 +25,55 @@ no per-host duplication.
### Three credentials are deliberately NOT in `common.yaml` ### Three credentials are deliberately NOT in `common.yaml`
`POSTGRES_PASSWORD`, `THERMOGRAPH_AUTH_SECRET` and `THERMOGRAPH_DATABASE_URL` `POSTGRES_PASSWORD`, `THERMOGRAPH_AUTH_SECRET` and `THERMOGRAPH_DATABASE_URL`
hold **identical values on prod and beta today**, so by the mechanical rule they hold **different values on prod and beta today** (beta has its own database
belong in `common.yaml`. They are kept per-host anyway. role/password on the shared instance, and its own auth secret), so unlike the
rest of `common.yaml` they were never candidates for sharing on values alone.
They stay per-environment for a reason that matters more now than it used to:
These are the credentials that let one environment act as another: with them, a **These are the credentials that let one environment act as another.** With
foothold on beta is a foothold on prod's database and prod's session signing. matching values, a foothold in one environment's rendered env file is a
Beta is the *more* exposed box — it serves public Forgejo and Grafana. They match foothold on the other's database and the other's session signing. That used
only because beta was seeded from prod, not because the two are meant to be one to matter because beta and prod were separate boxes; it matters for a
system. different reason now that they're co-resident on vps2 — keeping the
credentials themselves separate is what stops co-location on one host from
also becoming equivalence at the credential level. `deploy/db/provision-env-db.sh`
enforces the same boundary from the database side: beta's role
(`thermograph_beta`) is `NOSUPERUSER`/`NOCREATEDB`/`NOCREATEROLE`, owns only
its own database, and `CONNECT` is revoked from `PUBLIC` on it — so even a
leaked beta credential cannot reach prod's data, and the reverse.
Keeping them per-host costs nothing now (same values, one extra line each) and **What actually separates beta and prod now, stated plainly:** they share a
buys the ability to diverge: rotating prod's database password stops implying host by design (see `deploy/env-topology.sh`'s header), so a host-level
"and beta's too". Moving them into `common.yaml` would encode the equivalence as compromise of vps2 is shared between them — separate SSH credentials no
intentional and turn breaking it into a migration rather than an edit. longer buy a host boundary between beta and prod, the way they once did when
beta and prod were different machines. What remains is isolation at the
**database** level (separate roles, separate databases, `CONNECT` revoked from
`PUBLIC`) and the **file** level (separate checkouts, separate rendered env
files, separate stack env files, separate loopback ports). Keeping
`POSTGRES_PASSWORD`/`THERMOGRAPH_AUTH_SECRET`/`THERMOGRAPH_DATABASE_URL` out of
a shared file is part of that file-level isolation: nothing on vps2 would ever
hand you prod's auth secret merely because you have beta's.
If you ever *do* want them to differ, change one file. That is the whole point. **vps1 is the box that must never hold prod credentials.** It runs Forgejo,
Forgejo's CI runner, Grafana, and the `dev` environment — which runs whatever
branch is currently in flight, reviewed or not. That is the actual reason
`dev` renders `dev.yaml` **alone** and never layers `common.yaml` (see below):
not squeamishness about a dev box, but that vps1 is uniquely positioned to
leak whatever it's handed, and `common.yaml` is the fleet's shared production
credential set.
`dev.yaml` is the case that already differs: all three of its copies are its own, Keeping the three held-back credentials per-environment costs nothing (one
generated on the desktop. The rule earned its keep the first time it was used. line each) and buys the ability for them to diverge further, or for a future
environment to be added without inheriting anyone else's values by accident.
`dev.yaml` is the case that already differs completely: all three of its
values are its own, generated on vps1 and never shared with prod or beta.
At deploy the renderer concatenates `common.yaml` then `<env>.yaml`, so a **host At deploy the renderer concatenates `common.yaml` then `<env>.yaml`, so a **host
value wins** (env_file / `source` take the last occurrence of a duplicate key). value wins** (env_file / `source` take the last occurrence of a duplicate key).
`<env>` comes from `/etc/thermograph/secrets-env` on the box (`prod`/`beta`/`dev`). `<env>` comes from an explicit `THERMOGRAPH_ENV`/argument on vps2 (which runs
two environments) or `/etc/thermograph/secrets-env` as a fallback elsewhere —
see `deploy/env-topology.sh`.
### `dev` is vaulted, but renders `dev.yaml` **alone** ### `dev` is vaulted, but renders `dev.yaml` **alone**
@ -65,17 +92,17 @@ is a **plaintext concatenation**, so layering it would put every one of those in
`/etc/thermograph.env` on the dev box and, through `env_file:`, into the `/etc/thermograph.env` on the dev box and, through `env_file:`, into the
environment of every container in the dev stack. environment of every container in the dev stack.
The dev box is the operator's desktop. It is also the Forgejo CI runner, whose The dev box is **vps1** — the same box that runs Forgejo and its CI. Its
`docker`-labelled jobs get the host docker socket automounted, and the stack it `docker`-labelled runner jobs get the host docker socket automounted, and the
runs is the `dev` branch — code that has not been through the gate that guards stack it runs is the `dev` branch — code that has not been through the gate
prod. Handing that the estate's read-write object-storage keys and the push that guards prod. Handing that the estate's read-write object-storage keys and
signing key is a strictly larger blast radius than anything the vault buys back. the push signing key is a strictly larger blast radius than anything the vault
An override in `dev.yaml` would not help: last-wins governs *consumers*, but the buys back. An override in `dev.yaml` would not help: last-wins governs
production value is still physically a line in the file. *consumers*, but the production value is still physically a line in the file.
One value in `common.yaml` is also simply **wrong** for dev: One value in `common.yaml` is also simply **wrong** for dev:
`THERMOGRAPH_COOKIE_SECURE=1` is right for the TLS hosts and silently breaks login `THERMOGRAPH_COOKIE_SECURE=1` is right for the TLS hosts and silently breaks login
over dev's plain-HTTP LAN URL. `dev.yaml` sets `0`. over dev's plain-HTTP mesh-only URL. `dev.yaml` sets `0`.
So `dev.yaml` is self-contained. Where the value is shared-but-harmless So `dev.yaml` is self-contained. Where the value is shared-but-harmless
(`PORT`, `THERMOGRAPH_BASE`, `TIMESCALEDB_TAG`) it carries its own copy — a (`PORT`, `THERMOGRAPH_BASE`, `TIMESCALEDB_TAG`) it carries its own copy — a
@ -87,8 +114,8 @@ duplicated constant is the cheap half of this trade.
|-----|-----| |-----|-----|
| `POSTGRES_PASSWORD`, `THERMOGRAPH_DATABASE_URL` | dev's own. Deliberately the weak, well-known `thermograph-dev` — the value dev's `pgdata` volume is already initialized with, and the DB is never published off the compose network. Kept per-host for the reason below, and *not* shared with prod | | `POSTGRES_PASSWORD`, `THERMOGRAPH_DATABASE_URL` | dev's own. Deliberately the weak, well-known `thermograph-dev` — the value dev's `pgdata` volume is already initialized with, and the DB is never published off the compose network. Kept per-host for the reason below, and *not* shared with prod |
| `THERMOGRAPH_AUTH_SECRET` | dev's **own**, freshly generated. Fixes a live crash loop: the `daemon` service refuses to start without it (it derives the `/internal/*` token from it), so dev's daemon had been restarting every 60s. Prod's value must never be here — it signs sessions and verification links | | `THERMOGRAPH_AUTH_SECRET` | dev's **own**, freshly generated. Fixes a live crash loop: the `daemon` service refuses to start without it (it derives the `/internal/*` token from it), so dev's daemon had been restarting every 60s. Prod's value must never be here — it signs sessions and verification links |
| `THERMOGRAPH_BASE_URL` | `http://10.0.1.216:8137`. Unset, the app defaults to `https://thermograph.org`, so dev's IndexNow pings and Discord links claimed to be prod | | `THERMOGRAPH_BASE_URL` | `http://10.10.0.2:8137` (vps1's mesh address). Unset, the app defaults to `https://thermograph.org`, so dev's IndexNow pings and Discord links claimed to be prod |
| `THERMOGRAPH_COOKIE_SECURE=0` | plain HTTP on the LAN | | `THERMOGRAPH_COOKIE_SECURE=0` | plain HTTP on the mesh-only URL |
| `THERMOGRAPH_MAIL_BACKEND=console`, `THERMOGRAPH_DISCORD_BOT=0` | explicit fail-safes. Both are already the code defaults; stating them means dev cannot start mailing or open a Discord gateway by inheriting a value | | `THERMOGRAPH_MAIL_BACKEND=console`, `THERMOGRAPH_DISCORD_BOT=0` | explicit fail-safes. Both are already the code defaults; stating them means dev cannot start mailing or open a Discord gateway by inheriting a value |
| `PORT`, `THERMOGRAPH_BASE`, `TIMESCALEDB_TAG`, `DB_MEMORY`, `WORKERS` | non-secret config that would otherwise have come from `common.yaml` | | `PORT`, `THERMOGRAPH_BASE`, `TIMESCALEDB_TAG`, `DB_MEMORY`, `WORKERS` | non-secret config that would otherwise have come from `common.yaml` |
@ -104,8 +131,8 @@ duplicated constant is the cheap half of this trade.
(`backend/indexnow.py`). Prod's key is what authenticates URL submissions *for (`backend/indexnow.py`). Prod's key is what authenticates URL submissions *for
thermograph.org*. thermograph.org*.
- **`THERMOGRAPH_METRICS_TOKEN`.** With no token `/api/v2/metrics` is - **`THERMOGRAPH_METRICS_TOKEN`.** With no token `/api/v2/metrics` is
direct-loopback-only, which is exactly right on a LAN box; dev's Alloy agent direct-loopback-only, which is exactly right on a mesh-only box; dev's Alloy
ships logs, not metrics, so nothing scrapes it. agent ships logs, not metrics, so nothing scrapes it.
- **`REGISTRY_TOKEN`.** dev pulls with the persistent `docker login` credential - **`REGISTRY_TOKEN`.** dev pulls with the persistent `docker login` credential
already in the host's docker config, like the prod/beta CI paths. A vault copy already in the host's docker config, like the prod/beta CI paths. A vault copy
would only duplicate a credential the box already has into a more readable file. would only duplicate a credential the box already has into a more readable file.
@ -224,12 +251,17 @@ files), so layering would only push the VAPID private key, both S3 keypairs and
- `/etc/thermograph/age.key` (`0400`) on each VPS that renders at deploy time. - `/etc/thermograph/age.key` (`0400`) on each VPS that renders at deploy time.
- **Back it up** in the password manager. The public recipient is in `.sops.yaml`. - **Back it up** in the password manager. The public recipient is in `.sops.yaml`.
On the **dev desktop** those two are the same file, on purpose. `render-secrets.sh` On **vps1** (dev) those two can end up being the same file. `render-secrets.sh`
falls back to `sudo cat` for a root-owned key, the desktop has no passwordless sudo, falls back to `sudo cat` for a root-owned key, and `deploy-dev.sh` still assumes
and the Forgejo runner is a `systemd --user` service with no tty — a `/etc` copy the account driving a CI-triggered dev deploy has no passwordless sudo and no
would make every CI-triggered dev deploy hang on a prompt it cannot answer. tty to answer a `sudo` prompt (a `systemd --user` Forgejo-runner service) —
`deploy-dev.sh` points `THERMOGRAPH_AGE_KEY` at the operator's keyring instead, so inherited from the old desktop setup, where that was true of the whole box.
the box keeps **one** copy of the recovery root rather than two. Whether it's still true of vps1's CI-runner account specifically (as opposed
to the `agent` login, which does have passwordless sudo there per `ACCESS.md`)
wasn't re-verified for this pass; `deploy-dev.sh` points `THERMOGRAPH_AGE_KEY`
at the operator's keyring as a fallback either way, so the box keeps **one**
copy of the recovery root rather than two regardless of which account ends up
mattering.
## Prerequisites (once per machine) ## Prerequisites (once per machine)
@ -246,8 +278,10 @@ go install github.com/getsops/sops/v3/cmd/sops@latest # if you have Go
```sh ```sh
sops deploy/secrets/common.yaml # opens DECRYPTED in $EDITOR; re-encrypts on save sops deploy/secrets/common.yaml # opens DECRYPTED in $EDITOR; re-encrypts on save
git commit -am "rotate <thing>" && git push forgejo git commit -am "rotate <thing>" && git push forgejo
# beta auto-deploys on push to main. prod (no CI) — one command: # beta and prod both auto-deploy on push (main -> beta, release -> prod) via
ssh agent@169.58.46.181 'cd /opt/thermograph && git pull && deploy/deploy.sh' # the Deploy workflow. To force it by hand on vps2, say which environment:
ssh agent@169.58.46.181 'cd /opt/thermograph && git pull && THERMOGRAPH_ENV=prod deploy/deploy.sh'
ssh agent@169.58.46.181 'cd /opt/thermograph-beta && git pull && THERMOGRAPH_ENV=beta deploy/deploy.sh'
``` ```
A host-specific value (e.g. `POSTGRES_PASSWORD`) is the same, editing `prod.yaml` / A host-specific value (e.g. `POSTGRES_PASSWORD`) is the same, editing `prod.yaml` /
@ -264,10 +298,13 @@ commit, and the next `deploy-dev.sh` run (merge to `dev`, or by hand) renders it
## Setting up a new dev machine ## Setting up a new dev machine
The desktop is hands-on: Centralis cannot SSH to it, so this is typed, not Dev is provisioned like any fleet host now (`deploy/provision-dev.sh`, run as
automated. `provision-secrets.sh` is **not** the right tool here — it installs the root on vps1), not the sudo-free desktop bootstrap this predates. The
age key root-owned at `/etc/thermograph/age.key`, which the runner cannot read secrets-specific steps below are still worth doing by hand rather than
(above). folding into that script: `provision-secrets.sh` is **not** the right tool
here — it installs the age key root-owned at `/etc/thermograph/age.key`,
which the CI-runner-driven render may not be able to read (see the age-key
note above).
```sh ```sh
# 1. sops + age on PATH, and the age private key in the operator's keyring: # 1. sops + age on PATH, and the age private key in the operator's keyring:
@ -293,17 +330,17 @@ SOPS_AGE_KEY_FILE=~/.config/sops/age/keys.txt \
# 5. Deploy. deploy-dev.sh refuses to run if render-secrets.sh in the checkout # 5. Deploy. deploy-dev.sh refuses to run if render-secrets.sh in the checkout
# cannot honour THERMOGRAPH_SECRETS_SKIP_COMMON, rather than leak. # cannot honour THERMOGRAPH_SECRETS_SKIP_COMMON, rather than leak.
APP_DIR=~/thermograph-dev bash deploy/deploy-dev.sh APP_DIR=/opt/thermograph-dev bash deploy/deploy-dev.sh
``` ```
To take a dev box **back** off the vault, delete `/etc/thermograph/secrets-env`: To take a dev box **back** off the vault, delete `/etc/thermograph/secrets-env`:
presence detection turns rendering off and `deploy-dev.sh` falls back to its presence detection turns rendering off and `deploy-dev.sh` falls back to its
built-in `POSTGRES_PASSWORD` default. built-in `POSTGRES_PASSWORD` default.
A *different* dev machine (a second desktop, a laptop) needs its own values, not a A *different* dev machine (a laptop running `make dev-up` locally, or a future
copy of this one's: give it its own `<env>.yaml` and its own marker name, so the two second dev host) needs its own values, not a copy of vps1's: give it its own
can never be confused for one host — the same argument as the three held-back `<env>.yaml` and its own marker name, so the two can never be confused for one
credentials above. host — the same argument as the three held-back credentials above.
## Add a new secret ## Add a new secret
@ -332,17 +369,24 @@ git commit -am "rotate age identity" && push + deploy
## First-time bootstrap (seed from the live boxes) ## First-time bootstrap (seed from the live boxes)
**Historical** — this ran once, before the vps1/vps2 split, when beta was its
own box at `75.119.132.91`. That address is **vps1** now (dev + Forgejo +
Grafana), not beta; re-running the beta line below as written would seed
`beta.yaml` from the wrong host. Kept for the shape of the process, which is
unchanged if a new environment is ever bootstrapped the same way — just point
`seed-from-live.sh` at wherever that environment actually lives today.
Order matters — values must match the live env exactly so `AUTH_SECRET` / VAPID / Order matters — values must match the live env exactly so `AUTH_SECRET` / VAPID /
`POSTGRES_PASSWORD` don't rotate unintentionally: `POSTGRES_PASSWORD` don't rotate unintentionally:
1. Install `sops`+`age` and drop `/etc/thermograph/age.key` (0400) + 1. Install `sops`+`age` and drop `/etc/thermograph/age.key` (0400) +
`/etc/thermograph/secrets-env` on prod & beta (`provision-secrets.sh`). `/etc/thermograph/secrets-env` on the target hosts (`provision-secrets.sh`).
2. Seed each host's file from its live env with the helper (pulls over SSH, encrypts, 2. Seed each host's file from its live env with the helper (pulls over SSH, encrypts,
and verifies the render round-trips — run it on your own machine, it reads live and verifies the render round-trips — run it on your own machine, it reads live
secrets): secrets), e.g. as originally run:
```sh ```sh
deploy/secrets/seed-from-live.sh prod agent@169.58.46.181 deploy/secrets/seed-from-live.sh prod agent@169.58.46.181
deploy/secrets/seed-from-live.sh beta agent@75.119.132.91 deploy/secrets/seed-from-live.sh beta agent@75.119.132.91 # beta's address AT THE TIME; now vps1's
``` ```
That writes exact per-host copies (`prod.yaml`/`beta.yaml`); factor shared values That writes exact per-host copies (`prod.yaml`/`beta.yaml`); factor shared values
into `common.yaml` later. Commit. into `common.yaml` later. Commit.

30
infra/deploy/secrets/beta.yaml
View file

@ -1,23 +1,23 @@
APP_CPUS: ENC[AES256_GCM,data:Wg==,iv:9kB0WruqLCIkJp7i+t5kGB4qv8NzHwgTQlFmYwhTEXc=,tag:q6cKDI86YrFJjGRnEkdweA==,type:str] APP_CPUS: ENC[AES256_GCM,data:qg==,iv:T3eyF+9ssUCWDiFr8Wg0tRpgeV3BSLIjOSBE2i7WOlc=,tag:wisw0gQ6E8QCNom/Zd3mWQ==,type:str]
DB_CPUS: ENC[AES256_GCM,data:7w==,iv:xDOeZo3dJ5gnXIEGyOBGZm4RUTNyzinMkvxWoY/EbQs=,tag:ikmAjzeM/U4w5HDZ+C7mgA==,type:str] POSTGRES_PASSWORD: ENC[AES256_GCM,data:r5ygtwgycLjjGXSehpLt/RpUkhLyZ4lHPt6KiqNd+ng=,iv:qgqfU1mtm7ZuqSbeUEB9cgT56bOAkc0UFb3BMxED+xg=,tag:WiEoVHrA96WNFROqmFyElA==,type:str]
DB_MEMORY: ENC[AES256_GCM,data:apg=,iv:4+GHbccVBFUGtrP12a2oEya7Hz0KUhJAFzdwpqQJ2sc=,tag:X0h2nYza80c9PjeNgqX7Uw==,type:str] THERMOGRAPH_AUTH_SECRET: ENC[AES256_GCM,data:aZySO3pAVB24S7uCsblRtd65U+OkBatK2tmuzeALdlPsqRhMqRioXQLgMQ==,iv:5zxNwj39E/AOJXzV/AEX9w1n3LSnl9PnS3cp5wVRi2E=,tag:jQRGr2436E4qqBE53iltqQ==,type:str]
POSTGRES_PASSWORD: ENC[AES256_GCM,data:X5HmOvHVtm2mjeLZO7FaEWP0Qq767D0quycr2iI3/Mw=,iv:BQsjJaDPW9RxnMldE1rCU3yP5WBOXvc9VjnTim3kMZs=,tag:yfdi+T7lBbBK544bSvEZtQ==,type:str] THERMOGRAPH_BASE_URL: ENC[AES256_GCM,data:9IxftZqwSBG/1MY+jcALQeeTbIk+tJ/8GXGUgQ==,iv:EvsDUExWQHpidriMDDj0flM8Rz25iFycVPJLrK240+s=,tag:7rKr5KUDMe4J7U4OHUvPRg==,type:str]
THERMOGRAPH_AUTH_SECRET: ENC[AES256_GCM,data:YjgknnUhAU0gW6YuMCmlZYfVhOgye52MLhVF7nKMuc0ToGn0OTqqoiXN0A==,iv:Q9XCoKa5Y/7V9b72tktGZ12jgPumC4kJKjrtCM39ihw=,tag:3kEpMjLCEFQcpSir3N9f6w==,type:str] THERMOGRAPH_DATABASE_URL: ENC[AES256_GCM,data:SnD/achZHduuWwwrQYOseqw0mquV1B4k4c2eom81dqtib6SxGj1OabHYj0rJd+o9Y5sH+kTh+zkFHpBi0J7c5GEGzhBqLctt530uMlOstvu1Hh7qL+DareyBF+SHwG4=,iv:qPFUnfRpQ6gIfBx6phyl5RWsNgEFYdNqW8oVzDZgXU8=,tag:djaqFRpDetNoH7Ux1PmQ3Q==,type:str]
THERMOGRAPH_BASE_URL: ENC[AES256_GCM,data:bus0/ibfLohpsszHZKKoGe5P0/Su5i36Hk2nzg==,iv:QI3crdbipse62xdakCGxPIA4wmdq/T4KzZSvi/NE1Xg=,tag:/S4Bzk2aDoA6ahDlW7qG8w==,type:str] WORKERS: ENC[AES256_GCM,data:gQ==,iv:jkwIzTpiEtEtbt7acP8DWRbl51baTFuQiw4oJxb3eyA=,tag:X2Gqy3icYRKrI6F99lnuyQ==,type:str]
THERMOGRAPH_DATABASE_URL: ENC[AES256_GCM,data:xMd8m6VajAjtwQy8DDqTs4VbyASG8ua8FUZHfPmmEzZEb4cTtYILJ+z6LXhcyvdenDsPg2/GJx5BApvVAKSrWrnSzNWnp+pAs7x+VnNzZ2giZ9meiQ==,iv:MOU+SB/taEc/ExRc07ChF1ksOQiaDj7TPMQq+aV4Ltg=,tag:XP2h9KBGJE7JzripO7yjcg==,type:str] THERMOGRAPH_MAIL_BACKEND: ENC[AES256_GCM,data:OrchUc9tpw==,iv:SmAZ+USGesaCyaOdKopqKmPeG7y/WtDry9G3aDVgP/Q=,tag:EGh5foV6uClrIJCdkjP3Hg==,type:str]
WORKERS: ENC[AES256_GCM,data:BQ==,iv:XYgQ+Sn9OnbUR8LVW+4r9vhnQQJZmliVomfyRmUqMsw=,tag:JzM3JsNR08z1pePSTl/dhw==,type:str] THERMOGRAPH_DISCORD_BOT: ENC[AES256_GCM,data:Yw==,iv:jsHid5lQ75shgfIJkrsLnlhmcJ1aJWuxkUyhmpg52Dc=,tag:UqtZjQpYwQH5uvEYC9mp7w==,type:str]
sops: sops:
age: age:
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBFWHVZU2xSS1pmcy91S2w5 YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBqdFZ3MWVCSDdvdnVLWlZH
MWNhQW90TDdQdDdJV1BHM2Y0a1d6aFJOS2hnCjRodHZlY0orODdpS3QwTVdTdXM5 My9JMzF2TmhiSlJjbWRYbUIzSm0zb1dtdTI4CmI4N2o0YzR1NGNoTmI5OXBJM1Bj
Z3RJOFhyR3AzbjljaExkQjN0YUJ2ckUKLS0tIGpua1Z2TDJrRzE3WFZlMWNpL1Ja OVF3WWEvYmlWczR2dUJpYTROR2pvK2sKLS0tIDM5blFFdnZwRHpDOEZ6aHhyL0dn
VG45eVM0MldLejRNd0pweWNna3JYWlUKWuNU+6PqKlbr7F0ckrNxsMF2OyXh1fMu d1BDWVV6YXNQNWo3dWRqN2FVUC85UmMK8jjGvQxKDsnlr7i95Ar509nihDgRb/JW
cLFBIQg/7vO7O7PJ0VIy0Ugfq6gj2Gv91qKJUGeOXOw3tv1Y6HHbjA== qzLM5fwMU16HOPMiAPd/zH3rUyKcZxMtITR6QLcuSNLByURg4ZMBnQ==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age1xx4dzs0dxlwvkv9sjuqzsphl7lfrxannkfken374yu2qvvcte9sqzktqt2 recipient: age1xx4dzs0dxlwvkv9sjuqzsphl7lfrxannkfken374yu2qvvcte9sqzktqt2
lastmodified: "2026-07-25T00:38:24Z" lastmodified: "2026-07-25T21:43:52Z"
mac: ENC[AES256_GCM,data:lTpEmA8gweS+QlAakIziXxAEiHBUsNDk5lGwVG+uaXNhifIYWbu+QWkklP4wES9o3pu4TNCzFboRWkBbnzhW8HYRglkSChsmmFXDChz67IYlaveCFwczxffp39iypnV/EKaBCa8m1eDJ5kugzr2s7+blagCH0TwpNXBqZShyGNE=,iv:KBF6CreB4oTJGwNetTj27Grox30X3s3qEEYb1cJXmIE=,tag:YfRb5yAoY8K/TXspoZYdcA==,type:str] mac: ENC[AES256_GCM,data:/+kmyMP20n7fQRO2FdiHWaLl8MAfnMjtF8OE5HEUCLpRGet3s+IqYb8DajFIF9SCibW7aEqb6ir9m0KBx0IRW4TTsvK6JYFyRZKKtt4oiCzRaGwUMwhI75DE6xYdHxEbdAC6Q+qFUKGtRcIGhSX1LrlGEloSJggdEFSgfTlJLNg=,iv:UX1zMFw70ODiPolKSjEj8adKxa49Kf5dzfDA7ePtxmE=,tag:dA0OqIPTvXBlMPNH4oCNtg==,type:str]
unencrypted_suffix: _unencrypted unencrypted_suffix: _unencrypted
version: 3.13.2 version: 3.13.2

38
infra/deploy/secrets/dev.yaml
View file

@ -1,27 +1,27 @@
PORT: ENC[AES256_GCM,data:0LjaWw==,iv:3e8Q71QFlMAhQStgoiVh/c2ELI3+7UkWseAFv8BnElU=,tag:/wYfKmqErJpsLBjg1gs4Tw==,type:str] PORT: ENC[AES256_GCM,data:VCheDw==,iv:u9IC7psxzcsLuuWhU89fUJfVsmCI7lECbjxzKmC/nGM=,tag:ZTG/kG2vSNvUUlItAjsCHg==,type:str]
THERMOGRAPH_BASE: ENC[AES256_GCM,data:Ww==,iv:e2yNIAAw4dtlahfbA+PCM/jyODKFwWFNfVpVJ1JoREc=,tag:QzoGwDgIwdA9OFAMIPej7Q==,type:str] THERMOGRAPH_BASE: ENC[AES256_GCM,data:Aw==,iv:Pcmbm6D126jncQnmBquZ/t6qMaaUF7UDnQV6q85CsrY=,tag:SGakFybX3TNKAWvFo9gsEw==,type:str]
TIMESCALEDB_TAG: ENC[AES256_GCM,data:7Oj7OoxoaG51whg=,iv:PPbfdn4DHMxTPA5FUqv+lEY29wQsVX+g0aVuWvraYrU=,tag:NAyJZlRXU1iKo4jwBctUtw==,type:str] TIMESCALEDB_TAG: ENC[AES256_GCM,data:ghewNqRFuYJfBgE=,iv:q1GsQM3QZBW4A95U9NY90RR9WRmPV6miIxGLgLmPTps=,tag:eJARZNyqnJbHhtXXUI1AWw==,type:str]
DB_MEMORY: ENC[AES256_GCM,data:/3U=,iv:jFI/9NQaz8oIqBWAKydWUEEI7bI3SmLnsoBlVgfKw7I=,tag:oBoFANuTgZV0jXC9qR5IHQ==,type:str] DB_MEMORY: ENC[AES256_GCM,data:k2U=,iv:ZRCjRUHbNvVrCdGWqmOoJBtsJLeLVSVm32szpQ23HN4=,tag:KE5jUadHZnKFPis0friNbg==,type:str]
WORKERS: ENC[AES256_GCM,data:VQ==,iv:+d84ByEU22XVQi43NcRn8cHmbW9XudJMj0guidsHxBQ=,tag:XM//7y0VQCwini4CGL7pSg==,type:str] WORKERS: ENC[AES256_GCM,data:yQ==,iv:6VE9mjS7QLuOWazReqyt1DhAhamkvWSm92B2d/ZCZYo=,tag:hDl1/E11xwUyPVhFNhzOuQ==,type:str]
THERMOGRAPH_BASE_URL: ENC[AES256_GCM,data:FK2kOaZG8ESwDi3VMOlo0XvBKvdADQ==,iv:c6GvCrAnAf+ZFh53xg3ZOlf934Ls+PyR/+CXRi933lE=,tag:cYLXvxP5hk7/6gmz/CCP0A==,type:str] THERMOGRAPH_BASE_URL: ENC[AES256_GCM,data:/sOgmM0eCf7dWldK0iZaDWE0W4aN,iv:dpiLeHLanO2zwtMmPMbsqR4aTVaXfT+V2Ux9K8Alh3w=,tag:PDd/L2nt4X750xCKjHnSPw==,type:str]
THERMOGRAPH_COOKIE_SECURE: ENC[AES256_GCM,data:zg==,iv:sVr33CvZoZmn0LcHlP+FVA8esjGy0aKe1A2iPrrrT0I=,tag:3YneNF7m05fCRGfMgYzByA==,type:str] THERMOGRAPH_COOKIE_SECURE: ENC[AES256_GCM,data:Bw==,iv:dsLcLu126A3d7dTqBgQY1SOvJfKOBT/ye4ucIB4w6B4=,tag:/G7Qvlzb6KKVVjqmn3d/Nw==,type:str]
THERMOGRAPH_MAIL_BACKEND: ENC[AES256_GCM,data:pRBtOu6cTw==,iv:88QgyY6rwwp/6Y32r6Us1M+b7YpSLPfKIsCL2j/YuIY=,tag:FrchZZrMcCSeOFRWbePM4Q==,type:str] THERMOGRAPH_MAIL_BACKEND: ENC[AES256_GCM,data:ryWhCkSdxg==,iv:Lpl1CIjTINV8rF8BW5MHZD4Ji802i96zcyqF3xRMLcY=,tag:I0nbb6IP6LM/Y+KaHjBDpg==,type:str]
THERMOGRAPH_DISCORD_BOT: ENC[AES256_GCM,data:NQ==,iv:Zl888EtEoQQd8FdqGb4/N7gGMT4yEymQfWd2fZZCNVA=,tag:Xspr8/taeQwh1qHhfFbLfw==,type:str] THERMOGRAPH_DISCORD_BOT: ENC[AES256_GCM,data:Rw==,iv:o5tr5lQGXFkB6kTJ/dDARBWECDYRdofFlZ7v/HdMe5k=,tag:Lv97fH6u/nIKZlJr5HjbZw==,type:str]
POSTGRES_PASSWORD: ENC[AES256_GCM,data:l076hUutOAKw7+X6wboX,iv:9WVz3SVUikWREVjB6GsJ+EXWO3PvX+g+OaZNGN9f+jU=,tag:OkVuEqEbTSikpCNEySPX5A==,type:str] POSTGRES_PASSWORD: ENC[AES256_GCM,data:IBVel17GKtrW5yzqDjBi,iv:yEPSSYR/Ng9rjSvSl0veNRDPLayF0SvWZ1T8BSY1l50=,tag:hbB4Uo6YluQPgl9XyvgfKw==,type:str]
THERMOGRAPH_DATABASE_URL: ENC[AES256_GCM,data:kXCYmZXewa8JSY5iJd0cHDtsKrxyKEl5tSVwWeQEd8u7XU+/dn4BExT4RDB3vUWVFVoHXYIXedMnDO8h0QfDCxSI13s=,iv:s1bDjdH4E4T+kF126saOrKlx3AnKEkRWF68KEPcCUe0=,tag:nCzD9U/7g28bJmUXkiYh3Q==,type:str] THERMOGRAPH_DATABASE_URL: ENC[AES256_GCM,data:KjsSo+IglfY5MhtEsX3HJpExAtBwzmSu3E+O7Y22B2WuIrTKGDxG81WwvQGABYfKVel6kzzv8uKczq1qEtOwHRdMsRI=,iv:VPJNe1GrJa7BeAA/N/fqyBjBF+ciF8btE4j/TwrV3cA=,tag:tkyzkR1LxY8z28bjb0bJ7w==,type:str]
THERMOGRAPH_AUTH_SECRET: ENC[AES256_GCM,data:EjwKGKh1od5vgvm8LLv1l6TXO7nshUwh5ihXDpETl+3ldtF8L4H3CWszsw==,iv:4mqwEXWpGpLYbIUotHj9Qf7FSD0ATwkxwhfuL7pG+Zc=,tag:0iLU0ee86H0CgmgPCCtatQ==,type:str] THERMOGRAPH_AUTH_SECRET: ENC[AES256_GCM,data:ccEQxcNxMEn8/Gf2knQaVMSoBp7BRJ7zWhr9KmawaSMrwHGU7kADDWCoMw==,iv:nDpMHhP0ubjzz2LIY0QMGFLGP1M10zOIJb6pPgri0PI=,tag:+Z2j5XLdV+LuSK7hWf5r0Q==,type:str]
sops: sops:
age: age:
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBzV0lrcUpjeU81TmJDRVpa YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAyb2dCeVgrbGxGSnNpaUE3
elppeU1mMUlpc3lKSTVtU3MxUi83bzN2dVVRCmxZbDJFUkJQVjhCSmljNFpPaWVu RlovN0RrejVmREY0ckNGaExpNWkvSjcyRW1RCjYvc1IrMWwvdmJGU2g5bU52Mmgw
dkNqTkJTd1ZEaHZsZitpTWlJdElCK28KLS0tIEdSTXN6S1Y2eE12U3o2RGxBNXNN SVY1dnFHd0cyNkM1ZWhJaWlIZldzZlEKLS0tIGtLZDRSMS82UjRFRVdZU3J1ckMz
bDBlYkNxMDVTVEFLYVIyVkhIRnNwOGMKv/TMt307XqvzBLCOCA5C4kXFV9iJeVBn T0g1WnRUUGFFRTVyMlF6WnZiSEoweEEKPLwcTNiE4s24J0kComnNJj5jXIBRBzMh
7gyzb5MRbHbDoNAY/5ckU3341uWUXUQ+IrPvVt2snAdXIlWghm+HwA== PVktsQm9m1ojaG0hgCZsFaMOOnE5O0Vi41jgG7WWLSNZRaVggvRbJg==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age1xx4dzs0dxlwvkv9sjuqzsphl7lfrxannkfken374yu2qvvcte9sqzktqt2 recipient: age1xx4dzs0dxlwvkv9sjuqzsphl7lfrxannkfken374yu2qvvcte9sqzktqt2
lastmodified: "2026-07-25T00:59:09Z" lastmodified: "2026-07-25T21:44:03Z"
mac: ENC[AES256_GCM,data:Ga1lMooQrBhU+YasAbN3THZvjrct21KRbuxOoXf+72K5mrgPmqZyFDdB71B8dOUzmz4cqTPw3+jQx2s2ZrZCEiR3qmb897I8hWbTiFWnURSC7rjytxIe6kLhDBjCtso8ThKUMqNjAglqjkytHXljP7Cg4zCuGLEgczoakEJVB7s=,iv:NFHljXdG619XJMjy+1OuUknu+QaG5nH68fvo/zAf/TE=,tag:MyFf/edujKzypOwloVUGAA==,type:str] mac: ENC[AES256_GCM,data:ok66XdbCyLyU/2UxQ6Fo63S16r6VETT7Qk/KVJ2b4rrdd4hfOMMAvmUfFmBNkxclxBp+UMOlGJ3XpGRZRzcmWR9o1eXpFlussgDbWHU+KLoIVqAVkUsltVM9aDkg+3dl6/vInftWGw24LbfDh1WhYQ9+OwyKxZntj0sNHAShc8E=,iv:eBTeVtkAv6qcldT0x+zyO2yWVrCm0nhRUJjzaAfxLBM=,tag:BtgIWfkLSoZDcbJML9uO7g==,type:str]
unencrypted_suffix: _unencrypted unencrypted_suffix: _unencrypted
version: 3.13.2 version: 3.13.2

22
infra/deploy/secrets/seed-encrypt-on-host.sh
View file

@ -1,5 +1,5 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# Seed deploy/secrets/<env>.yaml by encrypting a box's live /etc/thermograph.env # Seed deploy/secrets/<env>.yaml by encrypting an environment's live env file
# ENTIRELY ON THE BOX. The plaintext never leaves the box; only the already-encrypted # ENTIRELY ON THE BOX. The plaintext never leaves the box; only the already-encrypted
# ciphertext is written back here. Encryption uses the age PUBLIC key (safe to hardcode) # ciphertext is written back here. Encryption uses the age PUBLIC key (safe to hardcode)
# so the machine you run this from needs nothing but SSH access + the agent key — no # so the machine you run this from needs nothing but SSH access + the agent key — no
@ -7,7 +7,13 @@
# #
# Run from any checkout of this branch that can SSH to the target with the agent key: # Run from any checkout of this branch that can SSH to the target with the agent key:
# deploy/secrets/seed-encrypt-on-host.sh prod agent@169.58.46.181 # deploy/secrets/seed-encrypt-on-host.sh prod agent@169.58.46.181
# deploy/secrets/seed-encrypt-on-host.sh beta agent@75.119.132.91 # deploy/secrets/seed-encrypt-on-host.sh beta agent@169.58.46.181
# deploy/secrets/seed-encrypt-on-host.sh dev agent@75.119.132.91
#
# prod and beta share a box (vps2); they differ only by which env FILE is read
# (/etc/thermograph.env vs /etc/thermograph-beta.env), resolved from
# deploy/env-topology.sh. Hardcoding that path meant "seed beta" would encrypt
# PROD's live secrets into beta.yaml.
# Then commit + push the resulting deploy/secrets/<env>.yaml. Verify faithfulness with # Then commit + push the resulting deploy/secrets/<env>.yaml. Verify faithfulness with
# the key-gaps skill after. See README.md. # the key-gaps skill after. See README.md.
set -euo pipefail set -euo pipefail
@ -17,14 +23,18 @@ ENV_NAME="${1:?usage: seed-encrypt-on-host.sh <env> <ssh-target> [ssh-key]}"
SSH_TARGET="${2:?ssh target, e.g. agent@169.58.46.181}" SSH_TARGET="${2:?ssh target, e.g. agent@169.58.46.181}"
SSH_KEY="${3:-$HOME/.ssh/thermograph_agent_ed25519}" SSH_KEY="${3:-$HOME/.ssh/thermograph_agent_ed25519}"
OUT="deploy/secrets/${ENV_NAME}.yaml" OUT="deploy/secrets/${ENV_NAME}.yaml"
# shellcheck source=infra/deploy/env-topology.sh
. deploy/env-topology.sh
thermograph_topology "$ENV_NAME"
REMOTE_ENV_FILE="$TG_ENV_FILE"
# Public age recipient — NOT a secret (matches .sops.yaml). The private key never # Public age recipient — NOT a secret (matches .sops.yaml). The private key never
# leaves the operator's machine / the hosts' /etc/thermograph/age.key. # leaves the operator's machine / the hosts' /etc/thermograph/age.key.
AGE_PUB="age1xx4dzs0dxlwvkv9sjuqzsphl7lfrxannkfken374yu2qvvcte9sqzktqt2" AGE_PUB="age1xx4dzs0dxlwvkv9sjuqzsphl7lfrxannkfken374yu2qvvcte9sqzktqt2"
SOPS_URL="https://github.com/getsops/sops/releases/download/v3.13.2/sops-v3.13.2.linux.amd64" SOPS_URL="https://github.com/getsops/sops/releases/download/v3.13.2/sops-v3.13.2.linux.amd64"
echo "==> Encrypting ${SSH_TARGET}:/etc/thermograph.env on-host -> ${OUT}" echo "==> Encrypting ${SSH_TARGET}:${REMOTE_ENV_FILE} on-host -> ${OUT}"
ssh -i "$SSH_KEY" "$SSH_TARGET" "AGE_PUB='$AGE_PUB' SOPS_URL='$SOPS_URL' bash -s" > "$OUT" <<'REMOTE' ssh -i "$SSH_KEY" "$SSH_TARGET" "AGE_PUB='$AGE_PUB' SOPS_URL='$SOPS_URL' ENV_FILE='$REMOTE_ENV_FILE' bash -s" > "$OUT" <<'REMOTE'
set -euo pipefail set -euo pipefail
exec 3>&1 # real stdout carries ONLY the ciphertext exec 3>&1 # real stdout carries ONLY the ciphertext
{ # setup noise -> stderr, so it can't corrupt the file { # setup noise -> stderr, so it can't corrupt the file
@ -34,7 +44,9 @@ exec 3>&1 # real stdout carries ONLY the ciphertext
fi fi
} >&2 } >&2
tmp="$(mktemp)"; tmy="$(mktemp)"; trap 'rm -f "$tmp" "$tmy"' EXIT tmp="$(mktemp)"; tmy="$(mktemp)"; trap 'rm -f "$tmp" "$tmy"' EXIT
sudo cat /etc/thermograph.env > "$tmp" # plaintext stays on this box only # $ENV_FILE comes from the ssh command line above (env-topology.sh resolved it
# for the named environment) — NOT hardcoded, because vps2 holds two of them.
sudo cat "$ENV_FILE" > "$tmp" # plaintext stays on this box only
python3 -c ' python3 -c '
import json, sys import json, sys
for line in open(sys.argv[1], encoding="utf-8"): for line in open(sys.argv[1], encoding="utf-8"):

22
infra/deploy/secrets/seed-from-live.sh
View file

@ -1,11 +1,19 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# Seed (or re-seed) an encrypted secrets file from a box's live /etc/thermograph.env. # Seed (or re-seed) an encrypted secrets file from an environment's live env file.
# Run on the operator's machine — it reads production secrets, so it is deliberately # Run on the operator's machine — it reads production secrets, so it is deliberately
# NOT something the agent runs for you. Requires: sops + age on PATH, the age private # NOT something the agent runs for you. Requires: sops + age on PATH, the age private
# key at ~/.config/sops/age/keys.txt (or SOPS_AGE_KEY_FILE), and SSH access to the box. # key at ~/.config/sops/age/keys.txt (or SOPS_AGE_KEY_FILE), and SSH access to the box.
# #
# deploy/secrets/seed-from-live.sh prod agent@169.58.46.181 ~/.ssh/thermograph_agent_ed25519 # deploy/secrets/seed-from-live.sh prod agent@169.58.46.181 ~/.ssh/thermograph_agent_ed25519
# deploy/secrets/seed-from-live.sh beta agent@75.119.132.91 ~/.ssh/thermograph_agent_ed25519 # deploy/secrets/seed-from-live.sh beta agent@169.58.46.181 ~/.ssh/thermograph_agent_ed25519
# deploy/secrets/seed-from-live.sh dev agent@75.119.132.91 ~/.ssh/thermograph_agent_ed25519
#
# Note prod and beta share a box (vps2) and therefore differ only by which env
# FILE is read: /etc/thermograph.env vs /etc/thermograph-beta.env. That path is
# resolved from deploy/env-topology.sh rather than hardcoded — hardcoding it
# meant "seed beta" would read PROD's live secrets and write them into
# beta.yaml, which is how one environment silently ends up holding another's
# credentials.
# #
# Writes deploy/secrets/<env>.yaml ENCRYPTED (exact copy of the live env, so the # Writes deploy/secrets/<env>.yaml ENCRYPTED (exact copy of the live env, so the
# deploy render is faithful), then verifies the render round-trips to the same # deploy render is faithful), then verifies the render round-trips to the same
@ -21,13 +29,17 @@ ENV_NAME="${1:?usage: seed-from-live.sh <env> <ssh-target> [ssh-key]}"
SSH_TARGET="${2:?ssh target, e.g. agent@169.58.46.181}" SSH_TARGET="${2:?ssh target, e.g. agent@169.58.46.181}"
SSH_KEY="${3:-$HOME/.ssh/thermograph_agent_ed25519}" SSH_KEY="${3:-$HOME/.ssh/thermograph_agent_ed25519}"
OUT="deploy/secrets/${ENV_NAME}.yaml" OUT="deploy/secrets/${ENV_NAME}.yaml"
# shellcheck source=infra/deploy/env-topology.sh
. deploy/env-topology.sh
thermograph_topology "$ENV_NAME"
REMOTE_ENV_FILE="$TG_ENV_FILE"
command -v sops >/dev/null || { echo "!! sops not on PATH" >&2; exit 1; } command -v sops >/dev/null || { echo "!! sops not on PATH" >&2; exit 1; }
tmp="$(mktemp)"; trap 'rm -f "$tmp" "$tmp.env"' EXIT tmp="$(mktemp)"; trap 'rm -f "$tmp" "$tmp.env"' EXIT
umask 077 umask 077
echo "==> Pulling ${SSH_TARGET}:/etc/thermograph.env (sudo)" echo "==> Pulling ${SSH_TARGET}:${REMOTE_ENV_FILE} (sudo)"
ssh -i "$SSH_KEY" "$SSH_TARGET" 'sudo cat /etc/thermograph.env' > "$tmp" ssh -i "$SSH_KEY" "$SSH_TARGET" "sudo cat $REMOTE_ENV_FILE" > "$tmp"
n=$(grep -cE '^[A-Za-z_][A-Za-z0-9_]*=' "$tmp" || true) n=$(grep -cE '^[A-Za-z_][A-Za-z0-9_]*=' "$tmp" || true)
[ "$n" -gt 0 ] || { echo "!! no KEY=VALUE lines read (permission? path?)" >&2; exit 1; } [ "$n" -gt 0 ] || { echo "!! no KEY=VALUE lines read (permission? path?)" >&2; exit 1; }
echo " ${n} vars" echo " ${n} vars"
@ -47,7 +59,7 @@ for line in open(src, encoding="utf-8"):
continue continue
pairs[k] = v pairs[k] = v
with open(dst, "w", encoding="utf-8") as fh: with open(dst, "w", encoding="utf-8") as fh:
fh.write(f"# SOPS-encrypted — {env} host secrets, seeded from the live /etc/thermograph.env.\n") fh.write(f"# SOPS-encrypted — {env} host secrets, seeded from that environment’s live env file.\n")
for k, v in pairs.items(): for k, v in pairs.items():
fh.write(f"{k}: {json.dumps(v)}\n") # JSON-escaped scalar is valid YAML fh.write(f"{k}: {json.dumps(v)}\n") # JSON-escaped scalar is valid YAML
print(f" {len(pairs)} keys") print(f" {len(pairs)} keys")

146
infra/deploy/stack/deploy-stack.sh
View file

@ -18,8 +18,20 @@
# rehearsal on the same host that cannot touch live data or ports. # rehearsal on the same host that cannot touch live data or ports.
set -euo pipefail set -euo pipefail
APP_DIR="${APP_DIR:-/opt/thermograph}"
SERVICE="${SERVICE:-all}" SERVICE="${SERVICE:-all}"
# Two stacks now run on vps2 (prod and beta), so nothing here may assume "the"
# stack: the name, the stack file, the loopback ports, the env file, the
# database role and the service-name prefix all come from env-topology.sh,
# keyed by the environment deploy.sh already resolved and exported.
SELF_DIR=$(cd "$(dirname "$0")" && pwd)
# shellcheck source=infra/deploy/env-topology.sh
. "$SELF_DIR/../env-topology.sh"
ENV_NAME=$(thermograph_env_name)
[ -n "$ENV_NAME" ] || ENV_NAME=prod
thermograph_topology "$ENV_NAME"
APP_DIR="${APP_DIR:-$TG_APP_DIR}"
cd "$APP_DIR" cd "$APP_DIR"
case "$SERVICE" in case "$SERVICE" in
@ -27,10 +39,26 @@ case "$SERVICE" in
*) echo "!! SERVICE must be backend|frontend|all, got '$SERVICE'" >&2; exit 2 ;; *) echo "!! SERVICE must be backend|frontend|all, got '$SERVICE'" >&2; exit 2 ;;
esac esac
# Which stack-file services this environment's roll targets. Prod's are bare
# (web, worker, ...); beta's carry a `beta-` prefix so its short DNS names never
# collide with prod's on the shared overlay — see env-topology.sh.
SVC_WEB="${TG_SVC_PREFIX}web"
SVC_WORKER="${TG_SVC_PREFIX}worker"
SVC_FRONTEND="${TG_SVC_PREFIX}frontend"
SVC_LAKE="${TG_SVC_PREFIX}lake"
SVC_DAEMON="${TG_SVC_PREFIX}daemon"
STACK_FILE="$APP_DIR/infra/$TG_STACK_FILE"
ENV_FILE="$TG_ENV_FILE"
STACK_ENV_FILE="$TG_STACK_ENV_FILE"
if [ "${STACK_TEST:-0}" = "1" ]; then if [ "${STACK_TEST:-0}" = "1" ]; then
# Test mode rehearses PROD's stack only; it is not a second-environment path.
STACK_NAME="thermograph-test" STACK_NAME="thermograph-test"
LB_NAME="thermograph-test-lb" LB_NAME="thermograph-test-lb"
LB_HTTP_PORT=18137; LB_FE_PORT=18080 LB_HTTP_PORT=18137; LB_FE_PORT=18080
DATA_NETWORK="thermograph-test_internal"
DB_SERVICE="thermograph-test_db"
export PGDATA_VOLUME="thermograph-test_pgdata" export PGDATA_VOLUME="thermograph-test_pgdata"
export APPDATA_VOLUME="thermograph-test_appdata" export APPDATA_VOLUME="thermograph-test_appdata"
export APPLOGS_VOLUME="thermograph-test_applogs" export APPLOGS_VOLUME="thermograph-test_applogs"
@ -38,29 +66,42 @@ if [ "${STACK_TEST:-0}" = "1" ]; then
docker volume create "$APPDATA_VOLUME" >/dev/null docker volume create "$APPDATA_VOLUME" >/dev/null
docker volume create "$APPLOGS_VOLUME" >/dev/null docker volume create "$APPLOGS_VOLUME" >/dev/null
else else
STACK_NAME="${STACK_NAME:-thermograph}" STACK_NAME="${STACK_NAME:-$TG_STACK_NAME}"
LB_NAME="thermograph-lb" LB_NAME="$TG_LB_NAME"
LB_HTTP_PORT=8137; LB_FE_PORT=8080 LB_HTTP_PORT="$TG_LB_HTTP_PORT"; LB_FE_PORT="$TG_LB_FE_PORT"
# Where the database is. Prod owns it inside its own overlay; beta reaches
# the same service across that overlay, which its stack declares external.
DATA_NETWORK="$TG_DATA_NETWORK"
DB_SERVICE="$TG_DB_SERVICE"
fi fi
export STACK_NAME export STACK_NAME
echo "==> Environment: $ENV_NAME (stack $STACK_NAME on $TG_HOST, checkout $APP_DIR)"
# --- secrets ------------------------------------------------------------------ # --- secrets ------------------------------------------------------------------
# Render (SOPS) + source /etc/thermograph.env exactly like deploy.sh, then # Render (SOPS) + source /etc/thermograph.env exactly like deploy.sh, then
# install the uid-10001-readable copy the tasks' env-entrypoint shim sources. # install the uid-10001-readable copy the tasks' env-entrypoint shim sources.
# 10001 = the app images' `thermograph` user; the file is 0400 to that uid. # 10001 = the app images' `thermograph` user; the file is 0400 to that uid.
#
# Per-environment paths throughout: on vps2 prod uses /etc/thermograph.env +
# /etc/thermograph/stack.env while beta uses /etc/thermograph-beta.env +
# /etc/thermograph/beta-stack.env. Sharing either file would give beta prod's
# database credentials and vice versa.
if [ -f "$APP_DIR/infra/deploy/render-secrets.sh" ]; then if [ -f "$APP_DIR/infra/deploy/render-secrets.sh" ]; then
# shellcheck source=infra/deploy/render-secrets.sh # shellcheck source=infra/deploy/render-secrets.sh
. "$APP_DIR/infra/deploy/render-secrets.sh" . "$APP_DIR/infra/deploy/render-secrets.sh"
render_thermograph_secrets "$APP_DIR/infra" render_thermograph_secrets "$APP_DIR/infra" "$ENV_NAME" "$ENV_FILE"
fi fi
# /etc/thermograph.env is rendered at deploy time from the SOPS vault — it # $ENV_FILE is rendered at deploy time from the SOPS vault — it cannot exist at
# cannot exist at lint time, so don't ask shellcheck to follow it. # lint time, so don't ask shellcheck to follow it.
set -a set -a
# shellcheck source=/dev/null # shellcheck source=/dev/null
. /etc/thermograph.env 2>/dev/null || true . "$ENV_FILE" 2>/dev/null || true
set +a set +a
sudo install -o 10001 -g 0 -m 0400 /etc/thermograph.env /etc/thermograph/stack.env \ sudo install -o 10001 -g 0 -m 0400 "$ENV_FILE" "$STACK_ENV_FILE" \
|| install -o 10001 -g 0 -m 0400 /etc/thermograph.env /etc/thermograph/stack.env || install -o 10001 -g 0 -m 0400 "$ENV_FILE" "$STACK_ENV_FILE"
# The stack file bind-mounts this path into every task; export it so the
# `volumes:` interpolation resolves to the right environment's copy.
export STACK_ENV_FILE
# --- image tags ----------------------------------------------------------------- # --- image tags -----------------------------------------------------------------
# Same persisted-tags contract as deploy.sh: incoming env wins, the file # Same persisted-tags contract as deploy.sh: incoming env wins, the file
@ -90,7 +131,14 @@ FRONTEND_IMAGE="$REGISTRY_HOST/${FRONTEND_IMAGE_PATH:-emi/thermograph/frontend}:
# Hazard #7: the db image under an existing volume must never drift. Resolve # Hazard #7: the db image under an existing volume must never drift. Resolve
# the digest-pinned ref from whatever is running (stack task or compose # the digest-pinned ref from whatever is running (stack task or compose
# container), falling back to the local latest-pg18's digest on first bring-up. # container), falling back to the local latest-pg18's digest on first bring-up.
if [ -z "${TIMESCALEDB_IMAGE:-}" ]; then #
# Only the environment that OWNS the database needs this — i.e. the one whose
# own stack declares the db service. Beta's stack declares none (it uses prod's
# instance), so resolving an image pin there answers a question beta's stack
# file never asks.
OWNS_DB=0
[ "$DB_SERVICE" = "${STACK_NAME}_db" ] && OWNS_DB=1
if [ -z "${TIMESCALEDB_IMAGE:-}" ] && [ "$OWNS_DB" = 1 ]; then
cid=$(docker ps -q --filter "label=com.docker.swarm.service.name=${STACK_NAME}_db" | head -1) cid=$(docker ps -q --filter "label=com.docker.swarm.service.name=${STACK_NAME}_db" | head -1)
[ -z "$cid" ] && cid=$(docker ps -q --filter "name=thermograph-db-1" | head -1) [ -z "$cid" ] && cid=$(docker ps -q --filter "name=thermograph-db-1" | head -1)
if [ -n "$cid" ]; then if [ -n "$cid" ]; then
@ -102,7 +150,11 @@ if [ -z "${TIMESCALEDB_IMAGE:-}" ]; then
[ -n "$TIMESCALEDB_IMAGE" ] || TIMESCALEDB_IMAGE="$img" [ -n "$TIMESCALEDB_IMAGE" ] || TIMESCALEDB_IMAGE="$img"
fi fi
export TIMESCALEDB_IMAGE export TIMESCALEDB_IMAGE
echo "==> Images: web/worker=$BACKEND_IMAGE frontend=$FRONTEND_IMAGE db=$TIMESCALEDB_IMAGE" if [ "$OWNS_DB" = 1 ]; then
echo "==> Images: web/worker=$BACKEND_IMAGE frontend=$FRONTEND_IMAGE db=$TIMESCALEDB_IMAGE"
else
echo "==> Images: web/worker=$BACKEND_IMAGE frontend=$FRONTEND_IMAGE (db: shared $DB_SERVICE)"
fi
# --- registry ------------------------------------------------------------------ # --- registry ------------------------------------------------------------------
if [ -n "${REGISTRY_TOKEN:-}" ]; then if [ -n "${REGISTRY_TOKEN:-}" ]; then
@ -124,15 +176,22 @@ done
# in the stack). Runs on the stack's overlay so `db` resolves. First-ever # in the stack). Runs on the stack's overlay so `db` resolves. First-ever
# deploy: the network doesn't exist yet — create it exactly as the stack will # deploy: the network doesn't exist yet — create it exactly as the stack will
# (attachable overlay) so the name is adopted, then migrate, then deploy. # (attachable overlay) so the name is adopted, then migrate, then deploy.
NET="${STACK_NAME}_internal" #
# Never pre-create $NET: docker stack deploy must own it (a pre-existing # The migrate task joins the network the DATABASE is on, which for beta is
# unlabeled network makes it fail with "already exists"). On first deploy the # prod's overlay rather than beta's own — and it connects as this environment's
# migrate runs AFTER stack deploy instead (FIRST_DEPLOY_MIGRATE below). # own role to this environment's own database (beta: thermograph_beta on both
# counts), so a beta migration can never touch prod's schema.
NET="$DATA_NETWORK"
MIGRATE_URL="postgresql+asyncpg://${TG_DB_USER}:${POSTGRES_PASSWORD}@db:5432/${TG_DB_NAME}"
# Never pre-create $NET when this stack owns it: docker stack deploy must own it
# (a pre-existing unlabeled network makes it fail with "already exists"). On
# first deploy the migrate runs AFTER stack deploy instead (FIRST_DEPLOY_MIGRATE
# below). Beta's data network is prod's and always already exists.
if [ "$SERVICE" = "backend" ] || [ "$SERVICE" = "all" ]; then if [ "$SERVICE" = "backend" ] || [ "$SERVICE" = "all" ]; then
if docker service inspect "${STACK_NAME}_db" >/dev/null 2>&1; then if docker service inspect "$DB_SERVICE" >/dev/null 2>&1; then
echo "==> One-shot migrate ($BACKEND_IMAGE)" echo "==> One-shot migrate ($BACKEND_IMAGE -> ${TG_DB_NAME})"
docker run --rm --network "$NET" \ docker run --rm --network "$NET" \
-e THERMOGRAPH_DATABASE_URL="postgresql+asyncpg://thermograph:${POSTGRES_PASSWORD}@db:5432/thermograph" \ -e THERMOGRAPH_DATABASE_URL="$MIGRATE_URL" \
--entrypoint /app/deploy/entrypoint.sh "$BACKEND_IMAGE" migrate --entrypoint /app/deploy/entrypoint.sh "$BACKEND_IMAGE" migrate
else else
echo "==> First deploy: db not up yet; replicas will be rolled after stack deploy runs migrate below" echo "==> First deploy: db not up yet; replicas will be rolled after stack deploy runs migrate below"
@ -141,38 +200,38 @@ fi
# --- deploy ---------------------------------------------------------------------- # --- deploy ----------------------------------------------------------------------
FIRST_DEPLOY_MIGRATE=0 FIRST_DEPLOY_MIGRATE=0
docker service inspect "${STACK_NAME}_db" >/dev/null 2>&1 || FIRST_DEPLOY_MIGRATE=1 docker service inspect "$DB_SERVICE" >/dev/null 2>&1 || FIRST_DEPLOY_MIGRATE=1
if [ "$SERVICE" = "all" ] || ! docker service inspect "${STACK_NAME}_web" >/dev/null 2>&1; then if [ "$SERVICE" = "all" ] || ! docker service inspect "${STACK_NAME}_${SVC_WEB}" >/dev/null 2>&1; then
echo "==> docker stack deploy ($STACK_NAME)" echo "==> docker stack deploy ($STACK_NAME)"
docker stack deploy --with-registry-auth -c "$APP_DIR/infra/deploy/stack/thermograph-stack.yml" "$STACK_NAME" docker stack deploy --with-registry-auth -c "$STACK_FILE" "$STACK_NAME"
# First-ever deploy ran no migrate above (db didn't exist): wait for db, # First-ever deploy ran no migrate above (db didn't exist): wait for db,
# migrate, then force web/worker to restart cleanly against the schema. # migrate, then force web/worker to restart cleanly against the schema.
if [ "${FIRST_DEPLOY_MIGRATE:-0}" = "1" ]; then if [ "${FIRST_DEPLOY_MIGRATE:-0}" = "1" ]; then
echo "==> Waiting for db, then first-boot migrate" echo "==> Waiting for db, then first-boot migrate"
for i in $(seq 1 60); do for i in $(seq 1 60); do
cid=$(docker ps -q --filter "label=com.docker.swarm.service.name=${STACK_NAME}_db" | head -1) cid=$(docker ps -q --filter "label=com.docker.swarm.service.name=${DB_SERVICE}" | head -1)
[ -n "$cid" ] && docker exec "$cid" pg_isready -U thermograph -d thermograph >/dev/null 2>&1 && break [ -n "$cid" ] && docker exec "$cid" pg_isready -U "$TG_DB_USER" -d "$TG_DB_NAME" >/dev/null 2>&1 && break
sleep 5 sleep 5
done done
docker run --rm --network "$NET" \ docker run --rm --network "$NET" \
-e THERMOGRAPH_DATABASE_URL="postgresql+asyncpg://thermograph:${POSTGRES_PASSWORD}@db:5432/thermograph" \ -e THERMOGRAPH_DATABASE_URL="$MIGRATE_URL" \
--entrypoint /app/deploy/entrypoint.sh "$BACKEND_IMAGE" migrate --entrypoint /app/deploy/entrypoint.sh "$BACKEND_IMAGE" migrate
docker service update --force --detach=false "${STACK_NAME}_web" docker service update --force --detach=false "${STACK_NAME}_${SVC_WEB}"
docker service update --force --detach=false "${STACK_NAME}_worker" docker service update --force --detach=false "${STACK_NAME}_${SVC_WORKER}"
fi fi
else else
case "$SERVICE" in case "$SERVICE" in
backend) backend)
echo "==> Rolling web + worker + lake to $BACKEND_IMAGE" echo "==> Rolling $SVC_WEB + $SVC_WORKER + $SVC_LAKE to $BACKEND_IMAGE"
docker service update --with-registry-auth --detach=false --image "$BACKEND_IMAGE" "${STACK_NAME}_web" docker service update --with-registry-auth --detach=false --image "$BACKEND_IMAGE" "${STACK_NAME}_${SVC_WEB}"
docker service update --with-registry-auth --detach=false --image "$BACKEND_IMAGE" "${STACK_NAME}_worker" docker service update --with-registry-auth --detach=false --image "$BACKEND_IMAGE" "${STACK_NAME}_${SVC_WORKER}"
# lake and daemon ship in the same image; a stack file predating either # lake and daemon ship in the same image; a stack file predating either
# has no service yet — the next SERVICE=all stack deploy creates it, so # has no service yet — the next SERVICE=all stack deploy creates it, so
# don't fail here. The daemon especially must roll with web: they share # don't fail here. The daemon especially must roll with web: they share
# the /internal/* contract, and a version skew between them is exactly # the /internal/* contract, and a version skew between them is exactly
# what pinning one BACKEND_IMAGE_TAG exists to prevent (seen live: the # what pinning one BACKEND_IMAGE_TAG exists to prevent (seen live: the
# first post-creation backend roll left the daemon a release behind). # first post-creation backend roll left the daemon a release behind).
for extra in lake daemon; do for extra in "$SVC_LAKE" "$SVC_DAEMON"; do
if docker service inspect "${STACK_NAME}_${extra}" >/dev/null 2>&1; then if docker service inspect "${STACK_NAME}_${extra}" >/dev/null 2>&1; then
docker service update --with-registry-auth --detach=false --image "$BACKEND_IMAGE" "${STACK_NAME}_${extra}" docker service update --with-registry-auth --detach=false --image "$BACKEND_IMAGE" "${STACK_NAME}_${extra}"
else else
@ -181,8 +240,8 @@ else
done done
;; ;;
frontend) frontend)
echo "==> Rolling frontend to $FRONTEND_IMAGE" echo "==> Rolling $SVC_FRONTEND to $FRONTEND_IMAGE"
docker service update --with-registry-auth --detach=false --image "$FRONTEND_IMAGE" "${STACK_NAME}_frontend" docker service update --with-registry-auth --detach=false --image "$FRONTEND_IMAGE" "${STACK_NAME}_${SVC_FRONTEND}"
;; ;;
esac esac
fi fi
@ -192,13 +251,20 @@ fi
# 0.0.0.0) on the attachable overlay, proxying to the service VIPs. Recreated # 0.0.0.0) on the attachable overlay, proxying to the service VIPs. Recreated
# only when missing/dead — its config rarely changes; `docker rm -f $LB_NAME` # only when missing/dead — its config rarely changes; `docker rm -f $LB_NAME`
# to force a refresh after editing lb/Caddyfile. # to force a refresh after editing lb/Caddyfile.
#
# Note the network: the LB proxies to THIS stack's own service VIPs, so it joins
# this stack's own overlay — which for beta is beta's `internal`, not the shared
# data network its tasks also sit on. And its config is per-environment, because
# beta's services are named beta-web/beta-frontend.
LB_NETWORK="${STACK_NAME}_internal"
LB_CONFIG="$APP_DIR/infra/${TG_LB_CONFIG:-deploy/stack/lb/Caddyfile}"
if ! docker ps --format '{{.Names}}' | grep -qx "$LB_NAME"; then if ! docker ps --format '{{.Names}}' | grep -qx "$LB_NAME"; then
docker rm -f "$LB_NAME" >/dev/null 2>&1 || true docker rm -f "$LB_NAME" >/dev/null 2>&1 || true
echo "==> Starting loopback LB bridge $LB_NAME (127.0.0.1:$LB_HTTP_PORT, :$LB_FE_PORT)" echo "==> Starting loopback LB bridge $LB_NAME (127.0.0.1:$LB_HTTP_PORT, :$LB_FE_PORT)"
docker run -d --name "$LB_NAME" --restart unless-stopped \ docker run -d --name "$LB_NAME" --restart unless-stopped \
--network "$NET" \ --network "$LB_NETWORK" \
-p "127.0.0.1:${LB_HTTP_PORT}:8137" -p "127.0.0.1:${LB_FE_PORT}:8080" \ -p "127.0.0.1:${LB_HTTP_PORT}:8137" -p "127.0.0.1:${LB_FE_PORT}:8080" \
-v "$APP_DIR/infra/deploy/stack/lb/Caddyfile:/etc/caddy/Caddyfile:ro" \ -v "$LB_CONFIG:/etc/caddy/Caddyfile:ro" \
caddy:2-alpine >/dev/null caddy:2-alpine >/dev/null
fi fi
@ -233,8 +299,14 @@ docker images --format '{{.Repository}}:{{.Tag}}' \
# Post-deploy warm + IndexNow, via any web task (skip in test mode: no data, # Post-deploy warm + IndexNow, via any web task (skip in test mode: no data,
# and the warmer would burn upstream quota against an empty cache). # and the warmer would burn upstream quota against an empty cache).
if [ "${STACK_TEST:-0}" != "1" ] && { [ "$SERVICE" = backend ] || [ "$SERVICE" = all ]; }; then #
wcid=$(docker ps -q --filter "label=com.docker.swarm.service.name=${STACK_NAME}_web" | head -1) # TG_POST_DEPLOY gates this per environment: prod does both, beta does neither
# (IndexNow from beta would ask search engines to index beta.thermograph.org,
# and the warm would spend the shared upstream archive quota on a rehearsal
# cache). See env-topology.sh.
if [ "${STACK_TEST:-0}" != "1" ] && [ "${TG_POST_DEPLOY:-1}" = 1 ] \
&& { [ "$SERVICE" = backend ] || [ "$SERVICE" = all ]; }; then
wcid=$(docker ps -q --filter "label=com.docker.swarm.service.name=${STACK_NAME}_${SVC_WEB}" | head -1)
if [ -n "$wcid" ]; then if [ -n "$wcid" ]; then
echo "==> Warming city archives (detached) + IndexNow" echo "==> Warming city archives (detached) + IndexNow"
docker exec -d "$wcid" sh -c 'python warm_cities.py --pace 2 >> /app/logs/warm-cities.log 2>&1' || true docker exec -d "$wcid" sh -c 'python warm_cities.py --pace 2 >> /app/logs/warm-cities.log 2>&1' || true

34
infra/deploy/stack/lb/Caddyfile.beta Normal file
View file

@ -0,0 +1,34 @@
# Beta's loopback LB bridge config — the beta counterpart of ./Caddyfile.
#
# Two differences from prod's, both load-bearing:
#
# 1. It proxies to beta-web / beta-frontend, not web / frontend. Beta's Swarm
# services carry that prefix so their short DNS names cannot collide with
# prod's on the shared overlay (see thermograph-beta-stack.yml).
# 2. deploy-stack.sh publishes this container on 127.0.0.1:8237 and :8180,
# not 8137/8180 — prod's LB already owns 8137/8080 on this host. The
# LISTEN ports below stay 8137/8080: those are inside the container, and
# the host mapping is what differs. Keeping the container ports identical
# to prod's means the two configs differ only in the upstream names.
#
# The host Caddy on vps2 terminates TLS for beta.thermograph.org and proxies to
# 127.0.0.1:8237 / :8180 — see deploy/Caddyfile.
{
auto_https off
admin off
}
:8137 {
reverse_proxy beta-web:8137 {
# Fail fast to the client if the VIP has no healthy task; Swarm's own
# task healthchecks handle ejecting dead replicas from the VIP.
lb_try_duration 5s
}
}
:8080 {
reverse_proxy beta-frontend:8080 {
lb_try_duration 5s
}
}

278
infra/deploy/stack/thermograph-beta-stack.yml Normal file
View file

@ -0,0 +1,278 @@
# Docker Swarm stack for BETA, co-resident with prod on vps2.
#
# Deployed by the same deploy/stack/deploy-stack.sh as prod, which picks this
# file (and beta's ports, env file, LB and DB role) out of deploy/env-topology.sh
# when THERMOGRAPH_ENV=beta. Beta moved here from its own box so that a beta
# green light is evidence about prod: same orchestrator, same host kernel, same
# Postgres build, same Caddy, same mail path, same mesh position.
#
# ---------------------------------------------------------------------------
# WHY THIS IS A SEPARATE FILE AND NOT AN OVERLAY ON thermograph-stack.yml
# ---------------------------------------------------------------------------
# `docker stack deploy` accepts multiple -c files and MERGES them. Merging can
# add and override, but it cannot REMOVE a service — and the single most
# important fact about beta is a removal: it has no `db`. It uses prod's. An
# overlay would therefore still create a second Postgres, which is the exact
# thing this design exists to avoid. The same goes for the two autoscalers,
# which beta deliberately does not run.
#
# The cost is a file that must be kept roughly in step with prod's by hand.
# Keep them in step for anything that affects whether the APP works (env vars,
# entrypoints, healthchecks, the migrate contract). Do NOT keep them in step on
# scale, replicas or resource limits — those differ on purpose (below).
#
# ---------------------------------------------------------------------------
# THE THREE THINGS THAT MAKE CO-RESIDENCY SAFE
# ---------------------------------------------------------------------------
# 1. SERVICE NAMES ARE PREFIXED (beta-web, not web). Swarm registers a service's
# short name as a DNS alias on every network it joins. Beta's tasks share the
# `data` network with prod's, so two services both called `web` would make
# `web` ambiguous — prod's frontend could resolve a beta task, and vice
# versa. The prefix removes the collision without touching prod's stack file.
#
# 2. THE DATABASE IS SHARED, THE DATA IS NOT. Beta connects to prod's `db`
# service as the role `thermograph_beta`, to the database `thermograph_beta`.
# That role owns only its own database (see deploy/db/provision-env-db.sh),
# so a beta deploy running an unmerged branch — or a migration that goes
# wrong — cannot read or write production data. One server is a capacity
# decision, not a trust decision.
#
# 3. NOTHING ELSE IS SHARED BY ACCIDENT. Separate checkout (/opt/thermograph-beta),
# separate rendered env file (/etc/thermograph-beta.env), separate stack env
# (/etc/thermograph/beta-stack.env), separate volumes, separate loopback
# ports (8237/8180 — prod owns 8137/8080), separate LB container, separate
# deploy lock and image-tag file. Every one of those is derived in
# env-topology.sh rather than repeated here by hand.
#
# ---------------------------------------------------------------------------
# WHAT BETA DELIBERATELY DOES NOT DO
# ---------------------------------------------------------------------------
# - No autoscaling: fixed 1 replica per service. Beta exists to answer "does
# this code work", not "does it scale"; a second replica would only add a
# variable prod's rehearsal doesn't need, on a host prod is also using.
# - No IndexNow ping and no city-archive warm (deploy-stack.sh gates both on
# TG_POST_DEPLOY). Pinging IndexNow from beta asks Bing/DuckDuckGo/Yandex to
# index beta.thermograph.org; the warm spends the shared upstream archive
# quota to fill a cache only a rehearsal reads.
# - No real mail and no Discord gateway. Both are governed by beta's vault
# (THERMOGRAPH_MAIL_BACKEND=console, THERMOGRAPH_DISCORD_BOT=0) rather than
# pinned here, so the operator can opt in with `sops edit` if a release ever
# genuinely needs to rehearse them. Discord in particular allows ONE gateway
# connection per bot token — beta and prod must never both hold one.
services:
beta-web:
image: ${REGISTRY_HOST:-git.thermograph.org}/${BACKEND_IMAGE_PATH:-emi/thermograph/backend}:${BACKEND_IMAGE_TAG:?required}
entrypoint: ["/host/env-entrypoint.sh"]
environment:
# Beta's OWN role and OWN database on the shared instance. `db` resolves
# across the external `data` network to prod's db service.
THERMOGRAPH_DATABASE_URL: postgresql+asyncpg://thermograph_beta:${POSTGRES_PASSWORD}@db:5432/thermograph_beta
THERMOGRAPH_BASE: /
PORT: 8137
THERMOGRAPH_SERVICE_ROLE: backend
THERMOGRAPH_FRONTEND_BASE_INTERNAL: http://beta-frontend:8080
WORKERS: ${BETA_WEB_WORKERS:-2}
THERMOGRAPH_DATA_DIR: /state
# Never the notifier/scheduler — that is beta-worker's job, exactly as in
# prod, so the two files stay honest about which process owns what.
THERMOGRAPH_ROLE: web
# Migrations run as the one-shot task in deploy-stack.sh.
RUN_MIGRATIONS: "0"
# Overlay tasks reach the HOST's Postfix via the docker_gwbridge gateway.
# Same host, same Postfix as prod — but see the mail note in the header:
# beta's vault selects the console backend, so nothing is actually sent.
THERMOGRAPH_SMTP_HOST: ${STACK_SMTP_HOST:-172.18.0.1}
THERMOGRAPH_LAKE_URL: http://beta-lake:8141
volumes:
- appdata:/state
- applogs:/app/logs
- /opt/thermograph-beta/infra/deploy/stack/env-entrypoint.sh:/host/env-entrypoint.sh:ro
- /etc/thermograph/beta-stack.env:/host/thermograph.env:ro
networks:
- internal
- data
deploy:
replicas: 1
# vps2 is the Swarm manager and every volume here is local to it. The
# desktop is a worker on this mesh and must never be scheduled the app.
placement:
constraints: ["node.role == manager"]
resources:
limits:
cpus: "${BETA_WEB_CPUS:-2}"
restart_policy:
condition: on-failure
update_config:
order: start-first
failure_action: rollback
beta-worker:
image: ${REGISTRY_HOST:-git.thermograph.org}/${BACKEND_IMAGE_PATH:-emi/thermograph/backend}:${BACKEND_IMAGE_TAG:?required}
entrypoint: ["/host/env-entrypoint.sh"]
environment:
THERMOGRAPH_DATABASE_URL: postgresql+asyncpg://thermograph_beta:${POSTGRES_PASSWORD}@db:5432/thermograph_beta
THERMOGRAPH_BASE: /
PORT: 8137
THERMOGRAPH_SERVICE_ROLE: backend
THERMOGRAPH_FRONTEND_BASE_INTERNAL: http://beta-frontend:8080
WORKERS: "1"
THERMOGRAPH_DATA_DIR: /state
THERMOGRAPH_ROLE: worker
# The advisory lock is taken in beta's OWN database, so it can never
# contend with prod's worker despite the shared server.
THERMOGRAPH_SINGLETON_PG: "1"
RUN_MIGRATIONS: "0"
THERMOGRAPH_SMTP_HOST: ${STACK_SMTP_HOST:-172.18.0.1}
THERMOGRAPH_LAKE_URL: http://beta-lake:8141
volumes:
- appdata:/state
- applogs:/app/logs
- /opt/thermograph-beta/infra/deploy/stack/env-entrypoint.sh:/host/env-entrypoint.sh:ro
- /etc/thermograph/beta-stack.env:/host/thermograph.env:ro
networks:
- internal
- data
deploy:
replicas: 1
placement:
constraints: ["node.role == manager"]
resources:
limits:
cpus: "${BETA_WORKER_CPUS:-1}"
restart_policy:
condition: on-failure
beta-lake:
image: ${REGISTRY_HOST:-git.thermograph.org}/${BACKEND_IMAGE_PATH:-emi/thermograph/backend}:${BACKEND_IMAGE_TAG:?required}
entrypoint: ["/host/env-entrypoint.sh"]
environment:
THERMOGRAPH_ROLE: lake
PORT: 8141
THERMOGRAPH_SERVICE_ROLE: backend
WORKERS: "1"
THERMOGRAPH_LAKE_CACHE: /state/lake-cache
volumes:
# Beta's own cache volume. Deliberately not prod's: they are read caches
# of the same bucket, but sharing a volume across two stacks would couple
# their lifecycles for no gain.
- lakecache:/state
- /opt/thermograph-beta/infra/deploy/stack/env-entrypoint.sh:/host/env-entrypoint.sh:ro
- /etc/thermograph/beta-stack.env:/host/thermograph.env:ro
# No `data` network: the lake reads object storage, never Postgres.
networks:
- internal
deploy:
replicas: 1
placement:
constraints: ["node.role == manager"]
resources:
limits:
cpus: "${BETA_LAKE_CPUS:-1}"
restart_policy:
condition: on-failure
update_config:
order: start-first
failure_action: rollback
beta-daemon:
image: ${REGISTRY_HOST:-git.thermograph.org}/${BACKEND_IMAGE_PATH:-emi/thermograph/backend}:${BACKEND_IMAGE_TAG:?required}
# Same reasoning as prod's daemon: NOT env-entrypoint.sh, because that shim
# execs the image's own entrypoint (Alembic + uvicorn) and migrations belong
# to the one-shot task. Source the host-rendered env and exec the binary.
#
# Note the ordering consequence, which is load-bearing here: this sources
# the env file AFTER the `environment:` block is applied, so a key present
# in /etc/thermograph/beta-stack.env WINS over one set below. That is why
# THERMOGRAPH_DISCORD_BOT=0 lives in beta's vault and not in this file — a
# value set here would be silently overridden if the vault ever set one.
entrypoint:
- /bin/bash
- -c
- 'set -a; [ -f /host/thermograph.env ] && . /host/thermograph.env; set +a; exec /usr/local/bin/thermograph-daemon'
environment:
THERMOGRAPH_API_BASE_INTERNAL: http://beta-web:8137
volumes:
- /etc/thermograph/beta-stack.env:/host/thermograph.env:ro
# The image HEALTHCHECK curls /healthz on ${PORT}; the daemon serves
# nothing, so without this override Swarm restarts it forever.
healthcheck:
disable: true
networks:
- internal
deploy:
# EXACTLY 1, and in beta's case the Discord gateway is off entirely
# (vault: THERMOGRAPH_DISCORD_BOT=0) because prod's daemon holds the only
# permitted gateway connection for that bot token.
replicas: 1
placement:
constraints: ["node.role == manager"]
resources:
limits:
cpus: "0.5"
memory: 128m
restart_policy:
condition: on-failure
update_config:
order: stop-first
failure_action: rollback
beta-frontend:
image: ${REGISTRY_HOST:-git.thermograph.org}/${FRONTEND_IMAGE_PATH:-emi/thermograph/frontend}:${FRONTEND_IMAGE_TAG:?required}
entrypoint: ["/host/env-entrypoint.sh"]
# REQUIRED: overriding `entrypoint:` with no `command:` drops the image's
# CMD entirely, and env-entrypoint.sh's fallback (`exec uvicorn app:app`)
# does not exist in this Go image — the task would exit 127 every deploy.
command: ["/usr/local/bin/thermograph-frontend"]
environment:
THERMOGRAPH_BASE: /
PORT: 8080
THERMOGRAPH_SERVICE_ROLE: frontend
THERMOGRAPH_API_BASE_INTERNAL: http://beta-web:8137
volumes:
- /opt/thermograph-beta/infra/deploy/stack/env-entrypoint.sh:/host/env-entrypoint.sh:ro
- /etc/thermograph/beta-stack.env:/host/thermograph.env:ro
networks:
- internal
deploy:
replicas: 1
placement:
constraints: ["node.role == manager"]
resources:
limits:
cpus: "${BETA_FRONTEND_CPUS:-1}"
restart_policy:
condition: on-failure
update_config:
order: start-first
failure_action: rollback
networks:
# Beta's own east-west network: beta-web <-> beta-frontend <-> beta-lake, and
# the loopback LB bridge joins it (hence attachable). Keeping this separate
# from prod's overlay means beta's ordinary traffic never touches it.
internal:
driver: overlay
attachable: true
# Prod's overlay, joined ONLY to reach the shared `db` service. Declared
# external because prod's stack owns it: `docker stack deploy` of THIS file
# must never create, modify or (on `docker stack rm thermograph-beta`) remove
# the network prod's database is on.
#
# Consequence worth knowing before you tear anything down: `docker stack rm
# thermograph` would take this network with it and beta would lose its
# database link until prod is redeployed.
data:
external: true
name: thermograph_internal
volumes:
# Beta's own, created by this stack under the thermograph-beta_ prefix. Unlike
# prod's (which are `external` because they were inherited from the compose
# era and hold live data), these can be recreated: beta's appdata is a parquet
# cache plus derived files, and its DATABASE — the part that matters — lives
# on the shared instance, not here.
appdata: {}
applogs: {}
lakecache: {}

71
infra/deploy/swarm/README.md
View file

@ -1,36 +1,41 @@
# 3-node Docker Swarm (prod + beta + desktop), for hosting Forgejo # 3-node Docker Swarm (vps2 + vps1 + desktop)
This Swarm cluster's only job is to run Forgejo (`deploy/forgejo/`) — it does This Swarm mesh's only **Swarm-scheduled** workload is Forgejo
**not** orchestrate the Thermograph app itself, which stays on the (`deploy/forgejo/`), pinned to vps1. It does **not** orchestrate the
Terraform-managed `docker compose` deploys on prod/beta independently (see Thermograph app the same way — prod and beta each run as their own `docker
`terraform/README.md`). Keeping those separate means nothing here can strand stack deploy` (`deploy/stack/thermograph-stack.yml` /
or interfere with the app's already-working, single-writer Postgres/TimescaleDB `thermograph-beta-stack.yml`) that happens to land on this same manager node
deploys. (vps2) because their volumes are local to it today. Keeping Forgejo's stack
and the app stacks conceptually separate means nothing here can strand or
interfere with the app's single-writer Postgres/TimescaleDB.
This is the canonical topology from This is the canonical topology from
`thermograph-docs/runbooks/implementation-handoff.md` (Track B steps 2-3) — three nodes, `thermograph-docs/runbooks/implementation-handoff.md` (Track B steps 2-3) — three nodes,
not two. An earlier revision of this doc/scripts covered just prod+beta; not two. The desktop (formerly the LAN dev machine) joins as a worker for flex
the desktop (this LAN dev machine) joins too. capacity and AI-model hosting; it hosts no Thermograph environment.
**Nodes:** **Nodes:**
- **manager** — prod, the new 48 GB / 12-core box (more headroom). - **manager** — vps2 (`169.58.46.181`), the box with headroom for prod's and
- **worker** — beta, the old VPS (`75.119.132.91`). beta's Swarm stacks and their local volumes.
- **worker** — desktop, this LAN dev machine (also runs the Forgejo Actions - **worker** — vps1 (`75.119.132.91`), pinned to run Forgejo
runner as a plain systemd service — see `deploy/forgejo/README.md` — not as (`node.labels.role == forge`). Also runs Grafana/Loki/Alloy and the dev
a Swarm-scheduled container). environment, both outside this Swarm cluster (plain Docker/compose on the
same host's daemon, not Swarm-scheduled).
- **worker** — desktop (this machine), flex capacity plus AI-model hosting. No
Thermograph environment runs here.
One manager, not more: Raft needs 3 nodes for real quorum-based HA, and this One manager, not more: Raft needs 3 nodes for real quorum-based HA, and this
cluster only has 3 nodes total, so making even one more of them a manager cluster only has 3 nodes total, so making even one more of them a manager
would still fall short of real HA while adding split-brain risk. If the would still fall short of real HA while adding split-brain risk. If the
manager (prod) goes down, the workers keep running whatever was already manager (vps2) goes down, the workers keep running whatever was already
scheduled on them (Forgejo, pinned to beta) but the cluster can't reschedule scheduled on them (Forgejo, pinned to vps1) but the cluster can't reschedule
anything until prod's back — acceptable for a small cluster whose only job is anything until vps2's back — acceptable for a small cluster whose only
CI/CD. Swarm-scheduled job is CI/CD.
## Order of operations ## Order of operations
1. **Agent access first** (`deploy/provision-agent-access.sh`) on prod and 1. **Agent access first** (`deploy/provision-agent-access.sh`) on vps1 and
beta — everything below on those two boxes is run through that access. The vps2 — everything below on those two boxes is run through that access. The
desktop is wherever you're already working from; no separate access step desktop is wherever you're already working from; no separate access step
needed there. needed there.
2. **WireGuard mesh** (`setup-wireguard.sh <my_wg_ip> <peers_file>`) — run on 2. **WireGuard mesh** (`setup-wireguard.sh <my_wg_ip> <peers_file>`) — run on
@ -38,19 +43,21 @@ CI/CD.
the two-pass key-exchange dance (pubkeys aren't known until every node has the two-pass key-exchange dance (pubkeys aren't known until every node has
run it once). Verify with `ping <peer_wg_ip>` to each of the other two run it once). Verify with `ping <peer_wg_ip>` to each of the other two
before continuing. before continuing.
3. **Swarm init** (`init-swarm.sh <manager_wg_ip>`) on the manager (prod) only. 3. **Swarm init** (`init-swarm.sh <manager_wg_ip>`) on the manager (vps2) only.
4. **Swarm join** (`join-swarm.sh <manager_wg_ip> <token>`) on **each** of the 4. **Swarm join** (`join-swarm.sh <manager_wg_ip> <token>`) on **each** of the
two workers (beta, desktop) — same token for both. two workers (vps1, desktop) — same token for both.
5. **Firewall lockdown** (`firewall-swarm.sh`) on **all three** nodes — closes 5. **Firewall lockdown** (`firewall-swarm.sh`) on **all three** nodes — closes
2377/7946/4789 to everything except the WireGuard interface. Do this 2377/7946/4789 to everything except the WireGuard interface. Do this
*after* joining is confirmed working on all three, not before (locking the *after* joining is confirmed working on all three, not before (locking the
ports first would make the join itself fail). ports first would make the join itself fail).
6. **Label beta** (`label-forge-node.sh <beta-node-name>`) on the manager — 6. **Label vps1** (`label-forge-node.sh <vps1-node-name>`) on the manager —
`docker node ls` shows each node's name/ID. Only beta gets `role=forge`; `docker node ls` shows each node's name/ID. Only vps1 gets `role=forge`;
the desktop and prod don't need a Swarm label for anything in this setup. the desktop and vps2 don't need a Swarm label for anything in this setup.
7. Deploy Forgejo: see `deploy/forgejo/README.md`. 7. Deploy Forgejo: see `deploy/forgejo/README.md`.
8. Register the Actions runner **on the desktop** (not through Swarm): 8. Register the Actions runner: `deploy/forgejo/register-lan-runner.sh`. See
`deploy/forgejo/register-lan-runner.sh`. that document (and `DEPLOY-DEV.md`) for where it actually runs today — its
own header comment predates the vps1/vps2 rename and still describes "the
desktop" as the canonical placement.
## Why WireGuard instead of relying on Swarm's built-in TLS alone ## Why WireGuard instead of relying on Swarm's built-in TLS alone
@ -65,19 +72,19 @@ than trusting the public internet (or the desktop's home network) directly.
## Verifying ## Verifying
```bash ```bash
# On the manager: # On the manager (vps2):
docker node ls # all three nodes Ready docker node ls # all three nodes Ready
docker node inspect <beta-node> --format '{{.Spec.Labels}}' # role:forge docker node inspect <vps1-node> --format '{{.Spec.Labels}}' # role:forge
# From a FOURTH machine outside the mesh entirely, confirm the Swarm ports # From a FOURTH machine outside the mesh entirely, confirm the Swarm ports
# are NOT reachable on either VPS's public IP (the desktop has no public IP # are NOT reachable on either VPS's public IP (the desktop has no public IP
# to check this way): # to check this way):
nc -zv -w2 <prod_or_beta_public_ip> 2377 # should fail/timeout nc -zv -w2 <vps1_or_vps2_public_ip> 2377 # should fail/timeout
nc -zvu -w2 <prod_or_beta_public_ip> 4789 # should fail/timeout nc -zvu -w2 <vps1_or_vps2_public_ip> 4789 # should fail/timeout
``` ```
## Adding a node label back out (undo) ## Adding a node label back out (undo)
```bash ```bash
docker node update --label-rm role <beta-node> docker node update --label-rm role <vps1-node>
``` ```

14
infra/deploy/swarm/firewall-swarm.sh
View file

@ -1,8 +1,8 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# Locks the Swarm ports (2377 control, 7946 gossip, 4789 overlay VXLAN) to the # Locks the Swarm ports (2377 control, 7946 gossip, 4789 overlay VXLAN) to the
# WireGuard interface only — they must never be reachable from the public # WireGuard interface only — they must never be reachable from the public
# internet. Run on BOTH boxes after joining the swarm. Existing app-facing # internet. Run on ALL THREE nodes (vps2, vps1, desktop) after joining the
# rules (80/443, SSH, etc.) are untouched. # swarm. Existing app-facing rules (80/443, SSH, etc.) are untouched.
set -euo pipefail set -euo pipefail
WG_IFACE="${WG_IFACE:-wg0}" WG_IFACE="${WG_IFACE:-wg0}"
@ -34,8 +34,10 @@ if ! ufw status | grep -q "^Status: active"; then
echo "ufw is currently INACTIVE on this node — 'ufw enable' switches its" echo "ufw is currently INACTIVE on this node — 'ufw enable' switches its"
echo "default policy to deny-incoming for EVERYTHING, not just the Swarm" echo "default policy to deny-incoming for EVERYTHING, not just the Swarm"
echo "ports above. Before enabling, explicitly allow every port this node" echo "ports above. Before enabling, explicitly allow every port this node"
echo "already serves publicly (SSH at minimum; on beta specifically, also" echo "already serves publicly (SSH at minimum; on vps2 specifically, also"
echo "80/tcp and 443/tcp for the live thermograph.org Caddy) — check" echo "80/tcp and 443/tcp for the live thermograph.org/beta.thermograph.org"
echo "'ss -tlnp' for what's actually listening first. Enabling ufw without" echo "Caddy; on vps1, 80/tcp and 443/tcp for git.thermograph.org,"
echo "doing this WILL drop live traffic the moment it activates." echo "dashboard.thermograph.org and emigriffith.dev) — check 'ss -tlnp' for"
echo "what's actually listening first. Enabling ufw without doing this WILL"
echo "drop live traffic the moment it activates."
fi fi

11
infra/deploy/swarm/init-swarm.sh
View file

@ -1,8 +1,9 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# Run ONCE, on the manager node only (prod — the new 48 GB box). Initializes # Run ONCE, on the manager node only (vps2 — the box with headroom for prod's
# the Swarm advertising the WireGuard address, so cluster traffic never # and beta's Swarm stacks and their local volumes). Initializes the Swarm
# touches the public interface. Run setup-wireguard.sh on all THREE nodes # advertising the WireGuard address, so cluster traffic never touches the
# first (prod, beta, and the desktop — see # public interface. Run setup-wireguard.sh on all THREE nodes first (vps2,
# vps1, and the desktop — see
# docs/runbooks/implementation-handoff.md Track B steps 2-3). # docs/runbooks/implementation-handoff.md Track B steps 2-3).
set -euo pipefail set -euo pipefail
@ -21,7 +22,7 @@ echo "==> docker swarm init, advertising ${MY_WG_IP} (the WireGuard address, not
docker swarm init --advertise-addr "$MY_WG_IP" --listen-addr "${MY_WG_IP}:2377" docker swarm init --advertise-addr "$MY_WG_IP" --listen-addr "${MY_WG_IP}:2377"
echo echo
echo "==> Worker join command (run this on EACH of the two workers — beta and" echo "==> Worker join command (run this on EACH of the two workers — vps1 and"
echo " the desktop; the same token works for both):" echo " the desktop; the same token works for both):"
TOKEN="$(docker swarm join-token -q worker)" TOKEN="$(docker swarm join-token -q worker)"
echo " docker swarm join --token ${TOKEN} ${MY_WG_IP}:2377" echo " docker swarm join --token ${TOKEN} ${MY_WG_IP}:2377"

4
infra/deploy/swarm/join-swarm.sh
View file

@ -1,6 +1,6 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# Run ONCE on EACH worker node (beta, and the desktop — not the manager, # Run ONCE on EACH worker node (vps1, and the desktop — not the manager,
# prod). Joins the Swarm initialized by init-swarm.sh, over the WireGuard # vps2). Joins the Swarm initialized by init-swarm.sh, over the WireGuard
# tunnel. The join token is the same for both workers. # tunnel. The join token is the same for both workers.
set -euo pipefail set -euo pipefail

4
infra/deploy/swarm/label-forge-node.sh
View file

@ -1,6 +1,6 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# Run ONCE, on the manager node, after both boxes have joined the swarm. # Run ONCE, on the manager node (vps2), after both boxes have joined the
# Labels the worker (beta) so the Forgejo stack's placement constraint # swarm. Labels the worker (vps1) so the Forgejo stack's placement constraint
# (node.labels.role == forge) schedules there and nowhere else. # (node.labels.role == forge) schedules there and nowhere else.
set -euo pipefail set -euo pipefail

6
infra/deploy/swarm/setup-wireguard.sh
View file

@ -1,6 +1,6 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# Sets up this node's side of a full-mesh WireGuard tunnel across all of # Sets up this node's side of a full-mesh WireGuard tunnel across all of
# prod, beta, and the desktop (three nodes, not two — see # vps2, vps1, and the desktop (three nodes, not two — see
# docs/runbooks/implementation-handoff.md Track B step 2). Docker Swarm's # docs/runbooks/implementation-handoff.md Track B step 2). Docker Swarm's
# control plane (2377/tcp) is TLS-encrypted by default, but the overlay data # control plane (2377/tcp) is TLS-encrypted by default, but the overlay data
# plane (VXLAN, 4789/udp) is NOT — and it should never face the public # plane (VXLAN, 4789/udp) is NOT — and it should never face the public
@ -14,8 +14,8 @@
# --- Peer list format -------------------------------------------------- # --- Peer list format --------------------------------------------------
# A text file, one line per OTHER node (not including the one you're running # A text file, one line per OTHER node (not including the one you're running
# on), each line: <wg_ip> <public_ip> <pubkey-or-dash> # on), each line: <wg_ip> <public_ip> <pubkey-or-dash>
# 10.10.0.1 169.58.46.181 <prod's pubkey, or - if not yet known> # 10.10.0.1 169.58.46.181 <vps2's pubkey, or - if not yet known>
# 10.10.0.2 75.119.132.91 <beta's pubkey, or -> # 10.10.0.2 75.119.132.91 <vps1's pubkey, or ->
# 10.10.0.3 <desktop's public/reachable IP or a DDNS name> <desktop's pubkey, or -> # 10.10.0.3 <desktop's public/reachable IP or a DDNS name> <desktop's pubkey, or ->
# #
# First pass on any node: peer pubkeys you don't have yet are "-". Run this # First pass on any node: peer pubkeys you don't have yet are "-". Run this

18
infra/deploy/thermograph.env.example
View file

@ -46,8 +46,9 @@ POSTGRES_PASSWORD=change-me
# an empty value is honored as-is and would break the fallback to the public API. # an empty value is honored as-is and would break the fallback to the public API.
#THERMOGRAPH_ARCHIVE_URL=http://open-meteo-api:8080/v1/archive #THERMOGRAPH_ARCHIVE_URL=http://open-meteo-api:8080/v1/archive
# Mark the session cookie Secure — required behind Caddy's HTTPS. Set to 1 in prod; # Mark the session cookie Secure — required behind Caddy's HTTPS. Set to 1 for
# leave unset only for plain-HTTP LAN dev (a Secure cookie is never sent over HTTP). # prod and beta (both TLS-fronted); leave unset only for dev's plain-HTTP
# mesh-only URL (a Secure cookie is never sent over HTTP).
THERMOGRAPH_COOKIE_SECURE=1 THERMOGRAPH_COOKIE_SECURE=1
# Pin these too (see their own sections below), so container restarts don't rotate # Pin these too (see their own sections below), so container restarts don't rotate
@ -58,11 +59,14 @@ THERMOGRAPH_COOKIE_SECURE=1
# Number of uvicorn worker processes. More than 1 stops a single slow upstream fetch # Number of uvicorn worker processes. More than 1 stops a single slow upstream fetch
# (e.g. a cache-miss weather lookup) from blocking every other request — the cause of # (e.g. a cache-miss weather lookup) from blocking every other request — the cause of
# past brief outages. Prod runs 4 (the compose app service also defaults to WORKERS=4); # past brief outages. Prod's Swarm `web` service defaults to 4 (WEB_WORKERS, also the
# leave unset (defaults to 1) on a small box. Workers elect one leader for the # autoscaler's per-replica count); beta's defaults to 2 (BETA_WEB_WORKERS) — a
# subscription notifier via a lockfile (THERMOGRAPH_SINGLETON_LOCK, set by the compose # rehearsal environment, not a decision that beta needs less headroom per se. The
# app service to /app/data/notifier.lock) so its timer-driven upstream sweep runs once, # base compose file (dev) also defaults to 4. Leave unset (defaults to 1) on a small
# not once per worker. ~200 MB RAM per worker. # box. Workers elect one leader for the subscription notifier via a lockfile
# (THERMOGRAPH_SINGLETON_LOCK, set by the compose/stack app service to
# /app/data/notifier.lock) so its timer-driven upstream sweep runs once, not once per
# worker. ~200 MB RAM per worker.
WORKERS=4 WORKERS=4
# THERMOGRAPH_SINGLETON_LOCK arbitrates workers on ONE host. Under multi-host Swarm, # THERMOGRAPH_SINGLETON_LOCK arbitrates workers on ONE host. Under multi-host Swarm,

41
infra/docker-compose.dev.yml
View file

@ -13,11 +13,21 @@
# by deploy-dev.sh. This overlay must NOT reintroduce `build:`; it only relaxes # by deploy-dev.sh. This overlay must NOT reintroduce `build:`; it only relaxes
# resource caps and LAN-exposes a port, same as the monorepo overlay did. # resource caps and LAN-exposes a port, same as the monorepo overlay did.
# #
# Differences from the prod stack (unchanged intent from the monorepo overlay): # Differences from the prod stack:
# 1. backend is published on ALL interfaces (0.0.0.0:8137), not loopback, so # 1. backend is published on ${DEV_BIND_ADDR}, defaulting to LOOPBACK.
# phones and other devices on the Wi-Fi can reach the dev server directly -- #
# dev has no Caddy in front (prod does, which is why the base file binds # This used to be a flat 0.0.0.0:8137, from when dev ran on the operator's
# 127.0.0.1 only). # LAN box and the point was for phones on the Wi-Fi to reach it. Dev now
# runs on vps1, a public VPS, where 0.0.0.0 would publish whatever
# unreviewed branch is in flight to the entire internet — with no Caddy, no
# TLS and no auth in front of it.
#
# So the default is 127.0.0.1 (safe anywhere, including a laptop running
# `make dev-up`), and deploy.sh sets DEV_BIND_ADDR to the environment's mesh
# address from deploy/env-topology.sh — 10.10.0.2 on vps1. Dev is then
# reachable at http://10.10.0.2:8137 from anything on the WireGuard mesh and
# from nowhere else. There is deliberately no dev DNS record and no Caddy
# site block for it.
# 2. frontend's port publish is dropped entirely -- dev has no Caddy to reach it # 2. frontend's port publish is dropped entirely -- dev has no Caddy to reach it
# directly, so it stays compose-internal-only, reached solely through # directly, so it stays compose-internal-only, reached solely through
# backend's own reverse-proxy fallback (THERMOGRAPH_FRONTEND_BASE_INTERNAL, # backend's own reverse-proxy fallback (THERMOGRAPH_FRONTEND_BASE_INTERNAL,
@ -27,18 +37,21 @@
# prod's backend=4 / frontend=2 / db=2 allocation. `!reset` drops the base # prod's backend=4 / frontend=2 / db=2 allocation. `!reset` drops the base
# value (both the top-level `cpus:` and the Swarm-style `deploy.resources` # value (both the top-level `cpus:` and the Swarm-style `deploy.resources`
# block the base file carries for parity). # block the base file carries for parity).
# 4. backend/daemon/lake get a second env_file entry pointing at a copy of the # 4. backend/daemon/lake get a second env_file entry pointing at an OPTIONAL
# render under $APP_DIR (deploy-dev.sh sets THERMOGRAPH_SECRETS_ENV_FILE_MIRROR # copy of the render under $APP_DIR (`required: false`, so it is a no-op
# to produce it). This box's Docker is the snap package, confined to $HOME -- # when absent). It exists for snap-packaged Docker, which is confined to
# it can't see /etc/thermograph.env at all (not a permissions error, it just # $HOME and cannot see /etc/thermograph.env at all — not a permissions
# doesn't exist as far as snap-confined Docker is concerned), so the base # error; the file simply does not exist as far as snap-confined Docker is
# file's env_file entry silently loads nothing here. compose appends env_file # concerned, so the base file's env_file entry silently loads nothing.
# lists across overlays (last-wins on duplicate keys), so this is additive: # deploy-dev.sh now produces the mirror ONLY when it detects snap Docker,
# prod/beta never set the mirror var, so they only ever get the base entry. # which vps1 (ordinary apt Docker) is not — there, dev reads
# /etc/thermograph.env like beta and prod do, and no plaintext copy of the
# render is written into the checkout. compose appends env_file lists across
# overlays (last-wins on duplicate keys), so this stays additive.
services: services:
backend: backend:
ports: !override ports: !override
- "8137:8137" - "${DEV_BIND_ADDR:-127.0.0.1}:8137:8137"
cpus: !reset null cpus: !reset null
deploy: !reset null deploy: !reset null
env_file: env_file:

32
infra/ops/ICEBERG-HANDOFF.md
View file

@ -5,6 +5,17 @@ already shipped (`infra/ops/dbq.sh`) — **mirror it**. This document is the
contract, the environment facts you'll trip over, and how the result gets contract, the environment facts you'll trip over, and how the result gets
verified. verified.
**Historical note:** `infra/ops/iceberg.sh` now exists, originally built
against the environment facts below as they stood before the vps1/vps2 split
— `beta` at `75.119.132.91` and `prod` at `169.58.46.181`, each its own box.
`iceberg.sh` (and `dbq.sh`) have since been updated to resolve their SSH
target and host facts from `deploy/env-topology.sh` instead of a hardcoded
table, so they already reflect the current split (vps1 = `75.119.132.91` =
dev + Forgejo + Grafana; vps2 = `169.58.46.181` = prod AND beta). This
document is left as a historical record of the original design contract, not
a live description of where each environment's engine runs — see
`infra/ops/README.md` for the current behavior.
## Definition of done ## Definition of done
`infra/ops/iceberg.sh` exists and behaves like `dbq.sh`: `infra/ops/iceberg.sh` exists and behaves like `dbq.sh`:
@ -36,6 +47,9 @@ new network exposure and no interactive steps.
## Environment facts you need ## Environment facts you need
**As originally written** (pre vps1/vps2 split; kept for history — see the
note at the top of this document):
- Monorepo `emi/thermograph` (domain dirs: `backend/ frontend/ infra/ - Monorepo `emi/thermograph` (domain dirs: `backend/ frontend/ infra/
observability/`). Ops tooling lives in `infra/ops/`. observability/`). Ops tooling lives in `infra/ops/`.
- Environments: **dev** = LAN compose stack on the dev machine; **beta** = - Environments: **dev** = LAN compose stack on the dev machine; **beta** =
@ -53,6 +67,24 @@ new network exposure and no interactive steps.
services. Untested; verify before relying on it. services. Untested; verify before relying on it.
- WireGuard mesh: prod `10.10.0.1`, beta `10.10.0.2`, dev `10.10.0.3`. - WireGuard mesh: prod `10.10.0.1`, beta `10.10.0.2`, dev `10.10.0.3`.
- `rclone` and `age` are already installed on prod and beta. - `rclone` and `age` are already installed on prod and beta.
**Current facts, post vps1/vps2 split** (the boxes and mesh IPs above did not
move — only which environment runs where, and what the boxes are called):
- **vps2** = `169.58.46.181` = mesh `10.10.0.1` = the box called "prod" above.
Runs **both** prod and beta now, as two separate Docker Swarm stacks — beta
is a Swarm service too today, not a compose exception, so the tunnelling
constraint above applies equally to it: `ssh -L` is impossible for beta now
as well, not just prod.
- **vps1** = `75.119.132.91` = mesh `10.10.0.2` = the box called "beta" above.
Runs Forgejo, Grafana/Loki/Alloy, and **dev** (its own Postgres container,
reached over SSH like any fleet host — not the LAN-box-local case the
original facts describe).
- **desktop** = mesh `10.10.0.3` = "the dev machine" above. Hosts no
Thermograph environment any more; it's a Swarm worker for flex capacity and
AI-model hosting. The "dev machine is a Swarm worker in the prod cluster"
fact still holds structurally (the desktop is still a worker node), but
nothing about Iceberg or the app database routes through it any more.
- Reference implementation to copy: **`infra/ops/dbq.sh`** + `infra/ops/README.md`. - Reference implementation to copy: **`infra/ops/dbq.sh`** + `infra/ops/README.md`.
## Storage — almost certainly your warehouse ## Storage — almost certainly your warehouse

61
infra/ops/README.md
View file

@ -11,34 +11,56 @@ infra/ops/dbq.sh beta -c '\dt'
echo "select 1" | infra/ops/dbq.sh prod -f - echo "select 1" | infra/ops/dbq.sh prod -f -
``` ```
Environments: `dev` (LAN compose stack on the dev machine), `beta` Environments: `dev` (own compose stack + own Postgres container, on vps1,
(75.119.132.91), `prod` (169.58.46.181). Extra args pass straight through to `75.119.132.91`), `beta` and `prod` (both on vps2, `169.58.46.181` — separate
roles/databases on the ONE shared TimescaleDB instance there). Host, SSH
target and container filter all come from `deploy/env-topology.sh` now, never
a hardcoded table — `dbq.sh` SSHes to vps1 for `dev` and to vps2 for
`beta`/`prod`, and beta (which has no db container of its own) resolves
prod's shared `thermograph_db` task. Extra args pass straight through to
`psql` (`-tA`, `-x`, `--csv`, `-f`, …); stdin is forwarded. `psql` (`-tA`, `-x`, `--csv`, `-f`, …); stdin is forwarded.
**Known gap:** beta has no read-only role of its own yet — `deploy/db/provision-env-db.sh`
only provisions the environment's *owning* role (`thermograph_beta`), so a
`dbq.sh beta` query connects as `thermograph_beta` (which can write) rather
than a read-only role, unlike `dev`/`prod`. See `dbq.sh`'s own header for the
tracked follow-up (a `thermograph_beta_ro` role).
### Why it execs into the container instead of using a connection string ### Why it execs into the container instead of using a connection string
None of the databases are exposed over TCP — each listens only on its private None of the databases are exposed over TCP — each listens only on its private
docker network. Prod's is a Swarm **overlay** (`10.0.2.0/24`) that the host docker network. Prod's and beta's is a Swarm **overlay** (`thermograph_internal`)
cannot route to, so `ssh -L` works for beta but is *impossible* for prod; that the vps2 host itself cannot route to, so `ssh -L` is impossible for
publishing 5432 would mean new ufw rules plus a Swarm endpoint change on either of them — treat beta exactly like prod here, now that it's a Swarm
production. Running `psql` inside the db container works identically in all service too, not the compose-network exception it used to be. Publishing 5432
three environments with no ports, no tunnels, and no infra changes. would mean new ufw rules plus a Swarm endpoint change on a live environment.
Running `psql` inside the db container works identically in all three
environments with no ports, no tunnels, and no infra changes.
The prod container name is a Swarm task name that changes on every redeploy, so The prod container name is a Swarm task name that changes on every redeploy, so
it is resolved at call time via `docker ps --filter name=…`, never hardcoded. it is resolved at call time via `docker ps --filter name=…`, never hardcoded.
### Safety ### Safety
Queries connect as **`thermograph_ro`** — a `NOSUPERUSER` role granted only `dev` and `prod` connect as **`thermograph_ro`** — a `NOSUPERUSER` role granted
`pg_read_all_data`. Read-only is enforced by Postgres, not by convention: only `pg_read_all_data`. Read-only is enforced by Postgres, not by convention:
``` ```
$ infra/ops/dbq.sh prod -c "create table t(i int)" $ infra/ops/dbq.sh prod -c "create table t(i int)"
ERROR: permission denied for schema public ERROR: permission denied for schema public
``` ```
**`beta` is the exception, today:** `deploy/db/provision-env-db.sh` only ever
provisions the environment's *owning* role (`thermograph_beta` —
`NOSUPERUSER`/`NOCREATEDB`/`NOCREATEROLE`, but it owns its own database and
schema) and revokes `CONNECT` from `PUBLIC` on `thermograph_beta`, so
`thermograph_ro` was never granted `CONNECT` there. A `dbq.sh beta` query
therefore connects as `thermograph_beta` until a beta-scoped read-only role
exists (tracked in `dbq.sh`'s own header as a TODO) — it can write to beta's
database, though never to prod's.
The app's own `thermograph` role is a superuser and is deliberately **not** used The app's own `thermograph` role is a superuser and is deliberately **not** used
by this tool. If the role is ever missing (fresh database), recreate it with: by this tool. If `thermograph_ro` is ever missing (fresh instance), recreate it with:
```sql ```sql
CREATE ROLE thermograph_ro LOGIN; CREATE ROLE thermograph_ro LOGIN;
@ -68,15 +90,18 @@ The lake is **one shared warehouse** in Contabo object storage
same credentials — so unlike Postgres there is no per-env database to reach, same credentials — so unlike Postgres there is no per-env database to reach,
and the environments differ only in *where the engine runs*: a throwaway and the environments differ only in *where the engine runs*: a throwaway
`docker run` of a small DuckDB image (duckdb CLI + `httpfs`/`iceberg` `docker run` of a small DuckDB image (duckdb CLI + `httpfs`/`iceberg`
extensions pre-installed) on the target box — locally for LAN dev, over SSH extensions pre-installed) on the target box — vps1 for `dev`, vps2 for
for beta/prod. The image is built on the host the first time it's needed from `beta`/`prod`, resolved from `deploy/env-topology.sh` rather than a hardcoded
a Dockerfile embedded in the script; its tag is a hash of that Dockerfile, so table (beta and prod are two Swarm stacks on the same vps2 box now). The
editing the script rolls every host forward on the next call. image is built on the host the first time it's needed from a Dockerfile
embedded in the script; its tag is a hash of that Dockerfile, so editing the
script rolls every host forward on the next call.
A container per query means no daemon, no listening port, no tunnel (prod's A container per query means no daemon, no listening port, no tunnel (beta's
overlay cannot be tunnelled), no firewall or Swarm changes — `sudo ss -ltnp` and prod's Swarm overlay cannot be tunnelled), no firewall or Swarm changes —
is identical before and after a query. It also never execs into an app `sudo ss -ltnp` is identical before and after a query. It also never execs
container and resolves no container names, so prod redeploys can't break it. into an app container and resolves no container names, so a redeploy can't
break it.
There is no catalog service to run or expose: an Iceberg table's current There is no catalog service to run or expose: an Iceberg table's current
state is fully described by the newest metadata JSON under state is fully described by the newest metadata JSON under

69
infra/ops/dbq.sh
View file

@ -7,26 +7,43 @@
# echo "select 1" | infra/ops/dbq.sh prod -f - # echo "select 1" | infra/ops/dbq.sh prod -f -
# #
# Why exec-into-the-container instead of a connection string: # Why exec-into-the-container instead of a connection string:
# none of the databases are exposed over TCP. Each listens only on its private # none of the databases are exposed over TCP. Dev's listens only on its own
# docker network -- and prod's is a Swarm *overlay* (10.0.2.0/24) that the host # private docker network on vps1; prod and beta share ONE instance on prod's
# itself cannot route to, so `ssh -L` works for beta but is impossible for prod. # Swarm *overlay* (10.0.2.0/24) on vps2, which the host itself cannot route
# Publishing 5432 would mean new firewall + Swarm endpoint changes on production. # to, so `ssh -L` is impossible for either. Publishing 5432 would mean new
# Running psql *inside* the db container works identically in all three # firewall + Swarm endpoint changes on production. Running psql *inside* the
# environments with no ports, no tunnels and no infra changes -- locally for LAN # db container works identically in all three environments with no ports, no
# dev, over SSH for beta/prod. # tunnels and no infra changes -- over SSH to vps1 for dev, to vps2 for
# beta/prod.
# #
# Safety: always connects as `thermograph_ro`, a NOSUPERUSER role granted only # Host, SSH target and container filter all come from env-topology.sh, never
# pg_read_all_data. Read-only is enforced by Postgres itself, not by convention, # a hardcoded table -- beta and prod are two Swarm stacks on the SAME box
# so a stray INSERT/DDL fails with "permission denied" even against prod. (The # (vps2) now, and beta has no db container of its own: it reaches prod's
# app's own `thermograph` role is a superuser -- deliberately not used here.) # `thermograph_db` task, the one shared TimescaleDB instance.
#
# Safety: every environment connects as a READ-ONLY role, never as the role the
# app itself uses -- a stray INSERT/DDL fails with "permission denied" even
# against prod. The role is the environment's own owning role with an `_ro`
# suffix, which resolves to the long-standing `thermograph_ro` for prod and dev
# and to `thermograph_beta_ro` for beta.
#
# That suffix convention is why beta needed no special case. It would have been
# easy to give beta one: `CONNECT` is revoked from `PUBLIC` on
# `thermograph_beta`, so prod's `thermograph_ro` cannot reach it, and the
# tempting shortcut was to let beta queries connect as `thermograph_beta` --
# the role that OWNS beta's database. That would have quietly made ad-hoc
# queries read-write on beta alone. deploy/db/provision-env-db.sh provisions the
# `_ro` role for every environment instead.
# #
# Any extra arguments are passed straight through to psql, so -tA, -c, -f, -x, # Any extra arguments are passed straight through to psql, so -tA, -c, -f, -x,
# --csv etc. all work. Stdin is forwarded, so `-f -` reads piped SQL. # --csv etc. all work. Stdin is forwarded, so `-f -` reads piped SQL.
set -euo pipefail set -euo pipefail
SELF_DIR=$(cd "$(dirname "$0")" && pwd)
# shellcheck source=infra/deploy/env-topology.sh
. "$SELF_DIR/../deploy/env-topology.sh"
KEYFILE="${THERMOGRAPH_AGENT_KEY:-$HOME/.ssh/thermograph_agent_ed25519}" KEYFILE="${THERMOGRAPH_AGENT_KEY:-$HOME/.ssh/thermograph_agent_ed25519}"
DB_USER="${THERMOGRAPH_DB_QUERY_USER:-thermograph_ro}"
DB_NAME="${THERMOGRAPH_DB_NAME:-thermograph}"
usage() { usage() {
sed -n '2,9p' "$0" | sed 's/^# \{0,1\}//' >&2 sed -n '2,9p' "$0" | sed 's/^# \{0,1\}//' >&2
@ -39,13 +56,33 @@ env_name="${1:-}"
shift shift
case "$env_name" in case "$env_name" in
dev) filter=thermograph-dev-db ; ssh_target= ;; dev|beta|prod) ;;
beta) filter=thermograph-db-1 ; ssh_target=agent@75.119.132.91 ;;
prod) filter=thermograph_db ; ssh_target=agent@169.58.46.181 ;;
-h|--help) usage ;; -h|--help) usage ;;
*) echo "dbq: unknown environment '$env_name' (want: dev|beta|prod)" >&2; exit 2 ;; *) echo "dbq: unknown environment '$env_name' (want: dev|beta|prod)" >&2; exit 2 ;;
esac esac
thermograph_topology "$env_name"
ssh_target="$TG_SSH_TARGET"
# Container filter: a Swarm task name (shared across prod and beta -- beta has
# no db task of its own) or a compose container name, per the environment's
# deploy mode.
if [ "$TG_DEPLOY_MODE" = stack ]; then
filter="$TG_DB_SERVICE"
else
filter="${TG_COMPOSE_PROJECT}-${TG_DB_SERVICE}"
fi
DB_NAME="${THERMOGRAPH_DB_NAME:-$TG_DB_NAME}"
if [ -n "${THERMOGRAPH_DB_QUERY_USER:-}" ]; then
DB_USER="$THERMOGRAPH_DB_QUERY_USER"
else
# thermograph_ro for prod/dev, thermograph_beta_ro for beta -- provisioned by
# deploy/db/provision-env-db.sh. Never the owning role.
DB_USER="${TG_DB_USER}_ro"
fi
[ $# -gt 0 ] || usage [ $# -gt 0 ] || usage
# Quote the psql arguments so they survive the remote shell intact. # Quote the psql arguments so they survive the remote shell intact.

23
infra/ops/iceberg.sh
View file

@ -9,9 +9,12 @@
# Why an ephemeral container instead of a query service: the lake is one shared # Why an ephemeral container instead of a query service: the lake is one shared
# warehouse in Contabo object storage (s3://era5-thermograph/iceberg), readable # warehouse in Contabo object storage (s3://era5-thermograph/iceberg), readable
# from every environment, so the environments differ only in WHERE the engine # from every environment, so the environments differ only in WHERE the engine
# runs -- a throwaway `docker run` of a small DuckDB image on the target box. # runs -- a throwaway `docker run` of a small DuckDB image on the target box
# No daemon, no listening port, no tunnel (prod's overlay cannot be tunnelled), # (vps1 for dev, vps2 for beta/prod -- the SSH target comes from
# and no container names to resolve, so prod redeploys cannot break it. # env-topology.sh, never a hardcoded table; beta and prod are two Swarm stacks
# on the SAME vps2 box now). No daemon, no listening port, no tunnel (prod's
# overlay cannot be tunnelled), and no container names to resolve, so prod
# redeploys cannot break it.
# #
# There is no catalog service: a table's current state is its newest metadata # There is no catalog service: a table's current state is its newest metadata
# JSON under <warehouse>/<table>/metadata/, resolved at call time and exposed # JSON under <warehouse>/<table>/metadata/, resolved at call time and exposed
@ -34,6 +37,10 @@
# read as piped SQL. A single bare argument is treated as `-c` SQL. # read as piped SQL. A single bare argument is treated as `-c` SQL.
set -euo pipefail set -euo pipefail
SELF_DIR=$(cd "$(dirname "$0")" && pwd)
# shellcheck source=infra/deploy/env-topology.sh
. "$SELF_DIR/../deploy/env-topology.sh"
KEYFILE="${THERMOGRAPH_AGENT_KEY:-$HOME/.ssh/thermograph_agent_ed25519}" KEYFILE="${THERMOGRAPH_AGENT_KEY:-$HOME/.ssh/thermograph_agent_ed25519}"
usage() { usage() {
@ -47,13 +54,17 @@ env_name="${1:-}"
shift shift
case "$env_name" in case "$env_name" in
dev) ssh_target= ;; dev|beta|prod) ;;
beta) ssh_target=agent@75.119.132.91 ;;
prod) ssh_target=agent@169.58.46.181 ;;
-h|--help) usage ;; -h|--help) usage ;;
*) echo "iceberg: unknown environment '$env_name' (want: dev|beta|prod)" >&2; exit 2 ;; *) echo "iceberg: unknown environment '$env_name' (want: dev|beta|prod)" >&2; exit 2 ;;
esac esac
# SSH target comes from env-topology.sh, not a hardcoded table: beta and prod
# are two Swarm stacks on the SAME box (vps2) now, so a stale beta entry here
# would silently point a beta query at the wrong host.
thermograph_topology "$env_name"
ssh_target="$TG_SSH_TARGET"
[ $# -gt 0 ] || usage [ $# -gt 0 ] || usage
# The engine image: duckdb CLI with the httpfs + iceberg extensions installed # The engine image: duckdb CLI with the httpfs + iceberg extensions installed

75
infra/terraform/README.md
View file

@ -10,27 +10,46 @@ scaffold" below) — scaffold-only today, no live resources until you opt in.
## What it manages ## What it manages
One reusable module (`modules/thermograph-host`) is instantiated per host via One reusable module (`modules/thermograph-host`) is instantiated per
`for_each` (`main.tf`'s `local.all_hosts`, merging `var.hosts` — SSH-managed, **(host, environment)** pair via `for_each` (`main.tf`'s `local.all_hosts`,
already-existing boxes — with any `var.gcp_hosts` Terraform created itself). merging `var.hosts` — SSH-managed, already-existing boxes — with any
This config manages **two VPS hosts** today: `var.gcp_hosts` Terraform created itself). `var.hosts` is keyed by HOST, not by
environment, because **vps2 alone runs two environments** (prod and beta) as
separate Docker Swarm stacks on the same box — see the `hosts` variable's file
header in `variables.tf` for why that stopped being "one entry = one
environment" and what still needs to change in `main.tf` to consume the new
shape (flagged there as a `TODO(cutover)`). This config manages **two VPS
hosts**, three environments, today:
| key | role | VPS | branch | domain | notes | | host | environment | VPS | branch | domain | notes |
|--------|--------|-------------------------|-----------|-------------------|------------------------------------| |--------|-------------|--------------------------------------------|--------|------------------------|----------------------------------------------|
| `prod` | prod | NEW 48 GB / 12-core box (`169.58.46.181`) | `release` (of the APP repos — see `backend_image_tag` / `frontend_image_tag`) | `thermograph.org` | Caddy TLS; sized up (8/8/4/16g) | | `vps1` | `dev` | `75.119.132.91` | `dev` (of the APP repos — see `backend_image_tag` / `frontend_image_tag`) | none (mesh-only) | Also hosts Forgejo, Grafana/Loki/Alloy, emigriffith.dev — not Terraform-managed |
| `beta` | beta | old box `75.119.132.91` | `main` (of the APP repo) | `beta.thermograph.org` | Caddy TLS; also hosts Forgejo | | `vps2` | `prod` | 48 GB / 12-core box (`169.58.46.181`) | `main` | `thermograph.org` | Caddy TLS (this module owns `/etc/caddy/Caddyfile`); sized up (8/8/4/16g) |
| `vps2` | `beta` | the SAME box as prod (`169.58.46.181`) | `main` | none (see below) | Separate Swarm stack + checkout, shares prod's TimescaleDB instance on a separate database/role |
Each host's own checkout on disk (`app_dir`, `git_branch`) is **this infra Only ONE environment per host may set a `domain`: the module installs a full
repo**, not the app repo — the "branch" column above is which app-repo tag a `/etc/caddy/Caddyfile`, and a second `terraform apply` with its own `domain` set
host is meant to track conceptually; the actual pinned versions are would silently overwrite the first's site instead of adding to it (there is no
`var.hosts[*].backend_image_tag` / `frontend_image_tag` (e.g. `"sha-<12 hex>"` each — merge). Prod claims that slot on vps2 today, so beta's public reverse-proxy (if
the app is two separately-published images, `emi/thermograph-backend/app` and and when `beta.thermograph.org` is exposed) has to be a site block in vps2's
`emi/thermograph-frontend/app`), since the host has no app checkout to derive a tag shared, hand-maintained Caddy config instead — the same pattern
from. The LAN dev server is **out of scope here** — it `deploy/forgejo/docker-stack.yml` already uses to put `git.thermograph.org` in
builds from source via the app repo's `deploy/deploy-dev.sh` (a self-hosted front of Forgejo's loopback port on vps1, alongside that box's own Caddy-fronted
Forgejo Actions runner), not Terraform. `emigriffith.dev`.
Per host, over SSH provisioners, Terraform: Each environment's own checkout on disk (`app_dir`, `git_branch`) is **this
infra repo**, not the app repo — the "branch" column above is which app-repo
tag an environment is meant to track conceptually; the actual pinned versions
are `var.hosts[*].environments[*].backend_image_tag` / `frontend_image_tag`
(e.g. `"sha-<12 hex>"` each — the app is two separately-published images,
`emi/thermograph-backend/app` and `emi/thermograph-frontend/app`), since the
host has no app checkout to derive a tag from. `app_dir` is required with no
default specifically so vps2's two environments can never collide on the same
checkout path (prod: `/opt/thermograph`, beta: `/opt/thermograph-beta`). The
LAN-laptop dev rehearsal (`make dev-up`) is **out of scope here** — it's a
local-only compose overlay, not this Terraform-managed `dev` environment.
Per (host, environment), over SSH provisioners, Terraform:
- installs Docker + the compose plugin if missing; - installs Docker + the compose plugin if missing;
- configures a `ufw` firewall (22/80/443 always; on a host with **no** domain it also - configures a `ufw` firewall (22/80/443 always; on a host with **no** domain it also
@ -104,7 +123,11 @@ terraform plan
terraform apply terraform apply
``` ```
Target one host with `-target='module.host["beta"]'` if you want to apply to just one. Target one host with `-target='module.host["prod"]'` if you want to apply to just one
(the module's `for_each` key changes once `main.tf` is updated to flatten `var.hosts`'s
new per-host `environments` map — see the `TODO(cutover)` on the `hosts` variable in
`variables.tf` — at which point the key for beta becomes something like
`module.host["vps2-beta"]`, not `module.host["beta"]`).
## Secrets ## Secrets
@ -150,11 +173,15 @@ Terraform brings the stack up — don't let Terraform recreate containers mid-mi
## Assumptions / notes ## Assumptions / notes
- **beta has no public domain by default.** With `compose_files = ["docker-compose.yml"]` - **beta has no public domain here, and can't without a different Caddy strategy.**
the app binds `127.0.0.1:8137` (loopback), so opening the port in `ufw` alone does not With `compose_files = ["docker-compose.yml"]` the app binds `127.0.0.1:8137`
expose it. Reach beta via an SSH tunnel, or set `domain = "beta.thermograph.org"` (adds (loopback), so opening the port in `ufw` alone does not expose it — and setting
Caddy TLS) — or add the `0.0.0.0`-publishing dev overlay to `compose_files` — to make `domain` on beta the way prod does would make this module overwrite the SAME
it reachable. `COOKIE_SECURE` is auto-set to `0` when there's no domain (a Secure `/etc/caddy/Caddyfile` prod's apply just installed (see the table above). Reach
beta via an SSH tunnel, or front it with a site block in vps2's own
hand-maintained Caddy config (outside Terraform), the same way Forgejo's
`git.thermograph.org` reaches Forgejo's loopback port on vps1.
`COOKIE_SECURE` is auto-set to `0` when there's no domain (a Secure
cookie is never sent over plain HTTP) and `1` behind Caddy TLS. cookie is never sent over plain HTTP) and `1` behind Caddy TLS.
- The rendered Caddyfile only reverse-proxies the app. The repo's `deploy/Caddyfile` - The rendered Caddyfile only reverse-proxies the app. The repo's `deploy/Caddyfile`
additionally serves the `emigriffith.dev` portfolio and legacy redirects; those are additionally serves the `emigriffith.dev` portfolio and legacy redirects; those are

36
infra/terraform/main.tf
View file

@ -44,13 +44,39 @@ module "gcp_vm" {
} }
locals { locals {
# Every host this config manages, SSH-only (var.hosts) plus GCP-created (whose # ONE MODULE INSTANCE PER (HOST, ENVIRONMENT) PAIR, not per host.
# `host` is filled in from the VM Terraform just created) — one unified map so a #
# single `module.host` for_each below handles both without duplicating any # var.hosts is keyed by machine (vps1, vps2) and each machine carries an
# provisioning logic. GCP hosts always use a named size tier (simpler than # `environments` map, because vps2 runs prod AND beta. The module below
# provisions an ENVIRONMENT — a checkout, a branch, image tags, a Caddy site,
# container sizing — so it needs one instance per environment, with the two
# instances on vps2 sharing that machine's SSH identity.
#
# Flattened to keys like "vps2-prod" and "vps2-beta". Keeping them distinct
# module instances is what makes `app_dir` (required, no default in
# variables.tf) do its job: two environments on one box get two explicitly
# different checkouts, so neither apply can `git reset --hard` the other's
# tree.
ssh_environments = merge([
for hname, h in var.hosts : {
for ename, e in h.environments :
"${hname}-${ename}" => merge(e, {
host = h.host
ssh_user = h.ssh_user
ssh_private_key_path = h.ssh_private_key_path
})
}
]...)
# Every environment this config manages, SSH-only (flattened above) plus
# GCP-created (whose `host` is filled in from the VM Terraform just created) —
# one unified map so a single `module.host` for_each below handles both without
# duplicating any provisioning logic. GCP hosts stay one-entry-per-machine:
# nothing creates two environments on a created VM today, and the variable
# keeps its flat shape. They always use a named size tier (simpler than
# exposing the four raw sizing fields on that variable too). # exposing the four raw sizing fields on that variable too).
all_hosts = merge( all_hosts = merge(
var.hosts, local.ssh_environments,
{ {
for name, h in var.gcp_hosts : name => merge(h, { for name, h in var.gcp_hosts : name => merge(h, {
host = module.gcp_vm[name].external_ip host = module.gcp_vm[name].external_ip

2
infra/terraform/modules/thermograph-host/variables.tf
View file

@ -21,7 +21,7 @@ variable "ssh_private_key_path" {
} }
variable "role" { variable "role" {
description = "\"prod\" | \"dev\" — informational." description = "\"prod\" | \"beta\" | \"dev\" — informational. One module instance is one (host, environment) pair, so vps2 (which runs both prod and beta) gets two instances, each with its own role/app_dir/image tags -- see the root module's `hosts` variable."
type = string type = string
} }

2
infra/terraform/outputs.tf
View file

@ -4,7 +4,7 @@ output "prod_url" {
} }
output "hosts" { output "hosts" {
description = "Per-host summary: the address managed and its role." description = "Per-environment summary, keyed \"<host>-<environment>\" (vps2 carries two): the machine managed and the role running on it."
value = { value = {
for name, mod in module.host : name => { for name, mod in module.host : name => {
host = mod.host host = mod.host

168
infra/terraform/terraform.tfvars.example
View file

@ -1,8 +1,8 @@
# Copy to terraform.tfvars and fill in real IPs + credentials. # Copy to terraform.tfvars and fill in real credentials.
# cp terraform.tfvars.example terraform.tfvars # cp terraform.tfvars.example terraform.tfvars
# terraform.tfvars is gitignored (repo_url may carry a credential, and om_rclone_conf # terraform.tfvars is gitignored (repo_url may carry a credential, and om_rclone_conf
# always does — both land in local state). NEVER commit real values. # always does — both land in local state). NEVER commit real values.
#
# App secrets (POSTGRES_PASSWORD, THERMOGRAPH_AUTH_SECRET, VAPID keys, # App secrets (POSTGRES_PASSWORD, THERMOGRAPH_AUTH_SECRET, VAPID keys,
# REGISTRY_TOKEN, Discord/SMTP creds, ...) are NOT here anymore -- they live in # REGISTRY_TOKEN, Discord/SMTP creds, ...) are NOT here anymore -- they live in
# the SOPS+age vault (../deploy/secrets/*.yaml), rendered at deploy time. See # the SOPS+age vault (../deploy/secrets/*.yaml), rendered at deploy time. See
@ -11,54 +11,99 @@
# --------------------------------------------------------------------------------- # ---------------------------------------------------------------------------------
# Hosts # Hosts
# --------------------------------------------------------------------------------- # ---------------------------------------------------------------------------------
# Two VPS hosts. (The `dev` branch deploys to the LAN dev server via the app repo's # Two VPS boxes, keyed by HOST (vps1, vps2) — NOT by environment, because vps2
# deploy/deploy-dev.sh — that box is NOT managed by Terraform.) # alone carries two (prod AND beta). Each host lists its SSH identity once, then
# an `environments` map for whatever runs on it. (The `dev` branch deploys to
# vps1 the same way in reality, but the LAN-laptop `make dev-up` rehearsal is NOT
# managed by Terraform at all.)
hosts = { hosts = {
# Production: the NEW 48 GB / 12-core VPS serving thermograph.org. # vps1: Forgejo (git+CI+registry), Grafana/Loki/Alloy, emigriffith.dev, and the
prod = { # `dev` environment. One environment today; the shape still nests it so a
host = "REPLACE_WITH_NEW_VPS_IP" # <-- the new prod VPS IP/hostname # second one (there is none planned) would never have to fight this host's
ssh_user = "agent" # SSH identity.
ssh_private_key_path = "~/.ssh/thermograph_agent_ed25519" vps1 = {
role = "prod"
git_branch = "main" # this INFRA repo's branch -- see *_image_tag for the app versions
backend_image_tag = "REPLACE_WITH_BACKEND_IMAGE_TAG" # e.g. "sha-abcdef012345" -- from thermograph-backend build-push.yml
frontend_image_tag = "REPLACE_WITH_FRONTEND_IMAGE_TAG" # e.g. "sha-012345abcdef" -- from thermograph-frontend build-push.yml
domain = "thermograph.org" # Caddy TLS in front, app on loopback
compose_files = ["docker-compose.yml"]
app_dir = "/opt/thermograph"
# "large" is the named size tier for this box (locals.sizes in main.tf) — same
# numbers as hand-picking workers=8/app_cpus=8/db_cpus=4/db_memory="16g" below,
# via the shortcut. The Postgres internal budget scales from db_memory
# automatically (deploy/db/init/20-tuning.sh); no separate tuning edit.
size = "large"
# Self-host the ERA5 archive here: layers docker-compose.openmeteo.yml and
# provisions the rclone mount of the object-storage bucket (om_* vars below).
openmeteo = true
om_data_dir = "/mnt/om-archive"
}
# Beta / testing: the OLD VPS, repurposed.
beta = {
host = "75.119.132.91" host = "75.119.132.91"
ssh_user = "agent" ssh_user = "agent"
ssh_private_key_path = "~/.ssh/thermograph_agent_ed25519" ssh_private_key_path = "~/.ssh/thermograph_agent_ed25519"
role = "beta"
git_branch = "main" environments = {
backend_image_tag = "REPLACE_WITH_BACKEND_IMAGE_TAG" dev = {
frontend_image_tag = "REPLACE_WITH_FRONTEND_IMAGE_TAG" role = "dev"
# No public domain by default: no Caddy/TLS, firewall opens the app port. NOTE: git_branch = "dev" # the only environment that tracks `dev`; beta/prod track `main`
# with compose_files = ["docker-compose.yml"] the app binds 127.0.0.1 only, so backend_image_tag = "REPLACE_WITH_BACKEND_IMAGE_TAG"
# until you either set a domain (e.g. "beta.thermograph.org", which fronts it with frontend_image_tag = "REPLACE_WITH_FRONTEND_IMAGE_TAG"
# Caddy) or add the 0.0.0.0-publishing dev overlay, reach it via an SSH tunnel. # Mesh-only: no public domain, no Caddy/TLS. Reachable at
domain = "" # http://10.10.0.2:8137 from the WireGuard mesh only.
compose_files = ["docker-compose.yml"] domain = ""
app_dir = "/opt/thermograph" compose_files = ["docker-compose.yml"]
# Explicit numbers, not a size tier — both styles work on any host; a tier is app_dir = "/opt/thermograph-dev"
# purely an opt-in shortcut (see prod's `size = "large"` above). workers = 4
workers = 4 app_cpus = 4
app_cpus = 4 db_cpus = 2
db_cpus = 2 db_memory = "8g"
db_memory = "8g" }
}
}
# vps2: prod (thermograph.org) AND beta (beta.thermograph.org) as two separate
# Swarm stacks on the SAME 48 GB / 12-core box — plus Centralis, Postfix and
# the backups (not Terraform-managed). ONE shared TimescaleDB instance serves
# both app_dirs below, on separate databases/roles (deploy/db/provision-env-db.sh);
# each environment's db_cpus/db_memory here sizes only that environment's own
# app-container caps, not a second database.
vps2 = {
host = "169.58.46.181"
ssh_user = "agent"
ssh_private_key_path = "~/.ssh/thermograph_agent_ed25519"
environments = {
prod = {
role = "prod"
git_branch = "main" # this INFRA repo's branch -- see *_image_tag for the app versions
backend_image_tag = "REPLACE_WITH_BACKEND_IMAGE_TAG" # e.g. "sha-abcdef012345" -- from thermograph-backend build-push.yml
frontend_image_tag = "REPLACE_WITH_FRONTEND_IMAGE_TAG" # e.g. "sha-012345abcdef" -- from thermograph-frontend build-push.yml
domain = "thermograph.org" # Caddy TLS in front, app on loopback
compose_files = ["docker-compose.yml"]
# MUST differ from beta's app_dir below -- see variables.tf's file header.
app_dir = "/opt/thermograph"
# "large" is the named size tier for this box (locals.sizes in main.tf) — same
# numbers as hand-picking workers=8/app_cpus=8/db_cpus=4/db_memory="16g" below,
# via the shortcut. The Postgres internal budget scales from db_memory
# automatically (deploy/db/init/20-tuning.sh); no separate tuning edit.
size = "large"
# Self-host the ERA5 archive here: layers docker-compose.openmeteo.yml and
# provisions the rclone mount of the object-storage bucket (om_* vars below).
openmeteo = true
om_data_dir = "/mnt/om-archive"
}
beta = {
role = "beta"
git_branch = "main"
backend_image_tag = "REPLACE_WITH_BACKEND_IMAGE_TAG"
frontend_image_tag = "REPLACE_WITH_FRONTEND_IMAGE_TAG"
# "" ON PURPOSE, unlike prod above: modules/thermograph-host installs a
# FULL /etc/caddy/Caddyfile per environment with a domain set, and only
# ONE environment on a box can own that file. Prod already claims it on
# this host, so beta.thermograph.org's public reverse-proxy has to be a
# site block in vps2's shared, hand-maintained Caddy instead (the same
# pattern deploy/forgejo/docker-stack.yml uses for git.thermograph.org
# on vps1) -- not something this module can render for a second
# environment on the same host.
domain = ""
compose_files = ["docker-compose.yml"]
# MUST differ from prod's app_dir above -- a SECOND checkout on the same
# box, so a `git reset --hard` in one deploy can never yank the tree out
# from under the other's running deploy.
app_dir = "/opt/thermograph-beta"
# Explicit numbers, not a size tier — both styles work on any environment; a
# tier is purely an opt-in shortcut (see prod's `size = "large"` above).
workers = 4
app_cpus = 4
db_cpus = 2
db_memory = "8g"
}
}
} }
# UAT: an ephemeral, single-node environment — same images/topology shape as # UAT: an ephemeral, single-node environment — same images/topology shape as
@ -68,20 +113,27 @@ hosts = {
# host = "REPLACE_WITH_UAT_VM_IP" # host = "REPLACE_WITH_UAT_VM_IP"
# ssh_user = "agent" # ssh_user = "agent"
# ssh_private_key_path = "~/.ssh/thermograph_agent_ed25519" # ssh_private_key_path = "~/.ssh/thermograph_agent_ed25519"
# role = "uat" # environments = {
# git_branch = "main" # uat = {
# backend_image_tag = "REPLACE_WITH_BACKEND_IMAGE_TAG" # role = "uat"
# frontend_image_tag = "REPLACE_WITH_FRONTEND_IMAGE_TAG" # git_branch = "main"
# domain = "" # backend_image_tag = "REPLACE_WITH_BACKEND_IMAGE_TAG"
# compose_files = ["docker-compose.yml"] # frontend_image_tag = "REPLACE_WITH_FRONTEND_IMAGE_TAG"
# app_dir = "/opt/thermograph" # domain = ""
# size = "nano" # compose_files = ["docker-compose.yml"]
# app_dir = "/opt/thermograph"
# size = "nano"
# }
# }
# } # }
} }
# GCP-created hosts — SCAFFOLD ONLY, empty by default. Populate an entry to have # GCP-created hosts — SCAFFOLD ONLY, empty by default, and still keyed one entry
# Terraform actually create a GCP Compute Engine VM (see modules/gcp-host); until # per ENVIRONMENT (see variables.tf's TODO(cutover) note on gcp_hosts — these are
# then no google_* resource is planned and no GCP credentials are needed. Example: # hypothetical single-purpose boxes, not vps1/vps2, so they don't need the
# host/environment nesting above). Populate an entry to have Terraform actually
# create a GCP Compute Engine VM (see modules/gcp-host); until then no google_*
# resource is planned and no GCP credentials are needed. Example:
# gcp_hosts = { # gcp_hosts = {
# gcp-uat = { # gcp-uat = {
# project = "REPLACE_WITH_GCP_PROJECT_ID" # project = "REPLACE_WITH_GCP_PROJECT_ID"
@ -108,7 +160,7 @@ hosts = {
# repo_url = "https://deploy:REPLACE_WITH_TOKEN@git.thermograph.org/emi/thermograph-infra.git" # repo_url = "https://deploy:REPLACE_WITH_TOKEN@git.thermograph.org/emi/thermograph-infra.git"
# --------------------------------------------------------------------------------- # ---------------------------------------------------------------------------------
# Self-hosted Open-Meteo archive (only used by hosts with openmeteo = true) -------- # Self-hosted Open-Meteo archive (only used by environments with openmeteo = true)
# --------------------------------------------------------------------------------- # ---------------------------------------------------------------------------------
# The ERA5 .om archive lives in an object-storage bucket, rclone-mounted on the host. # The ERA5 .om archive lives in an object-storage bucket, rclone-mounted on the host.
# om_rclone_conf holds bucket credentials (sensitive; lands in state — keep out of git). # om_rclone_conf holds bucket credentials (sensitive; lands in state — keep out of git).

146
infra/terraform/variables.tf
View file

@ -1,69 +1,109 @@
# --------------------------------------------------------------------------------- # ---------------------------------------------------------------------------------
# Hosts # Hosts
# --------------------------------------------------------------------------------- # ---------------------------------------------------------------------------------
# One entry per VPS. The same module is instantiated for each (see main.tf's # Keyed by HOST (vps1, vps2), NOT by environment: vps2 alone runs TWO environments
# for_each). This config manages TWO hosts: prod (new 48 GB / 12-core VPS, branch # (prod AND beta) as separate Swarm stacks on the SAME box, so "one entry = one
# `release`, thermograph.org) and beta (the old VPS 75.119.132.91, branch `main`, # environment" broke the moment that became true — two entries that both defaulted
# no public domain by default). The `dev` branch deploys to the LAN dev server via # `app_dir` to "/opt/thermograph" would let a prod apply and a beta apply stomp the
# deploy/deploy-dev.sh (a self-hosted runner, in the app repo) and is NOT managed here. # identical checkout. Each host now carries its SSH identity ONCE (host IP,
# ssh_user, key) plus an `environments` map of everything that varies per
# environment running on it — checkout path, git branch, image tags, domain,
# sizing. `app_dir` is REQUIRED with no default for exactly this reason: two
# environments on one host must be given two explicitly DIFFERENT paths (see
# terraform.tfvars.example), so their checkouts — and a `git reset --hard` in one
# of them — can never collide.
# #
# A host with `domain = ""` gets no Caddy/TLS: the app port is opened on the firewall # vps1 (75.119.132.91): Forgejo (git+CI+registry), Grafana/Loki/Alloy,
# and the app is reached directly (beta does this by default; set a domain to front # emigriffith.dev, and the `dev` environment (its own Postgres, mesh-only, no
# it with Caddy TLS). # public DNS/Caddy site). vps2 (169.58.46.181): `prod` (thermograph.org) and
# `beta` (beta.thermograph.org) — Centralis, Postfix and the backups also live
# here — sharing the ONE TimescaleDB instance both app environments use, on
# separate databases/roles (deploy/db/provision-env-db.sh); this variable's
# per-environment db_cpus/db_memory size only that environment's app-container
# caps, not a second database (see deploy/db/init/20-tuning.sh). The LAN dev
# branch (`make dev-up`, a laptop-local rehearsal) deploys nowhere via Terraform
# and is NOT managed here.
# #
# Resource sizing (workers / app_cpus / db_cpus / db_memory) defaults to the historical # An environment with `domain = ""` gets no Caddy/TLS: its app port is opened on
# 4 / 4 / 2 / 8g budget (right for beta). The prod box is 48 GB / 12 cores — raise these # the firewall and it's reached directly.
# there (the example uses app_cpus 8, db_cpus 4, db_memory "16g"). The Postgres internal #
# budget scales from db_memory automatically (deploy/db/init/20-tuning.sh), so no # Resource sizing (workers / app_cpus / db_cpus / db_memory), per environment,
# separate tuning edit is needed to exploit the RAM. # defaults to the historical 4 / 4 / 2 / 8g budget (right for beta). vps2 is 48 GB
# / 12 cores — raise these for prod (the example uses app_cpus 8, db_cpus 4,
# db_memory "16g", via the "large" size tier). The Postgres internal budget scales
# from db_memory automatically (deploy/db/init/20-tuning.sh), so no separate
# tuning edit is needed to exploit the RAM.
#
# TODO(cutover): main.tf's `module.host` for_each (via `local.all_hosts`) still
# expects ONE flat entry per key, reading `each.value.host`/`.role`/`.app_dir`/…
# directly. It needs a flattening step ahead of that for_each — one derived entry
# per (host, environment) pair, keyed e.g. "<host>-<environment>" ("vps2-prod" and
# "vps2-beta" as two separate module instances sharing vps2's host/ssh_user/
# ssh_private_key_path but each with its own app_dir/role/image tags/sizing).
# This variable's new shape is not yet wired into main.tf's module block.
variable "hosts" { variable "hosts" {
description = "Map of hosts to manage, keyed by a short name (e.g. \"prod\", \"beta\")." description = "Map of VPS boxes to manage, keyed by a short host name (\"vps1\", \"vps2\") — NOT by environment. SSH identity is per-host; everything environment-specific lives in that host's `environments` map."
type = map(object({ type = map(object({
host = string # IP or hostname to SSH to host = string # IP or hostname to SSH to
ssh_user = optional(string, "deploy") # SSH login user ssh_user = optional(string, "deploy") # SSH login user
ssh_private_key_path = string # path to the private key for that user ssh_private_key_path = string # path to the private key for that user
role = string # "prod" | "beta" (informational + outputs) # One entry per environment running on this host, keyed by environment name
git_branch = string # this INFRA repo's branch the checkout is reset to # ("prod" | "beta" | "dev"). A host normally carries one; vps2 carries two.
# Which app images to run, e.g. "sha-<12 hex>" (each matching build-push.yml's tag environments = map(object({
# for the commit that app repo built) or a semver tag on a release push. The app is role = string # "prod" | "beta" | "dev" (informational + outputs; should match the map key above)
# TWO separately-published images now — emi/thermograph-backend/app and git_branch = string # this INFRA repo's branch the checkout is reset to
# emi/thermograph-frontend/app — pinned independently. Required, no default: the # Which app images to run, e.g. "sha-<12 hex>" (each matching build-push.yml's tag
# host's checkout is this infra repo, not the app repos, so there's no "current # for the commit that app repo built) or a semver tag on a release push. The app is
# commit" to derive a tag from; both must be explicit. Bump these (via a normal # TWO separately-published images now — emi/thermograph-backend/app and
# tfvars edit + apply) whenever an app repo ships a commit you want this host # emi/thermograph-frontend/app — pinned independently. Required, no default: the
# running; the infra repo's own git_branch is independent and rarely needs to change. # host's checkout is this infra repo, not the app repos, so there's no "current
backend_image_tag = string # commit" to derive a tag from; both must be explicit. Bump these (via a normal
frontend_image_tag = string # tfvars edit + apply) whenever an app repo ships a commit you want this
domain = optional(string, "") # public domain; "" => no Caddy/TLS # environment running; the infra repo's own git_branch is independent and rarely
compose_files = optional(list(string), ["docker-compose.yml"]) # needs to change.
app_dir = optional(string, "/opt/thermograph") # checkout path on the host backend_image_tag = string
workers = optional(number, 4) # uvicorn workers (WORKERS) frontend_image_tag = string
app_cpus = optional(number, 4) # app container CPU cap (APP_CPUS) domain = optional(string, "") # public domain for this environment; "" => no Caddy/TLS
db_cpus = optional(number, 2) # db container CPU cap (DB_CPUS) compose_files = optional(list(string), ["docker-compose.yml"])
db_memory = optional(string, "8g") # db container memory cap (DB_MEMORY) # REQUIRED, no default — see the file header: two environments on one host
# A named size tier (see locals.sizes in main.tf: nano/small/medium/large) — # must never be able to default to the same checkout path.
# when set, overrides the four fields above with the tier's preset. Leave app_dir = string # checkout path on the host
# null (default) to keep hand-picking workers/app_cpus/db_cpus/db_memory, workers = optional(number, 4) # uvicorn workers (WORKERS)
# as prod/beta already do below. app_cpus = optional(number, 4) # app container CPU cap (APP_CPUS)
size = optional(string, null) db_cpus = optional(number, 2) # db container CPU cap (DB_CPUS)
# The floating tag matches today's behavior everywhere until you pin it. Pin to db_memory = optional(string, "8g") # db container memory cap (DB_MEMORY)
# an exact minor (SELECT extversion FROM pg_extension WHERE extname='timescaledb' # A named size tier (see locals.sizes in main.tf: nano/small/medium/large) —
# on the live DB) before any host of this stack could ever replicate with # when set, overrides the four fields above with the tier's preset. Leave
# another — a floating tag risks mismatched extension minors, which blocks a # null (default) to keep hand-picking workers/app_cpus/db_cpus/db_memory,
# physical replica (hop-1 cutover runbook hazard #7). Use the SAME tag everywhere. # as prod/beta already do below.
timescaledb_tag = optional(string, "latest-pg18") size = optional(string, null)
# Self-host the ERA5 archive (docker-compose.openmeteo.yml + a host rclone mount # The floating tag matches today's behavior everywhere until you pin it. Pin to
# of the object-storage bucket). Only the self-hosting host (prod) sets true. # an exact minor (SELECT extversion FROM pg_extension WHERE extname='timescaledb'
openmeteo = optional(bool, false) # on the live DB) before any host of this stack could ever replicate with
om_data_dir = optional(string, "/mnt/om-archive") # host rclone mount point (OM_DATA_DIR) # another — a floating tag risks mismatched extension minors, which blocks a
# physical replica (hop-1 cutover runbook hazard #7). Use the SAME tag everywhere.
timescaledb_tag = optional(string, "latest-pg18")
# Self-host the ERA5 archive (docker-compose.openmeteo.yml + a host rclone mount
# of the object-storage bucket). Only the self-hosting environment (prod) sets true.
openmeteo = optional(bool, false)
om_data_dir = optional(string, "/mnt/om-archive") # host rclone mount point (OM_DATA_DIR)
}))
})) }))
} }
# GCP-created hosts, keyed the same way as `hosts`. Default {} => zero GCP resources # GCP-created hosts, keyed the same way `hosts` used to be — one entry per
# planned and the google provider is never actually invoked (see versions.tf and # environment, since these are hypothetical single-purpose boxes (e.g. a future
# modules/gcp-host). Populate an entry to have Terraform create the VM itself; its # UAT VM), not vps1/vps2, so they don't need the host/environment split above.
# output IP then feeds into the SAME thermograph-host module every SSH-managed host # Default {} => zero GCP resources planned and the google provider is never
# uses (main.tf), so provisioning logic is never duplicated between providers. # actually invoked (see versions.tf and modules/gcp-host). Populate an entry to
# have Terraform create the VM itself; its output IP then feeds into the SAME
# thermograph-host module every SSH-managed host uses (main.tf), so provisioning
# logic is never duplicated between providers.
# TODO(cutover): main.tf's `local.all_hosts = merge(var.hosts, ...)` currently
# merges this flat shape with `var.hosts` into one map for `module.host`'s
# for_each. Once `var.hosts` is nested (above), that merge needs to target the
# same flattened (host, environment) shape this variable already has — e.g. by
# treating each `gcp_hosts` entry as its own single-environment host.
variable "gcp_hosts" { variable "gcp_hosts" {
description = "Map of hosts for Terraform to CREATE on GCP (Compute Engine), keyed the same way as `hosts`. Empty by default -- no live GCP resources exist yet; this is a scaffold for future use. See modules/gcp-host." description = "Map of hosts for Terraform to CREATE on GCP (Compute Engine), keyed the same way as `hosts`. Empty by default -- no live GCP resources exist yet; this is a scaffold for future use. See modules/gcp-host."
type = map(object({ type = map(object({

11
observability/.env.example
View file

@ -1,7 +1,10 @@
# Copy to .env on the monitoring host (beta) before `docker compose up`. # Copy to .env on the monitoring host (vps1) before `docker compose up`.
# .env is gitignored — never commit real secrets. # .env is gitignored — never commit real secrets.
#
# vps1 is NOT the beta environment — that runs on vps2 now, as its own Swarm
# stack. Don't confuse the two when reading older docs that say "beta" here.
# The public hostname beta's Caddy serves Grafana at (see caddy-grafana.conf). # The public hostname vps1's Caddy serves Grafana at (see caddy-grafana.conf).
GRAFANA_DOMAIN=dashboard.thermograph.org GRAFANA_DOMAIN=dashboard.thermograph.org
# Break-glass local admin (Google SSO below is the primary login). Keep this # Break-glass local admin (Google SSO below is the primary login). Keep this
@ -22,8 +25,8 @@ GOOGLE_CLIENT_SECRET=
# --- Alerting (Discord) ---------------------------------------------------------- # --- Alerting (Discord) ----------------------------------------------------------
# Where every alert in grafana/provisioning/alerting/ is delivered: a webhook on # Where every alert in grafana/provisioning/alerting/ is delivered: a webhook on
# the private #ops-alerts channel (Owners category) of the Thermograph.org # the private #ops-alerts channel (Owners category) of the Thermograph.org
# Discord. NOT email — beta's Grafana relays SMTP through prod's Postfix, so # Discord. NOT email — vps1's Grafana relays SMTP through vps2's Postfix, so
# email alerts die exactly when prod does. # email alerts die exactly when prod (on vps2) does.
# #
# To mint one: Discord -> #ops-alerts -> Edit Channel -> Integrations -> # To mint one: Discord -> #ops-alerts -> Edit Channel -> Integrations ->
# Webhooks -> New Webhook -> Copy Webhook URL. Treat it like a password; anyone # Webhooks -> New Webhook -> Copy Webhook URL. Treat it like a password; anyone

35
observability/CLAUDE.md
View file

@ -1,28 +1,37 @@
# observability/ — agent instructions # observability/ — agent instructions
The **logging stack** for the fleet: Loki + Grafana on beta, with a Grafana Alloy The **logging stack** for the fleet: Loki + Grafana on **vps1** — the monitoring
agent on every node (prod, beta, dev) shipping container and app logs over the host, mesh `10.10.0.2`, and NOT the beta *environment* (that runs on vps2 now;
WireGuard mesh. Grafana is fronted by beta's Caddy at any doc that still says "beta" meaning this box is stale) — with a Grafana
**`dashboard.thermograph.org`** (Google SSO, pre-provisioned users only) — use Alloy agent on every node shipping container and app logs over the WireGuard
that hostname everywhere, never `grafana.thermograph.org`. mesh. vps1 also hosts Forgejo, the portfolio site, and the **dev** environment
(mesh-only, no public DNS). vps2 runs **prod and beta** as two Swarm stacks.
Grafana is fronted by vps1's Caddy at **`dashboard.thermograph.org`** (Google
SSO, pre-provisioned users only) — use that hostname everywhere, never
`grafana.thermograph.org`.
This is operational config, not application code: there is **no build** and no This is operational config, not application code: there is **no build** and no
deploy automation. It ships by hand — `docker compose up -d` on beta for deploy automation. It ships by hand — `docker compose up -d` on vps1 for
Loki+Grafana, and `alloy/docker-compose.agent.yml` on each node. Read the root Loki+Grafana, and `alloy/docker-compose.agent.yml` on each node. Read the root
`CLAUDE.md` first; the branch model there applies here too. `CLAUDE.md` first; the branch model there applies here too.
## Layout ## Layout
- `docker-compose.yml` — the Loki + Grafana stack (beta). - `docker-compose.yml` — the Loki + Grafana stack (vps1).
- `loki/config.yml` — mesh-only, filesystem storage. - `loki/config.yml` — mesh-only, filesystem storage.
- `grafana/provisioning/` — datasource + dashboard provider, auto-loaded at - `grafana/provisioning/` — datasource + dashboard provider, auto-loaded at
startup. `grafana/dashboards/*.json` — the dashboards. startup. `grafana/dashboards/*.json` — the dashboards.
- `grafana/provisioning/alerting/` — alert rules, the Discord contact point, and - `grafana/provisioning/alerting/` — alert rules, the Discord contact point, and
the notification policy. the notification policy.
- `alloy/config.alloy` + `alloy/docker-compose.agent.yml` — the per-node shipper. - `alloy/config.alloy` + `alloy/docker-compose.agent.yml` (+
Node name comes from `ALLOY_NODE`, set per host. `alloy/docker-compose.agent.beta.yml`, a vps2-only overlay adding beta's
- `caddy-grafana.conf` — the beta Caddy vhost, kept here for reference. `applogs` mount) — the per-node shipper. `ALLOY_NODE` is the MACHINE
- `.env.example` — copy to `.env` on beta. `.env` is gitignored; never commit (`vps1`|`vps2`|`desktop`) — that's a semantic change from when the fleet had
one environment per node; `host` (`prod`|`beta`|`dev`), the ENVIRONMENT
label every dashboard/alert slices on, is now derived per source instead,
since vps2 alone carries two of them.
- `caddy-grafana.conf` — the vps1 Caddy vhost, kept here for reference.
- `.env.example` — copy to `.env` on vps1. `.env` is gitignored; never commit
OAuth secrets or the admin password. OAuth secrets or the admin password.
## Rules ## Rules
@ -40,8 +49,8 @@ Loki+Grafana, and `alloy/docker-compose.agent.yml` on each node. Read the root
literal Discord webhook URL in the repo is a hard failure. A rule pointing at a literal Discord webhook URL in the repo is a hard failure. A rule pointing at a
missing refId is valid YAML, provisions cleanly, and then never fires — that missing refId is valid YAML, provisions cleanly, and then never fires — that
silent no-op is what this check exists to prevent. silent no-op is what this check exists to prevent.
- **Alerts go to Discord `#ops-alerts`, never email.** Beta's Grafana relays SMTP - **Alerts go to Discord `#ops-alerts`, never email.** vps1's Grafana relays
through prod's Postfix, so email dies exactly when prod does. SMTP through vps2's Postfix, so email dies exactly when prod (on vps2) does.
- **Every rule is LogQL** — there is no Prometheus anywhere in the fleet. - **Every rule is LogQL** — there is no Prometheus anywhere in the fleet.
Thresholds were derived from real Loki data and the working is in the comments Thresholds were derived from real Loki data and the working is in the comments
beside each rule; re-derive before changing a number rather than guessing. beside each rule; re-derive before changing a number rather than guessing.

150
observability/README.md
View file

@ -7,41 +7,61 @@ over the private WireGuard mesh. This replaces the old in-repo
persistent, queryable, fleet-wide log store and UI. persistent, queryable, fleet-wide log store and UI.
``` ```
prod (10.10.0.1) beta (10.10.0.2) desktop / LAN dev (10.10.0.3) vps1 (10.10.0.2) vps2 (10.10.0.1) desktop (10.10.0.3)
┌────────────┐ ┌──────────────────┐ ┌────────────┐ ┌───────────────────────┐ ┌───────────────────────┐ ┌──────────────┐
│ Alloy agent│──wg0──┐ │ Loki ◀── Alloy │ ┌──wg0│ Alloy agent│ │ Loki ◀───── Alloy │◀──wg0───┤ Alloy agent │◀─wg0──┤ Alloy agent │
└────────────┘ └───▶│ Grafana (Caddy) │◀──┘ └────────────┘ │ Grafana (Caddy) │ │ prod + beta (Swarm) │ │ (no app │
docker+caddy+app └──────────────────┘ docker+caddy+app │ dev (compose) │◀──wg0───┤ Postfix, backups │ │ stack) │
logs central store + UI logs │ Forgejo, portfolio │ └───────────────────────┘ └──────────────┘
└───────────────────────┘ docker+caddy+app logs ships no app logs
central store + UI + dev
``` ```
- **vps1** is the monitoring host: Loki, Grafana, Forgejo, the portfolio site,
and the **dev** environment (its own compose project, mesh-only, no public
DNS). It does **not** run the beta *environment* — that now lives on vps2 — so
any older doc that says "beta" meaning *this box* is stale and needs reading
as vps1.
- **vps2** runs **prod and beta** as two separate Docker Swarm stacks
(`thermograph`, `thermograph-beta`), plus Postfix and backups. One shared
TimescaleDB serves both, on separate roles/databases.
- **desktop** hosts no Thermograph environment now (AI-model hosting + flex
Swarm-worker capacity instead), so it ships no app logs.
- **Loki** stores logs (filesystem, 30-day retention). Listens on the mesh IP - **Loki** stores logs (filesystem, 30-day retention). Listens on the mesh IP
`10.10.0.2:3100` — never public. `10.10.0.2:3100` — never public.
- **Grafana** is the UI, fronted by beta's Caddy at `dashboard.thermograph.org`, - **Grafana** is the UI, fronted by vps1's Caddy at `dashboard.thermograph.org`,
login via **Google SSO** (with a break-glass local admin); Loki is not exposed. login via **Google SSO** (with a break-glass local admin); Loki is not exposed.
- **Alloy** runs on each node and ships three sources to Loki: every Docker - **Alloy** runs on each node and ships three sources to Loki: every Docker
container's stdout/stderr, Caddy's host access logs, and the app's structured container's stdout/stderr, Caddy's host access logs, and the app's structured
JSON logs (`errors`/`access`/`audit` `*.jsonl`, parsed so `level`/`tag`/`phase` JSON logs (`errors`/`access`/`audit` `*.jsonl`, parsed so `level`/`tag`/`phase`
become labels). Every line is tagged with its node (`host = prod|beta|dev`). become labels). Every line now carries two distinct labels: `host`
(`prod`|`beta`|`dev`) is the ENVIRONMENT, derived *per source* (a container's
stack prefix, the Caddy access-log filename, or which `applogs` volume a JSON
line came out of) — because vps2 alone carries two environments, and one
node-wide value would have stamped every beta container and log line as
`prod`. `node` (`vps1`|`vps2`|`desktop`) is the MACHINE, stamped externally on
everything an agent ships. See `alloy/config.alloy` for exactly how `host` is
derived per source.
## Layout ## Layout
| Path | What | | Path | What |
|------|------| |------|------|
| `docker-compose.yml` | the central Loki + Grafana stack (runs on beta) | | `docker-compose.yml` | the central Loki + Grafana stack (runs on vps1) |
| `loki/config.yml` | Loki config (filesystem, retention) | | `loki/config.yml` | Loki config (filesystem, retention) |
| `grafana/provisioning/` | auto-wired Loki datasource + dashboard provider | | `grafana/provisioning/` | auto-wired Loki datasource + dashboard provider |
| `grafana/provisioning/alerting/` | contact point, notification policy and alert rules — see [Alerting](#alerting) | | `grafana/provisioning/alerting/` | contact point, notification policy and alert rules — see [Alerting](#alerting) |
| `grafana/dashboards/thermograph-logs.json` | the fleet-logs dashboard | | `grafana/dashboards/thermograph-logs.json` | the fleet-logs dashboard |
| `alloy/config.alloy` | the per-node collector config | | `alloy/config.alloy` | the per-node collector config |
| `alloy/docker-compose.agent.yml` | runs the Alloy agent on a node | | `alloy/docker-compose.agent.yml` | runs the Alloy agent on a node |
| `caddy-grafana.conf` | beta Caddy site block for the Grafana UI | | `alloy/docker-compose.agent.beta.yml` | overlay adding beta's `applogs` mount — vps2 only |
| `caddy-grafana.conf` | vps1 Caddy site block for the Grafana UI |
## Deploy ## Deploy
> ### ⚠️ Merging is not deploying > ### ⚠️ Merging is not deploying
> >
> `/opt/observability` on beta (and prod) is a checkout of the **archived** > `/opt/observability` on vps1 (and vps2) is a checkout of the **archived**
> `emi/thermograph-observability` repo. It can never `git pull` again — the > `emi/thermograph-observability` repo. It can never `git pull` again — the
> content now lives here, in the mono repo, under `observability/`. A previous > content now lives here, in the mono repo, under `observability/`. A previous
> observability PR merged and had **zero live effect** for exactly this reason. > observability PR merged and had **zero live effect** for exactly this reason.
@ -49,22 +69,23 @@ persistent, queryable, fleet-wide log store and UI.
> Until someone re-points those checkouts at the mono repo, every change here > Until someone re-points those checkouts at the mono repo, every change here
> ships by hand: back up the live file with a UTC-timestamped suffix, `scp` the > ships by hand: back up the live file with a UTC-timestamped suffix, `scp` the
> new one up, `sudo cp` it into place, restart the specific container. Use > new one up, `sudo cp` it into place, restart the specific container. Use
> `sudo docker restart <container>` — **not `docker compose`**, which on prod > `sudo docker restart <container>` — **not `docker compose`**, which on vps2
> demands a `GF_SECURITY_ADMIN_PASSWORD` it has no `.env` to interpolate from. > demands a `GF_SECURITY_ADMIN_PASSWORD` it has no `.env` to interpolate from
> (Beta's stack does have an `.env`, so compose works there — and is required > (vps2 only runs the Alloy agent — there's no Grafana there to configure).
> (vps1's stack does have an `.env`, so compose works there — and is required
> when a change adds a new environment variable, since `docker restart` alone > when a change adds a new environment variable, since `docker restart` alone
> will not pick one up.) > will not pick one up.)
### 1. Central stack (on beta) ### 1. Central stack (on vps1)
```bash ```bash
# on beta (75.119.132.91): # on vps1 (75.119.132.91):
git clone <this repo> thermograph-observability && cd thermograph-observability git clone <this repo> thermograph-observability && cd thermograph-observability
cp .env.example .env # set GF_SECURITY_ADMIN_PASSWORD cp .env.example .env # set GF_SECURITY_ADMIN_PASSWORD
docker compose up -d # Loki on 10.10.0.2:3100 + 127.0.0.1:3100; Grafana on 127.0.0.1:3000 docker compose up -d # Loki on 10.10.0.2:3100 + 127.0.0.1:3100; Grafana on 127.0.0.1:3000
``` ```
Expose the Grafana UI: point `dashboard.thermograph.org` at beta, append Expose the Grafana UI: point `dashboard.thermograph.org` at vps1, append
`caddy-grafana.conf` to `/etc/caddy/Caddyfile`, `systemctl reload caddy`. `caddy-grafana.conf` to `/etc/caddy/Caddyfile`, `systemctl reload caddy`.
### Google SSO (Grafana + Forgejo) ### Google SSO (Grafana + Forgejo)
@ -80,7 +101,7 @@ OAuth 2.0 Client (type: *Web application*) and register both redirect URIs:
(`allow_sign_up=false`), so provision your email as an admin once: (`allow_sign_up=false`), so provision your email as an admin once:
```bash ```bash
# on beta, after the stack is up (uses the break-glass admin creds from .env): # on vps1, after the stack is up (uses the break-glass admin creds from .env):
curl -s -u admin:"$GF_SECURITY_ADMIN_PASSWORD" -H 'Content-Type: application/json' \ curl -s -u admin:"$GF_SECURITY_ADMIN_PASSWORD" -H 'Content-Type: application/json' \
-X POST http://127.0.0.1:3000/api/admin/users \ -X POST http://127.0.0.1:3000/api/admin/users \
-d '{"name":"you","login":"you@gmail.com","email":"you@gmail.com","password":"'"$(openssl rand -base64 24)"'"}' -d '{"name":"you","login":"you@gmail.com","email":"you@gmail.com","password":"'"$(openssl rand -base64 24)"'"}'
@ -97,28 +118,50 @@ docker exec -u git <forgejo-container> forgejo admin auth add-oauth \
--auto-discover-url https://accounts.google.com/.well-known/openid-configuration --auto-discover-url https://accounts.google.com/.well-known/openid-configuration
``` ```
### 2. Alloy agent (on every node — prod, beta, desktop) ### 2. Alloy agent (on vps1 and vps2)
`ALLOY_NODE` names the MACHINE now, not the environment — vps2 alone runs two
environments (prod and beta), so a node name can no longer stand in for a
`host` value. `ALLOY_ENV` supplies the default environment label for anything
on that node not attributable to a specific stack (Forgejo, Grafana, the
portfolio site). `APPLOGS_VOLUME` is the node's primary `applogs` volume; vps2
additionally needs `BETA_APPLOGS_VOLUME` and a second compose file, since it
carries both environments' structured logs side by side.
Every node pushes to Loki over the mesh IP — a containerized agent can't reach
the host's `127.0.0.1`, but it can reach vps1's wg0 address at
`10.10.0.2:3100`.
```bash ```bash
cd thermograph-observability/alloy cd thermograph-observability/alloy
# Every node (beta included) pushes to Loki over the mesh IP — a containerized
# agent can't reach the host's 127.0.0.1, but it can reach beta's wg0 address: # vps1 (dev + Forgejo + the monitoring stack itself):
# beta: LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push ALLOY_NODE=beta ALLOY_NODE=vps1 ALLOY_ENV=dev APPLOGS_VOLUME=thermograph-dev_applogs \
# prod: LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push ALLOY_NODE=prod LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push \
# desktop: LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push ALLOY_NODE=dev
ALLOY_NODE=prod LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push \
docker compose -f docker-compose.agent.yml up -d docker compose -f docker-compose.agent.yml up -d
# vps2 (prod + beta) — note the SECOND file, adding beta's log volume:
ALLOY_NODE=vps2 ALLOY_ENV=prod APPLOGS_VOLUME=thermograph_applogs \
BETA_APPLOGS_VOLUME=thermograph-beta_applogs \
LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push \
docker compose -f docker-compose.agent.yml -f docker-compose.agent.beta.yml up -d
``` ```
On the LAN dev node the app's log volume is `thermograph-dev_applogs` (not The mesh must already be up (see the main repo's `deploy/swarm/` / `INFRA.md`);
`thermograph_applogs`) — edit the two references in `docker-compose.agent.yml` Swarm is **not** required for the agent itself — it talks plain HTTP over wg0.
there. The mesh must already be up (see the main repo's `deploy/swarm/` /
`INFRA.md`); Swarm is **not** required — the agents talk plain HTTP over wg0. <!-- TODO(cutover): desktop no longer runs a Thermograph app stack and ships no
app logs, but it isn't clear whether it still runs an Alloy agent at all
(for its AI-model / flex-worker container logs) or has been dropped from
the fleet entirely. `ALLOY_NODE=desktop` is still a valid value in
alloy/config.alloy and docker-compose.agent.yml, but no desktop deploy
command is given here — confirm and fill in once decided. -->
## Verify ## Verify
```bash ```bash
# From any mesh node, confirm Loki is receiving from every host: # From any mesh node, confirm Loki is receiving from every environment
# (host = prod|beta|dev; a separate node/vps1|vps2|desktop label exists too):
curl -s 'http://10.10.0.2:3100/loki/api/v1/label/host/values' # -> {"data":["beta","dev","prod"]} curl -s 'http://10.10.0.2:3100/loki/api/v1/label/host/values' # -> {"data":["beta","dev","prod"]}
# Then open Grafana → the "Thermograph — Fleet Logs" dashboard. # Then open Grafana → the "Thermograph — Fleet Logs" dashboard.
``` ```
@ -130,6 +173,13 @@ log volume by service, app error rate by tag, upstream 429 count, Caddy 5xx
count, a per-node notifier-liveness indicator (from the heartbeat log), recent app errors, and a live all-container count, a per-node notifier-liveness indicator (from the heartbeat log), recent app errors, and a live all-container
tail. Edit it in Grafana and re-export the JSON here to version a change. tail. Edit it in Grafana and re-export the JSON here to version a change.
<!-- TODO(cutover): the dashboard JSON isn't in this pass's scope, but check
whether its "Node" selector still means `node` (the machine) now that the
label has that meaning, or whether some panels actually want to filter by
`host` (the environment) instead — vps2 mixes prod and beta under one
node value, so a machine-only selector can no longer isolate one
environment there. -->
## Alerting ## Alerting
Provisioned from `grafana/provisioning/alerting/`, mounted read-only into Grafana Provisioned from `grafana/provisioning/alerting/`, mounted read-only into Grafana
@ -146,8 +196,8 @@ Discord contact point. Deliberately **not email**:
- Grafana's factory default pointed at the literal string `<example@email.com>` — - Grafana's factory default pointed at the literal string `<example@email.com>` —
a paging path that had never delivered a message to anyone. a paging path that had never delivered a message to anyone.
- beta's Grafana relays SMTP through **prod's** Postfix at `10.10.0.1:25` (see - vps1's Grafana relays SMTP through **vps2's** Postfix at `10.10.0.1:25` (see
`docker-compose.override.yml` on beta). Email alerts therefore travel through `docker-compose.override.yml` on vps1). Email alerts therefore travel through
the box most likely to be on fire, and vanish exactly when they matter. the box most likely to be on fire, and vanish exactly when they matter.
- Discord is off-estate, works when prod is dead, and reaches a phone. - Discord is off-estate, works when prod is dead, and reaches a phone.
@ -157,7 +207,7 @@ Discord contact point. Deliberately **not email**:
`#notes` / `#staff-chat` are for humans. `#notes` / `#staff-chat` are for humans.
The webhook URL is a secret and is **not in this repo**: it comes from The webhook URL is a secret and is **not in this repo**: it comes from
`DISCORD_ALERT_WEBHOOK_URL` in beta's gitignored `.env`, which compose passes to `DISCORD_ALERT_WEBHOOK_URL` in vps1's gitignored `.env`, which compose passes to
Grafana and which Grafana interpolates when reading the provisioning file. Grafana and which Grafana interpolates when reading the provisioning file.
Grafana refuses to start if it is unset. Grafana refuses to start if it is unset.
@ -187,7 +237,7 @@ value is its absence**: nothing can report its own death, so if `#ops-alerts` ha
been silent for more than a day, alerting is broken rather than the estate been silent for more than a day, alerting is broken rather than the estate
healthy. Set `isPaused: true` on it to opt out of that guarantee. healthy. Set `isPaused: true` on it to opt out of that guarantee.
### Deploying an alerting change to beta ### Deploying an alerting change to vps1
Grafana already mounts `./grafana/provisioning` read-only, so new files under Grafana already mounts `./grafana/provisioning` read-only, so new files under
`alerting/` need no compose change — but the **first** deploy does, because it `alerting/` need no compose change — but the **first** deploy does, because it
@ -196,25 +246,26 @@ new environment variable.
```bash ```bash
# run from the mono-repo root # run from the mono-repo root
BETA="agent@75.119.132.91" VPS1="agent@75.119.132.91" # the monitoring host — NOT the beta environment,
# which now runs on vps2; don't repoint this at it.
SSH="ssh -i ~/.ssh/thermograph_agent_ed25519" SSH="ssh -i ~/.ssh/thermograph_agent_ed25519"
TS=$(date -u +%Y%m%dT%H%M%SZ) TS=$(date -u +%Y%m%dT%H%M%SZ)
# 1. put the secret in beta's .env (once), then confirm it took: # 1. put the secret in vps1's .env (once), then confirm it took:
$SSH $BETA "grep -q '^DISCORD_ALERT_WEBHOOK_URL=' /opt/observability/.env && echo present || echo MISSING" $SSH $VPS1 "grep -q '^DISCORD_ALERT_WEBHOOK_URL=' /opt/observability/.env && echo present || echo MISSING"
# 2. back up the live compose file, stage the new files, move them into place: # 2. back up the live compose file, stage the new files, move them into place:
$SSH $BETA "sudo cp -a /opt/observability/docker-compose.yml /opt/observability/docker-compose.yml.$TS \ $SSH $VPS1 "sudo cp -a /opt/observability/docker-compose.yml /opt/observability/docker-compose.yml.$TS \
&& rm -rf /tmp/obs-stage && mkdir -p /tmp/obs-stage/alerting" && rm -rf /tmp/obs-stage && mkdir -p /tmp/obs-stage/alerting"
scp -i ~/.ssh/thermograph_agent_ed25519 observability/docker-compose.yml $BETA:/tmp/obs-stage/ scp -i ~/.ssh/thermograph_agent_ed25519 observability/docker-compose.yml $VPS1:/tmp/obs-stage/
scp -i ~/.ssh/thermograph_agent_ed25519 observability/grafana/provisioning/alerting/*.yml $BETA:/tmp/obs-stage/alerting/ scp -i ~/.ssh/thermograph_agent_ed25519 observability/grafana/provisioning/alerting/*.yml $VPS1:/tmp/obs-stage/alerting/
$SSH $BETA "sudo mkdir -p /opt/observability/grafana/provisioning/alerting \ $SSH $VPS1 "sudo mkdir -p /opt/observability/grafana/provisioning/alerting \
&& sudo cp /tmp/obs-stage/docker-compose.yml /opt/observability/ \ && sudo cp /tmp/obs-stage/docker-compose.yml /opt/observability/ \
&& sudo cp /tmp/obs-stage/alerting/*.yml /opt/observability/grafana/provisioning/alerting/" && sudo cp /tmp/obs-stage/alerting/*.yml /opt/observability/grafana/provisioning/alerting/"
# 3. recreate Grafana so it reads the new env var AND the new provisioning. # 3. recreate Grafana so it reads the new env var AND the new provisioning.
# `docker restart` is NOT enough here — it will not pick up a new env var. # `docker restart` is NOT enough here — it will not pick up a new env var.
$SSH $BETA "cd /opt/observability && docker compose up -d grafana" $SSH $VPS1 "cd /opt/observability && docker compose up -d grafana"
``` ```
Subsequent rule-only changes are just step 2 plus Subsequent rule-only changes are just step 2 plus
@ -227,13 +278,14 @@ new environment variable is involved.
falls back to its default (email → `<example@email.com>`) root policy — i.e. no falls back to its default (email → `<example@email.com>`) root policy — i.e. no
alerting, which is where this started. alerting, which is where this started.
Alerting lives on **beta only** — prod and dev run Alloy agents, not Grafana, so Alerting lives on **vps1 only** — vps2 (prod + beta) and dev only run Alloy
nothing in `alerting/` ever ships to them. agents, not Grafana, so nothing in `alerting/` ever ships to them, even though
dev happens to share a box with vps1's Grafana.
### Verify alerting ### Verify alerting
```bash ```bash
# on beta, after deploying and restarting Grafana: # on vps1, after deploying and restarting Grafana:
GFP=$(grep '^GF_SECURITY_ADMIN_PASSWORD=' /opt/observability/.env | cut -d= -f2-) GFP=$(grep '^GF_SECURITY_ADMIN_PASSWORD=' /opt/observability/.env | cut -d= -f2-)
WH=$(grep '^DISCORD_ALERT_WEBHOOK_URL=' /opt/observability/.env | cut -d= -f2-) WH=$(grep '^DISCORD_ALERT_WEBHOOK_URL=' /opt/observability/.env | cut -d= -f2-)
# 1. all 12 rules loaded and healthy (health should be "ok", never "error"): # 1. all 12 rules loaded and healthy (health should be "ok", never "error"):
@ -270,7 +322,7 @@ delete a receiver a policy still references). Either use the UI — Alerting →
Contact points → `grafana-default-email` → Delete — or: Contact points → `grafana-default-email` → Delete — or:
```bash ```bash
# on beta. Reads the running Alertmanager config, drops the one receiver, # on vps1. Reads the running Alertmanager config, drops the one receiver,
# writes it back. Verified against Grafana 11.6.1 (returns 202). # writes it back. Verified against Grafana 11.6.1 (returns 202).
GFP=$(grep '^GF_SECURITY_ADMIN_PASSWORD=' /opt/observability/.env | cut -d= -f2-) GFP=$(grep '^GF_SECURITY_ADMIN_PASSWORD=' /opt/observability/.env | cut -d= -f2-)
AM=localhost:3000/api/alertmanager/grafana/config/api/v1/alerts AM=localhost:3000/api/alertmanager/grafana/config/api/v1/alerts
@ -289,7 +341,7 @@ This is cosmetic — leaving it costs nothing but a confusing row in the UI.
- **Loki is mesh-only** by design; only Grafana (with auth) is public. - **Loki is mesh-only** by design; only Grafana (with auth) is public.
- Alloy reads the Docker socket read-only to discover containers. - Alloy reads the Docker socket read-only to discover containers.
- Retention is 30 days on beta's disk — bump `loki/config.yml` if you want more. - Retention is 30 days on vps1's disk — bump `loki/config.yml` if you want more.
- The app emits a `tag=heartbeat` line for the subscription notifier every - The app emits a `tag=heartbeat` line for the subscription notifier every
~15m; the dashboard's "Notifier alive?" panel turns red if a node misses it. ~15m; the dashboard's "Notifier alive?" panel turns red if a node misses it.
Extend the same `audit.log_heartbeat()` to any future background daemon. Extend the same `audit.log_heartbeat()` to any future background daemon.
@ -319,6 +371,6 @@ This is cosmetic — leaving it costs nothing but a confusing row in the UI.
is for, but it is a *manual* dead-man's switch: it relies on somebody noticing is for, but it is a *manual* dead-man's switch: it relies on somebody noticing
that the daily message stopped. A real external watchdog (an uptime pinger that the daily message stopped. A real external watchdog (an uptime pinger
hitting a Grafana endpoint from off-estate) would close this properly. hitting a Grafana endpoint from off-estate) would close this properly.
- **beta runs a `docker-compose.override.yml` that is not in this repo**, wiring - **vps1 runs a `docker-compose.override.yml` that is not in this repo**, wiring
Grafana's SMTP to prod's Postfix. It should be mirrored into the tracked Grafana's SMTP to vps2's Postfix. It should be mirrored into the tracked
compose file or deleted; alerting no longer depends on it. compose file or deleted; alerting no longer depends on it.

159
observability/alloy/config.alloy
View file

@ -1,15 +1,39 @@
// Grafana Alloy — the log collector that runs on EVERY node (prod, beta, LAN // Grafana Alloy — the log collector that runs on EVERY node (vps1, vps2). It
// dev). It gathers three sources and ships them to the central Loki on beta: // gathers three sources and ships them to the central Loki on vps1:
// //
// 1. Every Docker container's stdout/stderr (app, db, and on beta forgejo) // 1. Every Docker container's stdout/stderr (app, db, and on vps1 forgejo +
// via the Docker socket. // the monitoring stack itself) via the Docker socket.
// 2. Caddy's host access logs (/var/log/caddy/*.log) — the reverse proxy runs // 2. Caddy's host access logs (/var/log/caddy/*.log) — the reverse proxy runs
// on the host, not in a container, so its logs aren't in Docker. // on the host, not in a container, so its logs aren't in Docker.
// 3. The app's structured JSON logs (errors/access/audit *.jsonl) from the // 3. The app's structured JSON logs (errors/access/audit *.jsonl) from the
// `applogs` Docker volume, parsed so `level`/`tag`/`phase` become labels. // `applogs` Docker volumes, parsed so `level`/`tag`/`phase` become labels.
//
// ---------------------------------------------------------------------------
// `host` IS THE ENVIRONMENT, AND IT IS NO LONGER THE NODE
// ---------------------------------------------------------------------------
// Every query, dashboard and alert in this estate slices on host="prod" |
// "beta" | "dev". That label used to be an EXTERNAL label — one value stamped
// on everything the agent shipped — which was exact while one node ran exactly
// one environment.
//
// vps2 now runs prod AND beta. A single external label there would have marked
// every beta container, every beta access log and every beta JSON line as
// `prod`: not a cosmetic problem, but wrong data feeding the prod alert rules,
// and beta's own alerts (NotifierHeartbeatMissingBeta) firing on prod's
// traffic or never firing at all.
//
// So `host` is now derived PER SOURCE — from the container's stack prefix, from
// the access-log filename, from which log volume a JSON line came out of — and
// ALLOY_ENV supplies the fallback for anything on the node that isn't
// environment-specific (Forgejo, Grafana, Loki, the portfolio site). The node
// itself is still labelled, as `node`, because "which machine" is a real
// question too — it is just a different question from "which environment".
// //
// Per-node settings come from the environment (see docker-compose.agent.yml): // Per-node settings come from the environment (see docker-compose.agent.yml):
// ALLOY_NODE — this node's name label (prod | beta | dev) // ALLOY_NODE — the MACHINE this agent runs on (vps1 | vps2 | desktop)
// ALLOY_ENV — the default environment label for logs on this node that
// aren't attributable to a specific stack (vps1 -> dev,
// vps2 -> prod)
// LOKI_URL — where to push (http://10.10.0.2:3100/loki/api/v1/push over wg0) // LOKI_URL — where to push (http://10.10.0.2:3100/loki/api/v1/push over wg0)
livedebugging { enabled = false } livedebugging { enabled = false }
@ -43,9 +67,48 @@ discovery.relabel "containers" {
source_labels = ["__meta_docker_container_label_com_docker_compose_service"] source_labels = ["__meta_docker_container_label_com_docker_compose_service"]
target_label = "service" target_label = "service"
} }
// --- environment attribution -------------------------------------------
// Relabel rules are applied in order and a later write to the same target
// wins, so this is "default, then override with anything more specific".
//
// The regexes are fully anchored (Prometheus relabel semantics), which is
// what keeps `/thermograph_web.1.x` and `/thermograph-beta_beta-web.1.x`
// from matching each other: prod's stack prefix ends in an underscore,
// beta's in `-beta_`.
rule {
source_labels = ["__meta_docker_container_name"]
regex = ".*"
target_label = "host"
replacement = sys.env("ALLOY_ENV")
}
rule {
source_labels = ["__meta_docker_container_name"]
regex = "/thermograph_.*"
target_label = "host"
replacement = "prod"
}
rule {
source_labels = ["__meta_docker_container_name"]
regex = "/thermograph-beta_.*"
target_label = "host"
replacement = "beta"
}
rule {
source_labels = ["__meta_docker_container_name"]
regex = "/thermograph-dev[-_].*"
target_label = "host"
replacement = "dev"
}
// Drop noise/duplicate containers, per environment. Beta's equivalents are
// listed explicitly: its LB is `thermograph-beta-lb` (which does NOT match
// `thermograph-lb.*`) and its worker is `thermograph-beta_beta-worker`, so
// omitting them would have quietly re-admitted exactly the healthz-poll
// noise the prod entries exist to exclude.
rule { rule {
source_labels = ["container"] source_labels = ["container"]
regex = "(alloy|autoscaler|thermograph-test_.*|thermograph-lb|thermograph_worker).*" regex = "(alloy|autoscaler|thermograph-test_.*|thermograph-lb|thermograph-beta-lb|thermograph_worker|thermograph-beta_beta-worker).*"
action = "drop" action = "drop"
} }
} }
@ -94,11 +157,48 @@ loki.process "caddy" {
expression = "(?i)(semrushbot|claudebot|ahrefsbot|yandexbot|bytespider|mj12bot|petalbot)" expression = "(?i)(semrushbot|claudebot|ahrefsbot|yandexbot|bytespider|mj12bot|petalbot)"
drop_counter_reason = "crawler" drop_counter_reason = "crawler"
} }
// Attribute each access log to an environment BY FILENAME. On vps2 one
// Caddy fronts both thermograph.org and beta.thermograph.org, so a single
// node-wide label would file every beta request under prod.
//
// By filename rather than by parsing `request.host` out of the JSON: the
// site block that writes the file is the same place the hostname is
// declared (see infra/deploy/Caddyfile.vps2), one label costs nothing to
// evaluate, and it keeps working for a non-JSON log format.
stage.static_labels {
values = { host = sys.env("ALLOY_ENV") }
}
stage.match {
selector = "{filename=~\".*/beta[.]log\"}"
stage.static_labels {
values = { host = "beta" }
}
}
stage.match {
selector = "{filename=~\".*/thermograph[.]log\"}"
stage.static_labels {
values = { host = "prod" }
}
}
} }
// --- 3. App structured JSON logs (errors / access / audit) ----------------------- // --- 3. App structured JSON logs (errors / access / audit) -----------------------
// Mounted read-only from the app's `applogs` volume at /applogs (see the agent // Mounted read-only from the app's `applogs` volumes (see the agent compose).
// compose). Lift `level`/`tag`/`phase` out of the JSON so they're queryable. // Lift `level`/`tag`/`phase` out of the JSON so they're queryable.
//
// TWO MOUNTS, NOT ONE. Each environment has its own applogs volume, and on vps2
// both exist side by side:
//
// /applogs this node's primary environment (vps2 -> prod, vps1 -> dev)
// /applogs-beta beta's, mounted on vps2 only
//
// A single mount was structurally unable to carry both: whichever volume the
// agent bound, the other environment's structured logs would never reach Loki
// at all — silently, since a missing file source is not an error. Every alert
// built on the JSON stream (the notifier heartbeats in particular) reads this.
local.file_match "app_jsonl" { local.file_match "app_jsonl" {
path_targets = [{ __path__ = "/applogs/**/*.jsonl", job = "app-json" }] path_targets = [{ __path__ = "/applogs/**/*.jsonl", job = "app-json" }]
sync_period = "1m" sync_period = "1m"
@ -122,7 +222,38 @@ loki.process "app_jsonl" {
} }
// A record with no explicit level: an error-folder line is an error, else info. // A record with no explicit level: an error-folder line is an error, else info.
stage.static_labels { stage.static_labels {
values = { source = "app" } values = { source = "app", host = sys.env("ALLOY_ENV") }
}
stage.labels {
values = { level = "", tag = "", phase = "" }
}
}
// Beta's structured logs. Absent on vps1 (the mount simply isn't there), where
// this matches nothing and costs nothing.
local.file_match "app_jsonl_beta" {
path_targets = [{ __path__ = "/applogs-beta/**/*.jsonl", job = "app-json" }]
sync_period = "1m"
}
loki.source.file "app_jsonl_beta" {
targets = local.file_match.app_jsonl_beta.targets
forward_to = [loki.process.app_jsonl_beta.receiver]
file_watch {
min_poll_frequency = "2s"
max_poll_frequency = "10s"
}
}
loki.process "app_jsonl_beta" {
forward_to = [loki.write.central.receiver]
stage.json {
expressions = { level = "level", tag = "tag", phase = "phase", status = "status" }
}
stage.static_labels {
values = { source = "app", host = "beta" }
} }
stage.labels { stage.labels {
values = { level = "", tag = "", phase = "" } values = { level = "", tag = "", phase = "" }
@ -134,7 +265,9 @@ loki.write "central" {
endpoint { endpoint {
url = sys.env("LOKI_URL") url = sys.env("LOKI_URL")
} }
// Every line from this node is stamped with its node name, so one Grafana // `node` — the MACHINE, stamped on everything this agent ships. `host` is
// view can slice prod vs beta vs dev. // deliberately NOT here: it is the ENVIRONMENT, set per source above,
external_labels = { host = sys.env("ALLOY_NODE") } // because vps2 carries two of them and an external label cannot vary per
// stream. Setting host here would override that work on every line.
external_labels = { node = sys.env("ALLOY_NODE") }
} }

24
observability/alloy/docker-compose.agent.beta.yml Normal file
View file

@ -0,0 +1,24 @@
# Overlay for vps2 ONLY — the node that runs two environments.
#
# docker compose -f docker-compose.agent.yml -f docker-compose.agent.beta.yml up -d
#
# Adds beta's applogs volume at /applogs-beta, which config.alloy reads as a
# separate source and labels host="beta". Everything else — the Docker socket,
# the Caddy logs, the primary /applogs mount — comes from the base file.
#
# This is an overlay rather than another variable in the base file because the
# /applogs-beta source is labelled beta UNCONDITIONALLY. On a node where beta
# does not run, there is no correct volume to point it at: aiming it at the
# node's own logs would republish them as beta, manufacturing data for an
# environment that isn't there. "Absent" is the only honest configuration, and
# a missing overlay expresses that exactly.
services:
alloy:
volumes:
- beta_applogs:/applogs-beta:ro
volumes:
beta_applogs:
external: true
name: ${BETA_APPLOGS_VOLUME:?set BETA_APPLOGS_VOLUME (thermograph-beta_applogs)}

53
observability/alloy/docker-compose.agent.yml
View file

@ -1,22 +1,41 @@
# The Alloy log-collector agent. Runs on EACH node (prod, beta, LAN dev) — one # The Alloy log-collector agent. Runs on EACH node (vps1, vps2) — one per node
# per node — reading that node's Docker containers, Caddy logs, and the app's # — reading that node's Docker containers, Caddy logs, and the app's JSON logs,
# JSON logs, and shipping them to the central Loki on beta over the mesh. # and shipping them to the central Loki on vps1 over the mesh.
# #
# Per-node config comes from the environment (an .env file next to this, or # Per-node config comes from the environment (an .env file next to this, or
# exported before `up`): # exported before `up`):
# ALLOY_NODE this node's label: prod | beta | dev # ALLOY_NODE the MACHINE: vps1 | vps2 | desktop
# ALLOY_ENV the default ENVIRONMENT label for logs on this node that
# aren't attributable to a specific stack (Forgejo, Grafana,
# the portfolio site): vps1 -> dev, vps2 -> prod
# APPLOGS_VOLUME this node's primary applogs volume
# LOKI_URL push endpoint. All nodes push over the mesh: # LOKI_URL push endpoint. All nodes push over the mesh:
# http://10.10.0.2:3100/loki/api/v1/push # http://10.10.0.2:3100/loki/api/v1/push
# #
# Deploy on a node: # Deploy on vps1 (dev + Forgejo + the monitoring stack itself):
# ALLOY_NODE=prod LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push \ # ALLOY_NODE=vps1 ALLOY_ENV=dev APPLOGS_VOLUME=thermograph-dev_applogs \
# LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push \
# docker compose -f docker-compose.agent.yml up -d # docker compose -f docker-compose.agent.yml up -d
# #
# The app's JSON logs are read from its `applogs` Docker volume, named # Deploy on vps2 (prod + beta) — note the SECOND file, which adds beta's log
# `<project>_applogs`. prod and beta run the app under the compose project # volume:
# `thermograph`, so the external volume below is `thermograph_applogs`. The LAN # ALLOY_NODE=vps2 ALLOY_ENV=prod APPLOGS_VOLUME=thermograph_applogs \
# dev stack uses the project `thermograph-dev` — on that node, change the two # BETA_APPLOGS_VOLUME=thermograph-beta_applogs \
# `thermograph_applogs` references below to `thermograph-dev_applogs`. # LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push \
# docker compose -f docker-compose.agent.yml -f docker-compose.agent.beta.yml up -d
#
# WHY THE VOLUME NAME IS A VARIABLE NOW, AND WHY BETA IS AN OVERLAY. The name
# used to be the literal `thermograph_applogs`, with a comment telling you to
# hand-edit two lines on the dev node. That was survivable while one node ran
# one environment. vps2 runs two, each with its own applogs volume, and a single
# hardcoded mount there would ship one environment's structured logs and
# silently drop the other's.
#
# Beta's mount is a separate overlay file rather than a second variable in this
# one because the second mount is labelled host="beta" unconditionally. Pointing
# it at dev's volume on vps1 just to satisfy compose would ship every dev log
# line a second time labelled as beta — inventing an environment's worth of
# fake data on a box beta does not run on.
services: services:
alloy: alloy:
@ -27,7 +46,8 @@ services:
- --storage.path=/var/lib/alloy/data - --storage.path=/var/lib/alloy/data
- --server.http.listen-addr=0.0.0.0:12345 - --server.http.listen-addr=0.0.0.0:12345
environment: environment:
ALLOY_NODE: ${ALLOY_NODE:?set ALLOY_NODE (prod|beta|dev)} ALLOY_NODE: ${ALLOY_NODE:?set ALLOY_NODE (vps1|vps2|desktop) — the MACHINE}
ALLOY_ENV: ${ALLOY_ENV:?set ALLOY_ENV (dev|beta|prod) — this node's default environment label}
LOKI_URL: ${LOKI_URL:?set LOKI_URL} LOKI_URL: ${LOKI_URL:?set LOKI_URL}
volumes: volumes:
- ./config.alloy:/etc/alloy/config.alloy:ro - ./config.alloy:/etc/alloy/config.alloy:ro
@ -36,13 +56,16 @@ services:
# Caddy's host access logs (runs on the host, not a container). # Caddy's host access logs (runs on the host, not a container).
- /var/log/caddy:/var/log/caddy:ro - /var/log/caddy:/var/log/caddy:ro
# The app's structured JSON logs, straight off its named volume. # The app's structured JSON logs, straight off its named volume.
- thermograph_applogs:/applogs:ro - applogs:/applogs:ro
- alloy_data:/var/lib/alloy/data - alloy_data:/var/lib/alloy/data
restart: unless-stopped restart: unless-stopped
volumes: volumes:
# The app's existing log volume (created by the app's own stack). external:true # The app's existing log volume (created by the app's own stack). external:true
# means compose references it, never creates or deletes it. # means compose references it, never creates or deletes it. The NAME varies by
thermograph_applogs: # node — see the header — while the mount path stays /applogs, which is what
# lets one config.alloy serve every node.
applogs:
external: true external: true
name: ${APPLOGS_VOLUME:?set APPLOGS_VOLUME (e.g. thermograph_applogs on vps2, thermograph-dev_applogs on vps1)}
alloy_data: {} alloy_data: {}

13
observability/caddy-grafana.conf
View file

@ -1,10 +1,11 @@
# Append to beta's /etc/caddy/Caddyfile (beta is the monitoring host), then # Append to vps1's /etc/caddy/Caddyfile (vps1 is the monitoring host — NOT the
# `systemctl reload caddy`. Grafana publishes to 127.0.0.1:3000; this fronts it # beta environment, which runs on vps2 now), then `systemctl reload caddy`.
# with TLS, same pattern as beta's other site blocks. Login is Google SSO (with a # Grafana publishes to 127.0.0.1:3000; this fronts it with TLS, same pattern as
# break-glass local admin); Loki (the log store, on 10.10.0.2:3100) is NOT proxied # vps1's other site blocks. Login is Google SSO (with a break-glass local
# here and stays mesh-only. # admin); Loki (the log store, on 10.10.0.2:3100) is NOT proxied here and stays
# mesh-only.
# #
# dashboard.thermograph.org's A record must already point at beta (75.119.132.91). # dashboard.thermograph.org's A record must already point at vps1 (75.119.132.91).
dashboard.thermograph.org { dashboard.thermograph.org {
encode zstd gzip encode zstd gzip

21
observability/docker-compose.yml
View file

@ -1,11 +1,14 @@
# The central observability stack: Loki (log store) + Grafana (UI). Runs on ONE # The central observability stack: Loki (log store) + Grafana (UI). Runs on ONE
# node — the monitoring host, `beta` (75.119.132.91 / mesh 10.10.0.2) — because # node — the monitoring host, `vps1` (75.119.132.91 / mesh 10.10.0.2) — because
# beta is an always-on VPS with a public Caddy already fronting it. Every node's # vps1 is an always-on VPS with a public Caddy already fronting it. This is NOT
# Alloy agent (see alloy/) ships logs here; agents on prod and the desktop reach # where the beta *environment* lives (that's vps2 now, alongside prod, as two
# Loki over the WireGuard mesh, so Loki must listen on the mesh interface. # Swarm stacks) — don't let "beta" in older docs send you looking for Grafana
# there. Every node's Alloy agent (see alloy/) ships logs here; agents on vps2
# and desktop reach Loki over the WireGuard mesh, so Loki must listen on the
# mesh interface.
# #
# Deploy (on beta): docker compose up -d # Deploy (on vps1): docker compose up -d
# Grafana is proxied by beta's Caddy at dashboard.thermograph.org (see README); # Grafana is proxied by vps1's Caddy at dashboard.thermograph.org (see README);
# Loki is NOT public — it binds the mesh IP only, reachable to agents over wg0. # Loki is NOT public — it binds the mesh IP only, reachable to agents over wg0.
services: services:
@ -19,7 +22,7 @@ services:
# Mesh-only: agents on prod (10.10.0.1) and desktop (10.10.0.3) push here # Mesh-only: agents on prod (10.10.0.1) and desktop (10.10.0.3) push here
# over wg0. Never published on the public interface. # over wg0. Never published on the public interface.
- "10.10.0.2:3100:3100" - "10.10.0.2:3100:3100"
# Also localhost, so the beta-local Alloy agent and curl checks can reach it. # Also localhost, so the vps1-local Alloy agent and curl checks can reach it.
- "127.0.0.1:3100:3100" - "127.0.0.1:3100:3100"
restart: unless-stopped restart: unless-stopped
@ -61,7 +64,7 @@ services:
# --- Alerting -------------------------------------------------------------- # --- Alerting --------------------------------------------------------------
# The Discord webhook that grafana/provisioning/alerting/contact-points.yml # The Discord webhook that grafana/provisioning/alerting/contact-points.yml
# interpolates as $DISCORD_ALERT_WEBHOOK_URL. It is a secret (holding it is # interpolates as $DISCORD_ALERT_WEBHOOK_URL. It is a secret (holding it is
# enough to post in the channel), so it lives only in beta's .env. # enough to post in the channel), so it lives only in vps1's .env.
# Required, not defaulted: a Grafana that comes up with an empty webhook # Required, not defaulted: a Grafana that comes up with an empty webhook
# looks perfectly healthy and pages nobody, which is the failure mode this # looks perfectly healthy and pages nobody, which is the failure mode this
# whole config exists to end. Better to refuse to start. # whole config exists to end. Better to refuse to start.
@ -71,7 +74,7 @@ services:
- ./grafana/dashboards:/var/lib/grafana/dashboards:ro - ./grafana/dashboards:/var/lib/grafana/dashboards:ro
- grafana_data:/var/lib/grafana - grafana_data:/var/lib/grafana
ports: ports:
# Host-local; beta's Caddy reverse-proxies to it. Not public directly. # Host-local; vps1's Caddy reverse-proxies to it. Not public directly.
- "127.0.0.1:3000:3000" - "127.0.0.1:3000:3000"
restart: unless-stopped restart: unless-stopped

9
observability/grafana/provisioning/alerting/contact-points.yml
View file

@ -3,21 +3,22 @@
# #
# WHY NOT EMAIL. The Grafana factory default routes to the literal string # WHY NOT EMAIL. The Grafana factory default routes to the literal string
# "<example@email.com>", i.e. nowhere. Fixing it by pointing at a real mailbox # "<example@email.com>", i.e. nowhere. Fixing it by pointing at a real mailbox
# would still be wrong: beta's Grafana relays SMTP through prod's Postfix at # would still be wrong: vps1's Grafana (the monitoring host — not the beta
# 10.10.0.1:25 (see the live docker-compose.override.yml on beta), so email # environment, which runs on vps2) relays SMTP through vps2's Postfix at
# 10.10.0.1:25 (see the live docker-compose.override.yml on vps1), so email
# alerts travel *through the box most likely to be on fire* and are silently # alerts travel *through the box most likely to be on fire* and are silently
# lost whenever Postfix is down — which is exactly when you need them. Discord # lost whenever Postfix is down — which is exactly when you need them. Discord
# is off-estate: it works when prod is dead, and it works from a phone. # is off-estate: it works when prod is dead, and it works from a phone.
# #
# THE WEBHOOK URL IS A SECRET. Anyone holding it can post into the channel, so # THE WEBHOOK URL IS A SECRET. Anyone holding it can post into the channel, so
# it is NOT in this repo. It is read from the Grafana process environment, which # it is NOT in this repo. It is read from the Grafana process environment, which
# docker-compose.yml feeds from beta's gitignored .env (see .env.example). # docker-compose.yml feeds from vps1's gitignored .env (see .env.example).
# Grafana expands $VAR / $__env{VAR} when it reads provisioning files; if the # Grafana expands $VAR / $__env{VAR} when it reads provisioning files; if the
# variable is unset the contact point ends up with a literal "$DISCORD_..." # variable is unset the contact point ends up with a literal "$DISCORD_..."
# string and every notification fails — see README "Verify alerting". # string and every notification fails — see README "Verify alerting".
# #
# To rotate: make a new webhook on the #ops-alerts channel, replace the value in # To rotate: make a new webhook on the #ops-alerts channel, replace the value in
# beta's .env, `docker restart observability-grafana-1`, delete the old webhook # vps1's .env, `docker restart observability-grafana-1`, delete the old webhook
# in Discord. # in Discord.
apiVersion: 1 apiVersion: 1

42
observability/grafana/provisioning/alerting/rules-prod.yml
View file

@ -33,7 +33,11 @@
# label — only `container`, which includes a per-task suffix that changes on # label — only `container`, which includes a per-task suffix that changes on
# every redeploy (thermograph_web.1.<taskid>). So prod services are matched as # every redeploy (thermograph_web.1.<taskid>). So prod services are matched as
# container=~"thermograph_<name>.+" with job="docker". Do not use `service=` # container=~"thermograph_<name>.+" with job="docker". Do not use `service=`
# here; it only works on the compose hosts (beta, dev). # here; it only works on dev, now the ONLY compose host left — beta is Swarm
# too (stack `thermograph-beta`, since both prod and beta run on vps2), so a
# beta-targeted rule needs its own regex against beta's prefixed container
# names (thermograph-beta_beta-web.1.<taskid> etc.), not a bare `service=`
# match either.
# #
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# NOT ALERTABLE, ON PURPOSE # NOT ALERTABLE, ON PURPOSE
@ -69,6 +73,10 @@ groups:
# DATA: 15-minute windows over 24h ranged 35 -> 301 lines, minimum 35, # DATA: 15-minute windows over 24h ranged 35 -> 301 lines, minimum 35,
# and never once reached zero — including *during* the outage (72), when # and never once reached zero — including *during* the outage (72), when
# Caddy was happily logging 503s. Zero is unambiguous. # Caddy was happily logging 503s. Zero is unambiguous.
#
# The description names vps1, not "beta": the mesh path that matters here
# is the one to the MONITORING host, which is vps1. Beta is on vps2 and
# has nothing to do with whether these logs arrive.
- uid: tg_prod_edge_dark - uid: tg_prod_edge_dark
title: ProdEdgeDark title: ProdEdgeDark
condition: C condition: C
@ -81,7 +89,8 @@ groups:
Nothing has reached thermograph.org's reverse proxy in 15 minutes Nothing has reached thermograph.org's reverse proxy in 15 minutes
(healthy floor is 35 lines/15m, and it stayed at 72 even during the (healthy floor is 35 lines/15m, and it stayed at 72 even during the
04:20Z outage). Either prod is down, Caddy is down, prod's Alloy 04:20Z outage). Either prod is down, Caddy is down, prod's Alloy
agent stopped shipping, or the WireGuard mesh to beta is broken. agent stopped shipping, or the WireGuard mesh to vps1 (where Loki
runs) is broken.
This rule also fires on Loki NoData/errors, so it doubles as the This rule also fires on Loki NoData/errors, so it doubles as the
"we have lost observability" alarm — if it is the only thing firing, "we have lost observability" alarm — if it is the only thing firing,
suspect the pipeline before suspecting the app. suspect the pipeline before suspecting the app.
@ -397,9 +406,10 @@ groups:
# means two consecutive empty evaluations, so roughly 25 minutes of real # means two consecutive empty evaluations, so roughly 25 minutes of real
# silence before it pages — about one and a half missed beats. # silence before it pages — about one and a half missed beats.
# #
# dev is deliberately excluded: the LAN box had zero heartbeats for the # dev is deliberately excluded: the dev box (mesh-only, on vps1) had zero
# first 10.5h of the backtest window because it was simply off. Alerting # heartbeats for the first 10.5h of the backtest window because it was
# on dev would mean an alert that is firing more often than not. # simply off. Alerting on dev would mean an alert that is firing more
# often than not.
# #
# MUST stay filtered to daemon="subscription-notifier". The selector was # MUST stay filtered to daemon="subscription-notifier". The selector was
# originally bare `tag="heartbeat"`, which was unambiguous while the # originally bare `tag="heartbeat"`, which was unambiguous while the
@ -833,10 +843,10 @@ groups:
# -- 12 ---------------------------------------------------------------- # -- 12 ----------------------------------------------------------------
# Nothing in this file can tell you that Grafana itself has stopped, or # Nothing in this file can tell you that Grafana itself has stopped, or
# that beta is down, or that the Discord webhook was revoked — a monitoring # that vps1 (the monitoring host) is down, or that the Discord webhook was
# system cannot report its own death. So this rule fires permanently and # revoked — a monitoring system cannot report its own death. So this rule
# by design, throttled by the notification policy to exactly one Discord # fires permanently and by design, throttled by the notification policy
# message per 24 hours. # to exactly one Discord message per 24 hours.
# #
# HOW TO USE IT: if #ops-alerts has been silent for more than a day, the # HOW TO USE IT: if #ops-alerts has been silent for more than a day, the
# alerting pipeline is broken, not the estate healthy. That one daily # alerting pipeline is broken, not the estate healthy. That one daily
@ -844,6 +854,7 @@ groups:
# rules are still capable of reaching you. # rules are still capable of reaching you.
# #
# To silence it (and lose that guarantee), set isPaused: true. # To silence it (and lose that guarantee), set isPaused: true.
#
- uid: tg_alerting_watchdog - uid: tg_alerting_watchdog
title: AlertingWatchdog title: AlertingWatchdog
condition: C condition: C
@ -854,13 +865,20 @@ groups:
summary: 'Alerting pipeline is alive (daily heartbeat — not a fault)' summary: 'Alerting pipeline is alive (daily heartbeat — not a fault)'
description: >- description: >-
This is the dead-man's switch and it is supposed to be firing. It This is the dead-man's switch and it is supposed to be firing. It
proves Grafana on beta is evaluating rules, Loki is answering, and proves Grafana on vps1 is evaluating rules, Loki is answering, and
the Discord webhook still works. Seeing it once a day is correct. the Discord webhook still works. Seeing it once a day is correct.
NOT seeing it for over a day means alerting itself is down and no NOT seeing it for over a day means alerting itself is down and no
other rule can reach you — check observability-grafana-1 on beta. other rule can reach you — check observability-grafana-1 on vps1.
labels: labels:
severity: watchdog severity: watchdog
host: beta # `node`, not `host`: every other rule's `host` names an ENVIRONMENT
# (prod/beta/dev), and this rule is about the alerting pipeline
# itself, which is a machine. It used to be filed `host: beta` back
# when the monitoring box and the beta environment were the same
# machine — they are not any more, and that label would now read as a
# claim about beta. Routing is unaffected: notification-policies.yml
# matches on `severity` only and uses `host` just for grouping.
node: vps1
class: meta class: meta
isPaused: false isPaused: false
data: data:

11
observability/loki/config.yml
View file

@ -1,8 +1,9 @@
# Loki — the single log store for the whole Thermograph fleet (prod, beta, LAN # Loki — the single log store for the whole Thermograph fleet: prod and beta
# dev). Every node's Alloy agent pushes here over the WireGuard mesh; Grafana # (both on vps2, as separate Swarm stacks) and dev (vps1, mesh-only, its own
# reads from here. Single-binary, filesystem-backed: no object storage, no # compose project). Every node's Alloy agent pushes here over the WireGuard
# clustering — right for a three-node hobby fleet, and everything lives on one # mesh; Grafana reads from here. Single-binary, filesystem-backed: no object
# volume you can back up or blow away. # storage, no clustering — right for a three-node hobby fleet, and everything
# lives on one volume you can back up or blow away.
auth_enabled: false auth_enabled: false
server: server: