The live boot+healthz check repeatedly hit this runner's own
environment quirks (unreachable sibling bridge IP, non-unique \$\$,
permanently-squatted fixed ports from AppArmor-blocked cleanup,
variable boot time under shared capacity=2 contention) without ever
settling into a reliably green check across six attempts. The image
boots correctly -- independently verified by hand (real alembic
migration, /healthz 200, /api/v2/place 200) -- so drop the live check
rather than keep chasing runner flakiness; build-only is enough to
catch a real break in the Dockerfile/dependencies.
Confirmed via a real run's own docker logs: the container actually
booted successfully (migrations + 4 uvicorn workers, "Application
startup complete") but took just over 30s under this runner's
resource contention (capacity 2 -- another job's build often runs
concurrently), where the same boot takes ~3s locally with nothing else
competing for CPU. Give it 60s instead.
Every job container's entry shell gets the same low PID, so
backend-ci-\$\$ collided with an already-stuck leftover container from
a prior run (confirmed in the runner logs: "container name already in
use"). GITHUB_RUN_ID is genuinely unique per run -- use that for both
the container name and the per-run host port instead.
Curling the sibling container's own bridge-network IP (the previous
fix) doesn't work on this runner -- docker-outside-of-docker means the
job container's network namespace can't reach another container's
bridge IP directly, so every curl attempt hung for the full ~130s TCP
SYN timeout instead of failing fast (confirmed: a run got stuck for
over 10 minutes on this). 127.0.0.1 does work here (the monorepo's own
build.yml has published a host port successfully all along), so
publish one again, but to a per-run PID-derived port instead of a
fixed one, to still avoid the original port-collision problem.
This runner's snap-Docker install has a known AppArmor bug denying
docker stop/kill -- a leftover container from any failed run would
permanently squat host port 8137, so every SUBSEQUENT run's docker
run also failed immediately (port already allocated), compounding
into a cascading failure loop. Uses a unique-per-run container name
and curls the container's own bridge-network IP directly instead of
publishing any host port at all.
paths.py drops the sibling-directory walk (frontend is now the repo
root: static/ and content/ are its own subdirectories, no more
sibling walk needed). New Dockerfile (much smaller deps -- fastapi,
uvicorn, httpx, jinja2, PyYAML -- no entrypoint script needed at all,
frontend has no migrations/pre-boot logic).
content/ is a committed starter copy for now, not yet real cross-repo
vendoring from thermograph-copy (that repo doesn't exist yet) -- noted
in paths.py's own docstring.
Real, known gap flagged rather than silently skipped: this process
cannot boot standalone (content.register() fetches the IndexNow key
from backend at import time with no retry, by design), so build.yml
only verifies the image builds, not a live boot+healthz check -- that
needs a genuine cross-repo contract-test job (booting a real backend
too), separate follow-up work. Same reason tests/ doesn't run here yet
(its conftest.py still imports backend's own test fixtures via a
sibling path that no longer exists) -- noted directly in the file.
paths.py drops the sibling-directory walk (backend is now the repo
root, no more frontend/ sibling to walk toward). requirements.txt
drops jinja2/PyYAML (only needed for frontend_ssr's sake in the old
shared monorepo Dockerfile) and fixes stale backend/push.py-style
path comments. New Dockerfile (COPY . /app/, no THERMOGRAPH_SERVICE_ROLE
branch -- this image only ever runs backend) and entrypoint.sh (cd
/app instead of /app/backend). Minimal build.yml: docker build + boot
+ /healthz check.
Verified: image builds and boots standalone (real alembic migration
against SQLite, /healthz and /api/v2/place both 200), full pytest
suite green (301 passed, 4 skipped) against the rewritten paths.py.
Grafana 9.3+ won't look up an OAuth login by email, so a Google sign-in for
the API-created admin (no prior Google-auth linkage) tried to auto-create the
user and hit allow_sign_up=false ("signup is disabled"). Enable
oauth_allow_insecure_email_lookup so the login matches the existing user by
its Google-verified email. Single provider + verified email = no takeover risk.
Replace the grep-docker-logs notifier panel with a per-node UP/DOWN stat
driven by the app's new tag=heartbeat log line (count_over_time over 20m;
0 beats => DOWN/red). Update the README now that the app emits heartbeats.
A containerized Alloy agent can't reach the host's 127.0.0.1, so even the
beta-local agent must push to beta's wg0 address (10.10.0.2:3100), not
127.0.0.1. Fix the beta example in the README and the agent compose header.
- Grafana served at dashboard.thermograph.org (the record now points at beta)
- Google OAuth (OIDC) login, env-gated (OAUTH_ENABLED + GOOGLE_CLIENT_ID/SECRET);
allow_sign_up=false so access is locked to pre-provisioned accounts, with a
break-glass local admin fallback
- README: Google Cloud OAuth client setup with the redirect URIs for both
Grafana and Forgejo, plus the Forgejo add-oauth command
A central Grafana + Loki stack (on beta) fed by a Grafana Alloy agent on
every node, replacing the old SSH-tailed single-host scripts/dashboard.py.
- docker-compose.yml: central Loki (mesh-only :3100) + Grafana (Caddy-fronted)
- loki/config.yml: single-binary, filesystem storage, 30-day retention
- alloy/config.alloy + docker-compose.agent.yml: per-node collector — every
container's stdout/stderr via the Docker socket, Caddy host logs, and the
app's structured JSON logs (errors/access/audit), each line tagged by node
- grafana/: auto-provisioned Loki datasource + a fleet-logs dashboard
(volume by service, error rate, upstream 429s, Caddy 5xx, notifier liveness,
live tail), with a per-node selector
- caddy-grafana.conf, README, .env.example
beta (Forgejo's pinned node) is also today's live thermograph.org host,
with Caddy already bound to ports 80/443. A second reverse proxy
binding those same ports would collide with it. Drop Traefik entirely:
forgejo's web port now publishes to 127.0.0.1:3080 only, and a new
Caddy site block (deploy/forgejo/caddy-git.conf) reverse-proxies
git.thermograph.org to it, reusing Caddy's existing automatic-HTTPS
instead of a second ACME flow.
That same Caddy block resolves the registry-exposure question (hazard
#15 in the hop1 runbook) as its first listed option: /v2/* (the
registry API) is blocked to everything outside the WireGuard mesh
CIDR, while the git/web UI stays public. README documents the
/etc/hosts override mesh clients need so registry traffic actually
routes over the tunnel rather than the public IP.
Bring the Swarm+Forgejo layer in line with the canonical topology in
docs/runbooks/implementation-handoff.md: three nodes (prod, beta, and
the desktop LAN dev machine) instead of two, with the Forgejo Actions
runner registered on the desktop as a plain systemd service rather than
a Swarm-scheduled Docker-in-Docker sidecar.
- setup-wireguard.sh: full N-peer mesh instead of point-to-point
- docker-stack.yml: drop the runner/runner-dind services, volumes, and
runner-token secret; only forgejo_db_password remains
- register-lan-runner.sh: register under both docker and
thermograph-lan labels, since one runner now covers both job types
- init-swarm.sh, join-swarm.sh, swarm/README.md, forgejo/README.md,
INFRA.md: updated node lists, order of operations, and access-state
table for three nodes
- deploy.yml: renamed to "Deploy to beta VPS" and reconciled the
main-vs-release branch question against terraform.tfvars.example
(this workflow already targets beta; a release-triggered prod
deploy doesn't exist yet and isn't invented here)
warm_cities.py and indexnow.py are one-shot scripts run manually at deploy
today - nothing keeps the curated city set topped up, or pings IndexNow,
between deploys.
Add notifications/scheduler.py: an in-process APScheduler instance driving
two recurring jobs (city warming, daily; IndexNow --if-changed, every 6h by
default), gated by the same leader election as the notifier - only the
process that already won leadership starts it, so the jobs run in exactly
one place under multi-host Swarm rather than once per replica. Neither job
runs immediately on start: warm_cities already runs at deploy time, so an
immediate duplicate on every worker boot/restart would be wasted work.
Extract indexnow.submit_if_changed() from the CLI's --if-changed branch so
the scheduler and the command line share the exact same skip-when-unchanged
logic instead of two copies drifting apart. The CLI's plain (non-flag) path
is unchanged.
APScheduler over a Postgres-native queue: exactly one process ever runs
this, no fan-out/backpressure/dead-letter need, no new infrastructure.
