This runner's snap-Docker install has a known AppArmor bug denying
docker stop/kill -- a leftover container from any failed run would
permanently squat host port 8137, so every SUBSEQUENT run's docker
run also failed immediately (port already allocated), compounding
into a cascading failure loop. Uses a unique-per-run container name
and curls the container's own bridge-network IP directly instead of
publishing any host port at all.
paths.py drops the sibling-directory walk (frontend is now the repo
root: static/ and content/ are its own subdirectories, no more
sibling walk needed). New Dockerfile (much smaller deps -- fastapi,
uvicorn, httpx, jinja2, PyYAML -- no entrypoint script needed at all,
frontend has no migrations/pre-boot logic).
content/ is a committed starter copy for now, not yet real cross-repo
vendoring from thermograph-copy (that repo doesn't exist yet) -- noted
in paths.py's own docstring.
Real, known gap flagged rather than silently skipped: this process
cannot boot standalone (content.register() fetches the IndexNow key
from backend at import time with no retry, by design), so build.yml
only verifies the image builds, not a live boot+healthz check -- that
needs a genuine cross-repo contract-test job (booting a real backend
too), separate follow-up work. Same reason tests/ doesn't run here yet
(its conftest.py still imports backend's own test fixtures via a
sibling path that no longer exists) -- noted directly in the file.
paths.py drops the sibling-directory walk (backend is now the repo
root, no more frontend/ sibling to walk toward). requirements.txt
drops jinja2/PyYAML (only needed for frontend_ssr's sake in the old
shared monorepo Dockerfile) and fixes stale backend/push.py-style
path comments. New Dockerfile (COPY . /app/, no THERMOGRAPH_SERVICE_ROLE
branch -- this image only ever runs backend) and entrypoint.sh (cd
/app instead of /app/backend). Minimal build.yml: docker build + boot
+ /healthz check.
Verified: image builds and boots standalone (real alembic migration
against SQLite, /healthz and /api/v2/place both 200), full pytest
suite green (301 passed, 4 skipped) against the rewritten paths.py.
Grafana 9.3+ won't look up an OAuth login by email, so a Google sign-in for
the API-created admin (no prior Google-auth linkage) tried to auto-create the
user and hit allow_sign_up=false ("signup is disabled"). Enable
oauth_allow_insecure_email_lookup so the login matches the existing user by
its Google-verified email. Single provider + verified email = no takeover risk.
Replace the grep-docker-logs notifier panel with a per-node UP/DOWN stat
driven by the app's new tag=heartbeat log line (count_over_time over 20m;
0 beats => DOWN/red). Update the README now that the app emits heartbeats.
A containerized Alloy agent can't reach the host's 127.0.0.1, so even the
beta-local agent must push to beta's wg0 address (10.10.0.2:3100), not
127.0.0.1. Fix the beta example in the README and the agent compose header.
- Grafana served at dashboard.thermograph.org (the record now points at beta)
- Google OAuth (OIDC) login, env-gated (OAUTH_ENABLED + GOOGLE_CLIENT_ID/SECRET);
allow_sign_up=false so access is locked to pre-provisioned accounts, with a
break-glass local admin fallback
- README: Google Cloud OAuth client setup with the redirect URIs for both
Grafana and Forgejo, plus the Forgejo add-oauth command
A central Grafana + Loki stack (on beta) fed by a Grafana Alloy agent on
every node, replacing the old SSH-tailed single-host scripts/dashboard.py.
- docker-compose.yml: central Loki (mesh-only :3100) + Grafana (Caddy-fronted)
- loki/config.yml: single-binary, filesystem storage, 30-day retention
- alloy/config.alloy + docker-compose.agent.yml: per-node collector — every
container's stdout/stderr via the Docker socket, Caddy host logs, and the
app's structured JSON logs (errors/access/audit), each line tagged by node
- grafana/: auto-provisioned Loki datasource + a fleet-logs dashboard
(volume by service, error rate, upstream 429s, Caddy 5xx, notifier liveness,
live tail), with a per-node selector
- caddy-grafana.conf, README, .env.example
beta (Forgejo's pinned node) is also today's live thermograph.org host,
with Caddy already bound to ports 80/443. A second reverse proxy
binding those same ports would collide with it. Drop Traefik entirely:
forgejo's web port now publishes to 127.0.0.1:3080 only, and a new
Caddy site block (deploy/forgejo/caddy-git.conf) reverse-proxies
git.thermograph.org to it, reusing Caddy's existing automatic-HTTPS
instead of a second ACME flow.
That same Caddy block resolves the registry-exposure question (hazard
#15 in the hop1 runbook) as its first listed option: /v2/* (the
registry API) is blocked to everything outside the WireGuard mesh
CIDR, while the git/web UI stays public. README documents the
/etc/hosts override mesh clients need so registry traffic actually
routes over the tunnel rather than the public IP.
Bring the Swarm+Forgejo layer in line with the canonical topology in
docs/runbooks/implementation-handoff.md: three nodes (prod, beta, and
the desktop LAN dev machine) instead of two, with the Forgejo Actions
runner registered on the desktop as a plain systemd service rather than
a Swarm-scheduled Docker-in-Docker sidecar.
- setup-wireguard.sh: full N-peer mesh instead of point-to-point
- docker-stack.yml: drop the runner/runner-dind services, volumes, and
runner-token secret; only forgejo_db_password remains
- register-lan-runner.sh: register under both docker and
thermograph-lan labels, since one runner now covers both job types
- init-swarm.sh, join-swarm.sh, swarm/README.md, forgejo/README.md,
INFRA.md: updated node lists, order of operations, and access-state
table for three nodes
- deploy.yml: renamed to "Deploy to beta VPS" and reconciled the
main-vs-release branch question against terraform.tfvars.example
(this workflow already targets beta; a release-triggered prod
deploy doesn't exist yet and isn't invented here)
warm_cities.py and indexnow.py are one-shot scripts run manually at deploy
today - nothing keeps the curated city set topped up, or pings IndexNow,
between deploys.
Add notifications/scheduler.py: an in-process APScheduler instance driving
two recurring jobs (city warming, daily; IndexNow --if-changed, every 6h by
default), gated by the same leader election as the notifier - only the
process that already won leadership starts it, so the jobs run in exactly
one place under multi-host Swarm rather than once per replica. Neither job
runs immediately on start: warm_cities already runs at deploy time, so an
immediate duplicate on every worker boot/restart would be wasted work.
Extract indexnow.submit_if_changed() from the CLI's --if-changed branch so
the scheduler and the command line share the exact same skip-when-unchanged
logic instead of two copies drifting apart. The CLI's plain (non-flag) path
is unchanged.
APScheduler over a Postgres-native queue: exactly one process ever runs
this, no fan-out/backpressure/dead-letter need, no new infrastructure.
warm_cities.py and indexnow.py are one-shot scripts run manually at deploy
today - nothing keeps the curated city set topped up, or pings IndexNow,
between deploys.
Add notifications/scheduler.py: an in-process APScheduler instance driving
two recurring jobs (city warming, daily; IndexNow --if-changed, every 6h by
default), gated by the same leader election as the notifier - only the
process that already won leadership starts it, so the jobs run in exactly
one place under multi-host Swarm rather than once per replica. Neither job
runs immediately on start: warm_cities already runs at deploy time, so an
immediate duplicate on every worker boot/restart would be wasted work.
Extract indexnow.submit_if_changed() from the CLI's --if-changed branch so
the scheduler and the command line share the exact same skip-when-unchanged
logic instead of two copies drifting apart. The CLI's plain (non-flag) path
is unchanged.
APScheduler over a Postgres-native queue: exactly one process ever runs
this, no fan-out/backpressure/dead-letter need, no new infrastructure.
* Add a Postgres advisory-lock leader election for multi-host deploys
The subscription notifier elects one leader via a host-local flock
(THERMOGRAPH_SINGLETON_LOCK) so multiple uvicorn workers on one host don't
each run it. Under multi-host Swarm that guard is insufficient: each host
would independently elect its own leader, multiplying Open-Meteo quota use
N-fold again.
Add claim_pg(key) alongside the existing claim(lock_path): a cluster-wide
Postgres advisory lock, visible to every host talking to the same database.
The holding connection is dedicated and kept for the process lifetime
(advisory locks are session-scoped); a dead connection is dropped and
re-election retried on the next call.
claim_leader() dispatches between the two mechanisms from env:
THERMOGRAPH_SINGLETON_PG (+ Postgres) -> claim_pg; else
THERMOGRAPH_SINGLETON_LOCK -> claim; else always leader, unchanged. Wired in
at web/app.py in place of the direct claim() call. Off by default, so
today's single-host behavior is unaffected.
* Split web/worker duties with THERMOGRAPH_ROLE
Background work (the subscription notifier) is welded to the same process
that serves requests, so scaling the web tier to N replicas would also scale
notifier instances unless something restricts it further than leader
election alone.
Add THERMOGRAPH_ROLE (web|worker|all, default all - unchanged single-process
behavior). Every replica runs the same image; ROLE only gates whether a
process is allowed to own the notifier at all, layered on top of the
existing leader election: web replicas never start it even if they'd win
leader election, worker replicas start it if they win. The decision is
pulled into _should_run_notifier() so it's unit-testable without booting the
full app (DB init, places index, neighbor warmer).
Add a minimal /healthz liveness route (no DB/upstream I/O, not under BASE)
so a worker replica - which serves no real traffic - still has something
Swarm can health-check.
* Retrigger CI (no prior check run was ever recorded for this PR)
* Add a Postgres advisory-lock leader election for multi-host deploys
The subscription notifier elects one leader via a host-local flock
(THERMOGRAPH_SINGLETON_LOCK) so multiple uvicorn workers on one host don't
each run it. Under multi-host Swarm that guard is insufficient: each host
would independently elect its own leader, multiplying Open-Meteo quota use
N-fold again.
Add claim_pg(key) alongside the existing claim(lock_path): a cluster-wide
Postgres advisory lock, visible to every host talking to the same database.
The holding connection is dedicated and kept for the process lifetime
(advisory locks are session-scoped); a dead connection is dropped and
re-election retried on the next call.
claim_leader() dispatches between the two mechanisms from env:
THERMOGRAPH_SINGLETON_PG (+ Postgres) -> claim_pg; else
THERMOGRAPH_SINGLETON_LOCK -> claim; else always leader, unchanged. Wired in
at web/app.py in place of the direct claim() call. Off by default, so
today's single-host behavior is unaffected.
* Split web/worker duties with THERMOGRAPH_ROLE
Background work (the subscription notifier) is welded to the same process
that serves requests, so scaling the web tier to N replicas would also scale
notifier instances unless something restricts it further than leader
election alone.
Add THERMOGRAPH_ROLE (web|worker|all, default all - unchanged single-process
behavior). Every replica runs the same image; ROLE only gates whether a
process is allowed to own the notifier at all, layered on top of the
existing leader election: web replicas never start it even if they'd win
leader election, worker replicas start it if they win. The decision is
pulled into _should_run_notifier() so it's unit-testable without booting the
full app (DB init, places index, neighbor warmer).
Add a minimal /healthz liveness route (no DB/upstream I/O, not under BASE)
so a worker replica - which serves no real traffic - still has something
Swarm can health-check.
* Retrigger CI (no prior check run was ever recorded for this PR)
The weather-terms glossary (9 entries) and four static pages' SEO title/
description were hardcoded directly in web/content.py - a 1019-line module
that also owns all SSR rendering logic - mixed in with code that changes on
a completely different cadence and for different reasons.
Add content/glossary.yaml and content/pages.yaml (a new repo-root content/
tree, a sibling of backend/ and frontend/ paths.py resolves the same way -
the seed of a future thermograph-copy repo per the architecture decision
doc's own §4) plus web/content_loader.py: a small loader that validates each
file's shape at load time (required fields present and non-empty, no
duplicate glossary slugs) and fails loudly on a malformed edit rather than
rendering a blank glossary card or an empty <title>. content.py's GLOSSARY
dict and the about/privacy/hub/glossary_index page_title/description
literals now come from the loader.
Scope: only content that is genuinely pure static data with no embedded
template logic. cities_flavor.json (Wikipedia extracts, already its own
generated file) and the homepage's title/description (embedded in
home.html.j2 as Jinja block overrides, a heavily-tested product-critical
template) are deliberately left as they are - a future pass, not required
for this one. UI microcopy bound to frontend logic stays in frontend/,
per the doc's own line between content and frontend.
Verified: content/glossary.yaml generated programmatically from the live
GLOSSARY dict (not hand-transcribed) and round-tripped byte-for-byte
identical against it; content/pages.yaml's four entries checked field-by-
field against the original hardcoded strings. Full backend suite green (362
passed, 4 skipped) with zero existing test changes needed beyond one
assertion made escaping-aware (a pre-existing Jinja double-escape quirk on
the one title containing "&", intentionally preserved not fixed). Built
and booted the real Docker image: content/ present at /app/content, and
curled /glossary, /glossary/percentine, /about, /privacy from inside the
running container - all four render with the exact expected title text.
The weather-terms glossary (9 entries) and four static pages' SEO title/
description were hardcoded directly in web/content.py - a 1019-line module
that also owns all SSR rendering logic - mixed in with code that changes on
a completely different cadence and for different reasons.
Add content/glossary.yaml and content/pages.yaml (a new repo-root content/
tree, a sibling of backend/ and frontend/ paths.py resolves the same way -
the seed of a future thermograph-copy repo per the architecture decision
doc's own §4) plus web/content_loader.py: a small loader that validates each
file's shape at load time (required fields present and non-empty, no
duplicate glossary slugs) and fails loudly on a malformed edit rather than
rendering a blank glossary card or an empty <title>. content.py's GLOSSARY
dict and the about/privacy/hub/glossary_index page_title/description
literals now come from the loader.
Scope: only content that is genuinely pure static data with no embedded
template logic. cities_flavor.json (Wikipedia extracts, already its own
generated file) and the homepage's title/description (embedded in
home.html.j2 as Jinja block overrides, a heavily-tested product-critical
template) are deliberately left as they are - a future pass, not required
for this one. UI microcopy bound to frontend logic stays in frontend/,
per the doc's own line between content and frontend.
Verified: content/glossary.yaml generated programmatically from the live
GLOSSARY dict (not hand-transcribed) and round-tripped byte-for-byte
identical against it; content/pages.yaml's four entries checked field-by-
field against the original hardcoded strings. Full backend suite green (362
passed, 4 skipped) with zero existing test changes needed beyond one
assertion made escaping-aware (a pre-existing Jinja double-escape quirk on
the one title containing "&", intentionally preserved not fixed). Built
and booted the real Docker image: content/ present at /app/content, and
curled /glossary, /glossary/percentine, /about, /privacy from inside the
running container - all four render with the exact expected title text.
Terraform generates the secrets that have no external meaning
(POSTGRES_PASSWORD, AUTH_SECRET, METRICS_TOKEN, INDEXNOW_KEY) via the random
provider instead of requiring the operator to hand-generate and paste each
into terraform.tfvars. Each is pinned with a static keepers value (secrets.tf)
so apply never regenerates a value already in use - the exact incident class
this guards against: every session invalidated, the app<->DB password
mismatched. Rotation is now a deliberate keepers edit, never a side effect.
postgres_password/auth_secret move from required inputs to optional (default
"") - explicit var wins when supplied (seeding an EXISTING live secret during
a migration onto Terraform, hop-1 cutover runbook Stage 0), else Terraform
generates and owns it. metrics_token/indexnow_key are new: neither existed in
Terraform before, both previously left for the app's own fallback generation.
VAPID deliberately stays a required, non-generated input - an EC keypair
where regeneration breaks every existing push subscription outright, unlike
an opaque token.
Sizing tiers: a locals.sizes t-shirt map (nano/small/medium/large ->
{workers, app_cpus, db_cpus, db_memory}), toward the target Proxmox
sizing-tier model (architecture doc SS6) ahead of actually provisioning VMs -
Proxmox itself stays deferred; today a tier just sizes container caps on the
existing SSH-managed hosts. A host can reference one by name (hosts.<name>.
size) or keep hand-picking the four fields, so existing tfvars are
unaffected; prod's example now uses size = "large" (identical numbers),
beta keeps explicit numbers, and a commented uat example demonstrates the
shortcut for a future ephemeral host.
Strengthened terraform/README.md's local-state caveat: more Terraform-
generated secrets landing in tfstate raises the stakes of the existing
never-commit-cleartext-state guidance, not just the sizing.
Verified: terraform validate + fmt clean. A real `terraform plan` against
fake hosts (prod/beta/uat, mixing size="large"/explicit-numbers/size="nano")
resolved every sizing correctly (prod 8/8/4/16g, beta 4/4/2/8g, uat
1/1/1/1g) and planned exactly one instance of each random_password/random_id
resource. Applied just those four resources (real generation, -target to
avoid touching the fake SSH-only host resources) and re-planned: "No
changes" - confirming the keepers pinning holds. Adding an explicit
postgres_password override afterward left the random_password resource
itself completely untouched (0 replace/destroy), confirming the override
path never disturbs the generated resource.
The Swarm/Forgejo standup (a parallel infra track, see INFRA.md) landed real
.forgejo/workflows/{build,pr-build,deploy,deploy-dev}.yml before this PR
merged, making ci.yml a redundant duplicate of their build.yml (same build
gate, same job). Drop it.
Fix the remaining two files to match the real, already-deployed conventions
those files established rather than the guesses this PR shipped with:
runs-on: [self-hosted, thermograph] -> docker (the actual Docker-in-Docker
Swarm-hosted runner label), and appleboy/ssh-action referenced by full GitHub
URL (confirmed not mirrored on this Forgejo instance's default action
registry, per deploy.yml's own header comment) rather than the short form.
actions/checkout needed no change - already confirmed to resolve unchanged
from Forgejo's default mirror.
build-push.yml (image registry push) and ops-cron.yml (backup + IndexNow)
stay: neither duplicates anything in the real mirror, which faithfully
replicates the OLD git-checkout-and-build-in-place deploy model rather than
the registry-based one, and has no scheduled ops jobs at all.
Records the concrete, live state of Track 1 rather than just the setup
procedure: both boxes' roles/IPs, the ssh invocation, where the private key
lives (~/.ssh/thermograph_agent_ed25519, same convention DEPLOY.md already
uses for the CI deploy key), and how to rotate it. Also notes that neither
box has Docker yet and Terraform hasn't been applied to either, which is why
Track 2 is currently paused.
Checks off the two verification-checklist items that are actually confirmed
now (SSH access, password auth dead) rather than leaving them unchecked.
Three workflows for the self-hosted Forgejo instance (Track A chunk 7 of the
infra-design implementation handoff), mirroring/extending the existing
.github/workflows without touching them - GitHub stays the primary repo and
its own build.yml/ci-cd.yml keep gating auto-merge.
ci.yml mirrors build.yml's build gate (deps, backend tests, frontend JS
syntax check, boot + page/API health check) on a Forgejo runner.
build-push.yml builds the app image and pushes it to Forgejo's built-in
registry, tagged by git SHA (every push) and semver (version tags) - the
registry half of the hop-1 cutover's build-once/deploy-everywhere model.
Runs alongside the existing deploy.yml/deploy.sh (git-checkout-and-build-in-
place) without touching it, per the cutover runbook's Stage C.
ops-cron.yml runs a daily pg_dump backup and an IndexNow --if-changed ping,
both via SSH into the prod host (the same SSH secrets deploy.yml already
uses) using the app's already-running compose stack - no new network
exposure, no separate dependency install. These are ops/infra concerns
(scheduled via Forgejo Actions cron), distinct from the app-domain jobs the
worker's own APScheduler runs.
All three need a runner registered with the `thermograph` label and the
registry's public/mesh exposure resolved (Track B).
Verified: raw YAML syntax valid; actionlint clean except the pre-existing,
already-present-on-.github/workflows "unknown custom runner label" warning
(not something these files introduce - the existing thermograph-lan label
triggers the identical warning). One real finding fixed pre-commit: an
unused shellcheck-flagged loop variable in the boot-check retry loop.
