Jinemi/thermograph
4
0
Fork 0
Code Issues 2 Pull requests 9 Projects Releases Packages Wiki Activity Actions

docs: rewrite the agent context layer to match the live system (#81)

Browse source
This commit is contained in:
emi 2026-07-25 07:08:54 +00:00
parent bee5b21587
commit 1acf8414ff
9 changed files with 322 additions and 627 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

98
CLAUDE.md
View file

@ -1,27 +1,81 @@
# thermograph monorepo — agent instructions # thermograph monorepo — agent instructions
Reunified monorepo (2026-07-22): `backend/`, `frontend/`, `infra/`, One repo, four domains: `backend/`, `frontend/`, `infra/`, `observability/`.
`observability/` — each a former split repo subtree-merged with history. Reunified 2026-07-22 from four split repos, with history, via subtree merges.
`thermograph-docs` remains a separate repo; cross-cutting decision docs and `thermograph-docs` is deliberately a separate repo — cross-cutting decision
operator runbooks belong there, not here. records and operator runbooks go there, not here.
- **Each domain keeps its own `CLAUDE.md`** — read the one for the domain you **Rule for this file and every domain `CLAUDE.md`: only statements that would
touch. They still use repo-era wording ("this repo", sibling-repo names); break CI if they became false, or that name a file that exists.** Background,
mentally map `thermograph-backend` → `backend/` etc., and fix wording as you history and rationale belong in `thermograph-docs`. These files are read before
touch files near it. every change, so a stale one is a correctness bug, not a documentation bug.
- **CI lives only in root `.forgejo/workflows/`**, path-filtered per domain
(see README). Never re-add workflows under a domain's own `.forgejo/` — they ## Branches and environments
are inert there and become a trap.
- Images are `emi/thermograph/backend` and `emi/thermograph/frontend` | Branch | Deploys to | Workflow |
(`sha-<12hex>`). Deploy = `infra/deploy/deploy.sh` over SSH, per-service. |---|---|---|
| feature branch | nothing | PR into `dev` |
| `dev` | LAN dev | `*-deploy-dev.yml` — **currently inert**, see below |
| `main` | beta (beta.thermograph.org) | `*-deploy.yml` |
| `release` | prod (thermograph.org) | `*-deploy-prod.yml` |
`dev`, `main` and `release` are protected: **everything is a PR**, for humans and
agents alike. Promotion is one PR per hop, `dev` → `main` → `release`.
The two `*-deploy-dev.yml` workflows are inert — the LAN box's
`~/thermograph-dev` is still a split-era `thermograph-infra` checkout, so the
monorepo path they call does not exist there. Use `infra/`'s `make dev-up`
locally instead.
## The deploy contract
One entry point, two modes, one contract:
```
SERVICE=backend|frontend|all BACKEND_IMAGE_TAG=sha-<12hex> FRONTEND_IMAGE_TAG=sha-<12hex> \
/opt/thermograph/infra/deploy/deploy.sh
```
`deploy.sh` resets the host checkout, renders secrets from the SOPS vault, then
either rolls compose services or — if `/etc/thermograph/deploy-mode` contains
`stack` — execs `infra/deploy/stack/deploy-stack.sh`.
- **prod runs Swarm.** Its stack is `infra/deploy/stack/thermograph-stack.yml`
(db, web, worker, lake, daemon, frontend, autoscaler, autoscaler-lake).
`deploy-stack.sh` also offers `STACK_TEST=1`: a full parallel rehearsal on
throwaway volumes and ports that cannot touch live data.
- **beta and LAN dev run compose**, from `infra/docker-compose.yml`
(db, backend, lake, daemon, frontend).
- Each service's live tag is persisted host-side, so a single-service roll never
disturbs the sibling's running tag.
Images are `emi/thermograph/backend` and `emi/thermograph/frontend`, tagged
`sha-<12hex>` on every push and by semver on `v*.*.*` tags. They build and deploy
**independently** — a backend change ships without a frontend deploy and vice
versa. That independence is the point of the FE/BE split and survived
reunification.
## Rules that bind across domains
- **CI lives only in root `.forgejo/workflows/`**, path-filtered per domain.
Never add workflows under a domain's own `.forgejo/` — they are inert there and
become a trap.
- **Secrets only via the SOPS vault** (`infra/deploy/secrets/`). Never hand-edit
`/etc/thermograph.env` on a host — it is a rendered artifact. `secrets-guard`
CI rejects plaintext. `seed-from-live.sh` reads production secrets and is
explicitly **not** for an agent to run.
- FE and BE ship out of lockstep, so the `/api/version` contract and
`PAYLOAD_VER` discipline are load-bearing — see `backend/CLAUDE.md`.
- Anything named "prefetch" must never spend the Open-Meteo quota.
Nominatim ≤ 1 req/s.
- The compose project name is pinned (`name: thermograph` in - The compose project name is pinned (`name: thermograph` in
`infra/docker-compose.yml`); LAN dev overrides via `infra/docker-compose.yml`); LAN dev overrides with
`COMPOSE_PROJECT_NAME=thermograph-dev`. Don't remove either half. `COMPOSE_PROJECT_NAME=thermograph-dev`. Don't remove either half — the pinned
- Cross-domain rules that still bind: FE/BE ship independently → keep the name is what makes the Swarm stack's external volume names line up.
`/api/version` contract and `PAYLOAD_VER` discipline (see
`backend/CLAUDE.md`); anything named "prefetch" must never spend the
Open-Meteo quota; Nominatim ≤ 1 req/s; secrets only via the SOPS vault
(`infra/deploy/secrets/`).
- `CUTOVER-NOTES.md` is the source of truth for what is and isn't live yet. - `CUTOVER-NOTES.md` is the source of truth for what is and isn't live yet.
- Commits/PRs: concise and technical; never mention AI/assistants/automated
authorship. ## Commits & PRs
Describe only the substance of the change; concise and technical. Never mention
AI, Claude, assistants or automated authorship anywhere — no trailers,
co-authors, emoji, or "as requested" narration.

2
README.md
View file

@ -10,7 +10,7 @@ for: **per-domain images, per-domain deploys, and an async FE/BE contract**.
|---|---|---| |---|---|---|
| `backend/` | FastAPI graded-climate API, accounts, notifications (Discord bot, push, mail), data pipeline | `backend-build-push` → image `emi/thermograph/backend`; `backend-deploy[-prod\|-dev]` | | `backend/` | FastAPI graded-climate API, accounts, notifications (Discord bot, push, mail), data pipeline | `backend-build-push` → image `emi/thermograph/backend`; `backend-deploy[-prod\|-dev]` |
| `frontend/` | Public client: static JS/CSS + SSR pages | `frontend-*` mirrors of the above; image `emi/thermograph/frontend` | | `frontend/` | Public client: static JS/CSS + SSR pages | `frontend-*` mirrors of the above; image `emi/thermograph/frontend` |
| `infra/` | Compose, deploy scripts, terraform, SOPS secrets vault, ops cron | `infra-sync` (host checkout + secrets render), `secrets-guard`, `ops-cron` | | `infra/` | Compose (beta, LAN dev) + the Swarm stack (prod), deploy scripts, terraform, SOPS secrets vault, ops cron | `infra-sync` (host checkout + secrets render), `secrets-guard`, `ops-cron` |
| `observability/` | Loki + Grafana + Alloy stack | `observability-validate` | | `observability/` | Loki + Grafana + Alloy stack | `observability-validate` |
`thermograph-docs` deliberately **stays its own repo** (ADRs + runbooks, no `thermograph-docs` deliberately **stays its own repo** (ADRs + runbooks, no

207
backend/CLAUDE.md
View file

@ -1,161 +1,66 @@
# thermograph-backend — agent instructions # backend/ — agent instructions
## What this repo is The Thermograph **API service**: a FastAPI app serving the graded-climate API
(`/api/v2/...`), user accounts, notifications, and the SSR-content JSON API the
frontend renders pages from. It owns the climate data pipeline and the
derived-payload cache. It serves **no HTML/CSS/JS** — that is `frontend/`.
The Thermograph **API service**: FastAPI app serving the graded-climate API Read the root `CLAUDE.md` first for the branch model, deploy contract and image
(`/api/v2/...`), user accounts (`accounts/`, fastapi-users + Postgres via names.
alembic), push/email/Discord notifications (`notifications/`), and the
SSR-content JSON API (`api/content_routes.py` + `api/content_payloads.py`) that
the separate `thermograph-frontend`'s server-rendered pages consume. It owns
the climate data pipeline (`data/` — grid snapping, Open-Meteo fetch + parquet
cache, percentile grading/scoring, places/cities) and the derived-payload
cache/ETag store (`data/store.py`, `api/payloads.py`). It does **not** serve
any HTML/CSS/JS itself — that moved to `thermograph-frontend` (repo-split
Stage 7a); this process is a pure JSON API + DB-backed service, launched as
`app:app` (a shim re-exporting `web.app:app`, kept stable for systemd/CI/`run.sh`
callers that still expect that target).
## Split topology ## Build & verify
Split out of the `emi/thermograph` monorepo (see - `make test` — builds a py3.12 venv and runs the hermetic pytest suite
`MONOREPO-CUTOVER-PLAN.md` at the workspace root during the transition). Sibling (`scripts/test.sh`; pass `ARGS=...`). Tests are hermetic per `tests/conftest.py`:
repos: no real Open-Meteo/Nominatim/GeoNames calls, throwaway SQLite, notifier thread
- **`thermograph-frontend`** — the client the public sees: static disabled.
JS/CSS/HTML + `frontend_ssr`'s server-rendered climate hub/city/month pages, - `make smoke` — boots the built image plus a throwaway db and asserts
calling this repo's `/api/v2/...` and `/content/...` JSON endpoints. `/healthz` + `/api/version`.
- **`thermograph-infra`** — deploy/terraform/Swarm-Compose plumbing: - CI runs this suite **inside the built image** (`build.yml`), so the exact
`docker-compose.yml`, `deploy/deploy.sh`, secrets (SOPS-encrypted interpreter and deps that ship are what gets tested.
`deploy/secrets/*.yaml`), Terraform for prod/beta provisioning. This repo has
no deploy scripts or terraform of its own — it only builds an image and hands
infra a tag.
- **`thermograph-docs`** — non-code planning layer: architecture decision
records and operator runbooks. Nothing here should duplicate that; if you're
writing a cross-cutting decision doc or a step-by-step operator runbook,
it belongs there, not in this repo's README/CLAUDE.md.
- **`thermograph-observability`** — monitoring/metrics stack, separate from
this repo's own lightweight in-process `core/metrics.py`.
## Branch, PR & deploy flow `app.py` at the root is a one-line re-export shim for `web/app.py`, kept stable
for systemd/CI callers. Entry target is `app:app`.
- `build-push.yml` builds **this repo's own image** ## Contracts that break silently if you change them
(`git.thermograph.org/emi/thermograph-backend/app`, tagged `sha-<12hex>` on
every push to `dev`/`main`/`release` and additionally by semver on `v*.*.*`
tags) — no more shared monorepo image. `build.yml` (push/PR to `main`) is a
build-only sanity check (`docker build`), not a boot/health check — repeated
attempts at a live boot+healthz check hit this runner's own quirks (bridge-IP
reachability, non-unique `$$`, squatted host ports) without ever going
reliably green; the image's actual boot is verified by hand (`docker run` +
alembic + `/healthz` 200 + a real `/api/v2/place` 200) instead.
- **`main` → beta**: `deploy.yml` SSHes to beta and runs
`thermograph-infra`'s `/opt/thermograph/deploy/deploy.sh` with
`SERVICE=backend` + `BACKEND_IMAGE_TAG=sha-<12hex>` (must match
build-push.yml's 12-hex SHA tag exactly — Actions expressions have no
substring function for the full 40-char SHA). `deploy.sh` rolls only the
`backend` compose service (`--no-deps`) and persists the tag in
`deploy/.image-tags.env` host-side, so a backend-only deploy never touches
the frontend's running container or tag.
- **`release` → prod**: `deploy-prod.yml` is the same shape against a
completely separate secret set (`PROD_SSH_HOST/USER/KEY/PORT`) so a beta
credential leak can't reach prod. Also `SERVICE=backend` +
`BACKEND_IMAGE_TAG`.
- The frontend deploys itself the same way, independently — that
independence (a backend change ships without a frontend deploy and vice
versa) is the entire point of the FE/BE CI-CD split.
- No `dev`-branch LAN deploy path exists in this repo (the monorepo's
`deploy-dev.yml`/`docker-compose.dev.yml` never made it across the split —
known gap, see the cutover plan if resurrecting it).
## Run / test locally - **`GET /api/version`** → `{backend_version, min_frontend, payload_ver}`, driven
by `API_CONTRACT_VERSION` and `MIN_SUPPORTED_FRONTEND` in `web/app.py`. A
There is **no Makefile in this repo** — the monorepo's app-level `make` breaking change ships as a new `v3` router mounted **alongside** `v2`,
targets (`lan-run`, `test`, `venv`, ...) haven't been given a new home yet re-registering only changed handlers, with the constant bumped in the same PR.
(tracked as a gap in the cutover plan). Until that lands, drive it directly: `/api` and `/api/v1` stay mounted as aliases.
- **`PAYLOAD_VER`** (`api/payloads.py`) is separate from URL versioning — it is
```bash the cache/ETag invalidation token. Bump it whenever a response payload's shape
python3 -m venv .venv && .venv/bin/pip install -r requirements.txt -r requirements-dev.txt changes, so one bump atomically orphans every pre-upgrade cached row.
.venv/bin/uvicorn app:app --host 0.0.0.0 --port 8137 # serve - **ETag / `If-None-Match`** — every graded payload is cached against a validity
.venv/bin/python -m pytest tests # test token that doubles as a weak ETag. `expose_headers=["ETag"]` in the CORS
``` middleware matters as much as `allow_origins`: without it the frontend's
`cache.js` reads `null` cross-origin and never revalidates. Don't change the
The monorepo's Makefile preferred `uv venv --python 3.12 .venv` (falling back ETag derivation or that list without checking `frontend/static/cache.js`.
to plain `python3 -m venv` only if `uv` isn't installed) — prefer `uv` here too - **`data/grading.py::pct_ordinal()`** is mirrored by `frontend`'s
if it's on the box. There is no committed `.venv` in this repo; if one exists `shared.js::pctOrd()` — floor a percentile into `1..99`, never 0 or 100.
locally and looks broken (import errors, wrong Python), just `rm -rf .venv` and `TEMP_BANDS`/`RAIN_BANDS` are the source of truth for tier names and
recreate rather than debugging it — it's gitignored, disposable state, not thresholds; changing them without the frontend produces tiers drawn in colours
something the repo depends on being a specific way. that disagree with the labels the API returns.
- **Fahrenheit country set** — `api/content_payloads.py`'s `F_COUNTRIES` must stay
Tests are hermetic (`tests/conftest.py`): no real Open-Meteo/Nominatim/GeoNames identical to `frontend`'s `format.py::F_COUNTRIES` and `static/units.js`'s
calls, a throwaway SQLite for accounts + the derived-payload store, and the `F_REGIONS`. There is a test asserting this.
notifier thread disabled. `THERMOGRAPH_FRONTEND_BASE_INTERNAL` is set to an - **`/healthz` and `/api/version` are deliberately I/O-free** so they stay cheap
unreachable placeholder in conftest — `web/app.py` fails loud at import if it's under tight healthcheck intervals.
unset in a real run (it's the internal URL back to `frontend_ssr` for the
non-API HTML routes this process no longer serves itself).
## API version contract
- Everything real lives under **`/api/v2/...`**; `/api/` and `/api/v1/` stay
mounted as aliases of the same handlers (they always were — not new).
- **`GET /api/version`** (`web/app.py`) is the capability/negotiation endpoint
a split-out frontend can call at boot: `{backend_version, min_frontend,
payload_ver}`, driven by two module constants next to it —
`API_CONTRACT_VERSION` (bump only on a breaking change to any `/api/v{N}`
route or payload shape) and `MIN_SUPPORTED_FRONTEND` (oldest frontend
contract version this backend still serves correctly). A genuine breaking
change ships as a new `v3` router mounted alongside `v2` re-registering only
the changed handlers, `v2` kept working, `API_CONTRACT_VERSION` bumped in the
same PR.
- **`PAYLOAD_VER`** (`api/payloads.py`, currently `"p2"`) is a *separate*
concern from URL versioning — it's the cache/ETag invalidation token baked
into every derived-store validity token (`history_token()`). Bump it
whenever a response payload's shape changes (new metric, renamed/removed
field) so one bump atomically orphans every pre-upgrade cached row instead
of a stale row's `history_end` happening to still validate under the old
shape.
- Both endpoints (`/healthz`, `/api/version`) are deliberately I/O-free so they
stay cheap under tight healthcheck intervals.
## Live cross-repo contracts (don't break silently)
- **`/cell` bundle + ETag/If-None-Match**: every graded payload (`grade`,
`calendar`, `day`) is cached in SQLite by `(kind, cell_id, key)` keyed to a
validity token that only advances when the underlying history actually
changes (`_etag_for` in `web/app.py`, `history_token`/`PAYLOAD_VER` in
`api/payloads.py`). The same token doubles as a weak ETag; a client sending
`If-None-Match` gets an empty 304 without the payload being rebuilt or even
loaded. `expose_headers=["ETag"]` in the CORS middleware matters as much as
`allow_origins` — without it the frontend's `cache.js` reading
`res.headers.get("ETag")` cross-origin silently gets `null` and never
revalidates correctly. Don't change the ETag derivation or the CORS
`expose_headers` list without checking the frontend's cache layer.
- **Percentile/unit parity with `thermograph-frontend`'s `static/shared.js`**:
the frontend's percentile-to-ordinal display logic explicitly mirrors this
repo's `data/grading.py::pct_ordinal()` (floors an empirical percentile into
1..99 — a rank against the sample can round to 0 or 100, and "100th
percentile" is never shown). `TEMP_BANDS`/`RAIN_BANDS` tier boundaries in
`data/grading.py` are the source of truth for the tier names/thresholds
(Near Record / High / Above Normal / Normal / Below Normal / Low, and the
five rain-intensity tiers) — changing them without a corresponding frontend
update produces a client that draws differently-colored tiers than the
labels the API returns.
- **SSR content API** (`api/content_routes.py`): a separate ETag'd JSON surface
(`/content/hub`, `/content/city/{slug}`, `/content/city/{slug}/month/{month}`,
`/content/city/{slug}/records`, `/content/home`, `/content/sitemap`,
`/content/indexnow-key`) that `frontend_ssr` calls in-process-free (no shared
Python import) to render pages. Same caching pattern as the main API; keep
payload shape changes coordinated with whatever consumes it on the frontend
side.
## Layout ## Layout
`accounts/` (fastapi-users models/schemas/db, `alembic/` migrations), `accounts/` (fastapi-users + alembic), `api/` (payload builders + SSR content
`api/` (versioned payload builders + SSR content routes), `core/` (metrics, routes), `core/` (metrics, audit, singleton helper), `data/` (grid, climate
audit logging, a singleton helper), `data/` (grid, climate fetch/cache, grading/ fetch/cache, grading/scoring, places/cities, derived-payload store),
scoring, places/cities, the derived-payload store), `notifications/` (push, `notifications/` (push, email, Discord bot, digest, scheduler), `web/app.py`
email, Discord bot + interactions + linking, digest, scheduler), `web/app.py` (the real app), `daemon/`.
(the actual FastAPI app; `app.py` at the repo root is a one-line re-export
shim). `paths.py` resolves every filesystem location (climate parquet cache, `paths.py` resolves every filesystem location from the repo root — don't
accounts DB, logs, bundled `cities.json`/`cities_flavor.json`) from the repo reintroduce `__file__`-relative paths in a module; it breaks the moment a module
root — don't reintroduce `__file__`-relative paths in a module, it breaks the moves. `deploy/entrypoint.sh` is the container entrypoint (alembic migrate, then
moment a module moves. `deploy/entrypoint.sh` is the container entrypoint serve, with a Swarm-secrets-as-files shim).
(alembic migrate-then-serve, with a Swarm-secrets-as-files shim); actual deploy
orchestration lives in `thermograph-infra`, not here. ## Commits & PRs
Concise and technical. Never mention AI, assistants or automated authorship.

217
frontend/CLAUDE.md
View file

@ -1,181 +1,72 @@
# Thermograph frontend — agent instructions # frontend/ — agent instructions
This repo is the **SSR + static-asset service** split out of the `emi/thermograph` The **SSR + static-asset service**: server-rendered crawlable pages, the
monorepo (repo-split Stage 7). It has no climate data, no DB, and does no interactive tool's SPA shells, and every static asset. No climate data, no DB,
polars/compute work — everything comes from the backend's content API over no compute — everything comes from `backend/`'s content API over HTTP.
HTTP. It is one of four sibling repos in `thermograph-repos/`:
- **`thermograph-backend`** — FastAPI API + grading/scoring/grid compute + DB. Read the root `CLAUDE.md` first for the branch model, deploy contract and image
This repo's only dependency. names.
- **`thermograph-frontend`** (this repo) — SSR content pages + the interactive
tool's SPA shells + every static asset.
- **`thermograph-infra`** — Terraform, `docker-compose*.yml`, `deploy/` (incl.
`deploy.sh`, the SOPS secrets vault), Caddy config. No app code.
- **`thermograph-docs`** — architecture decision records + operator runbooks.
No code.
See `MONOREPO-CUTOVER-PLAN.md` (one level up, in `thermograph-repos/`) for the ## Build & verify
full split status and the remaining gap list before the monorepo can be
archived.
## What this repo is - `make test` — whole suite (`scripts/test.sh`).
- `make test-unit` — hermetic unit tier: SSR rendering fed committed fixtures,
no Docker. **This is the tier CI runs.**
- `make test-integration` — pulls and runs the real backend image and tests the
live contract. Local only; CI does not run it.
- `make backend-up` / `make backend-down` — a local backend container for dev.
- `make capture-fixtures` — refresh `tests/fixtures/*.json` from a live backend.
- **`content.py`** — server-rendered, crawlable pages (climate hub, per-city, ## Running it
month, records, glossary, about, privacy) + `robots.txt` + `sitemap.xml`.
Every route fetches its data from the backend's content API
(`api_client.py`) instead of computing in-process; `content_payloads.py` on
the backend owns `page_title`/`canonical_path`/`breadcrumb`/`jsonld`, not
this repo.
- **`static/*.js`** — the interactive tool: `app.js` (map/search/graded
results + inline SVG chart), `calendar.js`/`day.js`/`score.js`/`compare.js`
(SPA shells served by `app.py`'s `_page()`), `account.js` (auth), `cache.js`
(IndexedDB response cache + `/cell` bundle prefetch), `shared.js` (format
helpers shared across views).
- **`static/style.css`** — the single hand-written stylesheet; all design
tokens live here (see `DESIGN.md`).
- **`templates/*.html.j2`** — Jinja templates `content.py` renders.
- **`content/*.yaml`** — structured SSR copy (glossary, static-page SEO meta),
loaded by `content_loader.py`. Committed here as a starter copy extracted
alongside the split; real cross-repo copy vendoring (a pinned
`thermograph-copy` checkout at build time) is deferred, unbuilt follow-up —
don't assume it exists.
## Branch/PR + deploy flow `THERMOGRAPH_API_BASE_INTERNAL` is **required** — `api_client.py` raises at
import if unset.
- Work happens on feature branches; PR into `main`. (The monorepo's
`dev`→`main`→`release` three-stage promotion and its LAN `deploy-dev.yml`
path do **not** exist in this repo yet — a documented gap, see the cutover
plan §4. Don't assume a `dev` branch here.)
- **`.forgejo/workflows/build-push.yml`** — on push to `dev`/`main`/`release`
or a `v*.*.*` tag: builds THIS repo's own `Dockerfile` and pushes
`git.thermograph.org/emi/thermograph-frontend/app`, tagged `sha-<12 hex>`
(every push) and the semver tag (tag pushes only). Backend publishes its own
separate image the same way — the two are no longer one shared
`emi/thermograph/app` image.
- **`.forgejo/workflows/build.yml`** — push/PR to `main`: proves the Dockerfile
builds. It is a **build check only**, not a boot/health check — booting the
real app crashes at import without a reachable backend (see API-version
section below), so a standalone boot check would need to check out
`thermograph-backend` too. Not yet built; flagged, not silently skipped.
- **`.forgejo/workflows/deploy.yml`** — push to `main`: SSH to beta, run
`SERVICE=frontend FRONTEND_IMAGE_TAG=sha-<12 hex> /opt/thermograph/deploy/deploy.sh`
(that script lives in `thermograph-infra`). Rolls **only** the frontend
container (`--no-deps`); backend is untouched and deployed independently by
its own repo's workflow. `deploy.sh` retries the image pull for ~5 min
(Forgejo has no cross-workflow `needs:`, so this deploy can race ahead of
`build-push.yml`) and waits for a healthy backend before declaring the roll
OK (frontend's boot fetches the IndexNow key from backend — see below).
- **`.forgejo/workflows/deploy-prod.yml`** — push to `release`: same shape,
targets prod via its own `PROD_SSH_*` secret set (fully separate from beta's,
so a beta credential leak can't touch prod). Nothing else deploys to prod;
there is no release-triggered Terraform apply from this repo.
- The `SERVICE=frontend` + `FRONTEND_IMAGE_TAG=sha-<12 hex>` pair is the entire
contract into `thermograph-infra/deploy/deploy.sh`: it persists each
service's live tag in `deploy/.image-tags.env` (host-side, untracked) so a
frontend-only roll never disturbs backend's currently-running tag, and
vice versa.
## How to run / test
There is no `run.sh`/`Makefile` in this repo yet (the monorepo's `make
lan-run`/`make run`/`make stop`/`venv` targets have no home here — see the
cutover plan's gap list). Run directly:
```bash ```bash
python3 -m venv .venv && .venv/bin/pip install -r requirements.txt THERMOGRAPH_API_BASE_INTERNAL=http://127.0.0.1:8137 THERMOGRAPH_BASE=/thermograph \
THERMOGRAPH_API_BASE_INTERNAL=http://127.0.0.1:8137 \
THERMOGRAPH_BASE=/thermograph \
.venv/bin/uvicorn app:app --host 0.0.0.0 --port 8080 .venv/bin/uvicorn app:app --host 0.0.0.0 --port 8080
``` ```
`THERMOGRAPH_API_BASE_INTERNAL` is **required** — `api_client.py` raises Other env: `THERMOGRAPH_BASE` (default `/thermograph`; the Dockerfile sets `/`),
`RuntimeError` at import if it's unset. Other env vars: `THERMOGRAPH_BASE` `THERMOGRAPH_API_VERSION` (default `v2`), `THERMOGRAPH_API_BASE_PUBLIC`
(default `/thermograph`; the Dockerfile sets `/` for the deployed clean-root (browser-facing backend origin when cross-origin; empty = same-origin, today's
topology), `THERMOGRAPH_API_VERSION` (default `v2`, see below), default), `THERMOGRAPH_SSR_CACHE_TTL` (default 600s).
`THERMOGRAPH_API_BASE_PUBLIC` (browser-facing backend origin for asset URLs
when frontend and backend are cross-origin; empty = same-origin, today's
default), `THERMOGRAPH_SSR_CACHE_TTL` (default 600s, the content-API response
cache in `api_client.py`), `THERMOGRAPH_GOOGLE_VERIFY`/`THERMOGRAPH_BING_VERIFY`
(search-console `<meta>` tags).
**Known gap — `tests/conftest.py` does not run standalone in this repo.** It ## Layout
still assumes the pre-split layout: a sibling `backend/` checkout with its own
`tests/conftest.py` (`make_history`/`make_recent`) and an `api.content_payloads`
module, neither of which exists here. It was extracted verbatim as reference
material for the real fix (a genuine cross-repo contract-test job, or a
self-contained set of fixtures owned in this repo) — not yet built. Do **not**
assume `pytest tests` passes here until that lands; `.forgejo/workflows/build.yml`
only proves the image builds, for the same reason. There is also no
`requirements-dev.txt` in this repo yet (pytest/httpx test deps aren't pinned).
## API-version pinning contract - **`content.py`** — SSR routes (climate hub, city, month, records, glossary,
about, privacy) + `robots.txt` + `sitemap.xml`. Each fetches from the backend
via `api_client.py`; the backend's `content_payloads.py` owns `page_title` /
`canonical_path` / `breadcrumb` / `jsonld`, not this domain.
- **`static/*.js`** — `app.js` (map/search/results + inline SVG chart),
`calendar.js`/`day.js`/`score.js`/`compare.js` (SPA shells), `account.js`
(auth), `cache.js` (IndexedDB cache + `/cell` bundle prefetch), `shared.js`.
- **`static/style.css`** — the single hand-written stylesheet; design tokens live
here, documented in `DESIGN.md`.
- **`templates/*.html.j2`**, **`content/*.yaml`** (structured SSR copy, loaded by
`content_loader.py`).
Every backend call in this repo goes through a single pinned constant instead ## Contracts with the backend
of scattered `api/v2/...` literals:
- **Python (SSR):** `api_client.py`'s module-level `API_VERSION` (default - **API version is pinned in exactly two places** — `api_client.py`'s
`"v2"`, overridable via `THERMOGRAPH_API_VERSION`). Every path builder `API_VERSION` (Python) and `static/account.js`'s exported `API_VERSION` plus the
(`hub()`, `sitemap()`, `indexnow_key()`, `home()`, `city()`, `city_month()`, `uv(path)` helper (JS). Never hardcode `api/v2/...` anywhere else. Bump both in
`city_records()`) reads it. one PR, and only after the target backend's `/api/version` confirms it serves
- **JS (interactive tool):** `static/account.js`'s exported `API_VERSION` that version and its `min_frontend` doesn't exclude the one you're leaving.
constant and the `uv(path)` helper — every fetch in `cache.js`, `account.js` - **`shared.js::pctOrd()` must mirror `backend`'s `data/grading.py::pct_ordinal()`**
itself, etc. is built by wrapping the path in `uv(...)` rather than in behaviour — floor into `1..99`, never 0 or 100. Every percentile on every
hardcoding a version. surface goes through one of the two; if they diverge, the same reading says two
different things in two places.
**Bump only in lockstep with a verified backend `/api/version` check** — the - **`/cell` bundle + ETag** — `cache.js` fetches `/api/v2/cell` once per view-set
backend exposes `GET {BASE}/api/version` → `{backend_version, min_frontend, and slices it, revalidating with `If-None-Match`. The slice→view map must track
payload_ver}` (`web/app.py`'s `API_CONTRACT_VERSION`/`MIN_SUPPORTED_FRONTEND`). the backend's payload shape.
Before bumping this repo's `API_VERSION`, confirm the target backend's - **Fahrenheit country set** — `format.py::F_COUNTRIES` and `static/units.js`'s
`backend_version` actually supports it and its `min_frontend` doesn't already `F_REGIONS` must stay identical to the backend's `F_COUNTRIES`.
exclude the version you're moving *away* from. - **Boot must survive an unreachable backend.** `content.py`'s `register()` tries
the IndexNow-key fetch once, catches failure, and falls back to a lazy
**A v3 cutover would work like this:** backend mounts a new `v3 = APIRouter()` per-request lookup. Asynchronous FE/BE deploys depend on this — don't make it
alongside `v2`, re-registering only the changed handlers (v2 keeps serving fatal.
old clients); backend bumps `API_CONTRACT_VERSION` to `"3"` in that same PR.
Only once that's deployed and verified does this repo bump `API_VERSION` to
`"v3"` in both `api_client.py` and `account.js` — one PR, both pins together,
never one without the other. `v2` (and the `/api`, `/api/v1` aliases) stay
mounted and working until no client depends on them.
The frontend's own boot is **resilient to the backend being down**: the
`indexnow_key()` fetch in `content.py`'s `register()` is tried once eagerly
(so the common case still serves `/<key>.txt` as a plain static route), but a
failure is caught, logged, and falls back to a lazy per-request lookup
instead of crashing boot — asynchronous frontend/backend deploys depend on
this (a briefly-unreachable backend at frontend boot must be survivable, not
fatal).
## Cross-repo contracts this repo depends on
These are **live contracts with the backend** — a change on either side
without the other breaks something silently, not loudly:
- **The `/cell` bundle + ETag/`If-None-Match`** — `static/cache.js` fetches
`/api/v2/cell` once per view-set and slices it for calendar/day/score/compare
instead of one request per view; conditional refetch is via `If-None-Match`
against the stored `ETag`, so an unchanged payload costs an empty 304. The
URL map (which slice serves which view) must stay in sync with
`thermograph-backend`'s route/payload shape.
- **`shared.js`'s `pctOrd()` must mirror `thermograph-backend`'s
`data/grading.py`'s `pct_ordinal()`** byte-for-byte in behavior: floor a
percentile into `1..99` (never round to 100/0 — "100th percentile" reads as
measurement error, not "as extreme as it has ever been"). Every percentile
shown anywhere (Day page, calendar tooltip, chart, city pages, homepage
strip) goes through one of these two functions; if they diverge, the same
reading says two different things on two different surfaces.
- **Unit/region logic** (`format.py`'s `F_COUNTRIES` / `static/units.js`'s
`F_REGIONS` / backend's `api/content_payloads.py`'s `F_COUNTRIES`) — the set
of Fahrenheit-using country codes must stay identical across all three;
backend has a test asserting this.
## Design & visual verification
Design tokens + conventions are documented in `DESIGN.md` (source of truth:
`static/style.css`). To see a change rendered, see `DESIGN.md`'s `make shots`
section (`tools/shoot.py`).
## Commits & PRs ## Commits & PRs
Describe only the substance of the change; concise, technical. Never mention Concise and technical. Never mention AI, assistants or automated authorship.
AI/Claude/assistants or automated authorship anywhere — no trailers,
co-authors, emoji, or "as requested"/"per the agent" narration.

82
infra/CLAUDE.md
View file

@ -1,42 +1,58 @@
# thermograph-infra — agent instructions # infra/ — agent instructions
How and where the already-built Thermograph app images run. This repo owns How and where the already-built app images run: Terraform, the SOPS+age secrets
Terraform, the SOPS+age secrets vault, compose files, deploy scripts, host vault, compose and Swarm stack files, deploy scripts, host provisioning, mail,
provisioning, and the ops cron (DB backup + IndexNow). The app repos and the ops cron (DB backup + IndexNow). Read the root `CLAUDE.md` first — it
(`thermograph-backend`, `thermograph-frontend`) own building and testing the owns the branch model, the deploy contract, and which environment runs which
images; this repo never checks out app source. The old monorepo orchestrator.
(`emi/thermograph`) is archived — do not point anything at it.
## The four machines ## The machines
Same as ACCESS.md: **dev machine** (operator's box, LAN dev server, CI runner), Per `ACCESS.md`: **dev** (operator's box — LAN dev server and CI runner),
**prod** (`169.58.46.181`, thermograph.org, `agent` user, passwordless sudo), **prod** (`169.58.46.181`, thermograph.org, `agent` user, passwordless sudo),
**beta** (`75.119.132.91`, beta.thermograph.org + Forgejo, `agent` user). Hosts' **beta** (`75.119.132.91`, beta.thermograph.org + Forgejo + Grafana/Loki,
`/opt/thermograph` (and LAN's `~/thermograph-dev`) are checkouts of THIS repo. `agent` user). All four are on a WireGuard mesh. Hosts' `/opt/thermograph` is a
checkout of **this monorepo**, not an infra-only repo.
## Deploys ## Deploy paths
- `deploy/deploy.sh` — the one deploy path (prod/beta): resets this repo's - **`deploy/deploy.sh`** — the single entry point for beta and prod. Resets the
checkout, renders secrets from the SOPS vault, pulls the per-service image host checkout to `BRANCH` (default `main`), renders secrets, then routes:
tags (`BACKEND_IMAGE_TAG`/`FRONTEND_IMAGE_TAG`, persisted in untracked if `/etc/thermograph/deploy-mode` says `stack` it execs
`deploy/.image-tags.env`), rolls the target service, health-checks via `deploy/stack/deploy-stack.sh`, otherwise it rolls compose services. Per-service
container healthchecks. Invoked over SSH by the app repos' deploy workflows. tags persist in untracked `deploy/.image-tags.env` (compose) and
- `deploy/deploy-dev.sh` — thin LAN-dev wrapper around deploy.sh (dev compose `deploy/.stack-image-tags.env` (stack) so the two never mix.
overlay, `~/thermograph-dev`, branch `dev`). - **`deploy/stack/deploy-stack.sh`** — the Swarm path, live on prod. `backend`
- Branch model: see README — infra `main` is live on prod+beta, `dev` on LAN; rolls web **and** worker; `frontend` rolls frontend; `all` runs a full
`release` is currently unused. App code is env-staged via image tags; infra `docker stack deploy`. Start-first, health-gated, auto-rollback.
is not. `STACK_TEST=1` rehearses the whole thing under stack name `thermograph-test`
on throwaway volumes and ports `18137`/`18080`.
- **`deploy/deploy-dev.sh`** — thin LAN-dev wrapper (dev compose overlay,
`~/thermograph-dev`). Its CI trigger is inert; see root `CLAUDE.md`.
- **`Makefile`** — compose orchestration only: `up`, `down`, `db-up`, `dev-up`,
`om-up`, `om-backfill`.
Volumes are the reason the compose project name is pinned: compose creates
`thermograph_pgdata`/`_appdata`/`_applogs`, and the Swarm stack declares those
same names as `external: true` at the same mount paths.
## Rules ## Rules
- **Never run `terraform apply` casually** — no tfstate exists anywhere (never - **Never run `terraform apply` casually.** No tfstate is persisted anywhere, so
persisted), so an apply would attempt full re-provisioning of live hosts. an apply would attempt full re-provisioning of live hosts. Terraform here is
Terraform here is executable documentation until state is bootstrapped. executable documentation until state is bootstrapped.
- Secrets: only via the SOPS vault (`deploy/secrets/*.yaml`, `sops edit` + - **Secrets: SOPS vault only** (`deploy/secrets/*.yaml`, `sops edit` → commit →
commit + deploy). Never hand-edit `/etc/thermograph.env` on a SOPS-enabled deploy). Never hand-edit `/etc/thermograph.env` — it is a rendered artifact.
host — it's a rendered artifact. `secrets-guard` CI rejects plaintext. `secrets-guard` CI rejects plaintext. `deploy/secrets/seed-from-live.sh` reads
- The ops cron (`.forgejo/workflows/ops-cron.yml`) is THE prod backup — it uses production secrets and is **not** for an agent to run.
this repo's `PROD_SSH_*` Actions secrets. If you touch it, verify a dump - **The ops cron (`.forgejo/workflows/ops-cron.yml`, at the repo root) is THE
actually lands in `agent@prod:~/thermograph-backups/`. prod backup.** It uses `PROD_SSH_*`, not `SSH_*` — an earlier revision reused
- Commits/PRs: concise and technical; never mention AI/assistants/automated `SSH_*` and silently dumped beta while prod had no backup at all. If you touch
authorship. it, verify a dump actually lands in `agent@prod:~/thermograph-backups/`.
- Shell here runs as root over SSH against live hosts with no test suite in
front of it. `shell-lint` CI (pinned shellcheck) is the only guard — keep the
tree at zero findings.
## Commits & PRs
Concise and technical. Never mention AI, assistants or automated authorship.

77
infra/README.md
View file

@ -1,46 +1,49 @@
# thermograph-infra # infra/
Infrastructure for [Thermograph](https://thermograph.org): Terraform host Infrastructure for [Thermograph](https://thermograph.org): Terraform host
provisioning, the SOPS+age secrets vault, Docker Swarm/WireGuard networking, provisioning, the SOPS+age secrets vault, WireGuard/Swarm networking, Forgejo,
Forgejo, Caddy, and the deploy scripts that run the already-built app image on Caddy, mail, and the deploy scripts that run the already-built app images on each
each host. Extracted from the app monorepo (`emi/thermograph`) — the app repo host. This is a domain of the `emi/thermograph` monorepo — hosts' `/opt/thermograph`
owns building and testing the app; this repo owns running it. is a checkout of the whole monorepo, and `infra/` never builds app source; it only
runs published images.
- **`terraform/`** — provisions/configures hosts (SSH-driven by default; an - **`terraform/`** — provisions/configures hosts (SSH-driven by default; an
optional GCP-creating module is scaffolded, no live resources yet) and optional GCP-creating module is scaffolded, no live resources yet). See
triggers each deploy. See `terraform/README.md`. `terraform/README.md`. No tfstate is persisted anywhere — treat `apply` as
- **`deploy/secrets/`** — the git-native SOPS+age secrets vault (every app executable documentation, not a routine operation.
secret, encrypted at rest, rendered at deploy time). See - **`deploy/secrets/`** — the git-native SOPS+age secrets vault (every app secret,
`deploy/secrets/README.md`. encrypted at rest, rendered at deploy time). See `deploy/secrets/README.md`.
- **`deploy/swarm/`, `deploy/forgejo/`** — the 3-node WireGuard/Swarm cluster - **`deploy/swarm/`, `deploy/forgejo/`** — the WireGuard/Swarm cluster hosting
that hosts Forgejo (git + CI + registry); does not run the app itself. See Forgejo (git + CI + registry). See `ACCESS.md`.
`ACCESS.md` and the READMEs under each directory. - **`deploy/deploy.sh`** — the single deploy entry point for beta and prod.
- **`deploy/deploy.sh`** — pulls the pinned app image (`IMAGE_TAG`) and rolls Takes `SERVICE=backend|frontend|all` plus `BACKEND_IMAGE_TAG`/`FRONTEND_IMAGE_TAG`,
the compose stack; invoked by Terraform and by the app repo's resets the host checkout, renders secrets, and routes to the right orchestrator.
`.forgejo/workflows/deploy.yml` over SSH. - **`deploy/stack/`** — the **Swarm** path, live on **prod**:
- **`docker-compose*.yml`, `docker-stack.yml`** — how the app image runs `thermograph-stack.yml` (db, web, worker, lake, daemon, frontend, autoscaler,
(compose in production today; `docker-stack.yml` is a design record for a autoscaler-lake), `deploy-stack.sh`, `autoscale.sh`, and the LB. Rolling updates
possible future Swarm-based app deploy, not currently live). are start-first, health-gated, with auto-rollback. `STACK_TEST=1` rehearses the
whole stack on throwaway volumes and ports.
- **`docker-compose*.yml`** — the **compose** path, live on **beta** and LAN dev
(db, backend, lake, daemon, frontend). `docker-compose.dev.yml` is the LAN
overlay; `docker-compose.openmeteo.yml` is the self-hosted Open-Meteo overlay.
The app's own source, `Dockerfile`, and build/test CI stay in the app repo — Which path a host takes is decided by `/etc/thermograph/deploy-mode`: the string
this repo never checks out app source; hosts only pull tagged images from the `stack` makes `deploy.sh` exec `deploy/stack/deploy-stack.sh`; anything else is
registry. See `ACCESS.md` for host access and the Swarm/Forgejo topology, and compose. The workflows never need to know which mode a host runs.
`terraform/README.md` for the day-to-day `plan`/`apply` workflow.
## Branches & how changes reach each environment ## Branches & how changes reach each environment
- **`main`** — what **prod and beta** run: their `/opt/thermograph` checkouts - **`main`** — what **prod and beta** run. `infra-sync.yml` fires on a push to
`git reset --hard origin/main` at the start of every deploy (`deploy/deploy.sh`). `main` touching `infra/**`, fast-forwards each host's `/opt/thermograph` checkout
A merge to `main` reaches those hosts on the next app deploy (or a by-hand and re-renders `/etc/thermograph.env` from the vault. It deliberately does
`deploy.sh` run); there is no separate infra deploy trigger. **not** roll any service: image tags are the app domains' axis, not infra's. A
- **`dev`** — what **LAN dev** runs: `~/thermograph-dev` resets to it via compose or stack change that must recreate containers takes effect on the next
`deploy/deploy-dev.sh`. Keep it fast-forwarded to `main` (infra changes are not app deploy, or a by-hand `SERVICE=all … deploy/deploy.sh`.
environment-staged today; the branches exist so LAN dev *can* trail or lead - **`dev`** — what LAN dev would run via `deploy/deploy-dev.sh`. The CI trigger
when needed). for this is currently inert (the LAN box still holds a split-era checkout); use
- **`release`** — currently consumed by nothing (prod tracks `main`, not `make dev-up` locally.
`release`). It exists to mirror the app repos' dev→main→release promotion - **`release`** — consumed by app deploys only. Both hosts track infra via `main`;
shape if per-environment infra staging is ever wanted; until then, treat prod's *app images* are staged by `release`, but its checkout follows `main`.
`main` as live-everywhere.
Note the asymmetry with the app repos: app code IS environment-staged Note the asymmetry with the app domains: app code IS environment-staged
(dev→main→release maps to LAN→beta→prod via image tags), infra is not. (`dev`→`main`→`release` maps to LAN→beta→prod via image tags); infra is not.

6
infra/docker-compose.yml
View file

@ -46,8 +46,10 @@ services:
# to an exact minor (e.g. 2.17.2-pg18) before any host of this stack could ever # to an exact minor (e.g. 2.17.2-pg18) before any host of this stack could ever
# replicate with another — a floating tag risks two hosts landing on different # replicate with another — a floating tag risks two hosts landing on different
# extension minors, which blocks a physical replica and risks compressed-chunk # extension minors, which blocks a physical replica and risks compressed-chunk
# corruption on restore. docker-stack.yml (the Swarm interim stack) REQUIRES an # corruption on restore. The Swarm path (deploy/stack/) enforces this a
# exact pin for exactly this reason; use the SAME tag on both once you set one. # different way: deploy-stack.sh resolves TIMESCALEDB_IMAGE to the digest of
# whatever db is ALREADY running -- including this compose stack's
# thermograph-db-1 -- so the image under an existing volume can never drift.
image: timescale/timescaledb:${TIMESCALEDB_TAG:-latest-pg18} image: timescale/timescaledb:${TIMESCALEDB_TAG:-latest-pg18}
environment: environment:
POSTGRES_USER: thermograph POSTGRES_USER: thermograph

173
infra/docker-stack.yml
View file

@ -1,173 +0,0 @@
# Docker Swarm stack for the hop-1 interim cutover — see
# thermograph-docs/runbooks/hop1-forgejo-registry-cutover.md (which this file implements) and
# thermograph-docs/architecture/repo-topology-and-infrastructure.md §6/§9.
#
# Distinct from docker-compose.yml (today's plain-compose deploy, unaffected by
# this file): Swarm pulls a pre-built image from the registry (IMAGE_TAG) rather
# than building in place, and needs Swarm-specific keys (deploy:, networks:,
# secrets:, no host-published ports on the overlay-fronted services).
#
# docker stack deploy -c docker-stack.yml thermograph
#
# `docker stack deploy` only interpolates from the invoking shell's environment,
# not a .env file — export the required vars first (or `set -a; . ./stack.env;
# set +a; docker stack deploy ...`). Required shell vars: IMAGE_TAG (the tag CI
# pushed to the registry), TIMESCALEDB_TAG (an EXACT pinned minor — see the db
# service below, hazard #7). POSTGRES_PASSWORD is NOT a shell var here: unlike
# docker-compose.yml (which interpolates it into THERMOGRAPH_DATABASE_URL at
# compose time), this file reads it as the `postgres_password` Swarm secret —
# POSTGRES_PASSWORD_FILE for db (native support in the postgres/timescaledb
# image), and the chunk-3 entrypoint shim builds THERMOGRAPH_DATABASE_URL for
# app/worker from the same secret file at container start. All `postgres_password`
# / `thermograph_*` secrets below must already exist in the Swarm (`docker secret
# create <name> -`) before the first deploy — Track B step 7 in
# thermograph-docs/runbooks/implementation-handoff.md.
#
# Migrations are NOT run inline by app/worker here (RUN_MIGRATIONS=0) — the
# runbook's Stage F brings the schema to head via a one-shot task BEFORE scaling
# app up, so multiple replicas never race Alembic and nothing runs DDL against a
# still-read-only standby mid-cutover (hazard #9):
#
# docker run --rm --network thermograph_internal \
# -e THERMOGRAPH_DATABASE_URL=... <image> migrate
services:
db:
# Pin the EXACT TimescaleDB minor — never latest-pg18 here. A floating tag can
# give two hosts different extension minors, which blocks a physical replica
# (a newer .so over an older catalog won't start) and risks compressed-chunk
# corruption on timescaledb_post_restore() (hazard #7). Get the exact X.Y.Z
# from the source database: SELECT extversion FROM pg_extension WHERE
# extname='timescaledb' — use the SAME tag docker-compose.yml's TIMESCALEDB_TAG
# is pinned to, on every host that could ever replicate with this one.
image: timescale/timescaledb:${TIMESCALEDB_TAG:?set TIMESCALEDB_TAG to an exact pinned minor, e.g. 2.17.2-pg18 -- not latest-pg18}
environment:
POSTGRES_USER: thermograph
POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password
POSTGRES_DB: thermograph
DB_MEMORY: ${DB_MEMORY:-8g}
volumes:
- pgdata:/var/lib/postgresql
- ./deploy/db/init:/docker-entrypoint-initdb.d
networks:
- internal
secrets:
- postgres_password
deploy:
# Pin to the labelled DB node so replication/IO never crosses the slow WG
# uplink (hazard #14): `docker node update --label-add db=true <node>` once,
# on whichever node holds pgdata (Track B).
placement:
constraints: ["node.labels.db == true"]
resources:
limits:
cpus: "${DB_CPUS:-2}"
memory: ${DB_MEMORY:-8g}
restart_policy:
condition: on-failure
# No published port: reachable only as db:5432 on the `internal` overlay —
# Postgres must never be public (design doc §6).
# Stateless, freely-replicable web tier. ROLE=web means _should_run_notifier()
# is False unconditionally (web/app.py) — it never starts the notifier or the
# worker scheduler even if it would otherwise win leader election.
app:
image: ${IMAGE_TAG:?set IMAGE_TAG to the image CI pushed, e.g. forge.example/thermograph/app:sha-abc123}
environment:
THERMOGRAPH_BASE: /
PORT: 8137
WORKERS: ${WORKERS:-4}
THERMOGRAPH_ROLE: web
RUN_MIGRATIONS: "0"
volumes:
- appdata:/app/data
- applogs:/app/logs
networks:
- internal
secrets:
- postgres_password
- thermograph_auth_secret
- thermograph_vapid_private_key
- thermograph_vapid_public_key
deploy:
replicas: 1 # single replica this hop -- homepage.json now lives in
# Postgres (chunk 4), but the notifier/scheduler split (chunk
# 2/5) is what actually lets this go multi-replica in Phase 2
resources:
limits:
cpus: "${APP_CPUS:-4}"
restart_policy:
condition: on-failure
update_config:
order: start-first # new task must pass the image's HEALTHCHECK (now
# GET /healthz) before the old one is stopped
# No `ports:` — 127.0.0.1:8137:8137 (docker-compose.yml) has no Swarm
# equivalent: Swarm's routing mesh publishes on 0.0.0.0, which would expose
# the plaintext app un-fronted (hazard #6). Only the host's Caddy reaches
# `app`, over the `internal` overlay network — see the Caddyfile templates'
# health-gated reverse_proxy.
# Owns the notifier + worker scheduler (chunks 1/2/5). Exactly one replica —
# the leader-election guard is belt-and-suspenders, not a substitute for it.
worker:
image: ${IMAGE_TAG:?set IMAGE_TAG to the image CI pushed, e.g. forge.example/thermograph/app:sha-abc123}
environment:
THERMOGRAPH_BASE: /
WORKERS: "1"
THERMOGRAPH_ROLE: worker
# Cluster-wide advisory lock, not the flock: the flock only arbitrates
# workers on ONE host, so under Swarm it guards nothing across replicas.
THERMOGRAPH_SINGLETON_PG: "1"
RUN_MIGRATIONS: "0"
volumes:
- appdata:/app/data
- applogs:/app/logs
networks:
- internal
secrets:
- postgres_password
- thermograph_auth_secret
- thermograph_vapid_private_key
- thermograph_vapid_public_key
- thermograph_discord_webhook
deploy:
replicas: 1
resources:
limits:
cpus: "${WORKER_CPUS:-1}"
restart_policy:
condition: on-failure
# No `ports:` — the worker serves no public traffic; GET /healthz on its own
# container port is for the image's own HEALTHCHECK (Swarm task health) only.
networks:
internal:
driver: overlay
driver_opts:
# VXLAN-over-WireGuard double encapsulation needs a lower MTU than the
# ~1450 overlay default, or large payloads (a /cell bundle, the homepage
# feed) silently stall while small packets (health checks) keep passing
# (hazard #13). Validate with a real large-payload transfer, not just a
# ping, once the mesh is up (Track B).
com.docker.network.driver.mtu: "1370"
volumes:
pgdata: {}
appdata: {}
applogs: {}
# Declared here, provisioned externally (docker secret create <name> - < file) —
# never by this stack, and never committed. See Stage 0 of the cutover runbook
# for which values are continuity-critical (AUTH_SECRET, VAPID, POSTGRES_PASSWORD
# must be the EXISTING live values, not freshly generated ones).
secrets:
postgres_password:
external: true
thermograph_auth_secret:
external: true
thermograph_vapid_private_key:
external: true
thermograph_vapid_public_key:
external: true
thermograph_discord_webhook:
external: true

87
observability/CLAUDE.md
View file

@ -1,55 +1,52 @@
# thermograph-observability — agent instructions # observability/ — agent instructions
The **logging/metrics stack** for the Thermograph fleet: Loki + Grafana on beta, The **logging stack** for the fleet: Loki + Grafana on beta, with a Grafana Alloy
with a Grafana Alloy agent on every node (prod, beta, dev) shipping container and agent on every node (prod, beta, dev) shipping container and app logs over the
app logs over the WireGuard mesh. Grafana is fronted by beta's Caddy at WireGuard mesh. Grafana is fronted by beta's Caddy at
`dashboard.thermograph.org` (Google SSO, pre-provisioned users only). **`dashboard.thermograph.org`** (Google SSO, pre-provisioned users only) — use
that hostname everywhere, never `grafana.thermograph.org`.
This is **operational infra config, not application code** — there is no build. This is operational config, not application code: there is **no build** and no
It deploys by hand: `docker compose up -d` on beta for Loki+Grafana, and the deploy automation. It ships by hand — `docker compose up -d` on beta for
Alloy agent (`alloy/docker-compose.agent.yml`) on each node. See the README for Loki+Grafana, and `alloy/docker-compose.agent.yml` on each node. Read the root
the full per-node procedure. `CLAUDE.md` first; the branch model there applies here too.
## Layout ## Layout
- `docker-compose.yml` — the Loki + Grafana stack (runs on beta). - `docker-compose.yml` — the Loki + Grafana stack (beta).
- `loki/config.yml` — Loki config (mesh-only, filesystem storage). - `loki/config.yml` — mesh-only, filesystem storage.
- `grafana/provisioning/` — datasource (Loki) + dashboard provider, auto-loaded - `grafana/provisioning/` — datasource + dashboard provider, auto-loaded at
at startup. `grafana/dashboards/*.json` — the dashboards themselves. startup. `grafana/dashboards/*.json` — the dashboards.
- `grafana/provisioning/alerting/` — the alert rules, the Discord contact point - `grafana/provisioning/alerting/` — alert rules, the Discord contact point, and
and the notification policy, also auto-loaded at startup. Same rule as the the notification policy.
dashboards: **the repo is the only durable path**, UI edits get overwritten. - `alloy/config.alloy` + `alloy/docker-compose.agent.yml` — the per-node shipper.
Alerts go to Discord `#ops-alerts`, never email — beta's Grafana relays SMTP Node name comes from `ALLOY_NODE`, set per host.
through prod's Postfix, so email dies exactly when prod does. Every rule is - `caddy-grafana.conf` — the beta Caddy vhost, kept here for reference.
LogQL (there is no Prometheus anywhere in the fleet). Thresholds were derived - `.env.example` — copy to `.env` on beta. `.env` is gitignored; never commit
from real Loki data and the working is in the comments beside each rule — OAuth secrets or the admin password.
re-derive before changing a number rather than guessing.
- `alloy/config.alloy` + `alloy/docker-compose.agent.yml` — the per-node log
shipper. The node name comes from `ALLOY_NODE` (set per host).
- `caddy-grafana.conf` — the Caddy vhost for `dashboard.thermograph.org` (lives
in beta's Caddy config, kept here for reference).
- `.env.example` — copy to `.env` on beta before `docker compose up`. `.env` is
gitignored; never commit real OAuth secrets or the admin password.
## Conventions ## Rules
- **The repo is the only durable path.** Dashboards and alerting are provisioned
from this directory at startup; edits made through Grafana's UI or API are
overwritten on the next provision. Changes go through a PR.
- **Every artifact that ships to a node is CI-validated** - **Every artifact that ships to a node is CI-validated**
(`.forgejo/workflows/observability-validate.yml`): both compose files parse, all (`.forgejo/workflows/observability-validate.yml`, at the repo root): both
dashboard JSON is valid, and the Loki/provisioning YAML parses. Keep new compose files parse, all dashboard JSON is valid, the Loki and provisioning
dashboards as valid JSON and new config as valid YAML or CI fails. The alerting YAML parse, and the Alloy config is checked with the pinned `alloy` binary
config gets a stricter third step — every rule's `condition` must name a refId (v1.9.1, matching what the fleet runs).
that exists, every policy must route to a receiver that exists, and a literal - **Alerting gets a stricter third step.** Every rule's `condition` must name a
Discord webhook URL in the repo is a hard failure. (A rule pointing at a missing refId that exists, every policy must route to a receiver that exists, and a
refId is valid YAML, provisions cleanly, and then never fires; that is precisely literal Discord webhook URL in the repo is a hard failure. A rule pointing at a
the silent-no-op this whole domain exists to prevent.) The Alloy config is still missing refId is valid YAML, provisions cleanly, and then never fires — that
not CI-validated (needs the `alloy` binary). silent no-op is what this check exists to prevent.
- The public hostname is **`dashboard.thermograph.org`** everywhere — not - **Alerts go to Discord `#ops-alerts`, never email.** Beta's Grafana relays SMTP
`grafana.thermograph.org`. Match it in any new comment/config. through prod's Postfix, so email dies exactly when prod does.
- Secrets (OAuth client id/secret, admin password) live only in the host `.env`, - **Every rule is LogQL** — there is no Prometheus anywhere in the fleet.
never in the repo. Thresholds were derived from real Loki data and the working is in the comments
beside each rule; re-derive before changing a number rather than guessing.
- Secrets (OAuth client id/secret, admin password) live only in the host `.env`.
## Branching ## Commits & PRs
Single `main` branch, no protection. Open a PR against `main`; the validator Concise and technical. Never mention AI, assistants or automated authorship.
gates it. Deploying the change to beta / the nodes is a separate manual step
(this repo has no deploy automation — a known gap).