diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..0bd2165 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,161 @@ +# thermograph-backend — agent instructions + +## What this repo is + +The Thermograph **API service**: FastAPI app serving the graded-climate API +(`/api/v2/...`), user accounts (`accounts/`, fastapi-users + Postgres via +alembic), push/email/Discord notifications (`notifications/`), and the +SSR-content JSON API (`api/content_routes.py` + `api/content_payloads.py`) that +the separate `thermograph-frontend`'s server-rendered pages consume. It owns +the climate data pipeline (`data/` — grid snapping, Open-Meteo fetch + parquet +cache, percentile grading/scoring, places/cities) and the derived-payload +cache/ETag store (`data/store.py`, `api/payloads.py`). It does **not** serve +any HTML/CSS/JS itself — that moved to `thermograph-frontend` (repo-split +Stage 7a); this process is a pure JSON API + DB-backed service, launched as +`app:app` (a shim re-exporting `web.app:app`, kept stable for systemd/CI/`run.sh` +callers that still expect that target). + +## Split topology + +Split out of the `emi/thermograph` monorepo (see +`MONOREPO-CUTOVER-PLAN.md` at the workspace root during the transition). Sibling +repos: +- **`thermograph-frontend`** — the client the public sees: static + JS/CSS/HTML + `frontend_ssr`'s server-rendered climate hub/city/month pages, + calling this repo's `/api/v2/...` and `/content/...` JSON endpoints. +- **`thermograph-infra`** — deploy/terraform/Swarm-Compose plumbing: + `docker-compose.yml`, `deploy/deploy.sh`, secrets (SOPS-encrypted + `deploy/secrets/*.yaml`), Terraform for prod/beta provisioning. This repo has + no deploy scripts or terraform of its own — it only builds an image and hands + infra a tag. +- **`thermograph-docs`** — non-code planning layer: architecture decision + records and operator runbooks. Nothing here should duplicate that; if you're + writing a cross-cutting decision doc or a step-by-step operator runbook, + it belongs there, not in this repo's README/CLAUDE.md. +- **`thermograph-observability`** — monitoring/metrics stack, separate from + this repo's own lightweight in-process `core/metrics.py`. + +## Branch, PR & deploy flow + +- `build-push.yml` builds **this repo's own image** + (`git.thermograph.org/emi/thermograph-backend/app`, tagged `sha-<12hex>` on + every push to `dev`/`main`/`release` and additionally by semver on `v*.*.*` + tags) — no more shared monorepo image. `build.yml` (push/PR to `main`) is a + build-only sanity check (`docker build`), not a boot/health check — repeated + attempts at a live boot+healthz check hit this runner's own quirks (bridge-IP + reachability, non-unique `$$`, squatted host ports) without ever going + reliably green; the image's actual boot is verified by hand (`docker run` + + alembic + `/healthz` 200 + a real `/api/v2/place` 200) instead. +- **`main` → beta**: `deploy.yml` SSHes to beta and runs + `thermograph-infra`'s `/opt/thermograph/deploy/deploy.sh` with + `SERVICE=backend` + `BACKEND_IMAGE_TAG=sha-<12hex>` (must match + build-push.yml's 12-hex SHA tag exactly — Actions expressions have no + substring function for the full 40-char SHA). `deploy.sh` rolls only the + `backend` compose service (`--no-deps`) and persists the tag in + `deploy/.image-tags.env` host-side, so a backend-only deploy never touches + the frontend's running container or tag. +- **`release` → prod**: `deploy-prod.yml` is the same shape against a + completely separate secret set (`PROD_SSH_HOST/USER/KEY/PORT`) so a beta + credential leak can't reach prod. Also `SERVICE=backend` + + `BACKEND_IMAGE_TAG`. +- The frontend deploys itself the same way, independently — that + independence (a backend change ships without a frontend deploy and vice + versa) is the entire point of the FE/BE CI-CD split. +- No `dev`-branch LAN deploy path exists in this repo (the monorepo's + `deploy-dev.yml`/`docker-compose.dev.yml` never made it across the split — + known gap, see the cutover plan if resurrecting it). + +## Run / test locally + +There is **no Makefile in this repo** — the monorepo's app-level `make` +targets (`lan-run`, `test`, `venv`, ...) haven't been given a new home yet +(tracked as a gap in the cutover plan). Until that lands, drive it directly: + +```bash +python3 -m venv .venv && .venv/bin/pip install -r requirements.txt -r requirements-dev.txt +.venv/bin/uvicorn app:app --host 0.0.0.0 --port 8137 # serve +.venv/bin/python -m pytest tests # test +``` + +The monorepo's Makefile preferred `uv venv --python 3.12 .venv` (falling back +to plain `python3 -m venv` only if `uv` isn't installed) — prefer `uv` here too +if it's on the box. There is no committed `.venv` in this repo; if one exists +locally and looks broken (import errors, wrong Python), just `rm -rf .venv` and +recreate rather than debugging it — it's gitignored, disposable state, not +something the repo depends on being a specific way. + +Tests are hermetic (`tests/conftest.py`): no real Open-Meteo/Nominatim/GeoNames +calls, a throwaway SQLite for accounts + the derived-payload store, and the +notifier thread disabled. `THERMOGRAPH_FRONTEND_BASE_INTERNAL` is set to an +unreachable placeholder in conftest — `web/app.py` fails loud at import if it's +unset in a real run (it's the internal URL back to `frontend_ssr` for the +non-API HTML routes this process no longer serves itself). + +## API version contract + +- Everything real lives under **`/api/v2/...`**; `/api/` and `/api/v1/` stay + mounted as aliases of the same handlers (they always were — not new). +- **`GET /api/version`** (`web/app.py`) is the capability/negotiation endpoint + a split-out frontend can call at boot: `{backend_version, min_frontend, + payload_ver}`, driven by two module constants next to it — + `API_CONTRACT_VERSION` (bump only on a breaking change to any `/api/v{N}` + route or payload shape) and `MIN_SUPPORTED_FRONTEND` (oldest frontend + contract version this backend still serves correctly). A genuine breaking + change ships as a new `v3` router mounted alongside `v2` re-registering only + the changed handlers, `v2` kept working, `API_CONTRACT_VERSION` bumped in the + same PR. +- **`PAYLOAD_VER`** (`api/payloads.py`, currently `"p2"`) is a *separate* + concern from URL versioning — it's the cache/ETag invalidation token baked + into every derived-store validity token (`history_token()`). Bump it + whenever a response payload's shape changes (new metric, renamed/removed + field) so one bump atomically orphans every pre-upgrade cached row instead + of a stale row's `history_end` happening to still validate under the old + shape. +- Both endpoints (`/healthz`, `/api/version`) are deliberately I/O-free so they + stay cheap under tight healthcheck intervals. + +## Live cross-repo contracts (don't break silently) + +- **`/cell` bundle + ETag/If-None-Match**: every graded payload (`grade`, + `calendar`, `day`) is cached in SQLite by `(kind, cell_id, key)` keyed to a + validity token that only advances when the underlying history actually + changes (`_etag_for` in `web/app.py`, `history_token`/`PAYLOAD_VER` in + `api/payloads.py`). The same token doubles as a weak ETag; a client sending + `If-None-Match` gets an empty 304 without the payload being rebuilt or even + loaded. `expose_headers=["ETag"]` in the CORS middleware matters as much as + `allow_origins` — without it the frontend's `cache.js` reading + `res.headers.get("ETag")` cross-origin silently gets `null` and never + revalidates correctly. Don't change the ETag derivation or the CORS + `expose_headers` list without checking the frontend's cache layer. +- **Percentile/unit parity with `thermograph-frontend`'s `static/shared.js`**: + the frontend's percentile-to-ordinal display logic explicitly mirrors this + repo's `data/grading.py::pct_ordinal()` (floors an empirical percentile into + 1..99 — a rank against the sample can round to 0 or 100, and "100th + percentile" is never shown). `TEMP_BANDS`/`RAIN_BANDS` tier boundaries in + `data/grading.py` are the source of truth for the tier names/thresholds + (Near Record / High / Above Normal / Normal / Below Normal / Low, and the + five rain-intensity tiers) — changing them without a corresponding frontend + update produces a client that draws differently-colored tiers than the + labels the API returns. +- **SSR content API** (`api/content_routes.py`): a separate ETag'd JSON surface + (`/content/hub`, `/content/city/{slug}`, `/content/city/{slug}/month/{month}`, + `/content/city/{slug}/records`, `/content/home`, `/content/sitemap`, + `/content/indexnow-key`) that `frontend_ssr` calls in-process-free (no shared + Python import) to render pages. Same caching pattern as the main API; keep + payload shape changes coordinated with whatever consumes it on the frontend + side. + +## Layout + +`accounts/` (fastapi-users models/schemas/db, `alembic/` migrations), +`api/` (versioned payload builders + SSR content routes), `core/` (metrics, +audit logging, a singleton helper), `data/` (grid, climate fetch/cache, grading/ +scoring, places/cities, the derived-payload store), `notifications/` (push, +email, Discord bot + interactions + linking, digest, scheduler), `web/app.py` +(the actual FastAPI app; `app.py` at the repo root is a one-line re-export +shim). `paths.py` resolves every filesystem location (climate parquet cache, +accounts DB, logs, bundled `cities.json`/`cities_flavor.json`) from the repo +root — don't reintroduce `__file__`-relative paths in a module, it breaks the +moment a module moves. `deploy/entrypoint.sh` is the container entrypoint +(alembic migrate-then-serve, with a Swarm-secrets-as-files shim); actual deploy +orchestration lives in `thermograph-infra`, not here. diff --git a/README.md b/README.md new file mode 100644 index 0000000..ce8ab21 --- /dev/null +++ b/README.md @@ -0,0 +1,104 @@ +# thermograph-backend + +The Thermograph API service: grades recent local weather against ~45 years of +climate history, and hosts the accounts, notification, and SSR-content +back-end that the rest of the split Thermograph stack (`thermograph-frontend`) +talks to over HTTP. Split from the `emi/thermograph` monorepo — this repo owns +the API/DB/accounts/notifications layer only; it renders no HTML/CSS/JS of its +own. + +See [`CLAUDE.md`](CLAUDE.md) for the full split topology, deploy flow, and +API version contract, and [`thermograph-docs`](../thermograph-docs) (a sibling +repo) for cross-cutting architecture decisions and operator runbooks. + +## How it works + +1. **Grid** — a lat/lon is snapped to a stable **~4 sq mi cell** + (`data/grid.py`); longitude spacing is scaled by `cos(latitude)` so cells + stay roughly square at any latitude. The cell id is the cache key, so the + same spot always resolves to the same data. +2. **Data (on-demand, cached to parquet)** — the first request for a cell + fetches the full **1980–present daily record** (max/min temp, precip) from + the free [Open-Meteo](https://open-meteo.com) archive (ERA5) and writes it + to `data/cache/.parquet` (zstd, ~200 KB for 45 years). Later + requests read the parquet directly. Recent days come from Open-Meteo's + forecast API (`past_days`), so history and grading share one source. +3. **Percentiles & grading** (`data/grading.py`) — each day is graded against + every historical day within **±7 days** of it (a 15-day seasonal window, + wrapping year-end), as an empirical mid-rank percentile. Temperature uses a + symmetric tier ladder (`TEMP_BANDS`: Near Record / High / Above Normal / + Normal / Below Normal / Low / Near Record); precipitation is graded + separately (`RAIN_BANDS`) since most days are dry — a rainy day is ranked + only among rain days in its window, dry days are colored by dry-streak + length instead. +4. **Caching & ETags** — every derived payload (grade/calendar/day/SSR + content) is cached in SQLite (`data/store.py`) under `(kind, cell_id, key)`, + validated by a token that only advances when the cell's history actually + changes. That token doubles as a weak ETag, so an unchanged request costs a + 304 with no payload rebuild (`api/payloads.py`, `web/app.py`). + +## Layout + +``` +accounts/ fastapi-users models/schemas/db + api_accounts routes +alembic/ Postgres schema migrations (alembic upgrade head on boot) +api/ versioned payload builders (payloads.py, content_payloads.py) + + route wiring (content_routes.py, sitemap.py, homepage.py) +core/ metrics, audit/access logging, a singleton helper +data/ grid snapping, climate fetch/cache, grading/scoring, + places/cities, the derived-payload store +notifications/ push (VAPID), email, Discord bot (interactions + linking), + monthly digest, the in-process scheduler (city warming, IndexNow) +web/app.py the FastAPI app (routes, CORS, ETag/versioning, middleware) +deploy/ container entrypoint.sh (alembic migrate, then serve) +app.py shim re-exporting web.app:app — keeps the launch target + `app:app` stable regardless of internal package layout +scripts/ one-off admin scripts (Discord slash-command registration) +tests/ pytest suite, hermetic (see tests/conftest.py) +cities.json, +cities_flavor.json bundled reference data (generated by gen_cities.py / + gen_flavor.py), not runtime state +``` + +## How it fits the split + +- **`thermograph-frontend`** calls this service's `GET /api/v2/...` endpoints + (grade, geocode, calendar) and the SSR content endpoints under + `/content/...`; it negotiates compatibility via `GET /api/version`. +- **`thermograph-infra`** owns the deploy/Compose/Terraform layer — this repo + only builds and publishes its own container image + (`git.thermograph.org/emi/thermograph-backend/app`) and hands infra a tag to + roll out (`SERVICE=backend` + `BACKEND_IMAGE_TAG` into infra's + `deploy/deploy.sh`). +- **`thermograph-docs`** holds the cross-repo architecture/runbook docs; this + README only covers what's local to this service. + +## Build & run + +```bash +docker build -t thermograph-backend . +docker run -p 8137:8137 --env-file .env thermograph-backend +``` + +The image runs `deploy/entrypoint.sh`: `alembic upgrade head` against +`THERMOGRAPH_DATABASE_URL` (retried, since a fresh Postgres volume can still be +starting up), then `uvicorn app:app` on `$PORT` (default `8137`) with +`$WORKERS` workers (default 4). `/healthz` is an I/O-free liveness probe. + +For local development without a container, see the "Run / test locally" +section of [`CLAUDE.md`](CLAUDE.md) — there is no Makefile in this repo yet, +so it's a plain venv + `pytest`/`uvicorn` invocation. + +## Notifications: Discord slash commands + +`notifications/discord_interactions.py` answers Discord's HTTP Interactions +endpoint (`/discord/interactions`) for the `/grade ` slash command. +Registering (or updating) the command definition with Discord's REST API is a +one-off admin action, not part of the running app: + +```bash +THERMOGRAPH_DISCORD_APP_ID=... THERMOGRAPH_DISCORD_BOT_TOKEN=... \ + python3 scripts/register_discord_commands.py +``` + +Global command changes can take up to an hour to propagate. diff --git a/scripts/register_discord_commands.py b/scripts/register_discord_commands.py new file mode 100644 index 0000000..38e0a4a --- /dev/null +++ b/scripts/register_discord_commands.py @@ -0,0 +1,49 @@ +#!/usr/bin/env python3 +"""Register (upsert) Thermograph's Discord slash commands. Run once, and again +whenever the command list below changes. + + THERMOGRAPH_DISCORD_APP_ID=... THERMOGRAPH_DISCORD_BOT_TOKEN=... \ + python scripts/register_discord_commands.py + +This is a one-off admin action, not part of the running app: it PUTs the full +command set to Discord's REST API (a global overwrite). The app itself answers the +commands over the HTTP interactions endpoint (notifications/discord_interactions.py) and +needs no bot process. Global commands can take up to an hour to propagate. +""" +import os +import sys + +import httpx + +APP_ID = os.environ.get("THERMOGRAPH_DISCORD_APP_ID", "").strip() +BOT_TOKEN = os.environ.get("THERMOGRAPH_DISCORD_BOT_TOKEN", "").strip() + +# CHAT_INPUT command (type 1) with one required STRING option (type 3). +COMMANDS = [ + { + "name": "grade", + "description": "How unusual is a city's weather today?", + "type": 1, + "options": [ + {"name": "city", "description": "City name, e.g. Tokyo", "type": 3, "required": True}, + ], + }, +] + + +def main() -> int: + if not APP_ID or not BOT_TOKEN: + print("Set THERMOGRAPH_DISCORD_APP_ID and THERMOGRAPH_DISCORD_BOT_TOKEN.", file=sys.stderr) + return 2 + url = f"https://discord.com/api/v10/applications/{APP_ID}/commands" + resp = httpx.put(url, headers={"Authorization": f"Bot {BOT_TOKEN}"}, json=COMMANDS, timeout=30.0) + if resp.status_code >= 300: + print(f"Discord rejected the registration ({resp.status_code}): {resp.text}", file=sys.stderr) + return 1 + names = ", ".join("/" + c["name"] for c in resp.json()) + print(f"Registered {len(COMMANDS)} command(s): {names}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main())