# 4. Backend deep dive FastAPI on Python 3.12, plus a Go daemon that ships in the same image. Owns every piece of climate data and every bit of grading logic in the project. Read [`backend/CLAUDE.md`](../../backend/CLAUDE.md) alongside this — it is the binding, terse version. ## The one-process-many-roles model The backend image runs **five different jobs** depending on `THERMOGRAPH_ROLE` and which binary compose/Swarm selects. One image means version skew between them is impossible. | Service | How it's selected | Does | |---|---|---| | **web** | `THERMOGRAPH_ROLE=web` | Serves requests. Never starts the notifier, even if it would win the election — so the web tier scales to N replicas without also scaling background sweeps. | | **worker** | `THERMOGRAPH_ROLE=worker` | Owns the subscription notifier. Still serves requests today. | | **all** | default | Both — the single-process dev default. | | **lake** | `THERMOGRAPH_ROLE=lake` | `deploy/entrypoint.sh` execs `uvicorn lake_app:app` on port 8141 instead. No database, no migrations. Runs in every environment — its own Swarm service on prod and beta (`lake` / `beta-lake`), a compose service on dev. | | **daemon** | compose/Swarm sets the command to `/usr/local/bin/thermograph-daemon` | The Go binary: Discord gateway + cron timers. | `deploy/entrypoint.sh` runs `alembic upgrade head` (retried — a fresh Postgres volume can still be starting) and then serves. `/healthz` and `/api/version` are deliberately **I/O-free** so they stay cheap under tight healthcheck intervals; they're also deliberately **not** under `THERMOGRAPH_BASE`, so a healthcheck needn't know the base path. ## The data pipeline ### Where history comes from — the source ladder `data/climate.py::_load_history` walks this ladder on a cache miss, and the order matters: 1. **The ERA5 lake** (`data/era5lake.py`) — our own parquet extract of the public ERA5 archive in Contabo object storage. True ERA5 with measured gusts, no third-party API in the path. Unconfigured or missing-point cells fall through at zero cost. 2. **The Open-Meteo archive** — the same ERA5 family, so a backup-sourced record grades consistently with lake-sourced neighbours. Guarded by its own rate-limit cooldown. 3. **Stale cache**, served without rewriting it (so `recent_stamp` stays old and the next request retries upstream first). 4. Otherwise `WeatherUnavailable` → a clean 503. **NASA POWER is retired from every serving path.** Its MERRA-2 record ran ~1.3 °F off ERA5 and carried no gusts. `drift_check.py` still fetches it, and only that. Every source must return a plausibly-full span (`MIN_ARCHIVE_DAYS`) before it's accepted and cached as complete — a short response is rejected, not stored. ### Where "today" comes from `_load_recent_forecast` is a separate ladder: 1. **Open-Meteo forecast API** — recent past *and* forward days in one call, with gusts, ERA5-consistent with the history record. Skipped while its own cooldown is active. 2. **Fallback** — an archive recent-past range plus **MET Norway** forward days. Costs no Open-Meteo forecast quota. 3. Stale cache, again without rewriting. Cached per cell for `FORECAST_TTL_HOURS`, so it tracks model updates without hourly upstream I/O. `RECENT_PAST_DAYS = 25`, `FORECAST_DAYS = 8`. ### Concurrency and quota guards - A per-cell lock serialises archive fetches, so N concurrent first-requests for one cell cost one fetch. - Rate limits set module-level cooldowns (`_archive_cooldown_until`, `_forecast_cooldown_until`) rather than retrying into the wall. - Reverse geocoding runs on **one dedicated worker thread** draining a queue at ~1 req/s. The earlier design slept inside the caller's thread — which, in the server, is one of Starlette's shared sync-threadpool threads. ## Storage: three tiers with three different contracts | Tier | Module | Backend | Contract | |---|---|---|---| | **Raw climate record** | `data/climate_store.py` | TimescaleDB `climate_history` hypertable (durable), `climate_recent` (rewritten each refresh), `climate_sync` (freshness) | **Source of truth.** Expensive to refetch. Fail-soft: a miss degrades to *fetch-from-upstream*. Parquet files instead when `THERMOGRAPH_DATABASE_URL` isn't Postgres (dev, tests, offline tooling). | | **Derived payloads** | `data/store.py` | SQLite WAL, or **UNLOGGED** Postgres tables | **Pure accelerator.** Every reader falls back to recomputing from the raw record; every helper swallows its own errors. Deleting the store is a safe full reset. | | **Accounts** | `accounts/db.py` | Postgres (or SQLite) via SQLAlchemy | **Not regenerable.** Foreign keys enforced, errors surface rather than get swallowed, must be backed up. | That third row is the important distinction: accounts, subscriptions and notifications have no source to recompute from, so they live under stricter rules than everything else in the process. On Postgres, `accounts/db.py` runs **two async engines** over one primary — read-only (`default_transaction_read_only`) for pure-GET endpoints, read-write for everything else — plus a separate **sync** psycopg engine for the notifier thread, which has no event loop. ### The validity token Derived rows are keyed `(kind, cell_id, request_key)` and guarded by a `token` that encodes everything the payload depends on: ```python PAYLOAD_VER = "p2" # api/payloads.py history_token(history) → f"{PAYLOAD_VER}:{hist_end(history)}" content_token(...) → f"{PAYLOAD_VER}:{CONTENT_VER}:{max_date}" ``` A stored token that doesn't match the caller's current token is a **miss**. There is no clock-based expiry anywhere. That's why one `PAYLOAD_VER` bump atomically orphans every pre-upgrade row — and why forgetting to bump it serves old-shaped JSON forever. `CONTENT_VER` is kept separate so a content-only shape change doesn't invalidate the whole grading cache. The same token doubles as a **weak ETag**. See [contracts](06-contracts.md) for what that means for CORS. ## Grading `data/grading.py` is the heart of the product: - For a target day of year, the reference distribution is every historical day within **±`HALF_WINDOW` (7) days**, wrapping the year boundary — a 15-day seasonal window. - The observed value is placed as an **empirical mid-rank percentile**, which handles ties correctly (crucial: most days have zero precipitation). - `TEMP_BANDS` is a symmetric seven-tier ladder; `RAIN_BANDS` is separate, and dry days are coloured by dry-streak length instead of rank. - `pct_ordinal()` floors a percentile into `1..99` — never 0, never 100. `frontend/static/shared.js::pctOrd()` mirrors it. If they diverge, the same reading says two different things on two surfaces. `data/scoring.py` is a different question: how far a cell's **recent years** have drifted from its full multi-decade baseline, per metric and per percentile category, expressed in unit-free percentile points. Note `baseline_overlaps_recent` in the payload — the baseline deliberately includes the recent years, which mildly and uniformly attenuates the divergence. ## The HTTP surface `web/app.py` mounts routers so that every version stays served simultaneously: ``` /api → v1 (legacy unversioned alias) /api/v1 → v1 geocode, grade /api/v2 → v2 geocode, suggest, place, grade, calendar, day, score, forecast, cell, metrics, event /api/v2 → content_routes: /content/hub, /content/sitemap, /content/indexnow-key, /content/home, /content/city/{slug}[/month/{month}|/records] /api/v2/auth, /users fastapi-users: login, logout, register, verify, me /api/v2/subscriptions, /notifications, /push/* accounts/api_accounts.py /api/v2/discord/* account linking (OAuth2) /internal/* the Go daemon's control surface — NOT under BASE /healthz, /api/version I/O-free /discord/interactions Discord slash-command webhook (Ed25519-verified) /digest the footer signup form /{path:path} catch-all proxy to the frontend ← registered LAST ``` Two routes worth understanding properly: **`GET /api/v2/cell`** — one bundle carrying every view's payload for a cell, so the client warms four views with one request. Each slice is the *exact* payload its per-view endpoint returns, built by the same builders under the same derived-store keys and tokens, paired with the ETag that endpoint would emit. The client seeds its per-view cache from the slices and later revalidates each view individually. `prefetch=1` is the warm-only mode with a hard guarantee: **it never spends weather-API quota** — a cold cell answers `204` with no body. `neighbors=1` additionally enqueues the 8 surrounding cells for background warming, also warm-only. **The catch-all proxy** — the frontend owns every page and asset. In prod and beta, Caddy path-splits directly, so this proxy is never exercised. It exists as the fallback for environments with no Caddy in front (dev, bare-metal). It forwards `X-Forwarded-Host`/`-Proto` so the frontend can build correct absolute URLs instead of resolving to the internal hop's own address. Because it's registered dead last, every real backend route wins first — `/internal/*` in particular is registered before it, deliberately. ## Background work: who runs what, and where This is where the project's hardest-won lessons live. - **Neighbour warmer** — a per-worker daemon thread draining a request-fed queue. Correct to run in every worker: each drains its own queue. - **Subscription notifier** (`notifications/notify.py`) — a timer-driven sweep that issues upstream fetches independent of any request. Running it in N workers multiplies Open-Meteo quota use N-fold. That is not hypothetical: it is the documented cause of overnight 429/503s after prod went to 3 workers. So it runs behind `singleton.claim_leader()`: - `claim()` — a non-blocking `flock` on a lockfile, arbitrating workers **on one machine**; - `claim_pg()` — a Postgres advisory lock, visible to every host talking to the same database, arbitrating **cluster-wide** under Swarm; - unset (dev, tests, single worker) ⇒ always the leader. - **Heartbeat** — deliberately **unguarded** across every worker and replica. A beat is one local file append (no quota, no DB, no lock), so duplicates are free, and only the process actually being dead stops them. Container-log silence can't distinguish a live web/worker process from a dead one, so this is the signal observability alerts on. - **Discord gateway + recurring jobs** — moved *out* of this process entirely into the Go daemon. See below. The notifier's own quiet guards: a `UNIQUE(subscription_id, event_date, metric, direction, kind)` row-level dedup, and a **weekly cap** — a subscription that notified within 7 days is skipped entirely. Archive reads come from cache; a freshly-subscribed cell is fetched **once**, budget-capped per pass. ## The Go daemon (`backend/daemon/`) One binary, built into the backend image, running as exactly one replica. **Why it exists:** the Discord gateway tolerates one session per token, and duplicated timers repeat every job. One replica makes that a *deployment fact* instead of a runtime election. **Why it's thin:** Go owns no climate or grading logic. Grading needs polars and the cache, and the slash-command path deliberately shares one grade builder with the API so bot grades and API grades can never drift. So anything data-shaped is a POST back into Python: | Route | Purpose | |---|---| | `POST /internal/discord/grade` | `{"query": "phoenix"}` → gateway-ready Discord message JSON, relayed verbatim | | `POST /internal/jobs/warm-cities` | trigger the warm-cities job | | `POST /internal/jobs/indexnow` | trigger IndexNow check-and-submit | Every request carries `X-Thermograph-Internal-Token`. **Both ends fail closed**: with no `THERMOGRAPH_INTERNAL_TOKEN`, Python answers 404 across the whole surface and the daemon refuses to start. Caddy never routes `/internal/*` publicly, so the token is defence in depth, not the only control. The cron half runs unconditionally (`warm-cities` every 24h, `indexnow` every 6h, both configurable). A fatal gateway error logs loudly and leaves cron running — a Discord misconfiguration must not become a crash loop that also stops the schedule. The cron scheduler defers the first run by one interval, never overlaps a job with itself, and drops ticks that fire mid-run. ## The lake (`lake_app.py` + `data/era5lake.py`) Runs as a Swarm service on both prod and beta (`lake` / `beta-lake`, same image, `THERMOGRAPH_ROLE=lake`), and as a compose service on dev. The lake is parquet in an S3-compatible bucket (Contabo, `era5-thermograph`), extracted once from the public Earthmover ERA5 Icechunk archive by `gen_era5_lake.py`. Two layouts under `era5/`: - `daily/tile=_/year=/month=/part.parquet` — the analytical hive table. The location grain is a 12×12-point tile (~3°×3°) aligned to the source's chunk grid; a per-point-per-month grain would be ~170M objects. - `points/lat=/lon=.parquet` — the serving projection: one file per grid point with its full history, so the hot path is a single GET. - `manifest.parquet` — an index of every point file; readers prune with it and the extractor resumes from it. Endpoints: `/healthz`, `GET /history?lat=&lon=` (the hot path — nearest point's history as parquet bytes, disk-cached per point), `POST /query` (SELECT-only DuckDB SQL over `era5_daily` and `manifest`). Two operational facts: **Contabo is path-style only** with `region=default` (virtual-host addressing fails outright), and DuckDB's `httpfs`/`iceberg` extensions are baked into the image **as uid 10001**, because extensions install to the invoking user's `~/.duckdb` and a root-time bake left the runtime user unable to `LOAD` them — seen live as prod's lake `/query` returning 500. Filter on partition columns (`tile`/`year`/`month`). That predicate is the difference between a pruned scan and reading the whole warehouse. ## Observability hooks inside the app - `core/audit.py` writes JSONL streams: `audit` (one line per graded run, with per-phase timings and a `full`/`partial` run type), `errors` (upstream failures tagged `retry` or `error`), plus access, activity and heartbeat. Alloy parses these so `level`/`tag`/`phase` become Loki labels. - `core/metrics.py` keeps since-start counters, shared across uvicorn workers via `THERMOGRAPH_METRICS_DB` (a SQLite file) — unset keeps the zero-dependency in-memory store. Read over the token-gated `GET /api/v2/metrics`. - Every unhandled 500 gets an explicit exception handler emitting `tag="http.error"`. Without it the exception propagates past the middleware and the 500 is invisible in both the access log and metrics. ## Working on it ```bash cd backend make test # hermetic pytest, ~12s make test ARGS='tests/data/test_grading.py -q' make smoke # build the image, boot it + a throwaway db ``` CI runs the suite **inside the built image**, so the exact interpreter and dependencies that ship are what gets tested. Next: [Frontend deep dive](05-frontend.md).