thermograph/docs/onboarding/04-backend.md

4. Backend deep dive

FastAPI on Python 3.12, plus a Go daemon that ships in the same image. Owns every piece of climate data and every bit of grading logic in the project.

Read backend/CLAUDE.md alongside this — it is the binding, terse version.

The one-process-many-roles model

The backend image runs five different jobs depending on THERMOGRAPH_ROLE and which binary compose/Swarm selects. One image means version skew between them is impossible.

Service	How it's selected	Does
web	THERMOGRAPH_ROLE=web	Serves requests. Never starts the notifier, even if it would win the election — so the web tier scales to N replicas without also scaling background sweeps.
worker	THERMOGRAPH_ROLE=worker	Owns the subscription notifier. Still serves requests today.
all	default	Both — the single-process dev default.
lake	THERMOGRAPH_ROLE=lake	deploy/entrypoint.sh execs uvicorn lake_app:app on port 8141 instead. No database, no migrations. Runs in every environment — its own Swarm service on prod and beta (lake / beta-lake), a compose service on dev.
daemon	compose/Swarm sets the command to /usr/local/bin/thermograph-daemon	The Go binary: Discord gateway + cron timers.

deploy/entrypoint.sh runs alembic upgrade head (retried — a fresh Postgres volume can still be starting) and then serves. /healthz and /api/version are deliberately I/O-free so they stay cheap under tight healthcheck intervals; they're also deliberately not under THERMOGRAPH_BASE, so a healthcheck needn't know the base path.

The data pipeline

Where history comes from — the source ladder

data/climate.py::_load_history walks this ladder on a cache miss, and the order matters:

1. The ERA5 lake (data/era5lake.py) — our own parquet extract of the public ERA5 archive in Contabo object storage. True ERA5 with measured gusts, no third-party API in the path. Unconfigured or missing-point cells fall through at zero cost.
2. The Open-Meteo archive — the same ERA5 family, so a backup-sourced record grades consistently with lake-sourced neighbours. Guarded by its own rate-limit cooldown.
3. Stale cache, served without rewriting it (so recent_stamp stays old and the next request retries upstream first).
4. Otherwise WeatherUnavailable → a clean 503.

NASA POWER is retired from every serving path. Its MERRA-2 record ran ~1.3 °F off ERA5 and carried no gusts. drift_check.py still fetches it, and only that.

Every source must return a plausibly-full span (MIN_ARCHIVE_DAYS) before it's accepted and cached as complete — a short response is rejected, not stored.

Where "today" comes from

_load_recent_forecast is a separate ladder:

1. Open-Meteo forecast API — recent past and forward days in one call, with gusts, ERA5-consistent with the history record. Skipped while its own cooldown is active.
2. Fallback — an archive recent-past range plus MET Norway forward days. Costs no Open-Meteo forecast quota.
3. Stale cache, again without rewriting.

Cached per cell for FORECAST_TTL_HOURS, so it tracks model updates without hourly upstream I/O. RECENT_PAST_DAYS = 25, FORECAST_DAYS = 8.

Concurrency and quota guards

• A per-cell lock serialises archive fetches, so N concurrent first-requests for one cell cost one fetch.
• Rate limits set module-level cooldowns (_archive_cooldown_until, _forecast_cooldown_until) rather than retrying into the wall.
• Reverse geocoding runs on one dedicated worker thread draining a queue at ~1 req/s. The earlier design slept inside the caller's thread — which, in the server, is one of Starlette's shared sync-threadpool threads.

Storage: three tiers with three different contracts

Tier	Module	Backend	Contract
Raw climate record	data/climate_store.py	TimescaleDB climate_history hypertable (durable), climate_recent (rewritten each refresh), climate_sync (freshness)	Source of truth. Expensive to refetch. Fail-soft: a miss degrades to fetch-from-upstream. Parquet files instead when THERMOGRAPH_DATABASE_URL isn't Postgres (dev, tests, offline tooling).
Derived payloads	data/store.py	SQLite WAL, or UNLOGGED Postgres tables	Pure accelerator. Every reader falls back to recomputing from the raw record; every helper swallows its own errors. Deleting the store is a safe full reset.
Accounts	accounts/db.py	Postgres (or SQLite) via SQLAlchemy	Not regenerable. Foreign keys enforced, errors surface rather than get swallowed, must be backed up.

That third row is the important distinction: accounts, subscriptions and notifications have no source to recompute from, so they live under stricter rules than everything else in the process.

On Postgres, accounts/db.py runs two async engines over one primary — read-only (default_transaction_read_only) for pure-GET endpoints, read-write for everything else — plus a separate sync psycopg engine for the notifier thread, which has no event loop.

The validity token

Derived rows are keyed (kind, cell_id, request_key) and guarded by a token that encodes everything the payload depends on:

PAYLOAD_VER = "p2"                       # api/payloads.py
history_token(history) → f"{PAYLOAD_VER}:{hist_end(history)}"
content_token(...)     → f"{PAYLOAD_VER}:{CONTENT_VER}:{max_date}"

A stored token that doesn't match the caller's current token is a miss. There is no clock-based expiry anywhere. That's why one PAYLOAD_VER bump atomically orphans every pre-upgrade row — and why forgetting to bump it serves old-shaped JSON forever. CONTENT_VER is kept separate so a content-only shape change doesn't invalidate the whole grading cache.

The same token doubles as a weak ETag. See contracts for what that means for CORS.

Grading

data/grading.py is the heart of the product:

• For a target day of year, the reference distribution is every historical day within ±HALF_WINDOW (7) days, wrapping the year boundary — a 15-day seasonal window.
• The observed value is placed as an empirical mid-rank percentile, which handles ties correctly (crucial: most days have zero precipitation).
• TEMP_BANDS is a symmetric seven-tier ladder; RAIN_BANDS is separate, and dry days are coloured by dry-streak length instead of rank.
• pct_ordinal() floors a percentile into 1..99 — never 0, never 100. frontend/static/shared.js::pctOrd() mirrors it. If they diverge, the same reading says two different things on two surfaces.

data/scoring.py is a different question: how far a cell's recent years have drifted from its full multi-decade baseline, per metric and per percentile category, expressed in unit-free percentile points. Note baseline_overlaps_recent in the payload — the baseline deliberately includes the recent years, which mildly and uniformly attenuates the divergence.

The HTTP surface

web/app.py mounts routers so that every version stays served simultaneously:

/api        → v1 (legacy unversioned alias)
/api/v1     → v1        geocode, grade
/api/v2     → v2        geocode, suggest, place, grade, calendar, day,
                        score, forecast, cell, metrics, event
/api/v2     → content_routes:  /content/hub, /content/sitemap,
                        /content/indexnow-key, /content/home,
                        /content/city/{slug}[/month/{month}|/records]
/api/v2/auth, /users    fastapi-users: login, logout, register, verify, me
/api/v2/subscriptions, /notifications, /push/*   accounts/api_accounts.py
/api/v2/discord/*       account linking (OAuth2)
/internal/*             the Go daemon's control surface — NOT under BASE
/healthz, /api/version  I/O-free
/discord/interactions   Discord slash-command webhook (Ed25519-verified)
/digest                 the footer signup form
/{path:path}            catch-all proxy to the frontend  ← registered LAST

Two routes worth understanding properly:

GET /api/v2/cell — one bundle carrying every view's payload for a cell, so the client warms four views with one request. Each slice is the exact payload its per-view endpoint returns, built by the same builders under the same derived-store keys and tokens, paired with the ETag that endpoint would emit. The client seeds its per-view cache from the slices and later revalidates each view individually. prefetch=1 is the warm-only mode with a hard guarantee: it never spends weather-API quota — a cold cell answers 204 with no body. neighbors=1 additionally enqueues the 8 surrounding cells for background warming, also warm-only.

The catch-all proxy — the frontend owns every page and asset. In prod and beta, Caddy path-splits directly, so this proxy is never exercised. It exists as the fallback for environments with no Caddy in front (dev, bare-metal). It forwards X-Forwarded-Host/-Proto so the frontend can build correct absolute URLs instead of resolving to the internal hop's own address. Because it's registered dead last, every real backend route wins first — /internal/* in particular is registered before it, deliberately.

Background work: who runs what, and where

This is where the project's hardest-won lessons live.

• Neighbour warmer — a per-worker daemon thread draining a request-fed queue. Correct to run in every worker: each drains its own queue.
• Subscription notifier (notifications/notify.py) — a timer-driven sweep that issues upstream fetches independent of any request. Running it in N workers multiplies Open-Meteo quota use N-fold. That is not hypothetical: it is the documented cause of overnight 429/503s after prod went to 3 workers. So it runs behind singleton.claim_leader():
  • claim() — a non-blocking flock on a lockfile, arbitrating workers on one machine;
  • claim_pg() — a Postgres advisory lock, visible to every host talking to the same database, arbitrating cluster-wide under Swarm;
  • unset (dev, tests, single worker) ⇒ always the leader.
• Heartbeat — deliberately unguarded across every worker and replica. A beat is one local file append (no quota, no DB, no lock), so duplicates are free, and only the process actually being dead stops them. Container-log silence can't distinguish a live web/worker process from a dead one, so this is the signal observability alerts on.
• Discord gateway + recurring jobs — moved out of this process entirely into the Go daemon. See below.

The notifier's own quiet guards: a UNIQUE(subscription_id, event_date, metric, direction, kind) row-level dedup, and a weekly cap — a subscription that notified within 7 days is skipped entirely. Archive reads come from cache; a freshly-subscribed cell is fetched once, budget-capped per pass.

The Go daemon (backend/daemon/)

One binary, built into the backend image, running as exactly one replica.

Why it exists: the Discord gateway tolerates one session per token, and duplicated timers repeat every job. One replica makes that a deployment fact instead of a runtime election.

Why it's thin: Go owns no climate or grading logic. Grading needs polars and the cache, and the slash-command path deliberately shares one grade builder with the API so bot grades and API grades can never drift. So anything data-shaped is a POST back into Python:

Route	Purpose
POST /internal/discord/grade	{"query": "phoenix"} → gateway-ready Discord message JSON, relayed verbatim
POST /internal/jobs/warm-cities	trigger the warm-cities job
POST /internal/jobs/indexnow	trigger IndexNow check-and-submit

Every request carries X-Thermograph-Internal-Token. Both ends fail closed: with no THERMOGRAPH_INTERNAL_TOKEN, Python answers 404 across the whole surface and the daemon refuses to start. Caddy never routes /internal/* publicly, so the token is defence in depth, not the only control.

The cron half runs unconditionally (warm-cities every 24h, indexnow every 6h, both configurable). A fatal gateway error logs loudly and leaves cron running — a Discord misconfiguration must not become a crash loop that also stops the schedule. The cron scheduler defers the first run by one interval, never overlaps a job with itself, and drops ticks that fire mid-run.

The lake (lake_app.py + data/era5lake.py)

Runs as a Swarm service on both prod and beta (lake / beta-lake, same image, THERMOGRAPH_ROLE=lake), and as a compose service on dev.

The lake is parquet in an S3-compatible bucket (Contabo, era5-thermograph), extracted once from the public Earthmover ERA5 Icechunk archive by gen_era5_lake.py. Two layouts under era5/:

• daily/tile=<ti>_<tj>/year=<Y>/month=<M>/part.parquet — the analytical hive table. The location grain is a 12×12-point tile (~3°×3°) aligned to the source's chunk grid; a per-point-per-month grain would be ~170M objects.
• points/lat=<i>/lon=<j>.parquet — the serving projection: one file per grid point with its full history, so the hot path is a single GET.
• manifest.parquet — an index of every point file; readers prune with it and the extractor resumes from it.

Endpoints: /healthz, GET /history?lat=&lon= (the hot path — nearest point's history as parquet bytes, disk-cached per point), POST /query (SELECT-only DuckDB SQL over era5_daily and manifest).

Two operational facts: Contabo is path-style only with region=default (virtual-host addressing fails outright), and DuckDB's httpfs/iceberg extensions are baked into the image as uid 10001, because extensions install to the invoking user's ~/.duckdb and a root-time bake left the runtime user unable to LOAD them — seen live as prod's lake /query returning 500.

Filter on partition columns (tile/year/month). That predicate is the difference between a pruned scan and reading the whole warehouse.

Observability hooks inside the app

• core/audit.py writes JSONL streams: audit (one line per graded run, with per-phase timings and a full/partial run type), errors (upstream failures tagged retry or error), plus access, activity and heartbeat. Alloy parses these so level/tag/phase become Loki labels.
• core/metrics.py keeps since-start counters, shared across uvicorn workers via THERMOGRAPH_METRICS_DB (a SQLite file) — unset keeps the zero-dependency in-memory store. Read over the token-gated GET /api/v2/metrics.
• Every unhandled 500 gets an explicit exception handler emitting tag="http.error". Without it the exception propagates past the middleware and the 500 is invisible in both the access log and metrics.

Working on it

cd backend
make test                  # hermetic pytest, ~12s
make test ARGS='tests/data/test_grading.py -q'
make smoke                 # build the image, boot it + a throwaway db

CI runs the suite inside the built image, so the exact interpreter and dependencies that ship are what gets tested.

Next: Frontend deep dive.