thermograph/docs/onboarding/04-backend.md

# 4. Backend deep dive

FastAPI on Python 3.12, plus a Go daemon that ships in the same image. Owns
every piece of climate data and every bit of grading logic in the project.

Read [`backend/CLAUDE.md`](../../backend/CLAUDE.md) alongside this — it is the
binding, terse version.

## The one-process-many-roles model

The backend image runs **five different jobs** depending on `THERMOGRAPH_ROLE`
and which binary compose/Swarm selects. One image means version skew between
them is impossible.

| Service | How it's selected | Does |
|---|---|---|
| **web** | `THERMOGRAPH_ROLE=web` | Serves requests. Never starts the notifier, even if it would win the election — so the web tier scales to N replicas without also scaling background sweeps. |
| **worker** | `THERMOGRAPH_ROLE=worker` | Owns the subscription notifier. Still serves requests today. |
| **all** | default | Both — the single-process dev default. |
| **lake** | `THERMOGRAPH_ROLE=lake` | `deploy/entrypoint.sh` execs `uvicorn lake_app:app` on port 8141 instead. No database, no migrations. Runs in every environment — its own Swarm service on prod and beta (`lake` / `beta-lake`), a compose service on dev. |
| **daemon** | compose/Swarm sets the command to `/usr/local/bin/thermograph-daemon` | The Go binary: Discord gateway + cron timers. |

`deploy/entrypoint.sh` runs `alembic upgrade head` (retried — a fresh Postgres
volume can still be starting) and then serves. `/healthz` and `/api/version`
are deliberately **I/O-free** so they stay cheap under tight healthcheck
intervals; they're also deliberately **not** under `THERMOGRAPH_BASE`, so a
healthcheck needn't know the base path.

## The data pipeline

### Where history comes from — the source ladder

`data/climate.py::_load_history` walks this ladder on a cache miss, and the
order matters:

1. **The ERA5 lake** (`data/era5lake.py`) — our own parquet extract of the
   public ERA5 archive in Contabo object storage. True ERA5 with measured
   gusts, no third-party API in the path. Unconfigured or missing-point cells
   fall through at zero cost.
2. **The Open-Meteo archive** — the same ERA5 family, so a backup-sourced
   record grades consistently with lake-sourced neighbours. Guarded by its own
   rate-limit cooldown.
3. **Stale cache**, served without rewriting it (so `recent_stamp` stays old
   and the next request retries upstream first).
4. Otherwise `WeatherUnavailable` → a clean 503.

**NASA POWER is retired from every serving path.** Its MERRA-2 record ran
~1.3 °F off ERA5 and carried no gusts. `drift_check.py` still fetches it, and
only that.

Every source must return a plausibly-full span (`MIN_ARCHIVE_DAYS`) before it's
accepted and cached as complete — a short response is rejected, not stored.

### Where "today" comes from

`_load_recent_forecast` is a separate ladder:

1. **Open-Meteo forecast API** — recent past *and* forward days in one call,
   with gusts, ERA5-consistent with the history record. Skipped while its own
   cooldown is active.
2. **Fallback** — an archive recent-past range plus **MET Norway** forward days.
   Costs no Open-Meteo forecast quota.
3. Stale cache, again without rewriting.

Cached per cell for `FORECAST_TTL_HOURS`, so it tracks model updates without
hourly upstream I/O. `RECENT_PAST_DAYS = 25`, `FORECAST_DAYS = 8`.

### Concurrency and quota guards

- A per-cell lock serialises archive fetches, so N concurrent first-requests for
  one cell cost one fetch.
- Rate limits set module-level cooldowns (`_archive_cooldown_until`,
  `_forecast_cooldown_until`) rather than retrying into the wall.
- Reverse geocoding runs on **one dedicated worker thread** draining a queue at
  ~1 req/s. The earlier design slept inside the caller's thread — which, in the
  server, is one of Starlette's shared sync-threadpool threads.

## Storage: three tiers with three different contracts

| Tier | Module | Backend | Contract |
|---|---|---|---|
| **Raw climate record** | `data/climate_store.py` | TimescaleDB `climate_history` hypertable (durable), `climate_recent` (rewritten each refresh), `climate_sync` (freshness) | **Source of truth.** Expensive to refetch. Fail-soft: a miss degrades to *fetch-from-upstream*. Parquet files instead when `THERMOGRAPH_DATABASE_URL` isn't Postgres (dev, tests, offline tooling). |
| **Derived payloads** | `data/store.py` | SQLite WAL, or **UNLOGGED** Postgres tables | **Pure accelerator.** Every reader falls back to recomputing from the raw record; every helper swallows its own errors. Deleting the store is a safe full reset. |
| **Accounts** | `accounts/db.py` | Postgres (or SQLite) via SQLAlchemy | **Not regenerable.** Foreign keys enforced, errors surface rather than get swallowed, must be backed up. |

That third row is the important distinction: accounts, subscriptions and
notifications have no source to recompute from, so they live under stricter
rules than everything else in the process.

On Postgres, `accounts/db.py` runs **two async engines** over one primary —
read-only (`default_transaction_read_only`) for pure-GET endpoints, read-write
for everything else — plus a separate **sync** psycopg engine for the notifier
thread, which has no event loop.

### The validity token

Derived rows are keyed `(kind, cell_id, request_key)` and guarded by a `token`
that encodes everything the payload depends on:

```python
PAYLOAD_VER = "p2"                       # api/payloads.py
history_token(history) → f"{PAYLOAD_VER}:{hist_end(history)}"
content_token(...)     → f"{PAYLOAD_VER}:{CONTENT_VER}:{max_date}"
```

A stored token that doesn't match the caller's current token is a **miss**.
There is no clock-based expiry anywhere. That's why one `PAYLOAD_VER` bump
atomically orphans every pre-upgrade row — and why forgetting to bump it serves
old-shaped JSON forever. `CONTENT_VER` is kept separate so a content-only shape
change doesn't invalidate the whole grading cache.

The same token doubles as a **weak ETag**. See [contracts](06-contracts.md) for
what that means for CORS.

## Grading

`data/grading.py` is the heart of the product:

- For a target day of year, the reference distribution is every historical day
  within **±`HALF_WINDOW` (7) days**, wrapping the year boundary — a 15-day
  seasonal window.
- The observed value is placed as an **empirical mid-rank percentile**, which
  handles ties correctly (crucial: most days have zero precipitation).
- `TEMP_BANDS` is a symmetric seven-tier ladder; `RAIN_BANDS` is separate, and
  dry days are coloured by dry-streak length instead of rank.
- `pct_ordinal()` floors a percentile into `1..99` — never 0, never 100.
  `frontend/static/shared.js::pctOrd()` mirrors it. If they diverge, the same
  reading says two different things on two surfaces.

`data/scoring.py` is a different question: how far a cell's **recent years**
have drifted from its full multi-decade baseline, per metric and per
percentile category, expressed in unit-free percentile points. Note
`baseline_overlaps_recent` in the payload — the baseline deliberately includes
the recent years, which mildly and uniformly attenuates the divergence.

## The HTTP surface

`web/app.py` mounts routers so that every version stays served simultaneously:

```
/api        → v1 (legacy unversioned alias)
/api/v1     → v1        geocode, grade
/api/v2     → v2        geocode, suggest, place, grade, calendar, day,
                        score, forecast, cell, metrics, event
/api/v2     → content_routes:  /content/hub, /content/sitemap,
                        /content/indexnow-key, /content/home,
                        /content/city/{slug}[/month/{month}|/records]
/api/v2/auth, /users    fastapi-users: login, logout, register, verify, me
/api/v2/subscriptions, /notifications, /push/*   accounts/api_accounts.py
/api/v2/discord/*       account linking (OAuth2)
/internal/*             the Go daemon's control surface — NOT under BASE
/healthz, /api/version  I/O-free
/discord/interactions   Discord slash-command webhook (Ed25519-verified)
/digest                 the footer signup form
/{path:path}            catch-all proxy to the frontend  ← registered LAST
```

Two routes worth understanding properly:

**`GET /api/v2/cell`** — one bundle carrying every view's payload for a cell, so
the client warms four views with one request. Each slice is the *exact* payload
its per-view endpoint returns, built by the same builders under the same
derived-store keys and tokens, paired with the ETag that endpoint would emit.
The client seeds its per-view cache from the slices and later revalidates each
view individually. `prefetch=1` is the warm-only mode with a hard guarantee:
**it never spends weather-API quota** — a cold cell answers `204` with no body.
`neighbors=1` additionally enqueues the 8 surrounding cells for background
warming, also warm-only.

**The catch-all proxy** — the frontend owns every page and asset. In prod and
beta, Caddy path-splits directly, so this proxy is never exercised. It exists as
the fallback for environments with no Caddy in front (dev, bare-metal). It
forwards `X-Forwarded-Host`/`-Proto` so the frontend can build correct absolute
URLs instead of resolving to the internal hop's own address. Because it's
registered dead last, every real backend route wins first —
`/internal/*` in particular is registered before it, deliberately.

## Background work: who runs what, and where

This is where the project's hardest-won lessons live.

- **Neighbour warmer** — a per-worker daemon thread draining a request-fed
  queue. Correct to run in every worker: each drains its own queue.
- **Subscription notifier** (`notifications/notify.py`) — a timer-driven sweep
  that issues upstream fetches independent of any request. Running it in N
  workers multiplies Open-Meteo quota use N-fold. That is not hypothetical: it
  is the documented cause of overnight 429/503s after prod went to 3 workers.
  So it runs behind `singleton.claim_leader()`:
  - `claim()` — a non-blocking `flock` on a lockfile, arbitrating workers **on
    one machine**;
  - `claim_pg()` — a Postgres advisory lock, visible to every host talking to
    the same database, arbitrating **cluster-wide** under Swarm;
  - unset (dev, tests, single worker) ⇒ always the leader.
- **Heartbeat** — deliberately **unguarded** across every worker and replica. A
  beat is one local file append (no quota, no DB, no lock), so duplicates are
  free, and only the process actually being dead stops them. Container-log
  silence can't distinguish a live web/worker process from a dead one, so this
  is the signal observability alerts on.
- **Discord gateway + recurring jobs** — moved *out* of this process entirely
  into the Go daemon. See below.

The notifier's own quiet guards: a `UNIQUE(subscription_id, event_date, metric,
direction, kind)` row-level dedup, and a **weekly cap** — a subscription that
notified within 7 days is skipped entirely. Archive reads come from cache; a
freshly-subscribed cell is fetched **once**, budget-capped per pass.

## The Go daemon (`backend/daemon/`)

One binary, built into the backend image, running as exactly one replica.

**Why it exists:** the Discord gateway tolerates one session per token, and
duplicated timers repeat every job. One replica makes that a *deployment fact*
instead of a runtime election.

**Why it's thin:** Go owns no climate or grading logic. Grading needs polars and
the cache, and the slash-command path deliberately shares one grade builder with
the API so bot grades and API grades can never drift. So anything data-shaped is
a POST back into Python:

| Route | Purpose |
|---|---|
| `POST /internal/discord/grade` | `{"query": "phoenix"}` → gateway-ready Discord message JSON, relayed verbatim |
| `POST /internal/jobs/warm-cities` | trigger the warm-cities job |
| `POST /internal/jobs/indexnow` | trigger IndexNow check-and-submit |

Every request carries `X-Thermograph-Internal-Token`. **Both ends fail closed**:
with no `THERMOGRAPH_INTERNAL_TOKEN`, Python answers 404 across the whole
surface and the daemon refuses to start. Caddy never routes `/internal/*`
publicly, so the token is defence in depth, not the only control.

The cron half runs unconditionally (`warm-cities` every 24h, `indexnow` every
6h, both configurable). A fatal gateway error logs loudly and leaves cron
running — a Discord misconfiguration must not become a crash loop that also
stops the schedule. The cron scheduler defers the first run by one interval,
never overlaps a job with itself, and drops ticks that fire mid-run.

## The lake (`lake_app.py` + `data/era5lake.py`)

Runs as a Swarm service on both prod and beta (`lake` / `beta-lake`, same
image, `THERMOGRAPH_ROLE=lake`), and as a compose service on dev.

The lake is parquet in an S3-compatible bucket (Contabo, `era5-thermograph`),
extracted once from the public Earthmover ERA5 Icechunk archive by
`gen_era5_lake.py`. Two layouts under `era5/`:

- `daily/tile=<ti>_<tj>/year=<Y>/month=<M>/part.parquet` — the analytical hive
  table. The location grain is a 12×12-point tile (~3°×3°) aligned to the
  source's chunk grid; a per-point-per-month grain would be ~170M objects.
- `points/lat=<i>/lon=<j>.parquet` — the serving projection: one file per grid
  point with its full history, so the hot path is a single GET.
- `manifest.parquet` — an index of every point file; readers prune with it and
  the extractor resumes from it.

Endpoints: `/healthz`, `GET /history?lat=&lon=` (the hot path — nearest point's
history as parquet bytes, disk-cached per point), `POST /query` (SELECT-only
DuckDB SQL over `era5_daily` and `manifest`).

Two operational facts: **Contabo is path-style only** with `region=default`
(virtual-host addressing fails outright), and DuckDB's `httpfs`/`iceberg`
extensions are baked into the image **as uid 10001**, because extensions install
to the invoking user's `~/.duckdb` and a root-time bake left the runtime user
unable to `LOAD` them — seen live as prod's lake `/query` returning 500.

Filter on partition columns (`tile`/`year`/`month`). That predicate is the
difference between a pruned scan and reading the whole warehouse.

## Observability hooks inside the app

- `core/audit.py` writes JSONL streams: `audit` (one line per graded run, with
  per-phase timings and a `full`/`partial` run type), `errors` (upstream
  failures tagged `retry` or `error`), plus access, activity and heartbeat.
  Alloy parses these so `level`/`tag`/`phase` become Loki labels.
- `core/metrics.py` keeps since-start counters, shared across uvicorn workers
  via `THERMOGRAPH_METRICS_DB` (a SQLite file) — unset keeps the
  zero-dependency in-memory store. Read over the token-gated
  `GET /api/v2/metrics`.
- Every unhandled 500 gets an explicit exception handler emitting
  `tag="http.error"`. Without it the exception propagates past the middleware
  and the 500 is invisible in both the access log and metrics.

## Working on it

```bash
cd backend
make test                  # hermetic pytest, ~12s
make test ARGS='tests/data/test_grading.py -q'
make smoke                 # build the image, boot it + a throwaway db
```

CI runs the suite **inside the built image**, so the exact interpreter and
dependencies that ship are what gets tested.

Next: [Frontend deep dive](05-frontend.md).