The derived store's key/token formats existed in three places — each
endpoint, api_cell's slice assembly, and migrate.py — where any drift
would silently split the cache (endpoints missing rows the bundle wrote,
migrate materializing rows nobody reads). They are now defined once in
views.py (grade_key/calendar_key/day_key/forecast_key, history_token/
recent_token/day_token) and consumed everywhere, with a pinning test so
a format change is always deliberate.
The four data endpoints shared two copy-pasted sequences, now helpers:
- _fetch_history: history (+ optional recent bundle) fetch with upstream
failures mapped to clean HTTP errors and an empty record to 404.
- _cached_response: If-None-Match 304 / token-valid store replay /
build + persist + serve, with calendar's dont-persist-placeless rule
as an explicit flag.
Each endpoint is now its audit run + identity + a build callback (~10
lines); api_day's hourly-token special case moved into day_token. New
tests: identity format pins, rate-limit 503 parametrized across all five
data routes, prefetch=1 never touching upstream (cold 204, warm
history-only slices), and calendar's placeless-payload retry behavior.
app.py had grown three co-resident strata: HTTP/caching plumbing, payload
assembly, and search policy. This moves the payload layer — the four
build_* functions, cal_span clamping, hist_end, PAYLOAD_VER, NullRun and
their helpers — into a new views.py with no web dependencies (pure moves,
public names). app.py keeps routing, ETag/derived-store plumbing, and
page serving; migrate.py imports views instead of reaching into app's
privates.
That import previously constructed the whole FastAPI app and — because
places.start_loading() ran at import time — kicked off a background
GeoNames download from an offline batch script. The index load now hangs
off the app's lifespan hook, so it fires when a server starts, not when
the module is imported (tests, migrate).
New tests: cal_span clamping (months-back default, record bounds, 2-year
cap, inversion), builder shapes (grade window ordering, day obs from the
recent bundle, forecast future-only), a regression test that build_day
degrades to climatology when the recent fetch fails, and a subprocess
layering guard that importing views/migrate pulls in neither FastAPI nor
the app module and starts no index download.
* Compress API responses and revalidate static assets instead of re-downloading
- Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x.
- Serve pages/assets with Cache-Control: no-cache instead of no-store, so
browsers revalidate via the ETag/Last-Modified that FileResponse and
StaticFiles already emit. Unchanged assets now cost an empty 304 rather
than a full transfer on every page navigation, while deploys still show
up immediately.
* Persist derived responses in SQLite so grading is computed once per cell, not per request
New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet
records — finished grade/calendar/day/forecast payloads and reverse-geocode
labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms)
becomes a ~5ms database read that survives restarts and is shared across views.
Raw parquet stays the source of truth; the store is a pure accelerator (every
reader falls back to recomputing on a miss, and deleting the db is a safe reset).
Freshness is token-driven, not clock-driven: each cached payload is validated by
a token encoding what it was computed from (payload schema version, the archive
record's end date, the recent-fetch stamp). The existing freshness drivers are
untouched — get_history still tops up the tail hourly and get_recent_forecast
still refetches hourly — and tokens are derived from what they return, so cached
payloads expire exactly when their inputs change. The same tokens double as weak
ETags: If-None-Match answers with an empty 304 without touching the payload.
- backend/store.py: derived-payload + revgeo tables, thread-local WAL conns,
every helper fail-soft.
- app.py: endpoints split into pure payload builders + HTTP/caching shells; the
in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store).
- climate.py: revgeo persisted through the store; recent_stamp() and
load_cached_history() (no-network read) helpers.
- backend/migrate.py + make migrate: idempotent, resumable backfill of the store
from existing parquet caches (default calendar span + latest-day detail +
revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather.
- grid.from_id(): rebuild a cell from its cache filename (migrate tooling).
* Add /api/v2/cell: one bundle carrying every view's payload
GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months)
and day (today) payloads in one response, each the exact payload its per-view
endpoint returns — built by the same builders and cached under the same
derived-store keys/tokens — paired with the etag that endpoint would emit. The
frontend can warm all views with a single request, seed its per-view cache from
the slices, and later revalidate each view individually with If-None-Match. The
bundle's own etag combines the slices', so an unchanged bundle is an empty 304.
prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard
guarantee: it never spends weather-API quota. A cell with no cached archive
answers 204, and only the history-derived slices (calendar + latest-day detail)
are built. At most one Nominatim lookup for a never-labeled cell.
* Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch
The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota,
which multi-year calendar payloads regularly blew through) to IndexedDB, with an
in-memory map in front. Entries carry the server's ETag, so anything stale
revalidates conditionally — unchanged data costs an empty 304 and a re-stamp,
never a re-transfer. On network failure the stale copy is served over an error.
getJSON gains an optional onUpdate callback opting into stale-while-revalidate:
the three views (weekly, day, calendar) now render a cached copy immediately —
spinners are delayed 150ms so warm loads never flash-blank — and repaint only if
background revalidation finds changed data. New data shows up the moment it
exists instead of waiting out a TTL.
Cross-view prefetch collapses from one request per view to a single /api/v2/cell
bundle, whose slices (exact per-view payloads + their etags) are seeded under the
URLs each view actually requests; the bundle call itself is conditional via a
remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side
with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one
possible Nominatim lookup each), so tapping nearby lands on already-graded data.
Legacy tg:* storage entries are cleared once; cache entries untouched for two
weeks are pruned on page load.
