thermograph/app.py

701 lines
31 KiB
Python
Raw Normal View History

"""Thermograph API — grade recent local weather against ~45 years of climatology."""
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21) * Compress API responses and revalidate static assets instead of re-downloading - Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x. - Serve pages/assets with Cache-Control: no-cache instead of no-store, so browsers revalidate via the ETag/Last-Modified that FileResponse and StaticFiles already emit. Unchanged assets now cost an empty 304 rather than a full transfer on every page navigation, while deploys still show up immediately. * Persist derived responses in SQLite so grading is computed once per cell, not per request New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet records — finished grade/calendar/day/forecast payloads and reverse-geocode labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms) becomes a ~5ms database read that survives restarts and is shared across views. Raw parquet stays the source of truth; the store is a pure accelerator (every reader falls back to recomputing on a miss, and deleting the db is a safe reset). Freshness is token-driven, not clock-driven: each cached payload is validated by a token encoding what it was computed from (payload schema version, the archive record's end date, the recent-fetch stamp). The existing freshness drivers are untouched — get_history still tops up the tail hourly and get_recent_forecast still refetches hourly — and tokens are derived from what they return, so cached payloads expire exactly when their inputs change. The same tokens double as weak ETags: If-None-Match answers with an empty 304 without touching the payload. - backend/store.py: derived-payload + revgeo tables, thread-local WAL conns, every helper fail-soft. - app.py: endpoints split into pure payload builders + HTTP/caching shells; the in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store). - climate.py: revgeo persisted through the store; recent_stamp() and load_cached_history() (no-network read) helpers. - backend/migrate.py + make migrate: idempotent, resumable backfill of the store from existing parquet caches (default calendar span + latest-day detail + revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather. - grid.from_id(): rebuild a cell from its cache filename (migrate tooling). * Add /api/v2/cell: one bundle carrying every view's payload GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months) and day (today) payloads in one response, each the exact payload its per-view endpoint returns — built by the same builders and cached under the same derived-store keys/tokens — paired with the etag that endpoint would emit. The frontend can warm all views with a single request, seed its per-view cache from the slices, and later revalidate each view individually with If-None-Match. The bundle's own etag combines the slices', so an unchanged bundle is an empty 304. prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard guarantee: it never spends weather-API quota. A cell with no cached archive answers 204, and only the history-derived slices (calendar + latest-day detail) are built. At most one Nominatim lookup for a never-labeled cell. * Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota, which multi-year calendar payloads regularly blew through) to IndexedDB, with an in-memory map in front. Entries carry the server's ETag, so anything stale revalidates conditionally — unchanged data costs an empty 304 and a re-stamp, never a re-transfer. On network failure the stale copy is served over an error. getJSON gains an optional onUpdate callback opting into stale-while-revalidate: the three views (weekly, day, calendar) now render a cached copy immediately — spinners are delayed 150ms so warm loads never flash-blank — and repaint only if background revalidation finds changed data. New data shows up the moment it exists instead of waiting out a TTL. Cross-view prefetch collapses from one request per view to a single /api/v2/cell bundle, whose slices (exact per-view payloads + their etags) are seeded under the URLs each view actually requests; the bundle call itself is conditional via a remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one possible Nominatim lookup each), so tapping nearby lands on already-graded data. Legacy tg:* storage entries are cleared once; cache entries untouched for two weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
import contextlib
import datetime
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21) * Compress API responses and revalidate static assets instead of re-downloading - Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x. - Serve pages/assets with Cache-Control: no-cache instead of no-store, so browsers revalidate via the ETag/Last-Modified that FileResponse and StaticFiles already emit. Unchanged assets now cost an empty 304 rather than a full transfer on every page navigation, while deploys still show up immediately. * Persist derived responses in SQLite so grading is computed once per cell, not per request New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet records — finished grade/calendar/day/forecast payloads and reverse-geocode labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms) becomes a ~5ms database read that survives restarts and is shared across views. Raw parquet stays the source of truth; the store is a pure accelerator (every reader falls back to recomputing on a miss, and deleting the db is a safe reset). Freshness is token-driven, not clock-driven: each cached payload is validated by a token encoding what it was computed from (payload schema version, the archive record's end date, the recent-fetch stamp). The existing freshness drivers are untouched — get_history still tops up the tail hourly and get_recent_forecast still refetches hourly — and tokens are derived from what they return, so cached payloads expire exactly when their inputs change. The same tokens double as weak ETags: If-None-Match answers with an empty 304 without touching the payload. - backend/store.py: derived-payload + revgeo tables, thread-local WAL conns, every helper fail-soft. - app.py: endpoints split into pure payload builders + HTTP/caching shells; the in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store). - climate.py: revgeo persisted through the store; recent_stamp() and load_cached_history() (no-network read) helpers. - backend/migrate.py + make migrate: idempotent, resumable backfill of the store from existing parquet caches (default calendar span + latest-day detail + revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather. - grid.from_id(): rebuild a cell from its cache filename (migrate tooling). * Add /api/v2/cell: one bundle carrying every view's payload GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months) and day (today) payloads in one response, each the exact payload its per-view endpoint returns — built by the same builders and cached under the same derived-store keys/tokens — paired with the etag that endpoint would emit. The frontend can warm all views with a single request, seed its per-view cache from the slices, and later revalidate each view individually with If-None-Match. The bundle's own etag combines the slices', so an unchanged bundle is an empty 304. prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard guarantee: it never spends weather-API quota. A cell with no cached archive answers 204, and only the history-derived slices (calendar + latest-day detail) are built. At most one Nominatim lookup for a never-labeled cell. * Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota, which multi-year calendar payloads regularly blew through) to IndexedDB, with an in-memory map in front. Entries carry the server's ETag, so anything stale revalidates conditionally — unchanged data costs an empty 304 and a re-stamp, never a re-transfer. On network failure the stale copy is served over an error. getJSON gains an optional onUpdate callback opting into stale-while-revalidate: the three views (weekly, day, calendar) now render a cached copy immediately — spinners are delayed 150ms so warm loads never flash-blank — and repaint only if background revalidation finds changed data. New data shows up the moment it exists instead of waiting out a TTL. Cross-view prefetch collapses from one request per view to a single /api/v2/cell bundle, whose slices (exact per-view payloads + their etags) are seeded under the URLs each view actually requests; the bundle call itself is conditional via a remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one possible Nominatim lookup each), so tapping nearby lands on already-graded data. Legacy tg:* storage entries are cleared once; cache entries untouched for two weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
import hashlib
import json
import os
import time
import pandas as pd
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21) * Compress API responses and revalidate static assets instead of re-downloading - Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x. - Serve pages/assets with Cache-Control: no-cache instead of no-store, so browsers revalidate via the ETag/Last-Modified that FileResponse and StaticFiles already emit. Unchanged assets now cost an empty 304 rather than a full transfer on every page navigation, while deploys still show up immediately. * Persist derived responses in SQLite so grading is computed once per cell, not per request New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet records — finished grade/calendar/day/forecast payloads and reverse-geocode labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms) becomes a ~5ms database read that survives restarts and is shared across views. Raw parquet stays the source of truth; the store is a pure accelerator (every reader falls back to recomputing on a miss, and deleting the db is a safe reset). Freshness is token-driven, not clock-driven: each cached payload is validated by a token encoding what it was computed from (payload schema version, the archive record's end date, the recent-fetch stamp). The existing freshness drivers are untouched — get_history still tops up the tail hourly and get_recent_forecast still refetches hourly — and tokens are derived from what they return, so cached payloads expire exactly when their inputs change. The same tokens double as weak ETags: If-None-Match answers with an empty 304 without touching the payload. - backend/store.py: derived-payload + revgeo tables, thread-local WAL conns, every helper fail-soft. - app.py: endpoints split into pure payload builders + HTTP/caching shells; the in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store). - climate.py: revgeo persisted through the store; recent_stamp() and load_cached_history() (no-network read) helpers. - backend/migrate.py + make migrate: idempotent, resumable backfill of the store from existing parquet caches (default calendar span + latest-day detail + revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather. - grid.from_id(): rebuild a cell from its cache filename (migrate tooling). * Add /api/v2/cell: one bundle carrying every view's payload GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months) and day (today) payloads in one response, each the exact payload its per-view endpoint returns — built by the same builders and cached under the same derived-store keys/tokens — paired with the etag that endpoint would emit. The frontend can warm all views with a single request, seed its per-view cache from the slices, and later revalidate each view individually with If-None-Match. The bundle's own etag combines the slices', so an unchanged bundle is an empty 304. prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard guarantee: it never spends weather-API quota. A cell with no cached archive answers 204, and only the history-derived slices (calendar + latest-day detail) are built. At most one Nominatim lookup for a never-labeled cell. * Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota, which multi-year calendar payloads regularly blew through) to IndexedDB, with an in-memory map in front. Entries carry the server's ETag, so anything stale revalidates conditionally — unchanged data costs an empty 304 and a re-stamp, never a re-transfer. On network failure the stale copy is served over an error. getJSON gains an optional onUpdate callback opting into stale-while-revalidate: the three views (weekly, day, calendar) now render a cached copy immediately — spinners are delayed 150ms so warm loads never flash-blank — and repaint only if background revalidation finds changed data. New data shows up the moment it exists instead of waiting out a TTL. Cross-view prefetch collapses from one request per view to a single /api/v2/cell bundle, whose slices (exact per-view payloads + their etags) are seeded under the URLs each view actually requests; the bundle call itself is conditional via a remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one possible Nominatim lookup each), so tapping nearby lands on already-graded data. Legacy tg:* storage entries are cleared once; cache entries untouched for two weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
from fastapi import APIRouter, FastAPI, HTTPException, Query, Request, Response
from fastapi.middleware.gzip import GZipMiddleware
from fastapi.responses import FileResponse, RedirectResponse
from fastapi.staticfiles import StaticFiles
import audit
import climate
import grading
import grid
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21) * Compress API responses and revalidate static assets instead of re-downloading - Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x. - Serve pages/assets with Cache-Control: no-cache instead of no-store, so browsers revalidate via the ETag/Last-Modified that FileResponse and StaticFiles already emit. Unchanged assets now cost an empty 304 rather than a full transfer on every page navigation, while deploys still show up immediately. * Persist derived responses in SQLite so grading is computed once per cell, not per request New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet records — finished grade/calendar/day/forecast payloads and reverse-geocode labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms) becomes a ~5ms database read that survives restarts and is shared across views. Raw parquet stays the source of truth; the store is a pure accelerator (every reader falls back to recomputing on a miss, and deleting the db is a safe reset). Freshness is token-driven, not clock-driven: each cached payload is validated by a token encoding what it was computed from (payload schema version, the archive record's end date, the recent-fetch stamp). The existing freshness drivers are untouched — get_history still tops up the tail hourly and get_recent_forecast still refetches hourly — and tokens are derived from what they return, so cached payloads expire exactly when their inputs change. The same tokens double as weak ETags: If-None-Match answers with an empty 304 without touching the payload. - backend/store.py: derived-payload + revgeo tables, thread-local WAL conns, every helper fail-soft. - app.py: endpoints split into pure payload builders + HTTP/caching shells; the in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store). - climate.py: revgeo persisted through the store; recent_stamp() and load_cached_history() (no-network read) helpers. - backend/migrate.py + make migrate: idempotent, resumable backfill of the store from existing parquet caches (default calendar span + latest-day detail + revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather. - grid.from_id(): rebuild a cell from its cache filename (migrate tooling). * Add /api/v2/cell: one bundle carrying every view's payload GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months) and day (today) payloads in one response, each the exact payload its per-view endpoint returns — built by the same builders and cached under the same derived-store keys/tokens — paired with the etag that endpoint would emit. The frontend can warm all views with a single request, seed its per-view cache from the slices, and later revalidate each view individually with If-None-Match. The bundle's own etag combines the slices', so an unchanged bundle is an empty 304. prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard guarantee: it never spends weather-API quota. A cell with no cached archive answers 204, and only the history-derived slices (calendar + latest-day detail) are built. At most one Nominatim lookup for a never-labeled cell. * Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota, which multi-year calendar payloads regularly blew through) to IndexedDB, with an in-memory map in front. Entries carry the server's ETag, so anything stale revalidates conditionally — unchanged data costs an empty 304 and a re-stamp, never a re-transfer. On network failure the stale copy is served over an error. getJSON gains an optional onUpdate callback opting into stale-while-revalidate: the three views (weekly, day, calendar) now render a cached copy immediately — spinners are delayed 150ms so warm loads never flash-blank — and repaint only if background revalidation finds changed data. New data shows up the moment it exists instead of waiting out a TTL. Cross-view prefetch collapses from one request per view to a single /api/v2/cell bundle, whose slices (exact per-view payloads + their etags) are seeded under the URLs each view actually requests; the bundle call itself is conditional via a remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one possible Nominatim lookup each), so tapping nearby lands on already-graded data. Legacy tg:* storage entries are cleared once; cache entries untouched for two weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
import store
FRONTEND_DIR = os.path.join(os.path.dirname(__file__), "..", "frontend")
# Everything (pages, assets, API) is served under this base path so the app can
# live at https://<host>/thermograph/ behind a shared domain. Override with the
# THERMOGRAPH_BASE env var. The frontend uses base-relative URLs, so it follows
# this automatically without hardcoding the prefix.
BASE = "/" + os.environ.get("THERMOGRAPH_BASE", "/thermograph").strip("/")
# Observed values pulled from a daily record row for grading. Includes the
# temperature-scale metrics (tmax/tmin/feels/wind/gust) plus precip; a column may
# be absent on an older cache, so only carry the ones present.
OBS_COLS = ("tmax", "tmin", "precip", "feels", "humid", "wind", "gust")
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21) * Compress API responses and revalidate static assets instead of re-downloading - Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x. - Serve pages/assets with Cache-Control: no-cache instead of no-store, so browsers revalidate via the ETag/Last-Modified that FileResponse and StaticFiles already emit. Unchanged assets now cost an empty 304 rather than a full transfer on every page navigation, while deploys still show up immediately. * Persist derived responses in SQLite so grading is computed once per cell, not per request New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet records — finished grade/calendar/day/forecast payloads and reverse-geocode labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms) becomes a ~5ms database read that survives restarts and is shared across views. Raw parquet stays the source of truth; the store is a pure accelerator (every reader falls back to recomputing on a miss, and deleting the db is a safe reset). Freshness is token-driven, not clock-driven: each cached payload is validated by a token encoding what it was computed from (payload schema version, the archive record's end date, the recent-fetch stamp). The existing freshness drivers are untouched — get_history still tops up the tail hourly and get_recent_forecast still refetches hourly — and tokens are derived from what they return, so cached payloads expire exactly when their inputs change. The same tokens double as weak ETags: If-None-Match answers with an empty 304 without touching the payload. - backend/store.py: derived-payload + revgeo tables, thread-local WAL conns, every helper fail-soft. - app.py: endpoints split into pure payload builders + HTTP/caching shells; the in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store). - climate.py: revgeo persisted through the store; recent_stamp() and load_cached_history() (no-network read) helpers. - backend/migrate.py + make migrate: idempotent, resumable backfill of the store from existing parquet caches (default calendar span + latest-day detail + revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather. - grid.from_id(): rebuild a cell from its cache filename (migrate tooling). * Add /api/v2/cell: one bundle carrying every view's payload GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months) and day (today) payloads in one response, each the exact payload its per-view endpoint returns — built by the same builders and cached under the same derived-store keys/tokens — paired with the etag that endpoint would emit. The frontend can warm all views with a single request, seed its per-view cache from the slices, and later revalidate each view individually with If-None-Match. The bundle's own etag combines the slices', so an unchanged bundle is an empty 304. prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard guarantee: it never spends weather-API quota. A cell with no cached archive answers 204, and only the history-derived slices (calendar + latest-day detail) are built. At most one Nominatim lookup for a never-labeled cell. * Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota, which multi-year calendar payloads regularly blew through) to IndexedDB, with an in-memory map in front. Entries carry the server's ETag, so anything stale revalidates conditionally — unchanged data costs an empty 304 and a re-stamp, never a re-transfer. On network failure the stale copy is served over an error. getJSON gains an optional onUpdate callback opting into stale-while-revalidate: the three views (weekly, day, calendar) now render a cached copy immediately — spinners are delayed 150ms so warm loads never flash-blank — and repaint only if background revalidation finds changed data. New data shows up the moment it exists instead of waiting out a TTL. Cross-view prefetch collapses from one request per view to a single /api/v2/cell bundle, whose slices (exact per-view payloads + their etags) are seeded under the URLs each view actually requests; the bundle call itself is conditional via a remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one possible Nominatim lookup each), so tapping nearby lands on already-graded data. Legacy tg:* storage entries are cleared once; cache entries untouched for two weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
# Bump when any response payload shape changes (new metrics, renamed fields…).
# The version is part of every derived-store validity token, so one bump
# atomically orphans all pre-upgrade cached payloads instead of letting a stale
# row whose history_end happens to match keep serving the old shape.
PAYLOAD_VER = "p1"
def _obs_from_row(row) -> dict:
return {k: row[k] for k in OBS_COLS if k in row}
def _weather_fetch_error(e) -> HTTPException:
"""A clean, retryable message for a rate limit; the raw error otherwise."""
low = str(e).lower()
reason = (climate._rate_limit_reason(e) or "").lower()
blob = low + " " + reason
if climate._is_rate_limit(e) or "429" in low or "rate-limited" in low or "limit" in reason:
if "daily" in blob or "tomorrow" in blob: # daily quota exhausted — resets tomorrow
return HTTPException(
status_code=503,
detail="Open-Meteo's daily request limit is exhausted — new locations will work again "
"tomorrow. Places you've already viewed still work.",
)
return HTTPException(
status_code=503,
detail="The weather service is rate-limited right now — please try again in a minute.",
)
return HTTPException(status_code=502, detail=f"weather data fetch failed: {e}")
def _grade_rows(history, rows_df) -> list[dict]:
"""Grade each daily row against its own ±7-day climatology window."""
return [grading.grade_day(history, row["date"], _obs_from_row(row))
for _, row in rows_df.iterrows()]
def _attach_dry_streaks(graded: list[dict], *precip_frames) -> None:
"""Attach `dsr` (days since last measurable rain) to each graded day, computed
over a combined, de-duplicated precip series so streaks stay continuous across
the history / recent / forecast sources."""
frames = [f[["date", "precip"]] for f in precip_frames if f is not None and not f.empty]
if not frames:
return
combined = (pd.concat(frames)
.drop_duplicates(subset="date", keep="last")
.sort_values("date"))
dsr_map = grading.dry_streaks(combined["date"].values, combined["precip"].values)
for g in graded:
g["dsr"] = dsr_map.get(g["date"])
app = FastAPI(title="Thermograph", version="0.2.0")
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21) * Compress API responses and revalidate static assets instead of re-downloading - Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x. - Serve pages/assets with Cache-Control: no-cache instead of no-store, so browsers revalidate via the ETag/Last-Modified that FileResponse and StaticFiles already emit. Unchanged assets now cost an empty 304 rather than a full transfer on every page navigation, while deploys still show up immediately. * Persist derived responses in SQLite so grading is computed once per cell, not per request New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet records — finished grade/calendar/day/forecast payloads and reverse-geocode labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms) becomes a ~5ms database read that survives restarts and is shared across views. Raw parquet stays the source of truth; the store is a pure accelerator (every reader falls back to recomputing on a miss, and deleting the db is a safe reset). Freshness is token-driven, not clock-driven: each cached payload is validated by a token encoding what it was computed from (payload schema version, the archive record's end date, the recent-fetch stamp). The existing freshness drivers are untouched — get_history still tops up the tail hourly and get_recent_forecast still refetches hourly — and tokens are derived from what they return, so cached payloads expire exactly when their inputs change. The same tokens double as weak ETags: If-None-Match answers with an empty 304 without touching the payload. - backend/store.py: derived-payload + revgeo tables, thread-local WAL conns, every helper fail-soft. - app.py: endpoints split into pure payload builders + HTTP/caching shells; the in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store). - climate.py: revgeo persisted through the store; recent_stamp() and load_cached_history() (no-network read) helpers. - backend/migrate.py + make migrate: idempotent, resumable backfill of the store from existing parquet caches (default calendar span + latest-day detail + revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather. - grid.from_id(): rebuild a cell from its cache filename (migrate tooling). * Add /api/v2/cell: one bundle carrying every view's payload GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months) and day (today) payloads in one response, each the exact payload its per-view endpoint returns — built by the same builders and cached under the same derived-store keys/tokens — paired with the etag that endpoint would emit. The frontend can warm all views with a single request, seed its per-view cache from the slices, and later revalidate each view individually with If-None-Match. The bundle's own etag combines the slices', so an unchanged bundle is an empty 304. prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard guarantee: it never spends weather-API quota. A cell with no cached archive answers 204, and only the history-derived slices (calendar + latest-day detail) are built. At most one Nominatim lookup for a never-labeled cell. * Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota, which multi-year calendar payloads regularly blew through) to IndexedDB, with an in-memory map in front. Entries carry the server's ETag, so anything stale revalidates conditionally — unchanged data costs an empty 304 and a re-stamp, never a re-transfer. On network failure the stale copy is served over an error. getJSON gains an optional onUpdate callback opting into stale-while-revalidate: the three views (weekly, day, calendar) now render a cached copy immediately — spinners are delayed 150ms so warm loads never flash-blank — and repaint only if background revalidation finds changed data. New data shows up the moment it exists instead of waiting out a TTL. Cross-view prefetch collapses from one request per view to a single /api/v2/cell bundle, whose slices (exact per-view payloads + their etags) are seeded under the URLs each view actually requests; the bundle call itself is conditional via a remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one possible Nominatim lookup each), so tapping nearby lands on already-graded data. Legacy tg:* storage entries are cleared once; cache entries untouched for two weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
# Compress every sizeable response (the 2-year calendar JSON shrinks ~6-8×).
# Applies to API JSON and static assets alike; tiny responses are left alone.
app.add_middleware(GZipMiddleware, minimum_size=1024)
@app.middleware("http")
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21) * Compress API responses and revalidate static assets instead of re-downloading - Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x. - Serve pages/assets with Cache-Control: no-cache instead of no-store, so browsers revalidate via the ETag/Last-Modified that FileResponse and StaticFiles already emit. Unchanged assets now cost an empty 304 rather than a full transfer on every page navigation, while deploys still show up immediately. * Persist derived responses in SQLite so grading is computed once per cell, not per request New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet records — finished grade/calendar/day/forecast payloads and reverse-geocode labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms) becomes a ~5ms database read that survives restarts and is shared across views. Raw parquet stays the source of truth; the store is a pure accelerator (every reader falls back to recomputing on a miss, and deleting the db is a safe reset). Freshness is token-driven, not clock-driven: each cached payload is validated by a token encoding what it was computed from (payload schema version, the archive record's end date, the recent-fetch stamp). The existing freshness drivers are untouched — get_history still tops up the tail hourly and get_recent_forecast still refetches hourly — and tokens are derived from what they return, so cached payloads expire exactly when their inputs change. The same tokens double as weak ETags: If-None-Match answers with an empty 304 without touching the payload. - backend/store.py: derived-payload + revgeo tables, thread-local WAL conns, every helper fail-soft. - app.py: endpoints split into pure payload builders + HTTP/caching shells; the in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store). - climate.py: revgeo persisted through the store; recent_stamp() and load_cached_history() (no-network read) helpers. - backend/migrate.py + make migrate: idempotent, resumable backfill of the store from existing parquet caches (default calendar span + latest-day detail + revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather. - grid.from_id(): rebuild a cell from its cache filename (migrate tooling). * Add /api/v2/cell: one bundle carrying every view's payload GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months) and day (today) payloads in one response, each the exact payload its per-view endpoint returns — built by the same builders and cached under the same derived-store keys/tokens — paired with the etag that endpoint would emit. The frontend can warm all views with a single request, seed its per-view cache from the slices, and later revalidate each view individually with If-None-Match. The bundle's own etag combines the slices', so an unchanged bundle is an empty 304. prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard guarantee: it never spends weather-API quota. A cell with no cached archive answers 204, and only the history-derived slices (calendar + latest-day detail) are built. At most one Nominatim lookup for a never-labeled cell. * Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota, which multi-year calendar payloads regularly blew through) to IndexedDB, with an in-memory map in front. Entries carry the server's ETag, so anything stale revalidates conditionally — unchanged data costs an empty 304 and a re-stamp, never a re-transfer. On network failure the stale copy is served over an error. getJSON gains an optional onUpdate callback opting into stale-while-revalidate: the three views (weekly, day, calendar) now render a cached copy immediately — spinners are delayed 150ms so warm loads never flash-blank — and repaint only if background revalidation finds changed data. New data shows up the moment it exists instead of waiting out a TTL. Cross-view prefetch collapses from one request per view to a single /api/v2/cell bundle, whose slices (exact per-view payloads + their etags) are seeded under the URLs each view actually requests; the bundle call itself is conditional via a remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one possible Nominatim lookup each), so tapping nearby lands on already-graded data. Legacy tg:* storage entries are cleared once; cache entries untouched for two weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
async def revalidate_static(request, call_next):
"""Serve the frontend with no-cache (NOT no-store): browsers may keep a copy
but must revalidate it on every use. Starlette's FileResponse/StaticFiles
already emit ETag + Last-Modified, so an unchanged asset costs one conditional
request answered with an empty 304 instead of a full re-download page-to-page
navigation stops re-transferring the JS/CSS/HTML while still picking up every
deploy immediately (no "my change isn't showing" bugs)."""
response = await call_next(request)
path = request.url.path
pages = (BASE, f"{BASE}/", f"{BASE}/calendar", f"{BASE}/day", f"{BASE}/compare", f"{BASE}/legend")
if path.endswith((".js", ".css", ".html")) or path in pages:
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21) * Compress API responses and revalidate static assets instead of re-downloading - Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x. - Serve pages/assets with Cache-Control: no-cache instead of no-store, so browsers revalidate via the ETag/Last-Modified that FileResponse and StaticFiles already emit. Unchanged assets now cost an empty 304 rather than a full transfer on every page navigation, while deploys still show up immediately. * Persist derived responses in SQLite so grading is computed once per cell, not per request New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet records — finished grade/calendar/day/forecast payloads and reverse-geocode labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms) becomes a ~5ms database read that survives restarts and is shared across views. Raw parquet stays the source of truth; the store is a pure accelerator (every reader falls back to recomputing on a miss, and deleting the db is a safe reset). Freshness is token-driven, not clock-driven: each cached payload is validated by a token encoding what it was computed from (payload schema version, the archive record's end date, the recent-fetch stamp). The existing freshness drivers are untouched — get_history still tops up the tail hourly and get_recent_forecast still refetches hourly — and tokens are derived from what they return, so cached payloads expire exactly when their inputs change. The same tokens double as weak ETags: If-None-Match answers with an empty 304 without touching the payload. - backend/store.py: derived-payload + revgeo tables, thread-local WAL conns, every helper fail-soft. - app.py: endpoints split into pure payload builders + HTTP/caching shells; the in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store). - climate.py: revgeo persisted through the store; recent_stamp() and load_cached_history() (no-network read) helpers. - backend/migrate.py + make migrate: idempotent, resumable backfill of the store from existing parquet caches (default calendar span + latest-day detail + revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather. - grid.from_id(): rebuild a cell from its cache filename (migrate tooling). * Add /api/v2/cell: one bundle carrying every view's payload GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months) and day (today) payloads in one response, each the exact payload its per-view endpoint returns — built by the same builders and cached under the same derived-store keys/tokens — paired with the etag that endpoint would emit. The frontend can warm all views with a single request, seed its per-view cache from the slices, and later revalidate each view individually with If-None-Match. The bundle's own etag combines the slices', so an unchanged bundle is an empty 304. prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard guarantee: it never spends weather-API quota. A cell with no cached archive answers 204, and only the history-derived slices (calendar + latest-day detail) are built. At most one Nominatim lookup for a never-labeled cell. * Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota, which multi-year calendar payloads regularly blew through) to IndexedDB, with an in-memory map in front. Entries carry the server's ETag, so anything stale revalidates conditionally — unchanged data costs an empty 304 and a re-stamp, never a re-transfer. On network failure the stale copy is served over an error. getJSON gains an optional onUpdate callback opting into stale-while-revalidate: the three views (weekly, day, calendar) now render a cached copy immediately — spinners are delayed 150ms so warm loads never flash-blank — and repaint only if background revalidation finds changed data. New data shows up the moment it exists instead of waiting out a TTL. Cross-view prefetch collapses from one request per view to a single /api/v2/cell bundle, whose slices (exact per-view payloads + their etags) are seeded under the URLs each view actually requests; the bundle call itself is conditional via a remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one possible Nominatim lookup each), so tapping nearby lands on already-graded data. Legacy tg:* storage entries are cleared once; cache entries untouched for two weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
response.headers["Cache-Control"] = "no-cache"
return response
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21) * Compress API responses and revalidate static assets instead of re-downloading - Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x. - Serve pages/assets with Cache-Control: no-cache instead of no-store, so browsers revalidate via the ETag/Last-Modified that FileResponse and StaticFiles already emit. Unchanged assets now cost an empty 304 rather than a full transfer on every page navigation, while deploys still show up immediately. * Persist derived responses in SQLite so grading is computed once per cell, not per request New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet records — finished grade/calendar/day/forecast payloads and reverse-geocode labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms) becomes a ~5ms database read that survives restarts and is shared across views. Raw parquet stays the source of truth; the store is a pure accelerator (every reader falls back to recomputing on a miss, and deleting the db is a safe reset). Freshness is token-driven, not clock-driven: each cached payload is validated by a token encoding what it was computed from (payload schema version, the archive record's end date, the recent-fetch stamp). The existing freshness drivers are untouched — get_history still tops up the tail hourly and get_recent_forecast still refetches hourly — and tokens are derived from what they return, so cached payloads expire exactly when their inputs change. The same tokens double as weak ETags: If-None-Match answers with an empty 304 without touching the payload. - backend/store.py: derived-payload + revgeo tables, thread-local WAL conns, every helper fail-soft. - app.py: endpoints split into pure payload builders + HTTP/caching shells; the in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store). - climate.py: revgeo persisted through the store; recent_stamp() and load_cached_history() (no-network read) helpers. - backend/migrate.py + make migrate: idempotent, resumable backfill of the store from existing parquet caches (default calendar span + latest-day detail + revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather. - grid.from_id(): rebuild a cell from its cache filename (migrate tooling). * Add /api/v2/cell: one bundle carrying every view's payload GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months) and day (today) payloads in one response, each the exact payload its per-view endpoint returns — built by the same builders and cached under the same derived-store keys/tokens — paired with the etag that endpoint would emit. The frontend can warm all views with a single request, seed its per-view cache from the slices, and later revalidate each view individually with If-None-Match. The bundle's own etag combines the slices', so an unchanged bundle is an empty 304. prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard guarantee: it never spends weather-API quota. A cell with no cached archive answers 204, and only the history-derived slices (calendar + latest-day detail) are built. At most one Nominatim lookup for a never-labeled cell. * Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota, which multi-year calendar payloads regularly blew through) to IndexedDB, with an in-memory map in front. Entries carry the server's ETag, so anything stale revalidates conditionally — unchanged data costs an empty 304 and a re-stamp, never a re-transfer. On network failure the stale copy is served over an error. getJSON gains an optional onUpdate callback opting into stale-while-revalidate: the three views (weekly, day, calendar) now render a cached copy immediately — spinners are delayed 150ms so warm loads never flash-blank — and repaint only if background revalidation finds changed data. New data shows up the moment it exists instead of waiting out a TTL. Cross-view prefetch collapses from one request per view to a single /api/v2/cell bundle, whose slices (exact per-view payloads + their etags) are seeded under the URLs each view actually requests; the bundle call itself is conditional via a remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one possible Nominatim lookup each), so tapping nearby lands on already-graded data. Legacy tg:* storage entries are cleared once; cache entries untouched for two weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
# --- derived-store plumbing --------------------------------------------------
# Every graded payload is cached in SQLite under (kind, cell, key) and validated
# by a token that encodes what it was computed from (see store.py). The freshness
# drivers stay where they were — climate.get_history tops up the archive tail
# hourly and get_recent_forecast refetches hourly — and the tokens are derived
# from what those return, so a cached payload expires exactly when its inputs
# change and never before. The same tokens double as ETags: a client sending
# If-None-Match gets an empty 304 without the payload even being loaded.
class _NullRun:
"""audit.RunAudit stand-in for offline callers (the migrate script)."""
def set(self, **kw):
return self
def phase(self, name):
return contextlib.nullcontext()
def _hist_end(history) -> str:
"""Last day in the cell's archive record — the freshness token for everything
derived purely from history. Advances via the hourly tail top-up inside
climate.get_history, which cached payloads follow automatically."""
return pd.Timestamp(history["date"].max()).date().isoformat()
def _etag_for(kind: str, cell_id: str, key: str, token: str) -> str:
"""Deterministic weak ETag from a payload's identity + validity token. Weak
because the same content may be served under different encodings (gzip)."""
h = hashlib.sha1(f"{kind}:{cell_id}:{key}:{token}".encode()).hexdigest()[:20]
return f'W/"{h}"'
def _not_modified(request: Request, etag: str) -> bool:
"""Does the client already hold this exact payload? Answerable from the token
alone no payload load needed."""
inm = request.headers.get("if-none-match")
if not inm:
return False
tags = {t.strip() for t in inm.split(",")}
return "*" in tags or etag in tags or etag.removeprefix("W/") in tags
def _encode(payload: dict) -> bytes:
"""Compact JSON bytes, matching store.put_payload's encoding (allow_nan=False
mirrors Starlette a NaN from grading should fail loudly, not reach a client)."""
return json.dumps(payload, default=str, allow_nan=False, separators=(",", ":")).encode()
def _json_response(body: bytes, etag: str | None = None) -> Response:
headers = {"ETag": etag} if etag else {}
return Response(content=body, media_type="application/json", headers=headers)
# --- payload builders --------------------------------------------------------
# Pure "inputs → response dict" functions shared by the per-view endpoints, the
# /cell bundle, and the offline migrate script. HTTP semantics (audit runs, cache
# lookups, etags) stay in the endpoints; `run` is the audit run or None.
def _build_grade(cell, target, days, history, recent, cache_meta, place, run=None) -> dict:
"""/grade payload: the last `days` observed days graded against their own
±7-day windows, plus the target day's climatology summary."""
run = run or _NullRun()
with run.phase("grading"):
# Only grade observed days up to and including the target date (last `days`).
recent = recent[recent["date"] <= target].sort_values("date").tail(days)
graded = _grade_rows(history, recent)
_attach_dry_streaks(graded, history, recent)
graded.reverse() # newest first for display
climo = grading.climatology(history, int(target.dayofyear))
# Summarize what this run actually covered. A fresh history fetch pulls the
# full ~45-year archive (run_type "full"); a cache hit only fetched the
# recent window (run_type "partial").
years = pd.to_datetime(history["date"]).dt.year
full = not cache_meta.get("cached", False)
run.set(
run_type="full" if full else "partial",
history_source="fetch" if full else "cache",
history_rows=int(len(history)),
history_years=int(years.max() - years.min() + 1),
history_span=[int(years.min()), int(years.max())],
cache_age_days=cache_meta.get("cache_age_days"),
graded_days=len(graded),
place_found=place is not None,
)
return {
"cell": cell,
"place": place,
"target_date": target.date().isoformat(),
"cache": cache_meta,
"climatology": climo,
"recent": graded,
}
CAL_MAX_SPAN_DAYS = 732 # ~2 years — the most a single calendar request will grade
# (the frontend splits longer spans into 2-year chunks)
def _cal_span(history, start, end, months) -> tuple[pd.Timestamp, pd.Timestamp]:
"""Clamp a requested calendar range to the available record (≤ ~2 years).
Without a start, defaults to whole months back landing on the SAME
day-of-month as the end, so the two range endpoints line up
(e.g. 2024-06-29 2026-06-29)."""
first = pd.Timestamp(history["date"].min()).normalize()
last = pd.Timestamp(history["date"].max()).normalize()
end_ts = min(pd.Timestamp(end).normalize(), last) if end else last
if start:
start_ts = pd.Timestamp(start).normalize()
else:
start_ts = (end_ts - pd.DateOffset(months=months)).normalize()
# Cap the graded span at ~2 years, and keep it inside the available record.
min_start = end_ts - pd.Timedelta(days=CAL_MAX_SPAN_DAYS - 1)
start_ts = max(start_ts, min_start, first)
start_ts = min(start_ts, end_ts)
return start_ts, end_ts
def _build_calendar(cell, history, start_ts, end_ts, months, place, run=None) -> dict:
"""/calendar payload: every day in [start_ts, end_ts] graded, compact shape."""
run = run or _NullRun()
with run.phase("grading"):
days = grading.grade_range(history, start_ts, end_ts)
run.set(graded_days=len(days))
return {
"api_version": "v2",
"cell": cell,
"place": place,
"range": {"start": start_ts.date().isoformat(), "end": end_ts.date().isoformat()},
"months": months,
"days": days,
}
def _build_day(cell, history, target, place, run=None, recent=None) -> dict:
"""/day payload: the full percentile breakdown for one day. Observed values
prefer the archive record; a date newer than it comes from the recent window
(passed pre-loaded by the bundle path, fetched here otherwise)."""
run = run or _NullRun()
last = pd.Timestamp(history["date"].max()).normalize()
obs = None
hit = history[history["date"] == target]
if not hit.empty:
obs = _obs_from_row(hit.iloc[0])
elif target > last:
try:
if recent is None:
with run.phase("recent"):
recent = climate.get_recent_forecast(cell)
rr = recent[recent["date"] == target]
if not rr.empty:
obs = _obs_from_row(rr.iloc[0])
except Exception: # noqa: BLE001 - detail page still works from climatology alone
obs = None
with run.phase("detail"):
detail = grading.day_detail(history, target, obs)
run.set(has_observation=obs is not None)
return {
"api_version": "v2",
"cell": cell,
"place": place,
"latest": last.date().isoformat(),
"detail": detail,
}
def _build_forecast(cell, days, history, fc, today, place, run=None) -> dict:
"""/forecast payload: the next `days` forecast days graded — same shape as
/grade so the frontend renders it identically, furthest-out day first."""
run = run or _NullRun()
# Strictly future days (tomorrow onward), earliest→latest, capped at `days`.
future = fc[fc["date"] > today].sort_values("date").head(days)
with run.phase("grading"):
graded = _grade_rows(history, future)
_attach_dry_streaks(graded, history, fc)
graded.reverse() # furthest-out first, so the chart reverses to L→R chronological
climo = grading.climatology(history, int(today.dayofyear))
run.set(graded_days=len(graded), place_found=place is not None)
return {
"api_version": "v2",
"cell": cell,
"place": place,
"target_date": today.date().isoformat(),
"forecast": True,
"climatology": climo,
"recent": graded,
}
# --- endpoints ----------------------------------------------------------------
def api_geocode(q: str = Query(..., min_length=1)):
try:
return {"results": climate.geocode(q)}
except Exception as e: # noqa: BLE001 - surface upstream failures to the client
raise HTTPException(status_code=502, detail=f"geocoding failed: {e}")
def api_grade(
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21) * Compress API responses and revalidate static assets instead of re-downloading - Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x. - Serve pages/assets with Cache-Control: no-cache instead of no-store, so browsers revalidate via the ETag/Last-Modified that FileResponse and StaticFiles already emit. Unchanged assets now cost an empty 304 rather than a full transfer on every page navigation, while deploys still show up immediately. * Persist derived responses in SQLite so grading is computed once per cell, not per request New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet records — finished grade/calendar/day/forecast payloads and reverse-geocode labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms) becomes a ~5ms database read that survives restarts and is shared across views. Raw parquet stays the source of truth; the store is a pure accelerator (every reader falls back to recomputing on a miss, and deleting the db is a safe reset). Freshness is token-driven, not clock-driven: each cached payload is validated by a token encoding what it was computed from (payload schema version, the archive record's end date, the recent-fetch stamp). The existing freshness drivers are untouched — get_history still tops up the tail hourly and get_recent_forecast still refetches hourly — and tokens are derived from what they return, so cached payloads expire exactly when their inputs change. The same tokens double as weak ETags: If-None-Match answers with an empty 304 without touching the payload. - backend/store.py: derived-payload + revgeo tables, thread-local WAL conns, every helper fail-soft. - app.py: endpoints split into pure payload builders + HTTP/caching shells; the in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store). - climate.py: revgeo persisted through the store; recent_stamp() and load_cached_history() (no-network read) helpers. - backend/migrate.py + make migrate: idempotent, resumable backfill of the store from existing parquet caches (default calendar span + latest-day detail + revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather. - grid.from_id(): rebuild a cell from its cache filename (migrate tooling). * Add /api/v2/cell: one bundle carrying every view's payload GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months) and day (today) payloads in one response, each the exact payload its per-view endpoint returns — built by the same builders and cached under the same derived-store keys/tokens — paired with the etag that endpoint would emit. The frontend can warm all views with a single request, seed its per-view cache from the slices, and later revalidate each view individually with If-None-Match. The bundle's own etag combines the slices', so an unchanged bundle is an empty 304. prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard guarantee: it never spends weather-API quota. A cell with no cached archive answers 204, and only the history-derived slices (calendar + latest-day detail) are built. At most one Nominatim lookup for a never-labeled cell. * Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota, which multi-year calendar payloads regularly blew through) to IndexedDB, with an in-memory map in front. Entries carry the server's ETag, so anything stale revalidates conditionally — unchanged data costs an empty 304 and a re-stamp, never a re-transfer. On network failure the stale copy is served over an error. getJSON gains an optional onUpdate callback opting into stale-while-revalidate: the three views (weekly, day, calendar) now render a cached copy immediately — spinners are delayed 150ms so warm loads never flash-blank — and repaint only if background revalidation finds changed data. New data shows up the moment it exists instead of waiting out a TTL. Cross-view prefetch collapses from one request per view to a single /api/v2/cell bundle, whose slices (exact per-view payloads + their etags) are seeded under the URLs each view actually requests; the bundle call itself is conditional via a remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one possible Nominatim lookup each), so tapping nearby lands on already-graded data. Legacy tg:* storage entries are cleared once; cache entries untouched for two weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
request: Request,
lat: float = Query(..., ge=-90, le=90),
lon: float = Query(..., ge=-180, le=180),
date: str | None = Query(None, description="target date YYYY-MM-DD (default today)"),
days: int = Query(14, ge=1, le=60, description="how many recent days to grade"),
):
if not grid.in_north_america(lat, lon):
raise HTTPException(
status_code=400,
detail="Location is outside the supported US + Canada region.",
)
target = (
pd.Timestamp(date).normalize()
if date
else pd.Timestamp(datetime.date.today())
)
cell = grid.snap(lat, lon)
with audit.RunAudit(
endpoint="grade",
lat=round(lat, 4),
lon=round(lon, 4),
target_date=target.date().isoformat(),
recent_days=days,
cell_id=cell["id"],
) as run:
try:
with run.phase("history"):
history, cache_meta = climate.get_history(cell)
with run.phase("recent"):
recent = climate.get_recent_forecast(cell)
except Exception as e: # noqa: BLE001
raise _weather_fetch_error(e)
if history.empty:
raise HTTPException(status_code=404, detail="No historical data for this cell.")
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21) * Compress API responses and revalidate static assets instead of re-downloading - Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x. - Serve pages/assets with Cache-Control: no-cache instead of no-store, so browsers revalidate via the ETag/Last-Modified that FileResponse and StaticFiles already emit. Unchanged assets now cost an empty 304 rather than a full transfer on every page navigation, while deploys still show up immediately. * Persist derived responses in SQLite so grading is computed once per cell, not per request New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet records — finished grade/calendar/day/forecast payloads and reverse-geocode labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms) becomes a ~5ms database read that survives restarts and is shared across views. Raw parquet stays the source of truth; the store is a pure accelerator (every reader falls back to recomputing on a miss, and deleting the db is a safe reset). Freshness is token-driven, not clock-driven: each cached payload is validated by a token encoding what it was computed from (payload schema version, the archive record's end date, the recent-fetch stamp). The existing freshness drivers are untouched — get_history still tops up the tail hourly and get_recent_forecast still refetches hourly — and tokens are derived from what they return, so cached payloads expire exactly when their inputs change. The same tokens double as weak ETags: If-None-Match answers with an empty 304 without touching the payload. - backend/store.py: derived-payload + revgeo tables, thread-local WAL conns, every helper fail-soft. - app.py: endpoints split into pure payload builders + HTTP/caching shells; the in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store). - climate.py: revgeo persisted through the store; recent_stamp() and load_cached_history() (no-network read) helpers. - backend/migrate.py + make migrate: idempotent, resumable backfill of the store from existing parquet caches (default calendar span + latest-day detail + revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather. - grid.from_id(): rebuild a cell from its cache filename (migrate tooling). * Add /api/v2/cell: one bundle carrying every view's payload GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months) and day (today) payloads in one response, each the exact payload its per-view endpoint returns — built by the same builders and cached under the same derived-store keys/tokens — paired with the etag that endpoint would emit. The frontend can warm all views with a single request, seed its per-view cache from the slices, and later revalidate each view individually with If-None-Match. The bundle's own etag combines the slices', so an unchanged bundle is an empty 304. prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard guarantee: it never spends weather-API quota. A cell with no cached archive answers 204, and only the history-derived slices (calendar + latest-day detail) are built. At most one Nominatim lookup for a never-labeled cell. * Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota, which multi-year calendar payloads regularly blew through) to IndexedDB, with an in-memory map in front. Entries carry the server's ETag, so anything stale revalidates conditionally — unchanged data costs an empty 304 and a re-stamp, never a re-transfer. On network failure the stale copy is served over an error. getJSON gains an optional onUpdate callback opting into stale-while-revalidate: the three views (weekly, day, calendar) now render a cached copy immediately — spinners are delayed 150ms so warm loads never flash-blank — and repaint only if background revalidation finds changed data. New data shows up the moment it exists instead of waiting out a TTL. Cross-view prefetch collapses from one request per view to a single /api/v2/cell bundle, whose slices (exact per-view payloads + their etags) are seeded under the URLs each view actually requests; the bundle call itself is conditional via a remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one possible Nominatim lookup each), so tapping nearby lands on already-graded data. Legacy tg:* storage entries are cleared once; cache entries untouched for two weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
key = f"{target.date().isoformat()}:{days}"
token = f"{PAYLOAD_VER}:{_hist_end(history)}:{climate.recent_stamp(cell['id'])}"
etag = _etag_for("grade", cell["id"], key, token)
if _not_modified(request, etag):
run.set(run_type="cache", not_modified=True)
return Response(status_code=304, headers={"ETag": etag})
body = store.get_payload("grade", cell["id"], key, token)
if body is not None:
run.set(run_type="cache", history_source="store")
return _json_response(body, etag)
with run.phase("reverse_geocode"):
place = climate.reverse_geocode(cell["center_lat"], cell["center_lon"])
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21) * Compress API responses and revalidate static assets instead of re-downloading - Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x. - Serve pages/assets with Cache-Control: no-cache instead of no-store, so browsers revalidate via the ETag/Last-Modified that FileResponse and StaticFiles already emit. Unchanged assets now cost an empty 304 rather than a full transfer on every page navigation, while deploys still show up immediately. * Persist derived responses in SQLite so grading is computed once per cell, not per request New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet records — finished grade/calendar/day/forecast payloads and reverse-geocode labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms) becomes a ~5ms database read that survives restarts and is shared across views. Raw parquet stays the source of truth; the store is a pure accelerator (every reader falls back to recomputing on a miss, and deleting the db is a safe reset). Freshness is token-driven, not clock-driven: each cached payload is validated by a token encoding what it was computed from (payload schema version, the archive record's end date, the recent-fetch stamp). The existing freshness drivers are untouched — get_history still tops up the tail hourly and get_recent_forecast still refetches hourly — and tokens are derived from what they return, so cached payloads expire exactly when their inputs change. The same tokens double as weak ETags: If-None-Match answers with an empty 304 without touching the payload. - backend/store.py: derived-payload + revgeo tables, thread-local WAL conns, every helper fail-soft. - app.py: endpoints split into pure payload builders + HTTP/caching shells; the in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store). - climate.py: revgeo persisted through the store; recent_stamp() and load_cached_history() (no-network read) helpers. - backend/migrate.py + make migrate: idempotent, resumable backfill of the store from existing parquet caches (default calendar span + latest-day detail + revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather. - grid.from_id(): rebuild a cell from its cache filename (migrate tooling). * Add /api/v2/cell: one bundle carrying every view's payload GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months) and day (today) payloads in one response, each the exact payload its per-view endpoint returns — built by the same builders and cached under the same derived-store keys/tokens — paired with the etag that endpoint would emit. The frontend can warm all views with a single request, seed its per-view cache from the slices, and later revalidate each view individually with If-None-Match. The bundle's own etag combines the slices', so an unchanged bundle is an empty 304. prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard guarantee: it never spends weather-API quota. A cell with no cached archive answers 204, and only the history-derived slices (calendar + latest-day detail) are built. At most one Nominatim lookup for a never-labeled cell. * Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota, which multi-year calendar payloads regularly blew through) to IndexedDB, with an in-memory map in front. Entries carry the server's ETag, so anything stale revalidates conditionally — unchanged data costs an empty 304 and a re-stamp, never a re-transfer. On network failure the stale copy is served over an error. getJSON gains an optional onUpdate callback opting into stale-while-revalidate: the three views (weekly, day, calendar) now render a cached copy immediately — spinners are delayed 150ms so warm loads never flash-blank — and repaint only if background revalidation finds changed data. New data shows up the moment it exists instead of waiting out a TTL. Cross-view prefetch collapses from one request per view to a single /api/v2/cell bundle, whose slices (exact per-view payloads + their etags) are seeded under the URLs each view actually requests; the bundle call itself is conditional via a remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one possible Nominatim lookup each), so tapping nearby lands on already-graded data. Legacy tg:* storage entries are cleared once; cache entries untouched for two weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
payload = _build_grade(cell, target, days, history, recent, cache_meta, place, run)
return _json_response(store.put_payload("grade", cell["id"], key, token, payload), etag)
def api_calendar(
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21) * Compress API responses and revalidate static assets instead of re-downloading - Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x. - Serve pages/assets with Cache-Control: no-cache instead of no-store, so browsers revalidate via the ETag/Last-Modified that FileResponse and StaticFiles already emit. Unchanged assets now cost an empty 304 rather than a full transfer on every page navigation, while deploys still show up immediately. * Persist derived responses in SQLite so grading is computed once per cell, not per request New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet records — finished grade/calendar/day/forecast payloads and reverse-geocode labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms) becomes a ~5ms database read that survives restarts and is shared across views. Raw parquet stays the source of truth; the store is a pure accelerator (every reader falls back to recomputing on a miss, and deleting the db is a safe reset). Freshness is token-driven, not clock-driven: each cached payload is validated by a token encoding what it was computed from (payload schema version, the archive record's end date, the recent-fetch stamp). The existing freshness drivers are untouched — get_history still tops up the tail hourly and get_recent_forecast still refetches hourly — and tokens are derived from what they return, so cached payloads expire exactly when their inputs change. The same tokens double as weak ETags: If-None-Match answers with an empty 304 without touching the payload. - backend/store.py: derived-payload + revgeo tables, thread-local WAL conns, every helper fail-soft. - app.py: endpoints split into pure payload builders + HTTP/caching shells; the in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store). - climate.py: revgeo persisted through the store; recent_stamp() and load_cached_history() (no-network read) helpers. - backend/migrate.py + make migrate: idempotent, resumable backfill of the store from existing parquet caches (default calendar span + latest-day detail + revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather. - grid.from_id(): rebuild a cell from its cache filename (migrate tooling). * Add /api/v2/cell: one bundle carrying every view's payload GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months) and day (today) payloads in one response, each the exact payload its per-view endpoint returns — built by the same builders and cached under the same derived-store keys/tokens — paired with the etag that endpoint would emit. The frontend can warm all views with a single request, seed its per-view cache from the slices, and later revalidate each view individually with If-None-Match. The bundle's own etag combines the slices', so an unchanged bundle is an empty 304. prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard guarantee: it never spends weather-API quota. A cell with no cached archive answers 204, and only the history-derived slices (calendar + latest-day detail) are built. At most one Nominatim lookup for a never-labeled cell. * Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota, which multi-year calendar payloads regularly blew through) to IndexedDB, with an in-memory map in front. Entries carry the server's ETag, so anything stale revalidates conditionally — unchanged data costs an empty 304 and a re-stamp, never a re-transfer. On network failure the stale copy is served over an error. getJSON gains an optional onUpdate callback opting into stale-while-revalidate: the three views (weekly, day, calendar) now render a cached copy immediately — spinners are delayed 150ms so warm loads never flash-blank — and repaint only if background revalidation finds changed data. New data shows up the moment it exists instead of waiting out a TTL. Cross-view prefetch collapses from one request per view to a single /api/v2/cell bundle, whose slices (exact per-view payloads + their etags) are seeded under the URLs each view actually requests; the bundle call itself is conditional via a remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one possible Nominatim lookup each), so tapping nearby lands on already-graded data. Legacy tg:* storage entries are cleared once; cache entries untouched for two weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
request: Request,
lat: float = Query(..., ge=-90, le=90),
lon: float = Query(..., ge=-180, le=180),
start: str | None = Query(None, description="first date YYYY-MM-DD (overrides months)"),
end: str | None = Query(None, description="last date YYYY-MM-DD (default = latest available)"),
months: int = Query(24, ge=1, le=24, description="months back to grade when no start given"),
):
"""Grade every day over a date range (default: last 2 years) for the calendar view.
A custom [start, end] range is supported and capped at ~2 years per request;
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21) * Compress API responses and revalidate static assets instead of re-downloading - Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x. - Serve pages/assets with Cache-Control: no-cache instead of no-store, so browsers revalidate via the ETag/Last-Modified that FileResponse and StaticFiles already emit. Unchanged assets now cost an empty 304 rather than a full transfer on every page navigation, while deploys still show up immediately. * Persist derived responses in SQLite so grading is computed once per cell, not per request New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet records — finished grade/calendar/day/forecast payloads and reverse-geocode labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms) becomes a ~5ms database read that survives restarts and is shared across views. Raw parquet stays the source of truth; the store is a pure accelerator (every reader falls back to recomputing on a miss, and deleting the db is a safe reset). Freshness is token-driven, not clock-driven: each cached payload is validated by a token encoding what it was computed from (payload schema version, the archive record's end date, the recent-fetch stamp). The existing freshness drivers are untouched — get_history still tops up the tail hourly and get_recent_forecast still refetches hourly — and tokens are derived from what they return, so cached payloads expire exactly when their inputs change. The same tokens double as weak ETags: If-None-Match answers with an empty 304 without touching the payload. - backend/store.py: derived-payload + revgeo tables, thread-local WAL conns, every helper fail-soft. - app.py: endpoints split into pure payload builders + HTTP/caching shells; the in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store). - climate.py: revgeo persisted through the store; recent_stamp() and load_cached_history() (no-network read) helpers. - backend/migrate.py + make migrate: idempotent, resumable backfill of the store from existing parquet caches (default calendar span + latest-day detail + revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather. - grid.from_id(): rebuild a cell from its cache filename (migrate tooling). * Add /api/v2/cell: one bundle carrying every view's payload GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months) and day (today) payloads in one response, each the exact payload its per-view endpoint returns — built by the same builders and cached under the same derived-store keys/tokens — paired with the etag that endpoint would emit. The frontend can warm all views with a single request, seed its per-view cache from the slices, and later revalidate each view individually with If-None-Match. The bundle's own etag combines the slices', so an unchanged bundle is an empty 304. prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard guarantee: it never spends weather-API quota. A cell with no cached archive answers 204, and only the history-derived slices (calendar + latest-day detail) are built. At most one Nominatim lookup for a never-labeled cell. * Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota, which multi-year calendar payloads regularly blew through) to IndexedDB, with an in-memory map in front. Entries carry the server's ETag, so anything stale revalidates conditionally — unchanged data costs an empty 304 and a re-stamp, never a re-transfer. On network failure the stale copy is served over an error. getJSON gains an optional onUpdate callback opting into stale-while-revalidate: the three views (weekly, day, calendar) now render a cached copy immediately — spinners are delayed 150ms so warm loads never flash-blank — and repaint only if background revalidation finds changed data. New data shows up the moment it exists instead of waiting out a TTL. Cross-view prefetch collapses from one request per view to a single /api/v2/cell bundle, whose slices (exact per-view payloads + their etags) are seeded under the URLs each view actually requests; the bundle call itself is conditional via a remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one possible Nominatim lookup each), so tapping nearby lands on already-graded data. Legacy tg:* storage entries are cleared once; cache entries untouched for two weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
the frontend splits longer spans into successive 2-year chunks. Grades against
the already-cached history record (which reaches to ~6 days ago), so no extra
fetch is needed. The graded payload is cached in the derived store keyed by
the clamped span and validated by the record's end date, so a repeat span is
a database read across restarts until new archive days arrive. Returns a
compact per-day shape (see grading.grade_range). New in API v2.
"""
if not grid.in_north_america(lat, lon):
raise HTTPException(
status_code=400,
detail="Location is outside the supported US + Canada region.",
)
cell = grid.snap(lat, lon)
with audit.RunAudit(
endpoint="calendar",
lat=round(lat, 4),
lon=round(lon, 4),
recent_days=months * 31, # approximate span graded
cell_id=cell["id"],
) as run:
try:
with run.phase("history"):
history, cache_meta = climate.get_history(cell)
except Exception as e: # noqa: BLE001
raise _weather_fetch_error(e)
if history.empty:
raise HTTPException(status_code=404, detail="No historical data for this cell.")
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21) * Compress API responses and revalidate static assets instead of re-downloading - Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x. - Serve pages/assets with Cache-Control: no-cache instead of no-store, so browsers revalidate via the ETag/Last-Modified that FileResponse and StaticFiles already emit. Unchanged assets now cost an empty 304 rather than a full transfer on every page navigation, while deploys still show up immediately. * Persist derived responses in SQLite so grading is computed once per cell, not per request New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet records — finished grade/calendar/day/forecast payloads and reverse-geocode labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms) becomes a ~5ms database read that survives restarts and is shared across views. Raw parquet stays the source of truth; the store is a pure accelerator (every reader falls back to recomputing on a miss, and deleting the db is a safe reset). Freshness is token-driven, not clock-driven: each cached payload is validated by a token encoding what it was computed from (payload schema version, the archive record's end date, the recent-fetch stamp). The existing freshness drivers are untouched — get_history still tops up the tail hourly and get_recent_forecast still refetches hourly — and tokens are derived from what they return, so cached payloads expire exactly when their inputs change. The same tokens double as weak ETags: If-None-Match answers with an empty 304 without touching the payload. - backend/store.py: derived-payload + revgeo tables, thread-local WAL conns, every helper fail-soft. - app.py: endpoints split into pure payload builders + HTTP/caching shells; the in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store). - climate.py: revgeo persisted through the store; recent_stamp() and load_cached_history() (no-network read) helpers. - backend/migrate.py + make migrate: idempotent, resumable backfill of the store from existing parquet caches (default calendar span + latest-day detail + revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather. - grid.from_id(): rebuild a cell from its cache filename (migrate tooling). * Add /api/v2/cell: one bundle carrying every view's payload GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months) and day (today) payloads in one response, each the exact payload its per-view endpoint returns — built by the same builders and cached under the same derived-store keys/tokens — paired with the etag that endpoint would emit. The frontend can warm all views with a single request, seed its per-view cache from the slices, and later revalidate each view individually with If-None-Match. The bundle's own etag combines the slices', so an unchanged bundle is an empty 304. prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard guarantee: it never spends weather-API quota. A cell with no cached archive answers 204, and only the history-derived slices (calendar + latest-day detail) are built. At most one Nominatim lookup for a never-labeled cell. * Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota, which multi-year calendar payloads regularly blew through) to IndexedDB, with an in-memory map in front. Entries carry the server's ETag, so anything stale revalidates conditionally — unchanged data costs an empty 304 and a re-stamp, never a re-transfer. On network failure the stale copy is served over an error. getJSON gains an optional onUpdate callback opting into stale-while-revalidate: the three views (weekly, day, calendar) now render a cached copy immediately — spinners are delayed 150ms so warm loads never flash-blank — and repaint only if background revalidation finds changed data. New data shows up the moment it exists instead of waiting out a TTL. Cross-view prefetch collapses from one request per view to a single /api/v2/cell bundle, whose slices (exact per-view payloads + their etags) are seeded under the URLs each view actually requests; the bundle call itself is conditional via a remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one possible Nominatim lookup each), so tapping nearby lands on already-graded data. Legacy tg:* storage entries are cleared once; cache entries untouched for two weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
start_ts, end_ts = _cal_span(history, start, end, months)
key = f"{start_ts.date().isoformat()}:{end_ts.date().isoformat()}:{months}"
token = f"{PAYLOAD_VER}:{_hist_end(history)}"
etag = _etag_for("calendar", cell["id"], key, token)
if _not_modified(request, etag):
run.set(run_type="cache", not_modified=True)
return Response(status_code=304, headers={"ETag": etag})
body = store.get_payload("calendar", cell["id"], key, token)
if body is not None:
run.set(run_type="cache", history_source="store")
return _json_response(body, etag)
with run.phase("reverse_geocode"):
place = climate.reverse_geocode(cell["center_lat"], cell["center_lon"])
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21) * Compress API responses and revalidate static assets instead of re-downloading - Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x. - Serve pages/assets with Cache-Control: no-cache instead of no-store, so browsers revalidate via the ETag/Last-Modified that FileResponse and StaticFiles already emit. Unchanged assets now cost an empty 304 rather than a full transfer on every page navigation, while deploys still show up immediately. * Persist derived responses in SQLite so grading is computed once per cell, not per request New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet records — finished grade/calendar/day/forecast payloads and reverse-geocode labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms) becomes a ~5ms database read that survives restarts and is shared across views. Raw parquet stays the source of truth; the store is a pure accelerator (every reader falls back to recomputing on a miss, and deleting the db is a safe reset). Freshness is token-driven, not clock-driven: each cached payload is validated by a token encoding what it was computed from (payload schema version, the archive record's end date, the recent-fetch stamp). The existing freshness drivers are untouched — get_history still tops up the tail hourly and get_recent_forecast still refetches hourly — and tokens are derived from what they return, so cached payloads expire exactly when their inputs change. The same tokens double as weak ETags: If-None-Match answers with an empty 304 without touching the payload. - backend/store.py: derived-payload + revgeo tables, thread-local WAL conns, every helper fail-soft. - app.py: endpoints split into pure payload builders + HTTP/caching shells; the in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store). - climate.py: revgeo persisted through the store; recent_stamp() and load_cached_history() (no-network read) helpers. - backend/migrate.py + make migrate: idempotent, resumable backfill of the store from existing parquet caches (default calendar span + latest-day detail + revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather. - grid.from_id(): rebuild a cell from its cache filename (migrate tooling). * Add /api/v2/cell: one bundle carrying every view's payload GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months) and day (today) payloads in one response, each the exact payload its per-view endpoint returns — built by the same builders and cached under the same derived-store keys/tokens — paired with the etag that endpoint would emit. The frontend can warm all views with a single request, seed its per-view cache from the slices, and later revalidate each view individually with If-None-Match. The bundle's own etag combines the slices', so an unchanged bundle is an empty 304. prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard guarantee: it never spends weather-API quota. A cell with no cached archive answers 204, and only the history-derived slices (calendar + latest-day detail) are built. At most one Nominatim lookup for a never-labeled cell. * Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota, which multi-year calendar payloads regularly blew through) to IndexedDB, with an in-memory map in front. Entries carry the server's ETag, so anything stale revalidates conditionally — unchanged data costs an empty 304 and a re-stamp, never a re-transfer. On network failure the stale copy is served over an error. getJSON gains an optional onUpdate callback opting into stale-while-revalidate: the three views (weekly, day, calendar) now render a cached copy immediately — spinners are delayed 150ms so warm loads never flash-blank — and repaint only if background revalidation finds changed data. New data shows up the moment it exists instead of waiting out a TTL. Cross-view prefetch collapses from one request per view to a single /api/v2/cell bundle, whose slices (exact per-view payloads + their etags) are seeded under the URLs each view actually requests; the bundle call itself is conditional via a remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one possible Nominatim lookup each), so tapping nearby lands on already-graded data. Legacy tg:* storage entries are cleared once; cache entries untouched for two weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
payload = _build_calendar(cell, history, start_ts, end_ts, months, place, run)
full = not cache_meta.get("cached", False)
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21) * Compress API responses and revalidate static assets instead of re-downloading - Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x. - Serve pages/assets with Cache-Control: no-cache instead of no-store, so browsers revalidate via the ETag/Last-Modified that FileResponse and StaticFiles already emit. Unchanged assets now cost an empty 304 rather than a full transfer on every page navigation, while deploys still show up immediately. * Persist derived responses in SQLite so grading is computed once per cell, not per request New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet records — finished grade/calendar/day/forecast payloads and reverse-geocode labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms) becomes a ~5ms database read that survives restarts and is shared across views. Raw parquet stays the source of truth; the store is a pure accelerator (every reader falls back to recomputing on a miss, and deleting the db is a safe reset). Freshness is token-driven, not clock-driven: each cached payload is validated by a token encoding what it was computed from (payload schema version, the archive record's end date, the recent-fetch stamp). The existing freshness drivers are untouched — get_history still tops up the tail hourly and get_recent_forecast still refetches hourly — and tokens are derived from what they return, so cached payloads expire exactly when their inputs change. The same tokens double as weak ETags: If-None-Match answers with an empty 304 without touching the payload. - backend/store.py: derived-payload + revgeo tables, thread-local WAL conns, every helper fail-soft. - app.py: endpoints split into pure payload builders + HTTP/caching shells; the in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store). - climate.py: revgeo persisted through the store; recent_stamp() and load_cached_history() (no-network read) helpers. - backend/migrate.py + make migrate: idempotent, resumable backfill of the store from existing parquet caches (default calendar span + latest-day detail + revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather. - grid.from_id(): rebuild a cell from its cache filename (migrate tooling). * Add /api/v2/cell: one bundle carrying every view's payload GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months) and day (today) payloads in one response, each the exact payload its per-view endpoint returns — built by the same builders and cached under the same derived-store keys/tokens — paired with the etag that endpoint would emit. The frontend can warm all views with a single request, seed its per-view cache from the slices, and later revalidate each view individually with If-None-Match. The bundle's own etag combines the slices', so an unchanged bundle is an empty 304. prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard guarantee: it never spends weather-API quota. A cell with no cached archive answers 204, and only the history-derived slices (calendar + latest-day detail) are built. At most one Nominatim lookup for a never-labeled cell. * Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota, which multi-year calendar payloads regularly blew through) to IndexedDB, with an in-memory map in front. Entries carry the server's ETag, so anything stale revalidates conditionally — unchanged data costs an empty 304 and a re-stamp, never a re-transfer. On network failure the stale copy is served over an error. getJSON gains an optional onUpdate callback opting into stale-while-revalidate: the three views (weekly, day, calendar) now render a cached copy immediately — spinners are delayed 150ms so warm loads never flash-blank — and repaint only if background revalidation finds changed data. New data shows up the moment it exists instead of waiting out a TTL. Cross-view prefetch collapses from one request per view to a single /api/v2/cell bundle, whose slices (exact per-view payloads + their etags) are seeded under the URLs each view actually requests; the bundle call itself is conditional via a remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one possible Nominatim lookup each), so tapping nearby lands on already-graded data. Legacy tg:* storage entries are cleared once; cache entries untouched for two weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
run.set(run_type="full" if full else "partial",
history_source="fetch" if full else "cache")
# Don't persist a payload whose place failed to resolve (a transient reverse-
# geocode miss) — otherwise the bare-coordinates fallback would stick for the
# life of the token. Compare loads several cells at once, so this is where a
# miss is most likely; leaving it uncached lets the next request retry.
return _json_response(
store.put_payload("calendar", cell["id"], key, token, payload,
cache=place is not None), etag)
def api_day(
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21) * Compress API responses and revalidate static assets instead of re-downloading - Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x. - Serve pages/assets with Cache-Control: no-cache instead of no-store, so browsers revalidate via the ETag/Last-Modified that FileResponse and StaticFiles already emit. Unchanged assets now cost an empty 304 rather than a full transfer on every page navigation, while deploys still show up immediately. * Persist derived responses in SQLite so grading is computed once per cell, not per request New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet records — finished grade/calendar/day/forecast payloads and reverse-geocode labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms) becomes a ~5ms database read that survives restarts and is shared across views. Raw parquet stays the source of truth; the store is a pure accelerator (every reader falls back to recomputing on a miss, and deleting the db is a safe reset). Freshness is token-driven, not clock-driven: each cached payload is validated by a token encoding what it was computed from (payload schema version, the archive record's end date, the recent-fetch stamp). The existing freshness drivers are untouched — get_history still tops up the tail hourly and get_recent_forecast still refetches hourly — and tokens are derived from what they return, so cached payloads expire exactly when their inputs change. The same tokens double as weak ETags: If-None-Match answers with an empty 304 without touching the payload. - backend/store.py: derived-payload + revgeo tables, thread-local WAL conns, every helper fail-soft. - app.py: endpoints split into pure payload builders + HTTP/caching shells; the in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store). - climate.py: revgeo persisted through the store; recent_stamp() and load_cached_history() (no-network read) helpers. - backend/migrate.py + make migrate: idempotent, resumable backfill of the store from existing parquet caches (default calendar span + latest-day detail + revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather. - grid.from_id(): rebuild a cell from its cache filename (migrate tooling). * Add /api/v2/cell: one bundle carrying every view's payload GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months) and day (today) payloads in one response, each the exact payload its per-view endpoint returns — built by the same builders and cached under the same derived-store keys/tokens — paired with the etag that endpoint would emit. The frontend can warm all views with a single request, seed its per-view cache from the slices, and later revalidate each view individually with If-None-Match. The bundle's own etag combines the slices', so an unchanged bundle is an empty 304. prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard guarantee: it never spends weather-API quota. A cell with no cached archive answers 204, and only the history-derived slices (calendar + latest-day detail) are built. At most one Nominatim lookup for a never-labeled cell. * Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota, which multi-year calendar payloads regularly blew through) to IndexedDB, with an in-memory map in front. Entries carry the server's ETag, so anything stale revalidates conditionally — unchanged data costs an empty 304 and a re-stamp, never a re-transfer. On network failure the stale copy is served over an error. getJSON gains an optional onUpdate callback opting into stale-while-revalidate: the three views (weekly, day, calendar) now render a cached copy immediately — spinners are delayed 150ms so warm loads never flash-blank — and repaint only if background revalidation finds changed data. New data shows up the moment it exists instead of waiting out a TTL. Cross-view prefetch collapses from one request per view to a single /api/v2/cell bundle, whose slices (exact per-view payloads + their etags) are seeded under the URLs each view actually requests; the bundle call itself is conditional via a remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one possible Nominatim lookup each), so tapping nearby lands on already-graded data. Legacy tg:* storage entries are cleared once; cache entries untouched for two weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
request: Request,
lat: float = Query(..., ge=-90, le=90),
lon: float = Query(..., ge=-180, le=180),
date: str | None = Query(None, description="target date YYYY-MM-DD (default = latest available)"),
):
"""Full percentile breakdown for a single day: the value at every tier boundary
in that day-of-year's ±7-day window, plus where the observed values land.
Powers the single-day detail page. Grades against the cached history record;
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21) * Compress API responses and revalidate static assets instead of re-downloading - Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x. - Serve pages/assets with Cache-Control: no-cache instead of no-store, so browsers revalidate via the ETag/Last-Modified that FileResponse and StaticFiles already emit. Unchanged assets now cost an empty 304 rather than a full transfer on every page navigation, while deploys still show up immediately. * Persist derived responses in SQLite so grading is computed once per cell, not per request New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet records — finished grade/calendar/day/forecast payloads and reverse-geocode labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms) becomes a ~5ms database read that survives restarts and is shared across views. Raw parquet stays the source of truth; the store is a pure accelerator (every reader falls back to recomputing on a miss, and deleting the db is a safe reset). Freshness is token-driven, not clock-driven: each cached payload is validated by a token encoding what it was computed from (payload schema version, the archive record's end date, the recent-fetch stamp). The existing freshness drivers are untouched — get_history still tops up the tail hourly and get_recent_forecast still refetches hourly — and tokens are derived from what they return, so cached payloads expire exactly when their inputs change. The same tokens double as weak ETags: If-None-Match answers with an empty 304 without touching the payload. - backend/store.py: derived-payload + revgeo tables, thread-local WAL conns, every helper fail-soft. - app.py: endpoints split into pure payload builders + HTTP/caching shells; the in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store). - climate.py: revgeo persisted through the store; recent_stamp() and load_cached_history() (no-network read) helpers. - backend/migrate.py + make migrate: idempotent, resumable backfill of the store from existing parquet caches (default calendar span + latest-day detail + revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather. - grid.from_id(): rebuild a cell from its cache filename (migrate tooling). * Add /api/v2/cell: one bundle carrying every view's payload GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months) and day (today) payloads in one response, each the exact payload its per-view endpoint returns — built by the same builders and cached under the same derived-store keys/tokens — paired with the etag that endpoint would emit. The frontend can warm all views with a single request, seed its per-view cache from the slices, and later revalidate each view individually with If-None-Match. The bundle's own etag combines the slices', so an unchanged bundle is an empty 304. prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard guarantee: it never spends weather-API quota. A cell with no cached archive answers 204, and only the history-derived slices (calendar + latest-day detail) are built. At most one Nominatim lookup for a never-labeled cell. * Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota, which multi-year calendar payloads regularly blew through) to IndexedDB, with an in-memory map in front. Entries carry the server's ETag, so anything stale revalidates conditionally — unchanged data costs an empty 304 and a re-stamp, never a re-transfer. On network failure the stale copy is served over an error. getJSON gains an optional onUpdate callback opting into stale-while-revalidate: the three views (weekly, day, calendar) now render a cached copy immediately — spinners are delayed 150ms so warm loads never flash-blank — and repaint only if background revalidation finds changed data. New data shows up the moment it exists instead of waiting out a TTL. Cross-view prefetch collapses from one request per view to a single /api/v2/cell bundle, whose slices (exact per-view payloads + their etags) are seeded under the URLs each view actually requests; the bundle call itself is conditional via a remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one possible Nominatim lookup each), so tapping nearby lands on already-graded data. Legacy tg:* storage entries are cleared once; cache entries untouched for two weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
for a date newer than the cache it pulls the recent window for the observation
(those payloads expire hourly the recent bundle's own cadence — while fully
archived days stay valid until the record itself advances). New in API v2.
"""
if not grid.in_north_america(lat, lon):
raise HTTPException(
status_code=400,
detail="Location is outside the supported US + Canada region.",
)
cell = grid.snap(lat, lon)
with audit.RunAudit(
endpoint="day",
lat=round(lat, 4),
lon=round(lon, 4),
cell_id=cell["id"],
) as run:
with run.phase("history"):
history, cache_meta = climate.get_history(cell)
if history.empty:
raise HTTPException(status_code=404, detail="No historical data for this cell.")
last = pd.Timestamp(history["date"].max()).normalize()
target = pd.Timestamp(date).normalize() if date else last
run.set(target_date=target.date().isoformat())
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21) * Compress API responses and revalidate static assets instead of re-downloading - Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x. - Serve pages/assets with Cache-Control: no-cache instead of no-store, so browsers revalidate via the ETag/Last-Modified that FileResponse and StaticFiles already emit. Unchanged assets now cost an empty 304 rather than a full transfer on every page navigation, while deploys still show up immediately. * Persist derived responses in SQLite so grading is computed once per cell, not per request New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet records — finished grade/calendar/day/forecast payloads and reverse-geocode labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms) becomes a ~5ms database read that survives restarts and is shared across views. Raw parquet stays the source of truth; the store is a pure accelerator (every reader falls back to recomputing on a miss, and deleting the db is a safe reset). Freshness is token-driven, not clock-driven: each cached payload is validated by a token encoding what it was computed from (payload schema version, the archive record's end date, the recent-fetch stamp). The existing freshness drivers are untouched — get_history still tops up the tail hourly and get_recent_forecast still refetches hourly — and tokens are derived from what they return, so cached payloads expire exactly when their inputs change. The same tokens double as weak ETags: If-None-Match answers with an empty 304 without touching the payload. - backend/store.py: derived-payload + revgeo tables, thread-local WAL conns, every helper fail-soft. - app.py: endpoints split into pure payload builders + HTTP/caching shells; the in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store). - climate.py: revgeo persisted through the store; recent_stamp() and load_cached_history() (no-network read) helpers. - backend/migrate.py + make migrate: idempotent, resumable backfill of the store from existing parquet caches (default calendar span + latest-day detail + revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather. - grid.from_id(): rebuild a cell from its cache filename (migrate tooling). * Add /api/v2/cell: one bundle carrying every view's payload GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months) and day (today) payloads in one response, each the exact payload its per-view endpoint returns — built by the same builders and cached under the same derived-store keys/tokens — paired with the etag that endpoint would emit. The frontend can warm all views with a single request, seed its per-view cache from the slices, and later revalidate each view individually with If-None-Match. The bundle's own etag combines the slices', so an unchanged bundle is an empty 304. prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard guarantee: it never spends weather-API quota. A cell with no cached archive answers 204, and only the history-derived slices (calendar + latest-day detail) are built. At most one Nominatim lookup for a never-labeled cell. * Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota, which multi-year calendar payloads regularly blew through) to IndexedDB, with an in-memory map in front. Entries carry the server's ETag, so anything stale revalidates conditionally — unchanged data costs an empty 304 and a re-stamp, never a re-transfer. On network failure the stale copy is served over an error. getJSON gains an optional onUpdate callback opting into stale-while-revalidate: the three views (weekly, day, calendar) now render a cached copy immediately — spinners are delayed 150ms so warm loads never flash-blank — and repaint only if background revalidation finds changed data. New data shows up the moment it exists instead of waiting out a TTL. Cross-view prefetch collapses from one request per view to a single /api/v2/cell bundle, whose slices (exact per-view payloads + their etags) are seeded under the URLs each view actually requests; the bundle call itself is conditional via a remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one possible Nominatim lookup each), so tapping nearby lands on already-graded data. Legacy tg:* storage entries are cleared once; cache entries untouched for two weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
key = target.date().isoformat()
token = f"{PAYLOAD_VER}:{_hist_end(history)}"
if target > last:
token += f":h{int(time.time() // 3600)}" # obs from the hourly recent bundle
etag = _etag_for("day", cell["id"], key, token)
if _not_modified(request, etag):
run.set(run_type="cache", not_modified=True)
return Response(status_code=304, headers={"ETag": etag})
body = store.get_payload("day", cell["id"], key, token)
if body is not None:
run.set(run_type="cache", history_source="store")
return _json_response(body, etag)
with run.phase("reverse_geocode"):
place = climate.reverse_geocode(cell["center_lat"], cell["center_lon"])
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21) * Compress API responses and revalidate static assets instead of re-downloading - Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x. - Serve pages/assets with Cache-Control: no-cache instead of no-store, so browsers revalidate via the ETag/Last-Modified that FileResponse and StaticFiles already emit. Unchanged assets now cost an empty 304 rather than a full transfer on every page navigation, while deploys still show up immediately. * Persist derived responses in SQLite so grading is computed once per cell, not per request New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet records — finished grade/calendar/day/forecast payloads and reverse-geocode labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms) becomes a ~5ms database read that survives restarts and is shared across views. Raw parquet stays the source of truth; the store is a pure accelerator (every reader falls back to recomputing on a miss, and deleting the db is a safe reset). Freshness is token-driven, not clock-driven: each cached payload is validated by a token encoding what it was computed from (payload schema version, the archive record's end date, the recent-fetch stamp). The existing freshness drivers are untouched — get_history still tops up the tail hourly and get_recent_forecast still refetches hourly — and tokens are derived from what they return, so cached payloads expire exactly when their inputs change. The same tokens double as weak ETags: If-None-Match answers with an empty 304 without touching the payload. - backend/store.py: derived-payload + revgeo tables, thread-local WAL conns, every helper fail-soft. - app.py: endpoints split into pure payload builders + HTTP/caching shells; the in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store). - climate.py: revgeo persisted through the store; recent_stamp() and load_cached_history() (no-network read) helpers. - backend/migrate.py + make migrate: idempotent, resumable backfill of the store from existing parquet caches (default calendar span + latest-day detail + revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather. - grid.from_id(): rebuild a cell from its cache filename (migrate tooling). * Add /api/v2/cell: one bundle carrying every view's payload GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months) and day (today) payloads in one response, each the exact payload its per-view endpoint returns — built by the same builders and cached under the same derived-store keys/tokens — paired with the etag that endpoint would emit. The frontend can warm all views with a single request, seed its per-view cache from the slices, and later revalidate each view individually with If-None-Match. The bundle's own etag combines the slices', so an unchanged bundle is an empty 304. prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard guarantee: it never spends weather-API quota. A cell with no cached archive answers 204, and only the history-derived slices (calendar + latest-day detail) are built. At most one Nominatim lookup for a never-labeled cell. * Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota, which multi-year calendar payloads regularly blew through) to IndexedDB, with an in-memory map in front. Entries carry the server's ETag, so anything stale revalidates conditionally — unchanged data costs an empty 304 and a re-stamp, never a re-transfer. On network failure the stale copy is served over an error. getJSON gains an optional onUpdate callback opting into stale-while-revalidate: the three views (weekly, day, calendar) now render a cached copy immediately — spinners are delayed 150ms so warm loads never flash-blank — and repaint only if background revalidation finds changed data. New data shows up the moment it exists instead of waiting out a TTL. Cross-view prefetch collapses from one request per view to a single /api/v2/cell bundle, whose slices (exact per-view payloads + their etags) are seeded under the URLs each view actually requests; the bundle call itself is conditional via a remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one possible Nominatim lookup each), so tapping nearby lands on already-graded data. Legacy tg:* storage entries are cleared once; cache entries untouched for two weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
payload = _build_day(cell, history, target, place, run)
full = not cache_meta.get("cached", False)
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21) * Compress API responses and revalidate static assets instead of re-downloading - Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x. - Serve pages/assets with Cache-Control: no-cache instead of no-store, so browsers revalidate via the ETag/Last-Modified that FileResponse and StaticFiles already emit. Unchanged assets now cost an empty 304 rather than a full transfer on every page navigation, while deploys still show up immediately. * Persist derived responses in SQLite so grading is computed once per cell, not per request New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet records — finished grade/calendar/day/forecast payloads and reverse-geocode labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms) becomes a ~5ms database read that survives restarts and is shared across views. Raw parquet stays the source of truth; the store is a pure accelerator (every reader falls back to recomputing on a miss, and deleting the db is a safe reset). Freshness is token-driven, not clock-driven: each cached payload is validated by a token encoding what it was computed from (payload schema version, the archive record's end date, the recent-fetch stamp). The existing freshness drivers are untouched — get_history still tops up the tail hourly and get_recent_forecast still refetches hourly — and tokens are derived from what they return, so cached payloads expire exactly when their inputs change. The same tokens double as weak ETags: If-None-Match answers with an empty 304 without touching the payload. - backend/store.py: derived-payload + revgeo tables, thread-local WAL conns, every helper fail-soft. - app.py: endpoints split into pure payload builders + HTTP/caching shells; the in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store). - climate.py: revgeo persisted through the store; recent_stamp() and load_cached_history() (no-network read) helpers. - backend/migrate.py + make migrate: idempotent, resumable backfill of the store from existing parquet caches (default calendar span + latest-day detail + revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather. - grid.from_id(): rebuild a cell from its cache filename (migrate tooling). * Add /api/v2/cell: one bundle carrying every view's payload GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months) and day (today) payloads in one response, each the exact payload its per-view endpoint returns — built by the same builders and cached under the same derived-store keys/tokens — paired with the etag that endpoint would emit. The frontend can warm all views with a single request, seed its per-view cache from the slices, and later revalidate each view individually with If-None-Match. The bundle's own etag combines the slices', so an unchanged bundle is an empty 304. prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard guarantee: it never spends weather-API quota. A cell with no cached archive answers 204, and only the history-derived slices (calendar + latest-day detail) are built. At most one Nominatim lookup for a never-labeled cell. * Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota, which multi-year calendar payloads regularly blew through) to IndexedDB, with an in-memory map in front. Entries carry the server's ETag, so anything stale revalidates conditionally — unchanged data costs an empty 304 and a re-stamp, never a re-transfer. On network failure the stale copy is served over an error. getJSON gains an optional onUpdate callback opting into stale-while-revalidate: the three views (weekly, day, calendar) now render a cached copy immediately — spinners are delayed 150ms so warm loads never flash-blank — and repaint only if background revalidation finds changed data. New data shows up the moment it exists instead of waiting out a TTL. Cross-view prefetch collapses from one request per view to a single /api/v2/cell bundle, whose slices (exact per-view payloads + their etags) are seeded under the URLs each view actually requests; the bundle call itself is conditional via a remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one possible Nominatim lookup each), so tapping nearby lands on already-graded data. Legacy tg:* storage entries are cleared once; cache entries untouched for two weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
run.set(run_type="full" if full else "partial",
history_source="fetch" if full else "cache")
return _json_response(store.put_payload("day", cell["id"], key, token, payload), etag)
def api_forecast(
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21) * Compress API responses and revalidate static assets instead of re-downloading - Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x. - Serve pages/assets with Cache-Control: no-cache instead of no-store, so browsers revalidate via the ETag/Last-Modified that FileResponse and StaticFiles already emit. Unchanged assets now cost an empty 304 rather than a full transfer on every page navigation, while deploys still show up immediately. * Persist derived responses in SQLite so grading is computed once per cell, not per request New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet records — finished grade/calendar/day/forecast payloads and reverse-geocode labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms) becomes a ~5ms database read that survives restarts and is shared across views. Raw parquet stays the source of truth; the store is a pure accelerator (every reader falls back to recomputing on a miss, and deleting the db is a safe reset). Freshness is token-driven, not clock-driven: each cached payload is validated by a token encoding what it was computed from (payload schema version, the archive record's end date, the recent-fetch stamp). The existing freshness drivers are untouched — get_history still tops up the tail hourly and get_recent_forecast still refetches hourly — and tokens are derived from what they return, so cached payloads expire exactly when their inputs change. The same tokens double as weak ETags: If-None-Match answers with an empty 304 without touching the payload. - backend/store.py: derived-payload + revgeo tables, thread-local WAL conns, every helper fail-soft. - app.py: endpoints split into pure payload builders + HTTP/caching shells; the in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store). - climate.py: revgeo persisted through the store; recent_stamp() and load_cached_history() (no-network read) helpers. - backend/migrate.py + make migrate: idempotent, resumable backfill of the store from existing parquet caches (default calendar span + latest-day detail + revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather. - grid.from_id(): rebuild a cell from its cache filename (migrate tooling). * Add /api/v2/cell: one bundle carrying every view's payload GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months) and day (today) payloads in one response, each the exact payload its per-view endpoint returns — built by the same builders and cached under the same derived-store keys/tokens — paired with the etag that endpoint would emit. The frontend can warm all views with a single request, seed its per-view cache from the slices, and later revalidate each view individually with If-None-Match. The bundle's own etag combines the slices', so an unchanged bundle is an empty 304. prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard guarantee: it never spends weather-API quota. A cell with no cached archive answers 204, and only the history-derived slices (calendar + latest-day detail) are built. At most one Nominatim lookup for a never-labeled cell. * Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota, which multi-year calendar payloads regularly blew through) to IndexedDB, with an in-memory map in front. Entries carry the server's ETag, so anything stale revalidates conditionally — unchanged data costs an empty 304 and a re-stamp, never a re-transfer. On network failure the stale copy is served over an error. getJSON gains an optional onUpdate callback opting into stale-while-revalidate: the three views (weekly, day, calendar) now render a cached copy immediately — spinners are delayed 150ms so warm loads never flash-blank — and repaint only if background revalidation finds changed data. New data shows up the moment it exists instead of waiting out a TTL. Cross-view prefetch collapses from one request per view to a single /api/v2/cell bundle, whose slices (exact per-view payloads + their etags) are seeded under the URLs each view actually requests; the bundle call itself is conditional via a remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one possible Nominatim lookup each), so tapping nearby lands on already-graded data. Legacy tg:* storage entries are cleared once; cache entries untouched for two weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
request: Request,
lat: float = Query(..., ge=-90, le=90),
lon: float = Query(..., ge=-180, le=180),
days: int = Query(7, ge=1, le=14, description="how many forecast days ahead to grade"),
):
"""Grade the next `days` of forecast weather against local climatology.
Same shape as /grade (so the frontend renders it identically), but `recent`
holds the forward forecast furthest-out day first. The forecast is fetched
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21) * Compress API responses and revalidate static assets instead of re-downloading - Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x. - Serve pages/assets with Cache-Control: no-cache instead of no-store, so browsers revalidate via the ETag/Last-Modified that FileResponse and StaticFiles already emit. Unchanged assets now cost an empty 304 rather than a full transfer on every page navigation, while deploys still show up immediately. * Persist derived responses in SQLite so grading is computed once per cell, not per request New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet records — finished grade/calendar/day/forecast payloads and reverse-geocode labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms) becomes a ~5ms database read that survives restarts and is shared across views. Raw parquet stays the source of truth; the store is a pure accelerator (every reader falls back to recomputing on a miss, and deleting the db is a safe reset). Freshness is token-driven, not clock-driven: each cached payload is validated by a token encoding what it was computed from (payload schema version, the archive record's end date, the recent-fetch stamp). The existing freshness drivers are untouched — get_history still tops up the tail hourly and get_recent_forecast still refetches hourly — and tokens are derived from what they return, so cached payloads expire exactly when their inputs change. The same tokens double as weak ETags: If-None-Match answers with an empty 304 without touching the payload. - backend/store.py: derived-payload + revgeo tables, thread-local WAL conns, every helper fail-soft. - app.py: endpoints split into pure payload builders + HTTP/caching shells; the in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store). - climate.py: revgeo persisted through the store; recent_stamp() and load_cached_history() (no-network read) helpers. - backend/migrate.py + make migrate: idempotent, resumable backfill of the store from existing parquet caches (default calendar span + latest-day detail + revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather. - grid.from_id(): rebuild a cell from its cache filename (migrate tooling). * Add /api/v2/cell: one bundle carrying every view's payload GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months) and day (today) payloads in one response, each the exact payload its per-view endpoint returns — built by the same builders and cached under the same derived-store keys/tokens — paired with the etag that endpoint would emit. The frontend can warm all views with a single request, seed its per-view cache from the slices, and later revalidate each view individually with If-None-Match. The bundle's own etag combines the slices', so an unchanged bundle is an empty 304. prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard guarantee: it never spends weather-API quota. A cell with no cached archive answers 204, and only the history-derived slices (calendar + latest-day detail) are built. At most one Nominatim lookup for a never-labeled cell. * Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota, which multi-year calendar payloads regularly blew through) to IndexedDB, with an in-memory map in front. Entries carry the server's ETag, so anything stale revalidates conditionally — unchanged data costs an empty 304 and a re-stamp, never a re-transfer. On network failure the stale copy is served over an error. getJSON gains an optional onUpdate callback opting into stale-while-revalidate: the three views (weekly, day, calendar) now render a cached copy immediately — spinners are delayed 150ms so warm loads never flash-blank — and repaint only if background revalidation finds changed data. New data shows up the moment it exists instead of waiting out a TTL. Cross-view prefetch collapses from one request per view to a single /api/v2/cell bundle, whose slices (exact per-view payloads + their etags) are seeded under the URLs each view actually requests; the bundle call itself is conditional via a remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one possible Nominatim lookup each), so tapping nearby lands on already-graded data. Legacy tg:* storage entries are cleared once; cache entries untouched for two weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
fresh and cached only ~1 hour (see climate.get_recent_forecast) to track
updates; the graded payload's validity is tied to that fetch stamp, so it
expires exactly when a new forecast lands. New in API v2.
"""
if not grid.in_north_america(lat, lon):
raise HTTPException(
status_code=400,
detail="Location is outside the supported US + Canada region.",
)
cell = grid.snap(lat, lon)
with audit.RunAudit(endpoint="forecast", lat=round(lat, 4), lon=round(lon, 4),
recent_days=days, cell_id=cell["id"]) as run:
try:
with run.phase("history"):
history, _ = climate.get_history(cell)
with run.phase("forecast"):
fc = climate.get_recent_forecast(cell)
except Exception as e: # noqa: BLE001
raise _weather_fetch_error(e)
if history.empty:
raise HTTPException(status_code=404, detail="No historical data for this cell.")
today = pd.Timestamp(datetime.date.today())
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21) * Compress API responses and revalidate static assets instead of re-downloading - Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x. - Serve pages/assets with Cache-Control: no-cache instead of no-store, so browsers revalidate via the ETag/Last-Modified that FileResponse and StaticFiles already emit. Unchanged assets now cost an empty 304 rather than a full transfer on every page navigation, while deploys still show up immediately. * Persist derived responses in SQLite so grading is computed once per cell, not per request New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet records — finished grade/calendar/day/forecast payloads and reverse-geocode labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms) becomes a ~5ms database read that survives restarts and is shared across views. Raw parquet stays the source of truth; the store is a pure accelerator (every reader falls back to recomputing on a miss, and deleting the db is a safe reset). Freshness is token-driven, not clock-driven: each cached payload is validated by a token encoding what it was computed from (payload schema version, the archive record's end date, the recent-fetch stamp). The existing freshness drivers are untouched — get_history still tops up the tail hourly and get_recent_forecast still refetches hourly — and tokens are derived from what they return, so cached payloads expire exactly when their inputs change. The same tokens double as weak ETags: If-None-Match answers with an empty 304 without touching the payload. - backend/store.py: derived-payload + revgeo tables, thread-local WAL conns, every helper fail-soft. - app.py: endpoints split into pure payload builders + HTTP/caching shells; the in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store). - climate.py: revgeo persisted through the store; recent_stamp() and load_cached_history() (no-network read) helpers. - backend/migrate.py + make migrate: idempotent, resumable backfill of the store from existing parquet caches (default calendar span + latest-day detail + revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather. - grid.from_id(): rebuild a cell from its cache filename (migrate tooling). * Add /api/v2/cell: one bundle carrying every view's payload GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months) and day (today) payloads in one response, each the exact payload its per-view endpoint returns — built by the same builders and cached under the same derived-store keys/tokens — paired with the etag that endpoint would emit. The frontend can warm all views with a single request, seed its per-view cache from the slices, and later revalidate each view individually with If-None-Match. The bundle's own etag combines the slices', so an unchanged bundle is an empty 304. prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard guarantee: it never spends weather-API quota. A cell with no cached archive answers 204, and only the history-derived slices (calendar + latest-day detail) are built. At most one Nominatim lookup for a never-labeled cell. * Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota, which multi-year calendar payloads regularly blew through) to IndexedDB, with an in-memory map in front. Entries carry the server's ETag, so anything stale revalidates conditionally — unchanged data costs an empty 304 and a re-stamp, never a re-transfer. On network failure the stale copy is served over an error. getJSON gains an optional onUpdate callback opting into stale-while-revalidate: the three views (weekly, day, calendar) now render a cached copy immediately — spinners are delayed 150ms so warm loads never flash-blank — and repaint only if background revalidation finds changed data. New data shows up the moment it exists instead of waiting out a TTL. Cross-view prefetch collapses from one request per view to a single /api/v2/cell bundle, whose slices (exact per-view payloads + their etags) are seeded under the URLs each view actually requests; the bundle call itself is conditional via a remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one possible Nominatim lookup each), so tapping nearby lands on already-graded data. Legacy tg:* storage entries are cleared once; cache entries untouched for two weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
key = f"{today.date().isoformat()}:{days}"
token = f"{PAYLOAD_VER}:{_hist_end(history)}:{climate.recent_stamp(cell['id'])}"
etag = _etag_for("forecast", cell["id"], key, token)
if _not_modified(request, etag):
run.set(run_type="cache", not_modified=True)
return Response(status_code=304, headers={"ETag": etag})
body = store.get_payload("forecast", cell["id"], key, token)
if body is not None:
run.set(run_type="cache", history_source="store")
return _json_response(body, etag)
with run.phase("reverse_geocode"):
place = climate.reverse_geocode(cell["center_lat"], cell["center_lon"])
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21) * Compress API responses and revalidate static assets instead of re-downloading - Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x. - Serve pages/assets with Cache-Control: no-cache instead of no-store, so browsers revalidate via the ETag/Last-Modified that FileResponse and StaticFiles already emit. Unchanged assets now cost an empty 304 rather than a full transfer on every page navigation, while deploys still show up immediately. * Persist derived responses in SQLite so grading is computed once per cell, not per request New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet records — finished grade/calendar/day/forecast payloads and reverse-geocode labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms) becomes a ~5ms database read that survives restarts and is shared across views. Raw parquet stays the source of truth; the store is a pure accelerator (every reader falls back to recomputing on a miss, and deleting the db is a safe reset). Freshness is token-driven, not clock-driven: each cached payload is validated by a token encoding what it was computed from (payload schema version, the archive record's end date, the recent-fetch stamp). The existing freshness drivers are untouched — get_history still tops up the tail hourly and get_recent_forecast still refetches hourly — and tokens are derived from what they return, so cached payloads expire exactly when their inputs change. The same tokens double as weak ETags: If-None-Match answers with an empty 304 without touching the payload. - backend/store.py: derived-payload + revgeo tables, thread-local WAL conns, every helper fail-soft. - app.py: endpoints split into pure payload builders + HTTP/caching shells; the in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store). - climate.py: revgeo persisted through the store; recent_stamp() and load_cached_history() (no-network read) helpers. - backend/migrate.py + make migrate: idempotent, resumable backfill of the store from existing parquet caches (default calendar span + latest-day detail + revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather. - grid.from_id(): rebuild a cell from its cache filename (migrate tooling). * Add /api/v2/cell: one bundle carrying every view's payload GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months) and day (today) payloads in one response, each the exact payload its per-view endpoint returns — built by the same builders and cached under the same derived-store keys/tokens — paired with the etag that endpoint would emit. The frontend can warm all views with a single request, seed its per-view cache from the slices, and later revalidate each view individually with If-None-Match. The bundle's own etag combines the slices', so an unchanged bundle is an empty 304. prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard guarantee: it never spends weather-API quota. A cell with no cached archive answers 204, and only the history-derived slices (calendar + latest-day detail) are built. At most one Nominatim lookup for a never-labeled cell. * Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota, which multi-year calendar payloads regularly blew through) to IndexedDB, with an in-memory map in front. Entries carry the server's ETag, so anything stale revalidates conditionally — unchanged data costs an empty 304 and a re-stamp, never a re-transfer. On network failure the stale copy is served over an error. getJSON gains an optional onUpdate callback opting into stale-while-revalidate: the three views (weekly, day, calendar) now render a cached copy immediately — spinners are delayed 150ms so warm loads never flash-blank — and repaint only if background revalidation finds changed data. New data shows up the moment it exists instead of waiting out a TTL. Cross-view prefetch collapses from one request per view to a single /api/v2/cell bundle, whose slices (exact per-view payloads + their etags) are seeded under the URLs each view actually requests; the bundle call itself is conditional via a remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one possible Nominatim lookup each), so tapping nearby lands on already-graded data. Legacy tg:* storage entries are cleared once; cache entries untouched for two weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
payload = _build_forecast(cell, days, history, fc, today, place, run)
run.set(run_type="partial")
return _json_response(store.put_payload("forecast", cell["id"], key, token, payload), etag)
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21) * Compress API responses and revalidate static assets instead of re-downloading - Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x. - Serve pages/assets with Cache-Control: no-cache instead of no-store, so browsers revalidate via the ETag/Last-Modified that FileResponse and StaticFiles already emit. Unchanged assets now cost an empty 304 rather than a full transfer on every page navigation, while deploys still show up immediately. * Persist derived responses in SQLite so grading is computed once per cell, not per request New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet records — finished grade/calendar/day/forecast payloads and reverse-geocode labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms) becomes a ~5ms database read that survives restarts and is shared across views. Raw parquet stays the source of truth; the store is a pure accelerator (every reader falls back to recomputing on a miss, and deleting the db is a safe reset). Freshness is token-driven, not clock-driven: each cached payload is validated by a token encoding what it was computed from (payload schema version, the archive record's end date, the recent-fetch stamp). The existing freshness drivers are untouched — get_history still tops up the tail hourly and get_recent_forecast still refetches hourly — and tokens are derived from what they return, so cached payloads expire exactly when their inputs change. The same tokens double as weak ETags: If-None-Match answers with an empty 304 without touching the payload. - backend/store.py: derived-payload + revgeo tables, thread-local WAL conns, every helper fail-soft. - app.py: endpoints split into pure payload builders + HTTP/caching shells; the in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store). - climate.py: revgeo persisted through the store; recent_stamp() and load_cached_history() (no-network read) helpers. - backend/migrate.py + make migrate: idempotent, resumable backfill of the store from existing parquet caches (default calendar span + latest-day detail + revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather. - grid.from_id(): rebuild a cell from its cache filename (migrate tooling). * Add /api/v2/cell: one bundle carrying every view's payload GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months) and day (today) payloads in one response, each the exact payload its per-view endpoint returns — built by the same builders and cached under the same derived-store keys/tokens — paired with the etag that endpoint would emit. The frontend can warm all views with a single request, seed its per-view cache from the slices, and later revalidate each view individually with If-None-Match. The bundle's own etag combines the slices', so an unchanged bundle is an empty 304. prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard guarantee: it never spends weather-API quota. A cell with no cached archive answers 204, and only the history-derived slices (calendar + latest-day detail) are built. At most one Nominatim lookup for a never-labeled cell. * Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota, which multi-year calendar payloads regularly blew through) to IndexedDB, with an in-memory map in front. Entries carry the server's ETag, so anything stale revalidates conditionally — unchanged data costs an empty 304 and a re-stamp, never a re-transfer. On network failure the stale copy is served over an error. getJSON gains an optional onUpdate callback opting into stale-while-revalidate: the three views (weekly, day, calendar) now render a cached copy immediately — spinners are delayed 150ms so warm loads never flash-blank — and repaint only if background revalidation finds changed data. New data shows up the moment it exists instead of waiting out a TTL. Cross-view prefetch collapses from one request per view to a single /api/v2/cell bundle, whose slices (exact per-view payloads + their etags) are seeded under the URLs each view actually requests; the bundle call itself is conditional via a remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one possible Nominatim lookup each), so tapping nearby lands on already-graded data. Legacy tg:* storage entries are cleared once; cache entries untouched for two weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
def api_cell(
request: Request,
lat: float = Query(..., ge=-90, le=90),
lon: float = Query(..., ge=-180, le=180),
prefetch: int = Query(0, ge=0, le=1,
description="1 = warm-only: never fetch weather upstream; 204 for a cold cell"),
):
"""One bundle carrying every view's payload for a cell, so the frontend warms
all views with a single request instead of four.
Each slice is the EXACT payload its per-view endpoint returns built by the
same builders and cached under the same derived-store keys/tokens paired
with the etag that endpoint would emit. The client seeds its per-view cache
from the slices and later revalidates each view individually with
If-None-Match, so the bundle and the per-view endpoints stay one cache.
prefetch=1 is the neighbor-warming mode with a hard guarantee: it never
spends weather-API quota. A cell with no cached archive answers 204 (no
body), and only the history-derived slices (calendar + latest-day detail)
are built grading recent/forecast days needs the hourly upstream bundle.
Reverse geocoding makes at most one Nominatim call for a never-labeled cell;
the client staggers neighbor prefetches to respect that service. New in API v2.
"""
if not grid.in_north_america(lat, lon):
raise HTTPException(
status_code=400,
detail="Location is outside the supported US + Canada region.",
)
cell = grid.snap(lat, lon)
today = pd.Timestamp(datetime.date.today())
with audit.RunAudit(endpoint="cell", lat=round(lat, 4), lon=round(lon, 4),
cell_id=cell["id"], prefetch=bool(prefetch)) as run:
recent = None
if prefetch:
history = climate.load_cached_history(cell)
if history is None or history.empty:
run.set(run_type="cold-skip")
return Response(status_code=204)
cache_meta = {"cached": True}
else:
try:
with run.phase("history"):
history, cache_meta = climate.get_history(cell)
with run.phase("recent"):
recent = climate.get_recent_forecast(cell)
except Exception as e: # noqa: BLE001
raise _weather_fetch_error(e)
if history.empty:
raise HTTPException(status_code=404, detail="No historical data for this cell.")
cid = cell["id"]
hist_end = _hist_end(history)
last = pd.Timestamp(history["date"].max()).normalize()
with run.phase("reverse_geocode"):
place = climate.reverse_geocode(cell["center_lat"], cell["center_lon"])
def slice_for(kind: str, key: str, token: str, build) -> dict:
"""The endpoint's derived row (or build + store it), paired with the
etag that endpoint would emit for the same request."""
etag = _etag_for(kind, cid, key, token)
data = store.get_json(kind, cid, key, token)
if data is None:
data = build()
store.put_payload(kind, cid, key, token, data)
return {"etag": etag, "data": data}
slices = {}
hist_token = f"{PAYLOAD_VER}:{hist_end}"
# Calendar: the default last-24-months span — what the calendar view (and
# the cross-view prefetch) requests first.
start_ts, end_ts = _cal_span(history, None, None, 24)
cal_key = f"{start_ts.date().isoformat()}:{end_ts.date().isoformat()}:24"
slices["calendar"] = slice_for(
"calendar", cal_key, hist_token,
lambda: _build_calendar(cell, history, start_ts, end_ts, 24, place, run))
if prefetch:
# Latest archived day: its observation comes from history alone, so
# it's buildable without the (skipped) recent bundle.
slices["day"] = slice_for(
"day", last.date().isoformat(), hist_token,
lambda: _build_day(cell, history, last, place, run))
else:
rf_token = f"{PAYLOAD_VER}:{hist_end}:{climate.recent_stamp(cid)}"
slices["grade"] = slice_for(
"grade", f"{today.date().isoformat()}:14", rf_token,
lambda: _build_grade(cell, today, 14, history, recent, cache_meta, place, run))
slices["forecast"] = slice_for(
"forecast", f"{today.date().isoformat()}:7", rf_token,
lambda: _build_forecast(cell, 7, history, recent, today, place, run))
day_token = hist_token
if today > last:
day_token += f":h{int(time.time() // 3600)}" # obs from the hourly recent bundle
slices["day"] = slice_for(
"day", today.date().isoformat(), day_token,
lambda: _build_day(cell, history, today, place, run, recent=recent))
# The bundle's identity is the combination of its slices' identities.
etag = _etag_for("cell", cid, "bundle" + (":p" if prefetch else ""),
"|".join(s["etag"] for s in slices.values()))
if _not_modified(request, etag):
run.set(run_type="cache", not_modified=True)
return Response(status_code=304, headers={"ETag": etag})
run.set(run_type="bundle", slices=sorted(slices))
payload = {
"api_version": "v2",
"cell": cell,
"place": place,
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21) * Compress API responses and revalidate static assets instead of re-downloading - Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x. - Serve pages/assets with Cache-Control: no-cache instead of no-store, so browsers revalidate via the ETag/Last-Modified that FileResponse and StaticFiles already emit. Unchanged assets now cost an empty 304 rather than a full transfer on every page navigation, while deploys still show up immediately. * Persist derived responses in SQLite so grading is computed once per cell, not per request New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet records — finished grade/calendar/day/forecast payloads and reverse-geocode labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms) becomes a ~5ms database read that survives restarts and is shared across views. Raw parquet stays the source of truth; the store is a pure accelerator (every reader falls back to recomputing on a miss, and deleting the db is a safe reset). Freshness is token-driven, not clock-driven: each cached payload is validated by a token encoding what it was computed from (payload schema version, the archive record's end date, the recent-fetch stamp). The existing freshness drivers are untouched — get_history still tops up the tail hourly and get_recent_forecast still refetches hourly — and tokens are derived from what they return, so cached payloads expire exactly when their inputs change. The same tokens double as weak ETags: If-None-Match answers with an empty 304 without touching the payload. - backend/store.py: derived-payload + revgeo tables, thread-local WAL conns, every helper fail-soft. - app.py: endpoints split into pure payload builders + HTTP/caching shells; the in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store). - climate.py: revgeo persisted through the store; recent_stamp() and load_cached_history() (no-network read) helpers. - backend/migrate.py + make migrate: idempotent, resumable backfill of the store from existing parquet caches (default calendar span + latest-day detail + revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather. - grid.from_id(): rebuild a cell from its cache filename (migrate tooling). * Add /api/v2/cell: one bundle carrying every view's payload GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months) and day (today) payloads in one response, each the exact payload its per-view endpoint returns — built by the same builders and cached under the same derived-store keys/tokens — paired with the etag that endpoint would emit. The frontend can warm all views with a single request, seed its per-view cache from the slices, and later revalidate each view individually with If-None-Match. The bundle's own etag combines the slices', so an unchanged bundle is an empty 304. prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard guarantee: it never spends weather-API quota. A cell with no cached archive answers 204, and only the history-derived slices (calendar + latest-day detail) are built. At most one Nominatim lookup for a never-labeled cell. * Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota, which multi-year calendar payloads regularly blew through) to IndexedDB, with an in-memory map in front. Entries carry the server's ETag, so anything stale revalidates conditionally — unchanged data costs an empty 304 and a re-stamp, never a re-transfer. On network failure the stale copy is served over an error. getJSON gains an optional onUpdate callback opting into stale-while-revalidate: the three views (weekly, day, calendar) now render a cached copy immediately — spinners are delayed 150ms so warm loads never flash-blank — and repaint only if background revalidation finds changed data. New data shows up the moment it exists instead of waiting out a TTL. Cross-view prefetch collapses from one request per view to a single /api/v2/cell bundle, whose slices (exact per-view payloads + their etags) are seeded under the URLs each view actually requests; the bundle call itself is conditional via a remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one possible Nominatim lookup each), so tapping nearby lands on already-graded data. Legacy tg:* storage entries are cleared once; cache entries untouched for two weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
"today": today.date().isoformat(),
"slices": slices,
}
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21) * Compress API responses and revalidate static assets instead of re-downloading - Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x. - Serve pages/assets with Cache-Control: no-cache instead of no-store, so browsers revalidate via the ETag/Last-Modified that FileResponse and StaticFiles already emit. Unchanged assets now cost an empty 304 rather than a full transfer on every page navigation, while deploys still show up immediately. * Persist derived responses in SQLite so grading is computed once per cell, not per request New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet records — finished grade/calendar/day/forecast payloads and reverse-geocode labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms) becomes a ~5ms database read that survives restarts and is shared across views. Raw parquet stays the source of truth; the store is a pure accelerator (every reader falls back to recomputing on a miss, and deleting the db is a safe reset). Freshness is token-driven, not clock-driven: each cached payload is validated by a token encoding what it was computed from (payload schema version, the archive record's end date, the recent-fetch stamp). The existing freshness drivers are untouched — get_history still tops up the tail hourly and get_recent_forecast still refetches hourly — and tokens are derived from what they return, so cached payloads expire exactly when their inputs change. The same tokens double as weak ETags: If-None-Match answers with an empty 304 without touching the payload. - backend/store.py: derived-payload + revgeo tables, thread-local WAL conns, every helper fail-soft. - app.py: endpoints split into pure payload builders + HTTP/caching shells; the in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store). - climate.py: revgeo persisted through the store; recent_stamp() and load_cached_history() (no-network read) helpers. - backend/migrate.py + make migrate: idempotent, resumable backfill of the store from existing parquet caches (default calendar span + latest-day detail + revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather. - grid.from_id(): rebuild a cell from its cache filename (migrate tooling). * Add /api/v2/cell: one bundle carrying every view's payload GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months) and day (today) payloads in one response, each the exact payload its per-view endpoint returns — built by the same builders and cached under the same derived-store keys/tokens — paired with the etag that endpoint would emit. The frontend can warm all views with a single request, seed its per-view cache from the slices, and later revalidate each view individually with If-None-Match. The bundle's own etag combines the slices', so an unchanged bundle is an empty 304. prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard guarantee: it never spends weather-API quota. A cell with no cached archive answers 204, and only the history-derived slices (calendar + latest-day detail) are built. At most one Nominatim lookup for a never-labeled cell. * Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota, which multi-year calendar payloads regularly blew through) to IndexedDB, with an in-memory map in front. Entries carry the server's ETag, so anything stale revalidates conditionally — unchanged data costs an empty 304 and a re-stamp, never a re-transfer. On network failure the stale copy is served over an error. getJSON gains an optional onUpdate callback opting into stale-while-revalidate: the three views (weekly, day, calendar) now render a cached copy immediately — spinners are delayed 150ms so warm loads never flash-blank — and repaint only if background revalidation finds changed data. New data shows up the moment it exists instead of waiting out a TTL. Cross-view prefetch collapses from one request per view to a single /api/v2/cell bundle, whose slices (exact per-view payloads + their etags) are seeded under the URLs each view actually requests; the bundle call itself is conditional via a remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one possible Nominatim lookup each), so tapping nearby lands on already-graded data. Legacy tg:* storage entries are cleared once; cache entries untouched for two weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
# Not stored as its own derived row — the slices already are.
return _json_response(_encode(payload), etag)
# --- API versioning --------------------------------------------------------
# Non-backward-compatible additions land in a new version; every version stays
# mounted and served simultaneously. v1 is the original contract (grade/geocode);
# v2 adds the calendar. The unversioned /api/* paths are kept as aliases of v1 so
# existing clients keep working.
v1 = APIRouter(tags=["v1"])
v1.add_api_route("/geocode", api_geocode, methods=["GET"])
v1.add_api_route("/grade", api_grade, methods=["GET"])
v2 = APIRouter(tags=["v2"])
v2.add_api_route("/geocode", api_geocode, methods=["GET"])
v2.add_api_route("/grade", api_grade, methods=["GET"])
v2.add_api_route("/calendar", api_calendar, methods=["GET"])
v2.add_api_route("/day", api_day, methods=["GET"])
v2.add_api_route("/forecast", api_forecast, methods=["GET"])
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21) * Compress API responses and revalidate static assets instead of re-downloading - Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x. - Serve pages/assets with Cache-Control: no-cache instead of no-store, so browsers revalidate via the ETag/Last-Modified that FileResponse and StaticFiles already emit. Unchanged assets now cost an empty 304 rather than a full transfer on every page navigation, while deploys still show up immediately. * Persist derived responses in SQLite so grading is computed once per cell, not per request New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet records — finished grade/calendar/day/forecast payloads and reverse-geocode labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms) becomes a ~5ms database read that survives restarts and is shared across views. Raw parquet stays the source of truth; the store is a pure accelerator (every reader falls back to recomputing on a miss, and deleting the db is a safe reset). Freshness is token-driven, not clock-driven: each cached payload is validated by a token encoding what it was computed from (payload schema version, the archive record's end date, the recent-fetch stamp). The existing freshness drivers are untouched — get_history still tops up the tail hourly and get_recent_forecast still refetches hourly — and tokens are derived from what they return, so cached payloads expire exactly when their inputs change. The same tokens double as weak ETags: If-None-Match answers with an empty 304 without touching the payload. - backend/store.py: derived-payload + revgeo tables, thread-local WAL conns, every helper fail-soft. - app.py: endpoints split into pure payload builders + HTTP/caching shells; the in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store). - climate.py: revgeo persisted through the store; recent_stamp() and load_cached_history() (no-network read) helpers. - backend/migrate.py + make migrate: idempotent, resumable backfill of the store from existing parquet caches (default calendar span + latest-day detail + revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather. - grid.from_id(): rebuild a cell from its cache filename (migrate tooling). * Add /api/v2/cell: one bundle carrying every view's payload GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months) and day (today) payloads in one response, each the exact payload its per-view endpoint returns — built by the same builders and cached under the same derived-store keys/tokens — paired with the etag that endpoint would emit. The frontend can warm all views with a single request, seed its per-view cache from the slices, and later revalidate each view individually with If-None-Match. The bundle's own etag combines the slices', so an unchanged bundle is an empty 304. prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard guarantee: it never spends weather-API quota. A cell with no cached archive answers 204, and only the history-derived slices (calendar + latest-day detail) are built. At most one Nominatim lookup for a never-labeled cell. * Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota, which multi-year calendar payloads regularly blew through) to IndexedDB, with an in-memory map in front. Entries carry the server's ETag, so anything stale revalidates conditionally — unchanged data costs an empty 304 and a re-stamp, never a re-transfer. On network failure the stale copy is served over an error. getJSON gains an optional onUpdate callback opting into stale-while-revalidate: the three views (weekly, day, calendar) now render a cached copy immediately — spinners are delayed 150ms so warm loads never flash-blank — and repaint only if background revalidation finds changed data. New data shows up the moment it exists instead of waiting out a TTL. Cross-view prefetch collapses from one request per view to a single /api/v2/cell bundle, whose slices (exact per-view payloads + their etags) are seeded under the URLs each view actually requests; the bundle call itself is conditional via a remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one possible Nominatim lookup each), so tapping nearby lands on already-graded data. Legacy tg:* storage entries are cleared once; cache entries untouched for two weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
v2.add_api_route("/cell", api_cell, methods=["GET"])
app.include_router(v1, prefix=f"{BASE}/api") # legacy unversioned == v1 (backward compatible)
app.include_router(v1, prefix=f"{BASE}/api/v1")
app.include_router(v2, prefix=f"{BASE}/api/v2")
# --- static frontend (served under BASE) -----------------------------------
def _page(name):
return FileResponse(os.path.join(FRONTEND_DIR, name))
# The un-slashed base redirects to BASE/ so the frontend's relative asset URLs
# resolve correctly. The app stays scoped to BASE and never claims "/", leaving
# the domain root free for another app (e.g. a portfolio) to own via the proxy.
app.add_api_route(BASE, lambda: RedirectResponse(url=f"{BASE}/"), methods=["GET"], include_in_schema=False)
app.add_api_route(f"{BASE}/", lambda: _page("index.html"), methods=["GET"], include_in_schema=False)
app.add_api_route(f"{BASE}/calendar", lambda: _page("calendar.html"), methods=["GET"], include_in_schema=False)
app.add_api_route(f"{BASE}/day", lambda: _page("day.html"), methods=["GET"], include_in_schema=False)
app.add_api_route(f"{BASE}/compare", lambda: _page("compare.html"), methods=["GET"], include_in_schema=False)
app.add_api_route(f"{BASE}/legend", lambda: _page("legend.html"), methods=["GET"], include_in_schema=False)
# Everything else under BASE (app.js, style.css, nav.js, …) is a static asset.
# Registered last so the explicit page routes above win.
app.mount(BASE, StaticFiles(directory=FRONTEND_DIR), name="static")