Emi Griffith
e01d2e0eb8
* Compress API responses and revalidate static assets instead of re-downloading - Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x. - Serve pages/assets with Cache-Control: no-cache instead of no-store, so browsers revalidate via the ETag/Last-Modified that FileResponse and StaticFiles already emit. Unchanged assets now cost an empty 304 rather than a full transfer on every page navigation, while deploys still show up immediately. * Persist derived responses in SQLite so grading is computed once per cell, not per request New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet records — finished grade/calendar/day/forecast payloads and reverse-geocode labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms) becomes a ~5ms database read that survives restarts and is shared across views. Raw parquet stays the source of truth; the store is a pure accelerator (every reader falls back to recomputing on a miss, and deleting the db is a safe reset). Freshness is token-driven, not clock-driven: each cached payload is validated by a token encoding what it was computed from (payload schema version, the archive record's end date, the recent-fetch stamp). The existing freshness drivers are untouched — get_history still tops up the tail hourly and get_recent_forecast still refetches hourly — and tokens are derived from what they return, so cached payloads expire exactly when their inputs change. The same tokens double as weak ETags: If-None-Match answers with an empty 304 without touching the payload. - backend/store.py: derived-payload + revgeo tables, thread-local WAL conns, every helper fail-soft. - app.py: endpoints split into pure payload builders + HTTP/caching shells; the in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store). - climate.py: revgeo persisted through the store; recent_stamp() and load_cached_history() (no-network read) helpers. - backend/migrate.py + make migrate: idempotent, resumable backfill of the store from existing parquet caches (default calendar span + latest-day detail + revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather. - grid.from_id(): rebuild a cell from its cache filename (migrate tooling). * Add /api/v2/cell: one bundle carrying every view's payload GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months) and day (today) payloads in one response, each the exact payload its per-view endpoint returns — built by the same builders and cached under the same derived-store keys/tokens — paired with the etag that endpoint would emit. The frontend can warm all views with a single request, seed its per-view cache from the slices, and later revalidate each view individually with If-None-Match. The bundle's own etag combines the slices', so an unchanged bundle is an empty 304. prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard guarantee: it never spends weather-API quota. A cell with no cached archive answers 204, and only the history-derived slices (calendar + latest-day detail) are built. At most one Nominatim lookup for a never-labeled cell. * Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota, which multi-year calendar payloads regularly blew through) to IndexedDB, with an in-memory map in front. Entries carry the server's ETag, so anything stale revalidates conditionally — unchanged data costs an empty 304 and a re-stamp, never a re-transfer. On network failure the stale copy is served over an error. getJSON gains an optional onUpdate callback opting into stale-while-revalidate: the three views (weekly, day, calendar) now render a cached copy immediately — spinners are delayed 150ms so warm loads never flash-blank — and repaint only if background revalidation finds changed data. New data shows up the moment it exists instead of waiting out a TTL. Cross-view prefetch collapses from one request per view to a single /api/v2/cell bundle, whose slices (exact per-view payloads + their etags) are seeded under the URLs each view actually requests; the bundle call itself is conditional via a remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one possible Nominatim lookup each), so tapping nearby lands on already-graded data. Legacy tg:* storage entries are cleared once; cache entries untouched for two weeks are pruned on page load.
91 lines
3.7 KiB
Python
91 lines
3.7 KiB
Python
"""Backfill the SQLite derived store from the existing parquet cell caches.
|
|
|
|
For every cell with a cached archive record, materialize the payloads that are
|
|
derivable offline — the default 2-year calendar and the latest-day detail — plus
|
|
its reverse-geocode label, so a server restarted (or freshly deployed) onto an
|
|
existing parquet cache serves those views from the database immediately instead
|
|
of regrading each cell on first touch.
|
|
|
|
Idempotent and resumable: cells whose derived rows already match the current
|
|
validity token are skipped, so re-running after an interruption (or after new
|
|
archive days arrive) only does the missing work. Never fetches weather data —
|
|
history comes strictly from the parquet cache (cells without one are skipped and
|
|
will materialize lazily on first request, as always). Reverse geocoding makes at
|
|
most one Nominatim call per unlabeled cell, throttled to ~1/s per the service's
|
|
policy. Safe to run alongside a live server: writes go through the same WAL
|
|
store the server reads, and readers fall back to recomputing on any miss.
|
|
|
|
Run via `make migrate`.
|
|
"""
|
|
import os
|
|
import sys
|
|
import time
|
|
|
|
import pandas as pd
|
|
|
|
import climate
|
|
import grid
|
|
import store
|
|
from app import PAYLOAD_VER, _build_calendar, _build_day, _cal_span, _hist_end
|
|
|
|
|
|
def migrate() -> int:
|
|
if not os.path.isdir(climate.CACHE_DIR):
|
|
print("No parquet cache directory yet — nothing to migrate.")
|
|
return 0
|
|
cells = sorted(
|
|
f[: -len(".parquet")]
|
|
for f in os.listdir(climate.CACHE_DIR)
|
|
if f.endswith(".parquet") and not f.endswith("_rf.parquet")
|
|
)
|
|
built = current = skipped = 0
|
|
for cell_id in cells:
|
|
try:
|
|
cell = grid.from_id(cell_id)
|
|
except ValueError:
|
|
print(f" {cell_id}: unrecognized cache filename — skipped")
|
|
skipped += 1
|
|
continue
|
|
history = climate.load_cached_history(cell)
|
|
if history is None or history.empty:
|
|
# Pre-current-schema record; the server refetches it lazily on first
|
|
# touch (see climate.NEW_COLS) — nothing to materialize offline.
|
|
print(f" {cell_id}: no schema-complete record — skipped")
|
|
skipped += 1
|
|
continue
|
|
|
|
token = f"{PAYLOAD_VER}:{_hist_end(history)}"
|
|
start_ts, end_ts = _cal_span(history, None, None, 24)
|
|
cal_key = f"{start_ts.date().isoformat()}:{end_ts.date().isoformat()}:24"
|
|
last = pd.Timestamp(history["date"].max()).normalize()
|
|
day_key = last.date().isoformat()
|
|
|
|
have_cal = store.get_payload("calendar", cell_id, cal_key, token) is not None
|
|
have_day = store.get_payload("day", cell_id, day_key, token) is not None
|
|
if have_cal and have_day:
|
|
current += 1
|
|
continue
|
|
|
|
found, place = climate.reverse_geocode_cached(cell["center_lat"], cell["center_lon"])
|
|
if not found:
|
|
place = climate.reverse_geocode(cell["center_lat"], cell["center_lon"])
|
|
time.sleep(1.1) # Nominatim usage policy: at most ~1 request/second
|
|
|
|
if not have_cal:
|
|
store.put_payload("calendar", cell_id, cal_key, token,
|
|
_build_calendar(cell, history, start_ts, end_ts, 24, place))
|
|
if not have_day:
|
|
store.put_payload("day", cell_id, day_key, token,
|
|
_build_day(cell, history, last, place))
|
|
built += 1
|
|
print(f" {cell_id}: materialized ({place or 'no label'})")
|
|
|
|
s = store.stats()
|
|
print(f"{built} cell(s) materialized, {current} already current, {skipped} skipped"
|
|
f" · derived rows: {s['derived']} · revgeo: {s['revgeo']}"
|
|
f" · db: {s['bytes'] / 1e6:.1f} MB → {s['db_path']}")
|
|
return 0
|
|
|
|
|
|
if __name__ == "__main__":
|
|
sys.exit(migrate())
|