thermograph/backend/data/grid.py

93 lines
3.6 KiB
Python
Raw Permalink Normal View History

"""Snap an arbitrary lat/lon to a stable ~4-square-mile grid cell.
The grid is defined by fixed latitude rows (~2 miles tall). Within each row the
longitude step is scaled by cos(latitude) so cells stay roughly square (~4 sq mi)
at every latitude instead of getting skinny toward the poles. Cell ids are
deterministic, so the same physical location always maps to the same cache file.
Worldwide coverage: grade any point on Earth (#32) Remove the US+Canada bounding box so every endpoint accepts any lat/lon. The grading pipeline was already global-ready (ERA5 archive, timezone=auto, day-of-year climatology), so opening it up is mostly deleting the guard — plus the edge cases that only exist once the whole globe is in play: - grid.py: snap() wraps longitude into [-180, 180) and clamps latitude, and cell centers are normalized so the polar row and the cells straddling the antimeridian always report valid coordinates to the weather/geocoding APIs. snap() and from_id() now share one _cell() builder, making id round-trips exact by construction (verified with a 300k-point global sweep). - nav.js: neighbor-cell prefetch skips rows past the poles and wraps longitudes across the dateline instead of sending out-of-range queries. - Nominatim reverse geocoding requests accept-language=en so place labels render in one script worldwide (matching the forward geocoder). - mappicker: search suggestions are no longer filtered to US/CA, the placeholder and default map view are worldwide. - calendar: season filter labels flip for southern-hemisphere locations (Dec-Feb shows as Summer); the underlying month groups are unchanged, so saved filter selections keep meaning the same months. Verified end-to-end on a scratch server: Tokyo and Sydney grade with real labels, a Fiji cell on the antimeridian's east edge builds and serves warm hits from the derived store, and prefetch=1 on a cold cell still answers 204 without spending weather-API quota.
2026-07-11 15:02:28 +00:00
Coverage is worldwide: any lat in [-90, 90] and any longitude (wrapped into
[-180, 180)) maps to a cell.
"""
import math
# 1 degree of latitude ~= 69 miles. ~2 miles -> ~0.029 deg gives a ~4 sq mi cell.
LAT_STEP = 1.0 / 34.5 # ~= 0.02899 deg (~2.0 miles)
def _lon_step(center_lat: float) -> float:
"""Longitude degrees that span ~2 miles at the given latitude."""
c = math.cos(math.radians(center_lat))
c = max(c, 0.05) # clamp near the poles to avoid a blow-up
return LAT_STEP / c
Worldwide coverage: grade any point on Earth (#32) Remove the US+Canada bounding box so every endpoint accepts any lat/lon. The grading pipeline was already global-ready (ERA5 archive, timezone=auto, day-of-year climatology), so opening it up is mostly deleting the guard — plus the edge cases that only exist once the whole globe is in play: - grid.py: snap() wraps longitude into [-180, 180) and clamps latitude, and cell centers are normalized so the polar row and the cells straddling the antimeridian always report valid coordinates to the weather/geocoding APIs. snap() and from_id() now share one _cell() builder, making id round-trips exact by construction (verified with a 300k-point global sweep). - nav.js: neighbor-cell prefetch skips rows past the poles and wraps longitudes across the dateline instead of sending out-of-range queries. - Nominatim reverse geocoding requests accept-language=en so place labels render in one script worldwide (matching the forward geocoder). - mappicker: search suggestions are no longer filtered to US/CA, the placeholder and default map view are worldwide. - calendar: season filter labels flip for southern-hemisphere locations (Dec-Feb shows as Summer); the underlying month groups are unchanged, so saved filter selections keep meaning the same months. Verified end-to-end on a scratch server: Tokyo and Sydney grade with real labels, a Fiji cell on the antimeridian's east edge builds and serves warm hits from the derived store, and prefetch=1 on a cold cell still answers 204 without spending weather-API quota.
2026-07-11 15:02:28 +00:00
def _cell(i: int, j: int) -> dict:
"""Build the cell dict for grid indices (i, j). Shared by snap()/from_id()
so an id always rebuilds to the exact same cell."""
center_lat = (i + 0.5) * LAT_STEP
lon_step = _lon_step(center_lat)
center_lon = (j + 0.5) * lon_step
Worldwide coverage: grade any point on Earth (#32) Remove the US+Canada bounding box so every endpoint accepts any lat/lon. The grading pipeline was already global-ready (ERA5 archive, timezone=auto, day-of-year climatology), so opening it up is mostly deleting the guard — plus the edge cases that only exist once the whole globe is in play: - grid.py: snap() wraps longitude into [-180, 180) and clamps latitude, and cell centers are normalized so the polar row and the cells straddling the antimeridian always report valid coordinates to the weather/geocoding APIs. snap() and from_id() now share one _cell() builder, making id round-trips exact by construction (verified with a 300k-point global sweep). - nav.js: neighbor-cell prefetch skips rows past the poles and wraps longitudes across the dateline instead of sending out-of-range queries. - Nominatim reverse geocoding requests accept-language=en so place labels render in one script worldwide (matching the forward geocoder). - mappicker: search suggestions are no longer filtered to US/CA, the placeholder and default map view are worldwide. - calendar: season filter labels flip for southern-hemisphere locations (Dec-Feb shows as Summer); the underlying month groups are unchanged, so saved filter selections keep meaning the same months. Verified end-to-end on a scratch server: Tokyo and Sydney grade with real labels, a Fiji cell on the antimeridian's east edge builds and serves warm hits from the derived store, and prefetch=1 on a cold cell still answers 204 without spending weather-API quota.
2026-07-11 15:02:28 +00:00
# Keep the reported center a valid coordinate for the upstream weather and
# geocoding APIs: the topmost row's center overshoots the pole, and a row's
# outermost cells can have centers just past the antimeridian on either side.
center_lat = min(max(center_lat, -90.0), 90.0)
if center_lon > 180.0:
center_lon -= 360.0
elif center_lon < -180.0:
center_lon += 360.0
# Approximate cell dimensions in miles for display.
height_mi = LAT_STEP * 69.0
width_mi = lon_step * 69.0 * math.cos(math.radians(center_lat))
return {
"id": f"{i}_{j}",
"center_lat": round(center_lat, 5),
"center_lon": round(center_lon, 5),
"lat_step": LAT_STEP,
"lon_step": lon_step,
"bounds": {
"south": round(i * LAT_STEP, 5),
"north": round((i + 1) * LAT_STEP, 5),
"west": round(j * lon_step, 5),
"east": round((j + 1) * lon_step, 5),
},
"area_sq_mi": round(height_mi * width_mi, 2),
}
Worldwide coverage: grade any point on Earth (#32) Remove the US+Canada bounding box so every endpoint accepts any lat/lon. The grading pipeline was already global-ready (ERA5 archive, timezone=auto, day-of-year climatology), so opening it up is mostly deleting the guard — plus the edge cases that only exist once the whole globe is in play: - grid.py: snap() wraps longitude into [-180, 180) and clamps latitude, and cell centers are normalized so the polar row and the cells straddling the antimeridian always report valid coordinates to the weather/geocoding APIs. snap() and from_id() now share one _cell() builder, making id round-trips exact by construction (verified with a 300k-point global sweep). - nav.js: neighbor-cell prefetch skips rows past the poles and wraps longitudes across the dateline instead of sending out-of-range queries. - Nominatim reverse geocoding requests accept-language=en so place labels render in one script worldwide (matching the forward geocoder). - mappicker: search suggestions are no longer filtered to US/CA, the placeholder and default map view are worldwide. - calendar: season filter labels flip for southern-hemisphere locations (Dec-Feb shows as Summer); the underlying month groups are unchanged, so saved filter selections keep meaning the same months. Verified end-to-end on a scratch server: Tokyo and Sydney grade with real labels, a Fiji cell on the antimeridian's east edge builds and serves warm hits from the derived store, and prefetch=1 on a cold cell still answers 204 without spending weather-API quota.
2026-07-11 15:02:28 +00:00
def snap(lat: float, lon: float) -> dict:
"""Return the grid cell (id + center + span) containing (lat, lon)."""
lat = min(max(lat, -90.0), 90.0)
lon = ((lon + 180.0) % 360.0) - 180.0 # wrap into [-180, 180)
i = math.floor(lat / LAT_STEP)
j = math.floor(lon / _lon_step((i + 0.5) * LAT_STEP))
return _cell(i, j)
def neighbors(cell: dict) -> list[dict]:
"""The up-to-8 cells surrounding one cell. Steps one cell width from the
center and re-snaps, so adjacent rows whose longitude step differs
resolve to whichever cell actually contains the stepped point. Rows past
the poles are skipped; longitude wraps across the antimeridian (both via
snap). Deduplicated (near the poles steps can collapse onto one cell)."""
out: dict[str, dict] = {}
for di in (-1, 0, 1):
for dj in (-1, 0, 1):
if not di and not dj:
continue
lat = cell["center_lat"] + di * LAT_STEP
if abs(lat) > 90.0:
continue
n = snap(lat, cell["center_lon"] + dj * cell["lon_step"])
if n["id"] != cell["id"]:
out[n["id"]] = n
return list(out.values())
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21) * Compress API responses and revalidate static assets instead of re-downloading - Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x. - Serve pages/assets with Cache-Control: no-cache instead of no-store, so browsers revalidate via the ETag/Last-Modified that FileResponse and StaticFiles already emit. Unchanged assets now cost an empty 304 rather than a full transfer on every page navigation, while deploys still show up immediately. * Persist derived responses in SQLite so grading is computed once per cell, not per request New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet records — finished grade/calendar/day/forecast payloads and reverse-geocode labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms) becomes a ~5ms database read that survives restarts and is shared across views. Raw parquet stays the source of truth; the store is a pure accelerator (every reader falls back to recomputing on a miss, and deleting the db is a safe reset). Freshness is token-driven, not clock-driven: each cached payload is validated by a token encoding what it was computed from (payload schema version, the archive record's end date, the recent-fetch stamp). The existing freshness drivers are untouched — get_history still tops up the tail hourly and get_recent_forecast still refetches hourly — and tokens are derived from what they return, so cached payloads expire exactly when their inputs change. The same tokens double as weak ETags: If-None-Match answers with an empty 304 without touching the payload. - backend/store.py: derived-payload + revgeo tables, thread-local WAL conns, every helper fail-soft. - app.py: endpoints split into pure payload builders + HTTP/caching shells; the in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store). - climate.py: revgeo persisted through the store; recent_stamp() and load_cached_history() (no-network read) helpers. - backend/migrate.py + make migrate: idempotent, resumable backfill of the store from existing parquet caches (default calendar span + latest-day detail + revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather. - grid.from_id(): rebuild a cell from its cache filename (migrate tooling). * Add /api/v2/cell: one bundle carrying every view's payload GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months) and day (today) payloads in one response, each the exact payload its per-view endpoint returns — built by the same builders and cached under the same derived-store keys/tokens — paired with the etag that endpoint would emit. The frontend can warm all views with a single request, seed its per-view cache from the slices, and later revalidate each view individually with If-None-Match. The bundle's own etag combines the slices', so an unchanged bundle is an empty 304. prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard guarantee: it never spends weather-API quota. A cell with no cached archive answers 204, and only the history-derived slices (calendar + latest-day detail) are built. At most one Nominatim lookup for a never-labeled cell. * Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota, which multi-year calendar payloads regularly blew through) to IndexedDB, with an in-memory map in front. Entries carry the server's ETag, so anything stale revalidates conditionally — unchanged data costs an empty 304 and a re-stamp, never a re-transfer. On network failure the stale copy is served over an error. getJSON gains an optional onUpdate callback opting into stale-while-revalidate: the three views (weekly, day, calendar) now render a cached copy immediately — spinners are delayed 150ms so warm loads never flash-blank — and repaint only if background revalidation finds changed data. New data shows up the moment it exists instead of waiting out a TTL. Cross-view prefetch collapses from one request per view to a single /api/v2/cell bundle, whose slices (exact per-view payloads + their etags) are seeded under the URLs each view actually requests; the bundle call itself is conditional via a remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one possible Nominatim lookup each), so tapping nearby lands on already-graded data. Legacy tg:* storage entries are cleared once; cache entries untouched for two weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
def from_id(cell_id: str) -> dict:
"""Rebuild the full cell dict from a cache id ("i_j") — the inverse of snap().
Lets offline tooling (the migrate script) recover a cell from its parquet
filename alone. Raises ValueError on a malformed id."""
i, j = (int(p) for p in cell_id.split("_"))
Worldwide coverage: grade any point on Earth (#32) Remove the US+Canada bounding box so every endpoint accepts any lat/lon. The grading pipeline was already global-ready (ERA5 archive, timezone=auto, day-of-year climatology), so opening it up is mostly deleting the guard — plus the edge cases that only exist once the whole globe is in play: - grid.py: snap() wraps longitude into [-180, 180) and clamps latitude, and cell centers are normalized so the polar row and the cells straddling the antimeridian always report valid coordinates to the weather/geocoding APIs. snap() and from_id() now share one _cell() builder, making id round-trips exact by construction (verified with a 300k-point global sweep). - nav.js: neighbor-cell prefetch skips rows past the poles and wraps longitudes across the dateline instead of sending out-of-range queries. - Nominatim reverse geocoding requests accept-language=en so place labels render in one script worldwide (matching the forward geocoder). - mappicker: search suggestions are no longer filtered to US/CA, the placeholder and default map view are worldwide. - calendar: season filter labels flip for southern-hemisphere locations (Dec-Feb shows as Summer); the underlying month groups are unchanged, so saved filter selections keep meaning the same months. Verified end-to-end on a scratch server: Tokyo and Sydney grade with real labels, a Fiji cell on the antimeridian's east edge builds and serves warm hits from the derived store, and prefetch=1 on a cold cell still answers 204 without spending weather-API quota.
2026-07-11 15:02:28 +00:00
return _cell(i, j)