2026-07-11 00:29:47 +00:00
|
|
|
"""Snap an arbitrary lat/lon to a stable ~4-square-mile grid cell.
|
|
|
|
|
|
|
|
|
|
The grid is defined by fixed latitude rows (~2 miles tall). Within each row the
|
|
|
|
|
longitude step is scaled by cos(latitude) so cells stay roughly square (~4 sq mi)
|
|
|
|
|
at every latitude instead of getting skinny toward the poles. Cell ids are
|
|
|
|
|
deterministic, so the same physical location always maps to the same cache file.
|
Worldwide coverage: grade any point on Earth (#32)
Remove the US+Canada bounding box so every endpoint accepts any lat/lon.
The grading pipeline was already global-ready (ERA5 archive, timezone=auto,
day-of-year climatology), so opening it up is mostly deleting the guard —
plus the edge cases that only exist once the whole globe is in play:
- grid.py: snap() wraps longitude into [-180, 180) and clamps latitude, and
cell centers are normalized so the polar row and the cells straddling the
antimeridian always report valid coordinates to the weather/geocoding APIs.
snap() and from_id() now share one _cell() builder, making id round-trips
exact by construction (verified with a 300k-point global sweep).
- nav.js: neighbor-cell prefetch skips rows past the poles and wraps
longitudes across the dateline instead of sending out-of-range queries.
- Nominatim reverse geocoding requests accept-language=en so place labels
render in one script worldwide (matching the forward geocoder).
- mappicker: search suggestions are no longer filtered to US/CA, the
placeholder and default map view are worldwide.
- calendar: season filter labels flip for southern-hemisphere locations
(Dec-Feb shows as Summer); the underlying month groups are unchanged, so
saved filter selections keep meaning the same months.
Verified end-to-end on a scratch server: Tokyo and Sydney grade with real
labels, a Fiji cell on the antimeridian's east edge builds and serves warm
hits from the derived store, and prefetch=1 on a cold cell still answers
204 without spending weather-API quota.
2026-07-11 15:02:28 +00:00
|
|
|
Coverage is worldwide: any lat in [-90, 90] and any longitude (wrapped into
|
|
|
|
|
[-180, 180)) maps to a cell.
|
2026-07-11 00:29:47 +00:00
|
|
|
"""
|
|
|
|
|
import math
|
|
|
|
|
|
|
|
|
|
# 1 degree of latitude ~= 69 miles. ~2 miles -> ~0.029 deg gives a ~4 sq mi cell.
|
|
|
|
|
LAT_STEP = 1.0 / 34.5 # ~= 0.02899 deg (~2.0 miles)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def _lon_step(center_lat: float) -> float:
|
|
|
|
|
"""Longitude degrees that span ~2 miles at the given latitude."""
|
|
|
|
|
c = math.cos(math.radians(center_lat))
|
|
|
|
|
c = max(c, 0.05) # clamp near the poles to avoid a blow-up
|
|
|
|
|
return LAT_STEP / c
|
|
|
|
|
|
|
|
|
|
|
Worldwide coverage: grade any point on Earth (#32)
Remove the US+Canada bounding box so every endpoint accepts any lat/lon.
The grading pipeline was already global-ready (ERA5 archive, timezone=auto,
day-of-year climatology), so opening it up is mostly deleting the guard —
plus the edge cases that only exist once the whole globe is in play:
- grid.py: snap() wraps longitude into [-180, 180) and clamps latitude, and
cell centers are normalized so the polar row and the cells straddling the
antimeridian always report valid coordinates to the weather/geocoding APIs.
snap() and from_id() now share one _cell() builder, making id round-trips
exact by construction (verified with a 300k-point global sweep).
- nav.js: neighbor-cell prefetch skips rows past the poles and wraps
longitudes across the dateline instead of sending out-of-range queries.
- Nominatim reverse geocoding requests accept-language=en so place labels
render in one script worldwide (matching the forward geocoder).
- mappicker: search suggestions are no longer filtered to US/CA, the
placeholder and default map view are worldwide.
- calendar: season filter labels flip for southern-hemisphere locations
(Dec-Feb shows as Summer); the underlying month groups are unchanged, so
saved filter selections keep meaning the same months.
Verified end-to-end on a scratch server: Tokyo and Sydney grade with real
labels, a Fiji cell on the antimeridian's east edge builds and serves warm
hits from the derived store, and prefetch=1 on a cold cell still answers
204 without spending weather-API quota.
2026-07-11 15:02:28 +00:00
|
|
|
def _cell(i: int, j: int) -> dict:
|
|
|
|
|
"""Build the cell dict for grid indices (i, j). Shared by snap()/from_id()
|
|
|
|
|
so an id always rebuilds to the exact same cell."""
|
2026-07-11 00:29:47 +00:00
|
|
|
center_lat = (i + 0.5) * LAT_STEP
|
|
|
|
|
lon_step = _lon_step(center_lat)
|
|
|
|
|
center_lon = (j + 0.5) * lon_step
|
Worldwide coverage: grade any point on Earth (#32)
Remove the US+Canada bounding box so every endpoint accepts any lat/lon.
The grading pipeline was already global-ready (ERA5 archive, timezone=auto,
day-of-year climatology), so opening it up is mostly deleting the guard —
plus the edge cases that only exist once the whole globe is in play:
- grid.py: snap() wraps longitude into [-180, 180) and clamps latitude, and
cell centers are normalized so the polar row and the cells straddling the
antimeridian always report valid coordinates to the weather/geocoding APIs.
snap() and from_id() now share one _cell() builder, making id round-trips
exact by construction (verified with a 300k-point global sweep).
- nav.js: neighbor-cell prefetch skips rows past the poles and wraps
longitudes across the dateline instead of sending out-of-range queries.
- Nominatim reverse geocoding requests accept-language=en so place labels
render in one script worldwide (matching the forward geocoder).
- mappicker: search suggestions are no longer filtered to US/CA, the
placeholder and default map view are worldwide.
- calendar: season filter labels flip for southern-hemisphere locations
(Dec-Feb shows as Summer); the underlying month groups are unchanged, so
saved filter selections keep meaning the same months.
Verified end-to-end on a scratch server: Tokyo and Sydney grade with real
labels, a Fiji cell on the antimeridian's east edge builds and serves warm
hits from the derived store, and prefetch=1 on a cold cell still answers
204 without spending weather-API quota.
2026-07-11 15:02:28 +00:00
|
|
|
# Keep the reported center a valid coordinate for the upstream weather and
|
|
|
|
|
# geocoding APIs: the topmost row's center overshoots the pole, and a row's
|
|
|
|
|
# outermost cells can have centers just past the antimeridian on either side.
|
|
|
|
|
center_lat = min(max(center_lat, -90.0), 90.0)
|
|
|
|
|
if center_lon > 180.0:
|
|
|
|
|
center_lon -= 360.0
|
|
|
|
|
elif center_lon < -180.0:
|
|
|
|
|
center_lon += 360.0
|
2026-07-11 00:29:47 +00:00
|
|
|
|
|
|
|
|
# Approximate cell dimensions in miles for display.
|
|
|
|
|
height_mi = LAT_STEP * 69.0
|
|
|
|
|
width_mi = lon_step * 69.0 * math.cos(math.radians(center_lat))
|
|
|
|
|
|
|
|
|
|
return {
|
|
|
|
|
"id": f"{i}_{j}",
|
|
|
|
|
"center_lat": round(center_lat, 5),
|
|
|
|
|
"center_lon": round(center_lon, 5),
|
|
|
|
|
"lat_step": LAT_STEP,
|
|
|
|
|
"lon_step": lon_step,
|
|
|
|
|
"bounds": {
|
|
|
|
|
"south": round(i * LAT_STEP, 5),
|
|
|
|
|
"north": round((i + 1) * LAT_STEP, 5),
|
|
|
|
|
"west": round(j * lon_step, 5),
|
|
|
|
|
"east": round((j + 1) * lon_step, 5),
|
|
|
|
|
},
|
|
|
|
|
"area_sq_mi": round(height_mi * width_mi, 2),
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
Worldwide coverage: grade any point on Earth (#32)
Remove the US+Canada bounding box so every endpoint accepts any lat/lon.
The grading pipeline was already global-ready (ERA5 archive, timezone=auto,
day-of-year climatology), so opening it up is mostly deleting the guard —
plus the edge cases that only exist once the whole globe is in play:
- grid.py: snap() wraps longitude into [-180, 180) and clamps latitude, and
cell centers are normalized so the polar row and the cells straddling the
antimeridian always report valid coordinates to the weather/geocoding APIs.
snap() and from_id() now share one _cell() builder, making id round-trips
exact by construction (verified with a 300k-point global sweep).
- nav.js: neighbor-cell prefetch skips rows past the poles and wraps
longitudes across the dateline instead of sending out-of-range queries.
- Nominatim reverse geocoding requests accept-language=en so place labels
render in one script worldwide (matching the forward geocoder).
- mappicker: search suggestions are no longer filtered to US/CA, the
placeholder and default map view are worldwide.
- calendar: season filter labels flip for southern-hemisphere locations
(Dec-Feb shows as Summer); the underlying month groups are unchanged, so
saved filter selections keep meaning the same months.
Verified end-to-end on a scratch server: Tokyo and Sydney grade with real
labels, a Fiji cell on the antimeridian's east edge builds and serves warm
hits from the derived store, and prefetch=1 on a cold cell still answers
204 without spending weather-API quota.
2026-07-11 15:02:28 +00:00
|
|
|
def snap(lat: float, lon: float) -> dict:
|
|
|
|
|
"""Return the grid cell (id + center + span) containing (lat, lon)."""
|
|
|
|
|
lat = min(max(lat, -90.0), 90.0)
|
|
|
|
|
lon = ((lon + 180.0) % 360.0) - 180.0 # wrap into [-180, 180)
|
|
|
|
|
i = math.floor(lat / LAT_STEP)
|
|
|
|
|
j = math.floor(lon / _lon_step((i + 0.5) * LAT_STEP))
|
|
|
|
|
return _cell(i, j)
|
|
|
|
|
|
|
|
|
|
|
2026-07-11 20:47:31 +00:00
|
|
|
def neighbors(cell: dict) -> list[dict]:
|
|
|
|
|
"""The up-to-8 cells surrounding one cell. Steps one cell width from the
|
|
|
|
|
center and re-snaps, so adjacent rows — whose longitude step differs —
|
|
|
|
|
resolve to whichever cell actually contains the stepped point. Rows past
|
|
|
|
|
the poles are skipped; longitude wraps across the antimeridian (both via
|
|
|
|
|
snap). Deduplicated (near the poles steps can collapse onto one cell)."""
|
|
|
|
|
out: dict[str, dict] = {}
|
|
|
|
|
for di in (-1, 0, 1):
|
|
|
|
|
for dj in (-1, 0, 1):
|
|
|
|
|
if not di and not dj:
|
|
|
|
|
continue
|
|
|
|
|
lat = cell["center_lat"] + di * LAT_STEP
|
|
|
|
|
if abs(lat) > 90.0:
|
|
|
|
|
continue
|
|
|
|
|
n = snap(lat, cell["center_lon"] + dj * cell["lon_step"])
|
|
|
|
|
if n["id"] != cell["id"]:
|
|
|
|
|
out[n["id"]] = n
|
|
|
|
|
return list(out.values())
|
|
|
|
|
|
|
|
|
|
|
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21)
* Compress API responses and revalidate static assets instead of re-downloading
- Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x.
- Serve pages/assets with Cache-Control: no-cache instead of no-store, so
browsers revalidate via the ETag/Last-Modified that FileResponse and
StaticFiles already emit. Unchanged assets now cost an empty 304 rather
than a full transfer on every page navigation, while deploys still show
up immediately.
* Persist derived responses in SQLite so grading is computed once per cell, not per request
New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet
records — finished grade/calendar/day/forecast payloads and reverse-geocode
labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms)
becomes a ~5ms database read that survives restarts and is shared across views.
Raw parquet stays the source of truth; the store is a pure accelerator (every
reader falls back to recomputing on a miss, and deleting the db is a safe reset).
Freshness is token-driven, not clock-driven: each cached payload is validated by
a token encoding what it was computed from (payload schema version, the archive
record's end date, the recent-fetch stamp). The existing freshness drivers are
untouched — get_history still tops up the tail hourly and get_recent_forecast
still refetches hourly — and tokens are derived from what they return, so cached
payloads expire exactly when their inputs change. The same tokens double as weak
ETags: If-None-Match answers with an empty 304 without touching the payload.
- backend/store.py: derived-payload + revgeo tables, thread-local WAL conns,
every helper fail-soft.
- app.py: endpoints split into pure payload builders + HTTP/caching shells; the
in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store).
- climate.py: revgeo persisted through the store; recent_stamp() and
load_cached_history() (no-network read) helpers.
- backend/migrate.py + make migrate: idempotent, resumable backfill of the store
from existing parquet caches (default calendar span + latest-day detail +
revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather.
- grid.from_id(): rebuild a cell from its cache filename (migrate tooling).
* Add /api/v2/cell: one bundle carrying every view's payload
GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months)
and day (today) payloads in one response, each the exact payload its per-view
endpoint returns — built by the same builders and cached under the same
derived-store keys/tokens — paired with the etag that endpoint would emit. The
frontend can warm all views with a single request, seed its per-view cache from
the slices, and later revalidate each view individually with If-None-Match. The
bundle's own etag combines the slices', so an unchanged bundle is an empty 304.
prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard
guarantee: it never spends weather-API quota. A cell with no cached archive
answers 204, and only the history-derived slices (calendar + latest-day detail)
are built. At most one Nominatim lookup for a never-labeled cell.
* Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch
The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota,
which multi-year calendar payloads regularly blew through) to IndexedDB, with an
in-memory map in front. Entries carry the server's ETag, so anything stale
revalidates conditionally — unchanged data costs an empty 304 and a re-stamp,
never a re-transfer. On network failure the stale copy is served over an error.
getJSON gains an optional onUpdate callback opting into stale-while-revalidate:
the three views (weekly, day, calendar) now render a cached copy immediately —
spinners are delayed 150ms so warm loads never flash-blank — and repaint only if
background revalidation finds changed data. New data shows up the moment it
exists instead of waiting out a TTL.
Cross-view prefetch collapses from one request per view to a single /api/v2/cell
bundle, whose slices (exact per-view payloads + their etags) are seeded under the
URLs each view actually requests; the bundle call itself is conditional via a
remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side
with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one
possible Nominatim lookup each), so tapping nearby lands on already-graded data.
Legacy tg:* storage entries are cleared once; cache entries untouched for two
weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
|
|
|
def from_id(cell_id: str) -> dict:
|
|
|
|
|
"""Rebuild the full cell dict from a cache id ("i_j") — the inverse of snap().
|
|
|
|
|
Lets offline tooling (the migrate script) recover a cell from its parquet
|
|
|
|
|
filename alone. Raises ValueError on a malformed id."""
|
|
|
|
|
i, j = (int(p) for p in cell_id.split("_"))
|
Worldwide coverage: grade any point on Earth (#32)
Remove the US+Canada bounding box so every endpoint accepts any lat/lon.
The grading pipeline was already global-ready (ERA5 archive, timezone=auto,
day-of-year climatology), so opening it up is mostly deleting the guard —
plus the edge cases that only exist once the whole globe is in play:
- grid.py: snap() wraps longitude into [-180, 180) and clamps latitude, and
cell centers are normalized so the polar row and the cells straddling the
antimeridian always report valid coordinates to the weather/geocoding APIs.
snap() and from_id() now share one _cell() builder, making id round-trips
exact by construction (verified with a 300k-point global sweep).
- nav.js: neighbor-cell prefetch skips rows past the poles and wraps
longitudes across the dateline instead of sending out-of-range queries.
- Nominatim reverse geocoding requests accept-language=en so place labels
render in one script worldwide (matching the forward geocoder).
- mappicker: search suggestions are no longer filtered to US/CA, the
placeholder and default map view are worldwide.
- calendar: season filter labels flip for southern-hemisphere locations
(Dec-Feb shows as Summer); the underlying month groups are unchanged, so
saved filter selections keep meaning the same months.
Verified end-to-end on a scratch server: Tokyo and Sydney grade with real
labels, a Fiji cell on the antimeridian's east edge builds and serves warm
hits from the derived store, and prefetch=1 on a cold cell still answers
204 without spending weather-API quota.
2026-07-11 15:02:28 +00:00
|
|
|
return _cell(i, j)
|