2026-07-11 00:29:47 +00:00
|
|
|
"""Fetch historical + recent daily weather from Open-Meteo and cache to parquet.
|
|
|
|
|
|
|
|
|
|
The full multi-decade daily record for a grid cell is fetched once and stored as
|
|
|
|
|
a zstd-compressed parquet file keyed by cell id. Percentiles are computed on the
|
|
|
|
|
fly from this raw record (see grading.py), which keeps the cache small and lets
|
|
|
|
|
us handle ties (e.g. many zero-precip days) correctly.
|
|
|
|
|
"""
|
|
|
|
|
import datetime
|
|
|
|
|
import os
|
|
|
|
|
import threading
|
|
|
|
|
import time
|
|
|
|
|
|
|
|
|
|
import httpx
|
|
|
|
|
import numpy as np
|
|
|
|
|
import pandas as pd
|
|
|
|
|
|
|
|
|
|
import audit
|
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21)
* Compress API responses and revalidate static assets instead of re-downloading
- Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x.
- Serve pages/assets with Cache-Control: no-cache instead of no-store, so
browsers revalidate via the ETag/Last-Modified that FileResponse and
StaticFiles already emit. Unchanged assets now cost an empty 304 rather
than a full transfer on every page navigation, while deploys still show
up immediately.
* Persist derived responses in SQLite so grading is computed once per cell, not per request
New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet
records — finished grade/calendar/day/forecast payloads and reverse-geocode
labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms)
becomes a ~5ms database read that survives restarts and is shared across views.
Raw parquet stays the source of truth; the store is a pure accelerator (every
reader falls back to recomputing on a miss, and deleting the db is a safe reset).
Freshness is token-driven, not clock-driven: each cached payload is validated by
a token encoding what it was computed from (payload schema version, the archive
record's end date, the recent-fetch stamp). The existing freshness drivers are
untouched — get_history still tops up the tail hourly and get_recent_forecast
still refetches hourly — and tokens are derived from what they return, so cached
payloads expire exactly when their inputs change. The same tokens double as weak
ETags: If-None-Match answers with an empty 304 without touching the payload.
- backend/store.py: derived-payload + revgeo tables, thread-local WAL conns,
every helper fail-soft.
- app.py: endpoints split into pure payload builders + HTTP/caching shells; the
in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store).
- climate.py: revgeo persisted through the store; recent_stamp() and
load_cached_history() (no-network read) helpers.
- backend/migrate.py + make migrate: idempotent, resumable backfill of the store
from existing parquet caches (default calendar span + latest-day detail +
revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather.
- grid.from_id(): rebuild a cell from its cache filename (migrate tooling).
* Add /api/v2/cell: one bundle carrying every view's payload
GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months)
and day (today) payloads in one response, each the exact payload its per-view
endpoint returns — built by the same builders and cached under the same
derived-store keys/tokens — paired with the etag that endpoint would emit. The
frontend can warm all views with a single request, seed its per-view cache from
the slices, and later revalidate each view individually with If-None-Match. The
bundle's own etag combines the slices', so an unchanged bundle is an empty 304.
prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard
guarantee: it never spends weather-API quota. A cell with no cached archive
answers 204, and only the history-derived slices (calendar + latest-day detail)
are built. At most one Nominatim lookup for a never-labeled cell.
* Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch
The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota,
which multi-year calendar payloads regularly blew through) to IndexedDB, with an
in-memory map in front. Entries carry the server's ETag, so anything stale
revalidates conditionally — unchanged data costs an empty 304 and a re-stamp,
never a re-transfer. On network failure the stale copy is served over an error.
getJSON gains an optional onUpdate callback opting into stale-while-revalidate:
the three views (weekly, day, calendar) now render a cached copy immediately —
spinners are delayed 150ms so warm loads never flash-blank — and repaint only if
background revalidation finds changed data. New data shows up the moment it
exists instead of waiting out a TTL.
Cross-view prefetch collapses from one request per view to a single /api/v2/cell
bundle, whose slices (exact per-view payloads + their etags) are seeded under the
URLs each view actually requests; the bundle call itself is conditional via a
remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side
with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one
possible Nominatim lookup each), so tapping nearby lands on already-graded data.
Legacy tg:* storage entries are cleared once; cache entries untouched for two
weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
|
|
|
import store
|
2026-07-11 00:29:47 +00:00
|
|
|
|
|
|
|
|
CACHE_DIR = os.path.join(os.path.dirname(__file__), "..", "data", "cache")
|
|
|
|
|
MAX_ATTEMPTS = 3 # per upstream call, before giving up
|
|
|
|
|
START_DATE = "1980-01-01" # ERA5 reaches back to 1940; 1980 = 45 yrs, fast + robust
|
|
|
|
|
ARCHIVE_LATENCY_DAYS = 6 # ERA5 archive lags real time by a few days
|
|
|
|
|
# History rarely changes, so the full multi-decade archive is fetched once and
|
|
|
|
|
# cached indefinitely (refetched only to add new metric columns). Only the recent
|
|
|
|
|
# tail is refreshed — a small incremental fetch, at most hourly.
|
|
|
|
|
HISTORY_TOPUP_INTERVAL = 3600 # seconds between recent-tail refresh attempts
|
|
|
|
|
FORECAST_TTL_HOURS = 1 # refetch the forward forecast hourly to track updates
|
|
|
|
|
|
|
|
|
|
ARCHIVE_URL = "https://archive-api.open-meteo.com/v1/archive"
|
|
|
|
|
FORECAST_URL = "https://api.open-meteo.com/v1/forecast"
|
|
|
|
|
# Backup archive when Open-Meteo is unavailable (e.g. its daily rate limit). NASA
|
|
|
|
|
# POWER is free + keyless, global, daily from 1981. It lacks gusts + apparent temp,
|
|
|
|
|
# so gusts read as unavailable and "feels like" is computed from heat index/chill.
|
|
|
|
|
NASA_POWER_URL = "https://power.larc.nasa.gov/api/temporal/daily/point"
|
|
|
|
|
NASA_START = "19810101"
|
|
|
|
|
NASA_FILL = -900.0 # POWER's missing-value sentinel is ~-999
|
|
|
|
|
|
|
|
|
|
DAILY_VARS = (
|
|
|
|
|
"temperature_2m_max,temperature_2m_min,precipitation_sum,"
|
|
|
|
|
"wind_speed_10m_max,wind_gusts_10m_max,"
|
|
|
|
|
"apparent_temperature_max,apparent_temperature_min,"
|
|
|
|
|
"relative_humidity_2m_mean"
|
|
|
|
|
)
|
|
|
|
|
|
|
|
|
|
# Columns added after the original tmax/tmin/precip schema. A cached parquet
|
|
|
|
|
# missing any of these predates the wind/feels/humidity features (or the split
|
|
|
|
|
# apparent high/low) and is refetched so the new metrics show up immediately
|
|
|
|
|
# instead of after the 30-day cache TTL.
|
|
|
|
|
NEW_COLS = ("wind", "gust", "feels", "humid", "fmax", "fmin")
|
|
|
|
|
|
|
|
|
|
# Thermoneutral baseline (°F). The combined "feels like" metric reports whichever
|
|
|
|
|
# apparent-temperature extreme — the heat-index-driven daily max or the
|
|
|
|
|
# wind-chill-driven daily min — sits further from this comfort point, so a single
|
|
|
|
|
# daily value captures both heat index (hot side) and wind chill (cold side).
|
|
|
|
|
COMFORT_F = 65.0
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def _cache_path(cell_id: str) -> str:
|
|
|
|
|
return os.path.join(CACHE_DIR, f"{cell_id}.parquet")
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
# One lock per cell so concurrent requests/refreshes don't each pull the full
|
|
|
|
|
# multi-decade archive (which quickly trips the upstream rate limit).
|
|
|
|
|
_LOCKS_GUARD = threading.Lock()
|
|
|
|
|
_CELL_LOCKS: dict[str, threading.Lock] = {}
|
|
|
|
|
|
|
|
|
|
# When the archive rate-limits us (429), back off globally for a bit rather than
|
|
|
|
|
# re-hitting it on every refresh — that only prolongs the limit.
|
|
|
|
|
ARCHIVE_COOLDOWN = 120 # seconds
|
|
|
|
|
_archive_cooldown_until = 0.0
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
_archive_limit_daily = False # is the current cooldown a daily-quota exhaustion?
|
|
|
|
|
|
|
|
|
|
|
2026-07-11 19:53:46 +00:00
|
|
|
class WeatherUnavailable(RuntimeError):
|
|
|
|
|
"""Weather data can't be fetched right now (rate limit, upstream outage with
|
|
|
|
|
no cached fallback). Carries user-facing text; ``daily`` marks a daily-quota
|
|
|
|
|
exhaustion (resets after UTC midnight) rather than a transient burst limit.
|
|
|
|
|
The API layer maps this to a retryable 503 — it never needs to parse the
|
|
|
|
|
message, and new upstream sources classify their own failures here."""
|
|
|
|
|
|
|
|
|
|
def __init__(self, message: str, daily: bool = False):
|
|
|
|
|
super().__init__(message)
|
|
|
|
|
self.daily = daily
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def limit_message(daily: bool) -> str:
|
|
|
|
|
"""The user-facing rate-limit copy, in one place."""
|
|
|
|
|
if daily:
|
|
|
|
|
return ("Open-Meteo's daily request limit is exhausted — new locations will work again "
|
|
|
|
|
"tomorrow. Places you've already viewed still work.")
|
|
|
|
|
return "The weather service is rate-limited right now — please try again in a minute."
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def is_rate_limit(e) -> bool:
|
2026-07-11 00:29:47 +00:00
|
|
|
return getattr(getattr(e, "response", None), "status_code", None) == 429
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def _rate_limit_reason(e) -> str:
|
|
|
|
|
"""Upstream's human-readable 429 reason, if present in the response body."""
|
|
|
|
|
resp = getattr(e, "response", None)
|
|
|
|
|
if resp is not None:
|
|
|
|
|
try:
|
|
|
|
|
j = resp.json()
|
|
|
|
|
if isinstance(j, dict) and j.get("reason"):
|
|
|
|
|
return str(j["reason"])
|
|
|
|
|
except Exception: # noqa: BLE001
|
|
|
|
|
pass
|
|
|
|
|
return ""
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def _seconds_to_utc_reset() -> float:
|
|
|
|
|
"""Seconds until just after the next UTC midnight (when daily quotas reset)."""
|
|
|
|
|
now = datetime.datetime.now(datetime.timezone.utc)
|
|
|
|
|
reset = (now + datetime.timedelta(days=1)).replace(hour=0, minute=10, second=0, microsecond=0)
|
|
|
|
|
return max((reset - now).total_seconds(), 600)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def _note_rate_limit(e) -> None:
|
|
|
|
|
"""Record a rate limit: back off ~2 min, or until tomorrow for a daily quota."""
|
|
|
|
|
global _archive_cooldown_until, _archive_limit_daily
|
|
|
|
|
reason = _rate_limit_reason(e).lower()
|
|
|
|
|
_archive_limit_daily = "daily" in reason or "tomorrow" in reason
|
|
|
|
|
_archive_cooldown_until = time.time() + (
|
|
|
|
|
_seconds_to_utc_reset() if _archive_limit_daily else ARCHIVE_COOLDOWN)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def _cell_lock(cell_id: str) -> threading.Lock:
|
|
|
|
|
with _LOCKS_GUARD:
|
|
|
|
|
lk = _CELL_LOCKS.get(cell_id)
|
|
|
|
|
if lk is None:
|
|
|
|
|
lk = _CELL_LOCKS[cell_id] = threading.Lock()
|
|
|
|
|
return lk
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def _request(url, params, timeout, *, phase, headers=None, attempts=MAX_ATTEMPTS):
|
|
|
|
|
"""GET with bounded retries; every retry and the final failure are logged to
|
|
|
|
|
the errors folder (tagged ``retry`` / ``error``). A 429 (rate limit) fails fast
|
|
|
|
|
without retrying, so we don't hammer the limit and make it worse."""
|
|
|
|
|
last = None
|
|
|
|
|
for attempt in range(1, attempts + 1):
|
|
|
|
|
try:
|
|
|
|
|
r = httpx.get(url, params=params, timeout=timeout, headers=headers)
|
|
|
|
|
r.raise_for_status()
|
|
|
|
|
return r
|
|
|
|
|
except Exception as e: # noqa: BLE001 - upstream/network failures are expected
|
|
|
|
|
last = e
|
|
|
|
|
status = getattr(getattr(e, "response", None), "status_code", None)
|
|
|
|
|
rate_limited = status == 429
|
|
|
|
|
final = rate_limited or attempt == attempts
|
|
|
|
|
audit.log_event(
|
|
|
|
|
"error" if final else "retry",
|
|
|
|
|
{"phase": phase, "attempt": attempt, "max_attempts": attempts,
|
|
|
|
|
"url": url, "status": status, "error": repr(e)},
|
|
|
|
|
)
|
|
|
|
|
if final:
|
|
|
|
|
break
|
|
|
|
|
time.sleep(min(0.5 * 2 ** (attempt - 1), 4.0))
|
|
|
|
|
raise last
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def _derive_humidity(df: pd.DataFrame) -> pd.DataFrame:
|
|
|
|
|
"""Replace the raw mean *relative* humidity column with *absolute* humidity
|
|
|
|
|
(grams of water vapor per m³), in place.
|
|
|
|
|
|
|
|
|
|
Absolute humidity is derived from the day's mean RH and its mean temperature
|
|
|
|
|
((tmax+tmin)/2) via the Magnus saturation-vapor-pressure formula. It's a far
|
|
|
|
|
more informative "how muggy was it" signal than relative humidity, which mostly
|
|
|
|
|
tracks the day/night temperature swing (cold nights read ~100% RH regardless of
|
|
|
|
|
actual moisture). Applied at the read boundary so the parquet cache keeps the
|
|
|
|
|
raw RH the archive returns — no refetch needed for existing cells."""
|
|
|
|
|
if "humid" not in df.columns:
|
|
|
|
|
return df
|
|
|
|
|
tmean_c = ((df["tmax"] + df["tmin"]) / 2.0 - 32.0) * 5.0 / 9.0
|
|
|
|
|
rh = pd.to_numeric(df["humid"], errors="coerce")
|
|
|
|
|
es = 6.112 * np.exp(17.67 * tmean_c / (tmean_c + 243.5)) # sat. vapor pressure, hPa
|
|
|
|
|
df["humid"] = (es * rh * 2.1674 / (273.15 + tmean_c)).round(1) # absolute humidity, g/m³
|
|
|
|
|
return df
|
|
|
|
|
|
|
|
|
|
|
2026-07-11 20:05:57 +00:00
|
|
|
def _with_doy(df: pd.DataFrame) -> pd.DataFrame:
|
|
|
|
|
"""(Re)attach the int16 day-of-year column the grading windows key on."""
|
|
|
|
|
df["doy"] = df["date"].dt.dayofyear.astype("int16")
|
|
|
|
|
return df
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def _write_cache(df: pd.DataFrame, path: str) -> None:
|
|
|
|
|
"""Persist a daily record to the parquet cache — the raw record only, never
|
|
|
|
|
the derived doy column (it's recomputed at read time)."""
|
|
|
|
|
os.makedirs(CACHE_DIR, exist_ok=True)
|
|
|
|
|
df.drop(columns=["doy"], errors="ignore").to_parquet(
|
|
|
|
|
path, compression="zstd", index=False)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def _om_daily_params(cell: dict, **window) -> dict:
|
|
|
|
|
"""The Open-Meteo daily-request params every fetch shares — location, the
|
|
|
|
|
variable set, imperial units. The date/window selectors (start_date/end_date
|
|
|
|
|
or past_days/forecast_days) come in as kwargs."""
|
|
|
|
|
return {
|
|
|
|
|
"latitude": cell["center_lat"],
|
|
|
|
|
"longitude": cell["center_lon"],
|
|
|
|
|
"daily": DAILY_VARS,
|
|
|
|
|
"timezone": "auto",
|
|
|
|
|
"temperature_unit": "fahrenheit",
|
|
|
|
|
"precipitation_unit": "inch",
|
|
|
|
|
"wind_speed_unit": "mph",
|
|
|
|
|
**window,
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def _finalize_frame(df: pd.DataFrame) -> pd.DataFrame:
|
|
|
|
|
"""Shared tail of every source→frame mapping: the combined feels-like, the
|
|
|
|
|
valid-day filter, and the day-of-year column. A usable climate day needs a
|
|
|
|
|
real high/low; other columns may be NaN and simply grade as None."""
|
|
|
|
|
df["feels"] = _combined_feels(df["fmax"], df["fmin"])
|
|
|
|
|
df = df.dropna(subset=["tmax", "tmin"]).reset_index(drop=True)
|
|
|
|
|
return _with_doy(df)
|
|
|
|
|
|
|
|
|
|
|
2026-07-11 00:29:47 +00:00
|
|
|
def _combined_feels(amax: pd.Series, amin: pd.Series) -> pd.Series:
|
|
|
|
|
"""One daily "feels like" value: the apparent-temperature extreme furthest from
|
|
|
|
|
the comfort baseline — the heat-index high on warm days, the wind-chill low on
|
|
|
|
|
cold ones. Falls back to whichever side is present if one is missing."""
|
|
|
|
|
hot = amax - COMFORT_F
|
|
|
|
|
cold = COMFORT_F - amin
|
|
|
|
|
feels = amax.where(hot >= cold, amin) # NaN comparisons pick the min side
|
|
|
|
|
return feels.fillna(amax).fillna(amin)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def _to_frame(daily: dict) -> pd.DataFrame:
|
|
|
|
|
n = len(daily["time"])
|
|
|
|
|
# Older upstream responses (or a narrowed variable set) may omit a series; fall
|
|
|
|
|
# back to an all-null column of the right length so the frame shape is stable.
|
|
|
|
|
def col(key):
|
|
|
|
|
vals = daily.get(key)
|
|
|
|
|
return vals if vals else [None] * n
|
|
|
|
|
|
|
|
|
|
df = pd.DataFrame(
|
|
|
|
|
{
|
|
|
|
|
"date": pd.to_datetime(daily["time"]),
|
|
|
|
|
"tmax": daily["temperature_2m_max"],
|
|
|
|
|
"tmin": daily["temperature_2m_min"],
|
|
|
|
|
"precip": daily["precipitation_sum"],
|
|
|
|
|
"wind": col("wind_speed_10m_max"),
|
|
|
|
|
"gust": col("wind_gusts_10m_max"),
|
|
|
|
|
"humid": col("relative_humidity_2m_mean"),
|
|
|
|
|
}
|
|
|
|
|
)
|
|
|
|
|
# The apparent (felt) high and low are kept as their own columns — graded
|
|
|
|
|
# independently — alongside the combined `feels` (whichever side is further
|
|
|
|
|
# from the fixed comfort baseline), which the weekly/day views still use.
|
2026-07-11 20:05:57 +00:00
|
|
|
df["fmax"] = pd.to_numeric(pd.Series(col("apparent_temperature_max")), errors="coerce")
|
|
|
|
|
df["fmin"] = pd.to_numeric(pd.Series(col("apparent_temperature_min")), errors="coerce")
|
|
|
|
|
return _finalize_frame(df)
|
2026-07-11 00:29:47 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
def _fetch_history(cell: dict) -> pd.DataFrame:
|
|
|
|
|
end = (datetime.date.today() - datetime.timedelta(days=ARCHIVE_LATENCY_DAYS)).isoformat()
|
2026-07-11 20:05:57 +00:00
|
|
|
params = _om_daily_params(cell, start_date=START_DATE, end_date=end)
|
2026-07-11 00:29:47 +00:00
|
|
|
r = _request(ARCHIVE_URL, params, 180, phase="history_fetch")
|
|
|
|
|
return _to_frame(r.json()["daily"])
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def _heat_index(t_f, rh):
|
|
|
|
|
"""NWS heat index (°F); below ~80°F it's just the air temperature."""
|
|
|
|
|
T, R = t_f, rh
|
|
|
|
|
hi = (-42.379 + 2.04901523 * T + 10.14333127 * R - 0.22475541 * T * R
|
|
|
|
|
- 0.00683783 * T * T - 0.05481717 * R * R + 0.00122874 * T * T * R
|
|
|
|
|
+ 0.00085282 * T * R * R - 0.00000199 * T * T * R * R)
|
|
|
|
|
return np.where(np.asarray(T, dtype="float64") >= 80, hi, T)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def _wind_chill(t_f, v_mph):
|
|
|
|
|
"""NWS wind chill (°F); applies only when cold + breezy, else the air temp."""
|
|
|
|
|
T = np.asarray(t_f, dtype="float64")
|
|
|
|
|
V = np.clip(np.asarray(v_mph, dtype="float64"), 0, None)
|
|
|
|
|
Vp = np.power(V, 0.16)
|
|
|
|
|
wc = 35.74 + 0.6215 * T - 35.75 * Vp + 0.4275 * T * Vp
|
|
|
|
|
return np.where((T <= 50) & (V >= 3), wc, T)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def _nasa_to_frame(param: dict) -> pd.DataFrame:
|
|
|
|
|
"""Map a NASA POWER daily response to our (tmax/tmin/precip/wind/gust/humid/feels)
|
|
|
|
|
schema, converting units (°C→°F, mm→in, m/s→mph) and computing feels-like."""
|
|
|
|
|
dates = sorted(param.get("T2M_MAX", {}).keys())
|
|
|
|
|
|
|
|
|
|
def col(key, transform):
|
|
|
|
|
d = param.get(key, {})
|
|
|
|
|
return [np.nan if (v is None or v <= NASA_FILL) else transform(v)
|
|
|
|
|
for v in (d.get(k) for k in dates)]
|
|
|
|
|
|
|
|
|
|
c2f = lambda c: c * 9.0 / 5.0 + 32.0
|
|
|
|
|
df = pd.DataFrame({
|
|
|
|
|
"date": pd.to_datetime(dates, format="%Y%m%d"),
|
|
|
|
|
"tmax": col("T2M_MAX", c2f),
|
|
|
|
|
"tmin": col("T2M_MIN", c2f),
|
|
|
|
|
"precip": col("PRECTOTCORR", lambda mm: mm / 25.4),
|
|
|
|
|
"wind": col("WS10M_MAX", lambda ms: ms * 2.2369362920544),
|
|
|
|
|
"gust": [np.nan] * len(dates), # POWER has no gusts
|
|
|
|
|
"humid": col("RH2M", lambda v: v),
|
|
|
|
|
})
|
|
|
|
|
# POWER has no apparent temperature; approximate the felt high with the NWS
|
|
|
|
|
# heat index and the felt low with NWS wind chill (each falls back to the air
|
|
|
|
|
# temperature outside its regime), mirroring the Open-Meteo columns.
|
|
|
|
|
df["fmax"] = pd.Series(_heat_index(df["tmax"], df["humid"]), index=df.index)
|
|
|
|
|
df["fmin"] = pd.Series(_wind_chill(df["tmin"], df["wind"]), index=df.index)
|
2026-07-11 20:05:57 +00:00
|
|
|
return _finalize_frame(df)
|
2026-07-11 00:29:47 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
def _fetch_history_nasa(cell: dict) -> pd.DataFrame:
|
|
|
|
|
"""Backup history fetch from NASA POWER (used when Open-Meteo is unavailable)."""
|
|
|
|
|
end = (datetime.date.today() - datetime.timedelta(days=ARCHIVE_LATENCY_DAYS)).strftime("%Y%m%d")
|
|
|
|
|
params = {
|
|
|
|
|
"parameters": "T2M_MAX,T2M_MIN,PRECTOTCORR,WS10M_MAX,RH2M",
|
|
|
|
|
"community": "RE",
|
|
|
|
|
"latitude": cell["center_lat"],
|
|
|
|
|
"longitude": cell["center_lon"],
|
|
|
|
|
"start": NASA_START,
|
|
|
|
|
"end": end,
|
|
|
|
|
"format": "JSON",
|
|
|
|
|
}
|
|
|
|
|
r = _request(NASA_POWER_URL, params, 180, phase="history_nasa")
|
|
|
|
|
return _nasa_to_frame(r.json()["properties"]["parameter"])
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def _fetch_history_range(cell: dict, start_date: str, end_date: str) -> pd.DataFrame:
|
|
|
|
|
"""Fetch just a date range of archive history (used to top up the recent tail)."""
|
2026-07-11 20:05:57 +00:00
|
|
|
params = _om_daily_params(cell, start_date=start_date, end_date=end_date)
|
2026-07-11 00:29:47 +00:00
|
|
|
r = _request(ARCHIVE_URL, params, 60, phase="history_topup")
|
|
|
|
|
return _to_frame(r.json()["daily"])
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def _read_history_cache(path):
|
|
|
|
|
"""Schema-complete cached frame (no doy) + its file age in seconds, or None.
|
|
|
|
|
|
|
|
|
|
No age expiry — history is cached indefinitely; a pre-wind/humidity schema is
|
|
|
|
|
the only reason to refetch (to add the new metric columns)."""
|
|
|
|
|
if not os.path.exists(path):
|
|
|
|
|
return None
|
|
|
|
|
df = pd.read_parquet(path)
|
|
|
|
|
if not all(c in df.columns for c in NEW_COLS):
|
|
|
|
|
return None
|
|
|
|
|
return df, time.time() - os.path.getmtime(path)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def _topup_tail(cell: dict, df: pd.DataFrame, path: str) -> pd.DataFrame:
|
|
|
|
|
"""Append newly-available archive days to a cached record. Best-effort and
|
|
|
|
|
serialized per cell; a small incremental fetch, not the full multi-decade pull."""
|
|
|
|
|
global _archive_cooldown_until
|
|
|
|
|
with _cell_lock(cell["id"]):
|
|
|
|
|
fresh = _read_history_cache(path) # another thread may have just refreshed it
|
|
|
|
|
if fresh is not None:
|
|
|
|
|
df, age_s = fresh
|
|
|
|
|
if age_s < HISTORY_TOPUP_INTERVAL:
|
|
|
|
|
return df
|
|
|
|
|
if time.time() < _archive_cooldown_until:
|
|
|
|
|
return df
|
|
|
|
|
expected = datetime.date.today() - datetime.timedelta(days=ARCHIVE_LATENCY_DAYS)
|
|
|
|
|
cached_max = pd.Timestamp(df["date"].max()).date()
|
|
|
|
|
if cached_max >= expected:
|
|
|
|
|
try: os.utime(path, None) # tail already current — reset the hourly timer
|
|
|
|
|
except OSError: pass
|
|
|
|
|
return df
|
|
|
|
|
try:
|
|
|
|
|
recent = _fetch_history_range(
|
|
|
|
|
cell, (cached_max + datetime.timedelta(days=1)).isoformat(), expected.isoformat())
|
|
|
|
|
except Exception as e: # noqa: BLE001 - keep the cached record on failure
|
2026-07-11 19:53:46 +00:00
|
|
|
if is_rate_limit(e):
|
2026-07-11 00:29:47 +00:00
|
|
|
_note_rate_limit(e)
|
|
|
|
|
return df
|
|
|
|
|
merged = (pd.concat([df, recent.drop(columns=["doy"], errors="ignore")])
|
|
|
|
|
.drop_duplicates(subset="date", keep="last")
|
|
|
|
|
.sort_values("date").reset_index(drop=True))
|
2026-07-11 20:05:57 +00:00
|
|
|
_write_cache(merged, path)
|
2026-07-11 00:29:47 +00:00
|
|
|
return merged
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def get_history(cell: dict) -> tuple[pd.DataFrame, dict]:
|
|
|
|
|
"""Return (daily history frame, cache metadata) for a cell, with humidity as
|
|
|
|
|
absolute humidity (g/m³). Thin wrapper over the raw loader (see below)."""
|
|
|
|
|
df, meta = _load_history(cell)
|
|
|
|
|
_derive_humidity(df)
|
|
|
|
|
return df, meta
|
|
|
|
|
|
|
|
|
|
|
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21)
* Compress API responses and revalidate static assets instead of re-downloading
- Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x.
- Serve pages/assets with Cache-Control: no-cache instead of no-store, so
browsers revalidate via the ETag/Last-Modified that FileResponse and
StaticFiles already emit. Unchanged assets now cost an empty 304 rather
than a full transfer on every page navigation, while deploys still show
up immediately.
* Persist derived responses in SQLite so grading is computed once per cell, not per request
New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet
records — finished grade/calendar/day/forecast payloads and reverse-geocode
labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms)
becomes a ~5ms database read that survives restarts and is shared across views.
Raw parquet stays the source of truth; the store is a pure accelerator (every
reader falls back to recomputing on a miss, and deleting the db is a safe reset).
Freshness is token-driven, not clock-driven: each cached payload is validated by
a token encoding what it was computed from (payload schema version, the archive
record's end date, the recent-fetch stamp). The existing freshness drivers are
untouched — get_history still tops up the tail hourly and get_recent_forecast
still refetches hourly — and tokens are derived from what they return, so cached
payloads expire exactly when their inputs change. The same tokens double as weak
ETags: If-None-Match answers with an empty 304 without touching the payload.
- backend/store.py: derived-payload + revgeo tables, thread-local WAL conns,
every helper fail-soft.
- app.py: endpoints split into pure payload builders + HTTP/caching shells; the
in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store).
- climate.py: revgeo persisted through the store; recent_stamp() and
load_cached_history() (no-network read) helpers.
- backend/migrate.py + make migrate: idempotent, resumable backfill of the store
from existing parquet caches (default calendar span + latest-day detail +
revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather.
- grid.from_id(): rebuild a cell from its cache filename (migrate tooling).
* Add /api/v2/cell: one bundle carrying every view's payload
GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months)
and day (today) payloads in one response, each the exact payload its per-view
endpoint returns — built by the same builders and cached under the same
derived-store keys/tokens — paired with the etag that endpoint would emit. The
frontend can warm all views with a single request, seed its per-view cache from
the slices, and later revalidate each view individually with If-None-Match. The
bundle's own etag combines the slices', so an unchanged bundle is an empty 304.
prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard
guarantee: it never spends weather-API quota. A cell with no cached archive
answers 204, and only the history-derived slices (calendar + latest-day detail)
are built. At most one Nominatim lookup for a never-labeled cell.
* Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch
The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota,
which multi-year calendar payloads regularly blew through) to IndexedDB, with an
in-memory map in front. Entries carry the server's ETag, so anything stale
revalidates conditionally — unchanged data costs an empty 304 and a re-stamp,
never a re-transfer. On network failure the stale copy is served over an error.
getJSON gains an optional onUpdate callback opting into stale-while-revalidate:
the three views (weekly, day, calendar) now render a cached copy immediately —
spinners are delayed 150ms so warm loads never flash-blank — and repaint only if
background revalidation finds changed data. New data shows up the moment it
exists instead of waiting out a TTL.
Cross-view prefetch collapses from one request per view to a single /api/v2/cell
bundle, whose slices (exact per-view payloads + their etags) are seeded under the
URLs each view actually requests; the bundle call itself is conditional via a
remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side
with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one
possible Nominatim lookup each), so tapping nearby lands on already-graded data.
Legacy tg:* storage entries are cleared once; cache entries untouched for two
weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
|
|
|
def load_cached_history(cell: dict) -> pd.DataFrame | None:
|
|
|
|
|
"""History from the parquet cache ONLY — never fetches upstream and never tops
|
|
|
|
|
up the tail. Powers the warm-only prefetch path (which must not spend upstream
|
|
|
|
|
quota) and the offline migrate script. None when the cell has no
|
|
|
|
|
(schema-complete) cached record."""
|
|
|
|
|
hit = _read_history_cache(_cache_path(cell["id"]))
|
|
|
|
|
if hit is None:
|
|
|
|
|
return None
|
2026-07-11 20:05:57 +00:00
|
|
|
df = _with_doy(hit[0].copy())
|
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21)
* Compress API responses and revalidate static assets instead of re-downloading
- Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x.
- Serve pages/assets with Cache-Control: no-cache instead of no-store, so
browsers revalidate via the ETag/Last-Modified that FileResponse and
StaticFiles already emit. Unchanged assets now cost an empty 304 rather
than a full transfer on every page navigation, while deploys still show
up immediately.
* Persist derived responses in SQLite so grading is computed once per cell, not per request
New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet
records — finished grade/calendar/day/forecast payloads and reverse-geocode
labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms)
becomes a ~5ms database read that survives restarts and is shared across views.
Raw parquet stays the source of truth; the store is a pure accelerator (every
reader falls back to recomputing on a miss, and deleting the db is a safe reset).
Freshness is token-driven, not clock-driven: each cached payload is validated by
a token encoding what it was computed from (payload schema version, the archive
record's end date, the recent-fetch stamp). The existing freshness drivers are
untouched — get_history still tops up the tail hourly and get_recent_forecast
still refetches hourly — and tokens are derived from what they return, so cached
payloads expire exactly when their inputs change. The same tokens double as weak
ETags: If-None-Match answers with an empty 304 without touching the payload.
- backend/store.py: derived-payload + revgeo tables, thread-local WAL conns,
every helper fail-soft.
- app.py: endpoints split into pure payload builders + HTTP/caching shells; the
in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store).
- climate.py: revgeo persisted through the store; recent_stamp() and
load_cached_history() (no-network read) helpers.
- backend/migrate.py + make migrate: idempotent, resumable backfill of the store
from existing parquet caches (default calendar span + latest-day detail +
revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather.
- grid.from_id(): rebuild a cell from its cache filename (migrate tooling).
* Add /api/v2/cell: one bundle carrying every view's payload
GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months)
and day (today) payloads in one response, each the exact payload its per-view
endpoint returns — built by the same builders and cached under the same
derived-store keys/tokens — paired with the etag that endpoint would emit. The
frontend can warm all views with a single request, seed its per-view cache from
the slices, and later revalidate each view individually with If-None-Match. The
bundle's own etag combines the slices', so an unchanged bundle is an empty 304.
prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard
guarantee: it never spends weather-API quota. A cell with no cached archive
answers 204, and only the history-derived slices (calendar + latest-day detail)
are built. At most one Nominatim lookup for a never-labeled cell.
* Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch
The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota,
which multi-year calendar payloads regularly blew through) to IndexedDB, with an
in-memory map in front. Entries carry the server's ETag, so anything stale
revalidates conditionally — unchanged data costs an empty 304 and a re-stamp,
never a re-transfer. On network failure the stale copy is served over an error.
getJSON gains an optional onUpdate callback opting into stale-while-revalidate:
the three views (weekly, day, calendar) now render a cached copy immediately —
spinners are delayed 150ms so warm loads never flash-blank — and repaint only if
background revalidation finds changed data. New data shows up the moment it
exists instead of waiting out a TTL.
Cross-view prefetch collapses from one request per view to a single /api/v2/cell
bundle, whose slices (exact per-view payloads + their etags) are seeded under the
URLs each view actually requests; the bundle call itself is conditional via a
remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side
with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one
possible Nominatim lookup each), so tapping nearby lands on already-graded data.
Legacy tg:* storage entries are cleared once; cache entries untouched for two
weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
|
|
|
_derive_humidity(df)
|
|
|
|
|
return df
|
|
|
|
|
|
|
|
|
|
|
2026-07-11 00:29:47 +00:00
|
|
|
def _load_history(cell: dict) -> tuple[pd.DataFrame, dict]:
|
|
|
|
|
"""Return (daily history frame, cache metadata) for a cell.
|
|
|
|
|
|
|
|
|
|
The full archive is cached indefinitely (fetched once); only the recent tail is
|
|
|
|
|
topped up, at most hourly. Concurrent callers are serialized so only one archive
|
|
|
|
|
fetch happens; on an upstream failure (e.g. 429) any existing cache is served."""
|
|
|
|
|
path = _cache_path(cell["id"])
|
|
|
|
|
|
|
|
|
|
hit = _read_history_cache(path)
|
|
|
|
|
if hit is not None:
|
|
|
|
|
df, age_s = hit
|
|
|
|
|
if age_s > HISTORY_TOPUP_INTERVAL:
|
|
|
|
|
df = _topup_tail(cell, df, path) # refresh just the recent days
|
2026-07-11 20:05:57 +00:00
|
|
|
return _with_doy(df.copy()), {"cached": True, "cache_age_days": round(age_s / 86400.0, 1)}
|
2026-07-11 00:29:47 +00:00
|
|
|
|
|
|
|
|
global _archive_cooldown_until
|
|
|
|
|
|
|
|
|
|
with _cell_lock(cell["id"]):
|
|
|
|
|
hit = _read_history_cache(path) # another thread may have populated it while we waited
|
|
|
|
|
if hit is not None:
|
|
|
|
|
df, age_s = hit
|
2026-07-11 20:05:57 +00:00
|
|
|
return _with_doy(df), {"cached": True, "cache_age_days": round(age_s / 86400.0, 1)}
|
2026-07-11 00:29:47 +00:00
|
|
|
|
|
|
|
|
stale = pd.read_parquet(path) if os.path.exists(path) else None
|
|
|
|
|
|
|
|
|
|
def _serve_stale():
|
2026-07-11 20:05:57 +00:00
|
|
|
return _with_doy(stale), {"cached": True, "stale": True, "cache_age_days": None}
|
2026-07-11 00:29:47 +00:00
|
|
|
|
|
|
|
|
# Fetch. Open-Meteo is primary (richer: gusts + apparent temp); NASA POWER is
|
|
|
|
|
# the backup when Open-Meteo is unavailable (network error or its daily limit).
|
|
|
|
|
# Skip Open-Meteo entirely while it's in a rate-limit cooldown.
|
|
|
|
|
df = None
|
|
|
|
|
source = None
|
|
|
|
|
if time.time() >= _archive_cooldown_until:
|
|
|
|
|
try:
|
|
|
|
|
df = _fetch_history(cell)
|
|
|
|
|
source = "open-meteo"
|
|
|
|
|
except Exception as e: # noqa: BLE001
|
2026-07-11 19:53:46 +00:00
|
|
|
if is_rate_limit(e):
|
2026-07-11 00:29:47 +00:00
|
|
|
_note_rate_limit(e)
|
|
|
|
|
if df is None:
|
|
|
|
|
try:
|
|
|
|
|
df = _fetch_history_nasa(cell)
|
|
|
|
|
source = "nasa-power"
|
|
|
|
|
except Exception: # noqa: BLE001 - backup unavailable too
|
|
|
|
|
df = None
|
|
|
|
|
if df is None:
|
|
|
|
|
if stale is not None:
|
|
|
|
|
return _serve_stale()
|
2026-07-11 19:53:46 +00:00
|
|
|
raise WeatherUnavailable(limit_message(_archive_limit_daily),
|
|
|
|
|
daily=_archive_limit_daily)
|
2026-07-11 00:29:47 +00:00
|
|
|
|
|
|
|
|
# Store the raw record; percentiles are derived at request time.
|
2026-07-11 20:05:57 +00:00
|
|
|
_write_cache(df, path)
|
2026-07-11 00:29:47 +00:00
|
|
|
return df, {"cached": False, "cache_age_days": 0, "source": source}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
RECENT_PAST_DAYS = 25 # recent observations window (covers the ~2-week graded view)
|
|
|
|
|
FORECAST_DAYS = 8 # today + 7 days ahead
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def _rf_cache_path(cell_id: str) -> str:
|
|
|
|
|
return os.path.join(CACHE_DIR, f"{cell_id}_rf.parquet")
|
|
|
|
|
|
|
|
|
|
|
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21)
* Compress API responses and revalidate static assets instead of re-downloading
- Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x.
- Serve pages/assets with Cache-Control: no-cache instead of no-store, so
browsers revalidate via the ETag/Last-Modified that FileResponse and
StaticFiles already emit. Unchanged assets now cost an empty 304 rather
than a full transfer on every page navigation, while deploys still show
up immediately.
* Persist derived responses in SQLite so grading is computed once per cell, not per request
New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet
records — finished grade/calendar/day/forecast payloads and reverse-geocode
labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms)
becomes a ~5ms database read that survives restarts and is shared across views.
Raw parquet stays the source of truth; the store is a pure accelerator (every
reader falls back to recomputing on a miss, and deleting the db is a safe reset).
Freshness is token-driven, not clock-driven: each cached payload is validated by
a token encoding what it was computed from (payload schema version, the archive
record's end date, the recent-fetch stamp). The existing freshness drivers are
untouched — get_history still tops up the tail hourly and get_recent_forecast
still refetches hourly — and tokens are derived from what they return, so cached
payloads expire exactly when their inputs change. The same tokens double as weak
ETags: If-None-Match answers with an empty 304 without touching the payload.
- backend/store.py: derived-payload + revgeo tables, thread-local WAL conns,
every helper fail-soft.
- app.py: endpoints split into pure payload builders + HTTP/caching shells; the
in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store).
- climate.py: revgeo persisted through the store; recent_stamp() and
load_cached_history() (no-network read) helpers.
- backend/migrate.py + make migrate: idempotent, resumable backfill of the store
from existing parquet caches (default calendar span + latest-day detail +
revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather.
- grid.from_id(): rebuild a cell from its cache filename (migrate tooling).
* Add /api/v2/cell: one bundle carrying every view's payload
GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months)
and day (today) payloads in one response, each the exact payload its per-view
endpoint returns — built by the same builders and cached under the same
derived-store keys/tokens — paired with the etag that endpoint would emit. The
frontend can warm all views with a single request, seed its per-view cache from
the slices, and later revalidate each view individually with If-None-Match. The
bundle's own etag combines the slices', so an unchanged bundle is an empty 304.
prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard
guarantee: it never spends weather-API quota. A cell with no cached archive
answers 204, and only the history-derived slices (calendar + latest-day detail)
are built. At most one Nominatim lookup for a never-labeled cell.
* Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch
The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota,
which multi-year calendar payloads regularly blew through) to IndexedDB, with an
in-memory map in front. Entries carry the server's ETag, so anything stale
revalidates conditionally — unchanged data costs an empty 304 and a re-stamp,
never a re-transfer. On network failure the stale copy is served over an error.
getJSON gains an optional onUpdate callback opting into stale-while-revalidate:
the three views (weekly, day, calendar) now render a cached copy immediately —
spinners are delayed 150ms so warm loads never flash-blank — and repaint only if
background revalidation finds changed data. New data shows up the moment it
exists instead of waiting out a TTL.
Cross-view prefetch collapses from one request per view to a single /api/v2/cell
bundle, whose slices (exact per-view payloads + their etags) are seeded under the
URLs each view actually requests; the bundle call itself is conditional via a
remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side
with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one
possible Nominatim lookup each), so tapping nearby lands on already-graded data.
Legacy tg:* storage entries are cleared once; cache entries untouched for two
weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
|
|
|
def recent_stamp(cell_id: str) -> int:
|
|
|
|
|
"""Identity stamp (mtime, whole seconds) of the cell's cached recent+forecast
|
|
|
|
|
parquet; 0 when absent. Changes exactly when the recent/forecast data does, so
|
|
|
|
|
it's the freshness token for every payload that grades recent or future days."""
|
|
|
|
|
try:
|
|
|
|
|
return int(os.path.getmtime(_rf_cache_path(cell_id)))
|
|
|
|
|
except OSError:
|
|
|
|
|
return 0
|
|
|
|
|
|
|
|
|
|
|
2026-07-11 00:29:47 +00:00
|
|
|
def get_recent_forecast(cell: dict) -> pd.DataFrame:
|
|
|
|
|
"""Recent observations + forward forecast, with humidity as absolute humidity
|
|
|
|
|
(g/m³). Thin wrapper over the raw loader (see below)."""
|
|
|
|
|
df = _load_recent_forecast(cell)
|
|
|
|
|
_derive_humidity(df)
|
|
|
|
|
return df
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def _load_recent_forecast(cell: dict) -> pd.DataFrame:
|
|
|
|
|
"""Recent observations AND the forward forecast in ONE forecast-API call.
|
|
|
|
|
|
|
|
|
|
Both the recent (past) view and the forecast (future) view slice from this, so
|
|
|
|
|
a cell needs just one upstream forecast request per hour (plus the ~monthly
|
|
|
|
|
archive fetch). Cached per cell for one hour so it still follows model updates.
|
|
|
|
|
"""
|
|
|
|
|
path = _rf_cache_path(cell["id"])
|
|
|
|
|
if os.path.exists(path):
|
|
|
|
|
age_h = (time.time() - os.path.getmtime(path)) / 3600.0
|
|
|
|
|
if age_h < FORECAST_TTL_HOURS:
|
2026-07-11 20:05:57 +00:00
|
|
|
return _with_doy(pd.read_parquet(path))
|
2026-07-11 00:29:47 +00:00
|
|
|
|
|
|
|
|
params = {
|
|
|
|
|
"latitude": cell["center_lat"],
|
|
|
|
|
"longitude": cell["center_lon"],
|
|
|
|
|
"daily": DAILY_VARS,
|
|
|
|
|
"timezone": "auto",
|
|
|
|
|
"temperature_unit": "fahrenheit",
|
|
|
|
|
"precipitation_unit": "inch",
|
|
|
|
|
"wind_speed_unit": "mph",
|
|
|
|
|
"past_days": RECENT_PAST_DAYS,
|
|
|
|
|
"forecast_days": FORECAST_DAYS,
|
|
|
|
|
}
|
2026-07-11 19:53:46 +00:00
|
|
|
try:
|
|
|
|
|
r = _request(FORECAST_URL, params, 60, phase="recent_forecast_fetch")
|
|
|
|
|
except Exception as e: # noqa: BLE001
|
|
|
|
|
# Classify a forecast-API rate limit here so callers get the typed error
|
|
|
|
|
# (the archive path classifies its own inside _load_history).
|
|
|
|
|
if is_rate_limit(e):
|
|
|
|
|
daily = "daily" in _rate_limit_reason(e).lower()
|
|
|
|
|
raise WeatherUnavailable(limit_message(daily), daily=daily) from e
|
|
|
|
|
raise
|
2026-07-11 00:29:47 +00:00
|
|
|
df = _to_frame(r.json()["daily"])
|
2026-07-11 20:05:57 +00:00
|
|
|
_write_cache(df, path)
|
2026-07-11 00:29:47 +00:00
|
|
|
return df
|
|
|
|
|
|
|
|
|
|
|
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21)
* Compress API responses and revalidate static assets instead of re-downloading
- Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x.
- Serve pages/assets with Cache-Control: no-cache instead of no-store, so
browsers revalidate via the ETag/Last-Modified that FileResponse and
StaticFiles already emit. Unchanged assets now cost an empty 304 rather
than a full transfer on every page navigation, while deploys still show
up immediately.
* Persist derived responses in SQLite so grading is computed once per cell, not per request
New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet
records — finished grade/calendar/day/forecast payloads and reverse-geocode
labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms)
becomes a ~5ms database read that survives restarts and is shared across views.
Raw parquet stays the source of truth; the store is a pure accelerator (every
reader falls back to recomputing on a miss, and deleting the db is a safe reset).
Freshness is token-driven, not clock-driven: each cached payload is validated by
a token encoding what it was computed from (payload schema version, the archive
record's end date, the recent-fetch stamp). The existing freshness drivers are
untouched — get_history still tops up the tail hourly and get_recent_forecast
still refetches hourly — and tokens are derived from what they return, so cached
payloads expire exactly when their inputs change. The same tokens double as weak
ETags: If-None-Match answers with an empty 304 without touching the payload.
- backend/store.py: derived-payload + revgeo tables, thread-local WAL conns,
every helper fail-soft.
- app.py: endpoints split into pure payload builders + HTTP/caching shells; the
in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store).
- climate.py: revgeo persisted through the store; recent_stamp() and
load_cached_history() (no-network read) helpers.
- backend/migrate.py + make migrate: idempotent, resumable backfill of the store
from existing parquet caches (default calendar span + latest-day detail +
revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather.
- grid.from_id(): rebuild a cell from its cache filename (migrate tooling).
* Add /api/v2/cell: one bundle carrying every view's payload
GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months)
and day (today) payloads in one response, each the exact payload its per-view
endpoint returns — built by the same builders and cached under the same
derived-store keys/tokens — paired with the etag that endpoint would emit. The
frontend can warm all views with a single request, seed its per-view cache from
the slices, and later revalidate each view individually with If-None-Match. The
bundle's own etag combines the slices', so an unchanged bundle is an empty 304.
prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard
guarantee: it never spends weather-API quota. A cell with no cached archive
answers 204, and only the history-derived slices (calendar + latest-day detail)
are built. At most one Nominatim lookup for a never-labeled cell.
* Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch
The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota,
which multi-year calendar payloads regularly blew through) to IndexedDB, with an
in-memory map in front. Entries carry the server's ETag, so anything stale
revalidates conditionally — unchanged data costs an empty 304 and a re-stamp,
never a re-transfer. On network failure the stale copy is served over an error.
getJSON gains an optional onUpdate callback opting into stale-while-revalidate:
the three views (weekly, day, calendar) now render a cached copy immediately —
spinners are delayed 150ms so warm loads never flash-blank — and repaint only if
background revalidation finds changed data. New data shows up the moment it
exists instead of waiting out a TTL.
Cross-view prefetch collapses from one request per view to a single /api/v2/cell
bundle, whose slices (exact per-view payloads + their etags) are seeded under the
URLs each view actually requests; the bundle call itself is conditional via a
remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side
with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one
possible Nominatim lookup each), so tapping nearby lands on already-graded data.
Legacy tg:* storage entries are cleared once; cache entries untouched for two
weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
|
|
|
_REVGEO_CACHE: dict[str, str | None] = {}
|
|
|
|
|
|
2026-07-11 09:03:57 +00:00
|
|
|
# Nominatim's usage policy allows ~1 request/second and rejects bursts. The compare
|
|
|
|
|
# page loads several locations at once, so serialize the reverse lookups here and
|
|
|
|
|
# space them out — otherwise the burst is rate-limited, a null label gets cached, and
|
|
|
|
|
# those locations are left showing bare coordinates.
|
|
|
|
|
_REVGEO_LOCK = threading.Lock()
|
|
|
|
|
_REVGEO_MIN_INTERVAL = 1.1 # seconds between successive Nominatim reverse calls
|
|
|
|
|
_revgeo_last = 0.0
|
|
|
|
|
|
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21)
* Compress API responses and revalidate static assets instead of re-downloading
- Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x.
- Serve pages/assets with Cache-Control: no-cache instead of no-store, so
browsers revalidate via the ETag/Last-Modified that FileResponse and
StaticFiles already emit. Unchanged assets now cost an empty 304 rather
than a full transfer on every page navigation, while deploys still show
up immediately.
* Persist derived responses in SQLite so grading is computed once per cell, not per request
New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet
records — finished grade/calendar/day/forecast payloads and reverse-geocode
labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms)
becomes a ~5ms database read that survives restarts and is shared across views.
Raw parquet stays the source of truth; the store is a pure accelerator (every
reader falls back to recomputing on a miss, and deleting the db is a safe reset).
Freshness is token-driven, not clock-driven: each cached payload is validated by
a token encoding what it was computed from (payload schema version, the archive
record's end date, the recent-fetch stamp). The existing freshness drivers are
untouched — get_history still tops up the tail hourly and get_recent_forecast
still refetches hourly — and tokens are derived from what they return, so cached
payloads expire exactly when their inputs change. The same tokens double as weak
ETags: If-None-Match answers with an empty 304 without touching the payload.
- backend/store.py: derived-payload + revgeo tables, thread-local WAL conns,
every helper fail-soft.
- app.py: endpoints split into pure payload builders + HTTP/caching shells; the
in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store).
- climate.py: revgeo persisted through the store; recent_stamp() and
load_cached_history() (no-network read) helpers.
- backend/migrate.py + make migrate: idempotent, resumable backfill of the store
from existing parquet caches (default calendar span + latest-day detail +
revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather.
- grid.from_id(): rebuild a cell from its cache filename (migrate tooling).
* Add /api/v2/cell: one bundle carrying every view's payload
GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months)
and day (today) payloads in one response, each the exact payload its per-view
endpoint returns — built by the same builders and cached under the same
derived-store keys/tokens — paired with the etag that endpoint would emit. The
frontend can warm all views with a single request, seed its per-view cache from
the slices, and later revalidate each view individually with If-None-Match. The
bundle's own etag combines the slices', so an unchanged bundle is an empty 304.
prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard
guarantee: it never spends weather-API quota. A cell with no cached archive
answers 204, and only the history-derived slices (calendar + latest-day detail)
are built. At most one Nominatim lookup for a never-labeled cell.
* Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch
The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota,
which multi-year calendar payloads regularly blew through) to IndexedDB, with an
in-memory map in front. Entries carry the server's ETag, so anything stale
revalidates conditionally — unchanged data costs an empty 304 and a re-stamp,
never a re-transfer. On network failure the stale copy is served over an error.
getJSON gains an optional onUpdate callback opting into stale-while-revalidate:
the three views (weekly, day, calendar) now render a cached copy immediately —
spinners are delayed 150ms so warm loads never flash-blank — and repaint only if
background revalidation finds changed data. New data shows up the moment it
exists instead of waiting out a TTL.
Cross-view prefetch collapses from one request per view to a single /api/v2/cell
bundle, whose slices (exact per-view payloads + their etags) are seeded under the
URLs each view actually requests; the bundle call itself is conditional via a
remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side
with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one
possible Nominatim lookup each), so tapping nearby lands on already-graded data.
Legacy tg:* storage entries are cleared once; cache entries untouched for two
weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
|
|
|
|
|
|
|
|
def reverse_geocode_cached(lat: float, lon: float) -> tuple[bool, str | None]:
|
|
|
|
|
"""(found, label) from the in-memory + SQLite revgeo caches only — never calls
|
|
|
|
|
Nominatim. Used by paths that must not add upstream traffic (prefetch)."""
|
|
|
|
|
key = store.revgeo_key(lat, lon)
|
|
|
|
|
if key in _REVGEO_CACHE:
|
|
|
|
|
return (True, _REVGEO_CACHE[key])
|
|
|
|
|
found, label = store.get_revgeo(key)
|
|
|
|
|
if found:
|
|
|
|
|
_REVGEO_CACHE[key] = label
|
|
|
|
|
return (found, label)
|
2026-07-11 00:29:47 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
def reverse_geocode(lat: float, lon: float) -> str | None:
|
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21)
* Compress API responses and revalidate static assets instead of re-downloading
- Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x.
- Serve pages/assets with Cache-Control: no-cache instead of no-store, so
browsers revalidate via the ETag/Last-Modified that FileResponse and
StaticFiles already emit. Unchanged assets now cost an empty 304 rather
than a full transfer on every page navigation, while deploys still show
up immediately.
* Persist derived responses in SQLite so grading is computed once per cell, not per request
New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet
records — finished grade/calendar/day/forecast payloads and reverse-geocode
labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms)
becomes a ~5ms database read that survives restarts and is shared across views.
Raw parquet stays the source of truth; the store is a pure accelerator (every
reader falls back to recomputing on a miss, and deleting the db is a safe reset).
Freshness is token-driven, not clock-driven: each cached payload is validated by
a token encoding what it was computed from (payload schema version, the archive
record's end date, the recent-fetch stamp). The existing freshness drivers are
untouched — get_history still tops up the tail hourly and get_recent_forecast
still refetches hourly — and tokens are derived from what they return, so cached
payloads expire exactly when their inputs change. The same tokens double as weak
ETags: If-None-Match answers with an empty 304 without touching the payload.
- backend/store.py: derived-payload + revgeo tables, thread-local WAL conns,
every helper fail-soft.
- app.py: endpoints split into pure payload builders + HTTP/caching shells; the
in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store).
- climate.py: revgeo persisted through the store; recent_stamp() and
load_cached_history() (no-network read) helpers.
- backend/migrate.py + make migrate: idempotent, resumable backfill of the store
from existing parquet caches (default calendar span + latest-day detail +
revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather.
- grid.from_id(): rebuild a cell from its cache filename (migrate tooling).
* Add /api/v2/cell: one bundle carrying every view's payload
GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months)
and day (today) payloads in one response, each the exact payload its per-view
endpoint returns — built by the same builders and cached under the same
derived-store keys/tokens — paired with the etag that endpoint would emit. The
frontend can warm all views with a single request, seed its per-view cache from
the slices, and later revalidate each view individually with If-None-Match. The
bundle's own etag combines the slices', so an unchanged bundle is an empty 304.
prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard
guarantee: it never spends weather-API quota. A cell with no cached archive
answers 204, and only the history-derived slices (calendar + latest-day detail)
are built. At most one Nominatim lookup for a never-labeled cell.
* Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch
The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota,
which multi-year calendar payloads regularly blew through) to IndexedDB, with an
in-memory map in front. Entries carry the server's ETag, so anything stale
revalidates conditionally — unchanged data costs an empty 304 and a re-stamp,
never a re-transfer. On network failure the stale copy is served over an error.
getJSON gains an optional onUpdate callback opting into stale-while-revalidate:
the three views (weekly, day, calendar) now render a cached copy immediately —
spinners are delayed 150ms so warm loads never flash-blank — and repaint only if
background revalidation finds changed data. New data shows up the moment it
exists instead of waiting out a TTL.
Cross-view prefetch collapses from one request per view to a single /api/v2/cell
bundle, whose slices (exact per-view payloads + their etags) are seeded under the
URLs each view actually requests; the bundle call itself is conditional via a
remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side
with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one
possible Nominatim lookup each), so tapping nearby lands on already-graded data.
Legacy tg:* storage entries are cleared once; cache entries untouched for two
weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
|
|
|
"""Best-effort neighbourhood/city label for a point (OpenStreetMap Nominatim).
|
2026-07-11 00:29:47 +00:00
|
|
|
|
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21)
* Compress API responses and revalidate static assets instead of re-downloading
- Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x.
- Serve pages/assets with Cache-Control: no-cache instead of no-store, so
browsers revalidate via the ETag/Last-Modified that FileResponse and
StaticFiles already emit. Unchanged assets now cost an empty 304 rather
than a full transfer on every page navigation, while deploys still show
up immediately.
* Persist derived responses in SQLite so grading is computed once per cell, not per request
New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet
records — finished grade/calendar/day/forecast payloads and reverse-geocode
labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms)
becomes a ~5ms database read that survives restarts and is shared across views.
Raw parquet stays the source of truth; the store is a pure accelerator (every
reader falls back to recomputing on a miss, and deleting the db is a safe reset).
Freshness is token-driven, not clock-driven: each cached payload is validated by
a token encoding what it was computed from (payload schema version, the archive
record's end date, the recent-fetch stamp). The existing freshness drivers are
untouched — get_history still tops up the tail hourly and get_recent_forecast
still refetches hourly — and tokens are derived from what they return, so cached
payloads expire exactly when their inputs change. The same tokens double as weak
ETags: If-None-Match answers with an empty 304 without touching the payload.
- backend/store.py: derived-payload + revgeo tables, thread-local WAL conns,
every helper fail-soft.
- app.py: endpoints split into pure payload builders + HTTP/caching shells; the
in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store).
- climate.py: revgeo persisted through the store; recent_stamp() and
load_cached_history() (no-network read) helpers.
- backend/migrate.py + make migrate: idempotent, resumable backfill of the store
from existing parquet caches (default calendar span + latest-day detail +
revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather.
- grid.from_id(): rebuild a cell from its cache filename (migrate tooling).
* Add /api/v2/cell: one bundle carrying every view's payload
GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months)
and day (today) payloads in one response, each the exact payload its per-view
endpoint returns — built by the same builders and cached under the same
derived-store keys/tokens — paired with the etag that endpoint would emit. The
frontend can warm all views with a single request, seed its per-view cache from
the slices, and later revalidate each view individually with If-None-Match. The
bundle's own etag combines the slices', so an unchanged bundle is an empty 304.
prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard
guarantee: it never spends weather-API quota. A cell with no cached archive
answers 204, and only the history-derived slices (calendar + latest-day detail)
are built. At most one Nominatim lookup for a never-labeled cell.
* Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch
The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota,
which multi-year calendar payloads regularly blew through) to IndexedDB, with an
in-memory map in front. Entries carry the server's ETag, so anything stale
revalidates conditionally — unchanged data costs an empty 304 and a re-stamp,
never a re-transfer. On network failure the stale copy is served over an error.
getJSON gains an optional onUpdate callback opting into stale-while-revalidate:
the three views (weekly, day, calendar) now render a cached copy immediately —
spinners are delayed 150ms so warm loads never flash-blank — and repaint only if
background revalidation finds changed data. New data shows up the moment it
exists instead of waiting out a TTL.
Cross-view prefetch collapses from one request per view to a single /api/v2/cell
bundle, whose slices (exact per-view payloads + their etags) are seeded under the
URLs each view actually requests; the bundle call itself is conditional via a
remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side
with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one
possible Nominatim lookup each), so tapping nearby lands on already-graded data.
Legacy tg:* storage entries are cleared once; cache entries untouched for two
weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
|
|
|
Cached per ~cell — in memory for the process and in SQLite across restarts —
|
|
|
|
|
so panning around (or redeploying) doesn't re-hit the service, and failures
|
2026-07-11 00:29:47 +00:00
|
|
|
return None so the caller can fall back to bare coordinates.
|
|
|
|
|
"""
|
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21)
* Compress API responses and revalidate static assets instead of re-downloading
- Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x.
- Serve pages/assets with Cache-Control: no-cache instead of no-store, so
browsers revalidate via the ETag/Last-Modified that FileResponse and
StaticFiles already emit. Unchanged assets now cost an empty 304 rather
than a full transfer on every page navigation, while deploys still show
up immediately.
* Persist derived responses in SQLite so grading is computed once per cell, not per request
New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet
records — finished grade/calendar/day/forecast payloads and reverse-geocode
labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms)
becomes a ~5ms database read that survives restarts and is shared across views.
Raw parquet stays the source of truth; the store is a pure accelerator (every
reader falls back to recomputing on a miss, and deleting the db is a safe reset).
Freshness is token-driven, not clock-driven: each cached payload is validated by
a token encoding what it was computed from (payload schema version, the archive
record's end date, the recent-fetch stamp). The existing freshness drivers are
untouched — get_history still tops up the tail hourly and get_recent_forecast
still refetches hourly — and tokens are derived from what they return, so cached
payloads expire exactly when their inputs change. The same tokens double as weak
ETags: If-None-Match answers with an empty 304 without touching the payload.
- backend/store.py: derived-payload + revgeo tables, thread-local WAL conns,
every helper fail-soft.
- app.py: endpoints split into pure payload builders + HTTP/caching shells; the
in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store).
- climate.py: revgeo persisted through the store; recent_stamp() and
load_cached_history() (no-network read) helpers.
- backend/migrate.py + make migrate: idempotent, resumable backfill of the store
from existing parquet caches (default calendar span + latest-day detail +
revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather.
- grid.from_id(): rebuild a cell from its cache filename (migrate tooling).
* Add /api/v2/cell: one bundle carrying every view's payload
GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months)
and day (today) payloads in one response, each the exact payload its per-view
endpoint returns — built by the same builders and cached under the same
derived-store keys/tokens — paired with the etag that endpoint would emit. The
frontend can warm all views with a single request, seed its per-view cache from
the slices, and later revalidate each view individually with If-None-Match. The
bundle's own etag combines the slices', so an unchanged bundle is an empty 304.
prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard
guarantee: it never spends weather-API quota. A cell with no cached archive
answers 204, and only the history-derived slices (calendar + latest-day detail)
are built. At most one Nominatim lookup for a never-labeled cell.
* Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch
The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota,
which multi-year calendar payloads regularly blew through) to IndexedDB, with an
in-memory map in front. Entries carry the server's ETag, so anything stale
revalidates conditionally — unchanged data costs an empty 304 and a re-stamp,
never a re-transfer. On network failure the stale copy is served over an error.
getJSON gains an optional onUpdate callback opting into stale-while-revalidate:
the three views (weekly, day, calendar) now render a cached copy immediately —
spinners are delayed 150ms so warm loads never flash-blank — and repaint only if
background revalidation finds changed data. New data shows up the moment it
exists instead of waiting out a TTL.
Cross-view prefetch collapses from one request per view to a single /api/v2/cell
bundle, whose slices (exact per-view payloads + their etags) are seeded under the
URLs each view actually requests; the bundle call itself is conditional via a
remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side
with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one
possible Nominatim lookup each), so tapping nearby lands on already-graded data.
Legacy tg:* storage entries are cleared once; cache entries untouched for two
weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
|
|
|
key = store.revgeo_key(lat, lon)
|
|
|
|
|
found, label = reverse_geocode_cached(lat, lon)
|
|
|
|
|
if found:
|
|
|
|
|
return label
|
2026-07-11 09:03:57 +00:00
|
|
|
# Serialize + rate-limit the upstream call. Concurrent callers (the compare page
|
|
|
|
|
# loads several locations at once) would otherwise burst past Nominatim's ~1/sec
|
|
|
|
|
# limit and get rate-limited, caching a null label. Under the lock we re-check the
|
|
|
|
|
# cache (a peer may have just resolved this cell) and space successive calls out.
|
|
|
|
|
global _revgeo_last
|
|
|
|
|
with _REVGEO_LOCK:
|
|
|
|
|
found, label = reverse_geocode_cached(lat, lon)
|
|
|
|
|
if found:
|
|
|
|
|
return label
|
|
|
|
|
wait = _REVGEO_MIN_INTERVAL - (time.monotonic() - _revgeo_last)
|
|
|
|
|
if wait > 0:
|
|
|
|
|
time.sleep(wait)
|
2026-07-11 00:29:47 +00:00
|
|
|
label = None
|
2026-07-11 09:03:57 +00:00
|
|
|
try:
|
|
|
|
|
r = _request(
|
|
|
|
|
"https://nominatim.openstreetmap.org/reverse",
|
|
|
|
|
# zoom 14 resolves to the suburb/neighbourhood level so we can lead
|
|
|
|
|
# with it when OSM has one (zoom 10 only ever returns the city).
|
Worldwide coverage: grade any point on Earth (#32)
Remove the US+Canada bounding box so every endpoint accepts any lat/lon.
The grading pipeline was already global-ready (ERA5 archive, timezone=auto,
day-of-year climatology), so opening it up is mostly deleting the guard —
plus the edge cases that only exist once the whole globe is in play:
- grid.py: snap() wraps longitude into [-180, 180) and clamps latitude, and
cell centers are normalized so the polar row and the cells straddling the
antimeridian always report valid coordinates to the weather/geocoding APIs.
snap() and from_id() now share one _cell() builder, making id round-trips
exact by construction (verified with a 300k-point global sweep).
- nav.js: neighbor-cell prefetch skips rows past the poles and wraps
longitudes across the dateline instead of sending out-of-range queries.
- Nominatim reverse geocoding requests accept-language=en so place labels
render in one script worldwide (matching the forward geocoder).
- mappicker: search suggestions are no longer filtered to US/CA, the
placeholder and default map view are worldwide.
- calendar: season filter labels flip for southern-hemisphere locations
(Dec-Feb shows as Summer); the underlying month groups are unchanged, so
saved filter selections keep meaning the same months.
Verified end-to-end on a scratch server: Tokyo and Sydney grade with real
labels, a Fiji cell on the antimeridian's east edge builds and serves warm
hits from the derived store, and prefetch=1 on a cold cell still answers
204 without spending weather-API quota.
2026-07-11 15:02:28 +00:00
|
|
|
# accept-language=en keeps labels in one script worldwide (matches
|
|
|
|
|
# the forward geocoder's language=en).
|
2026-07-11 09:03:57 +00:00
|
|
|
{"lat": lat, "lon": lon, "format": "jsonv2", "zoom": 14,
|
Worldwide coverage: grade any point on Earth (#32)
Remove the US+Canada bounding box so every endpoint accepts any lat/lon.
The grading pipeline was already global-ready (ERA5 archive, timezone=auto,
day-of-year climatology), so opening it up is mostly deleting the guard —
plus the edge cases that only exist once the whole globe is in play:
- grid.py: snap() wraps longitude into [-180, 180) and clamps latitude, and
cell centers are normalized so the polar row and the cells straddling the
antimeridian always report valid coordinates to the weather/geocoding APIs.
snap() and from_id() now share one _cell() builder, making id round-trips
exact by construction (verified with a 300k-point global sweep).
- nav.js: neighbor-cell prefetch skips rows past the poles and wraps
longitudes across the dateline instead of sending out-of-range queries.
- Nominatim reverse geocoding requests accept-language=en so place labels
render in one script worldwide (matching the forward geocoder).
- mappicker: search suggestions are no longer filtered to US/CA, the
placeholder and default map view are worldwide.
- calendar: season filter labels flip for southern-hemisphere locations
(Dec-Feb shows as Summer); the underlying month groups are unchanged, so
saved filter selections keep meaning the same months.
Verified end-to-end on a scratch server: Tokyo and Sydney grade with real
labels, a Fiji cell on the antimeridian's east edge builds and serves warm
hits from the derived store, and prefetch=1 on a cold cell still answers
204 without spending weather-API quota.
2026-07-11 15:02:28 +00:00
|
|
|
"addressdetails": 1, "accept-language": "en"},
|
2026-07-11 09:03:57 +00:00
|
|
|
15,
|
|
|
|
|
phase="reverse_geocode",
|
|
|
|
|
headers={"User-Agent": "Thermograph/0.1 (local weather grading app)"},
|
|
|
|
|
)
|
|
|
|
|
a = r.json().get("address", {}) or {}
|
|
|
|
|
# Lead with the neighbourhood when available, then the city, then region.
|
|
|
|
|
neighborhood = (a.get("neighbourhood") or a.get("suburb")
|
|
|
|
|
or a.get("quarter") or a.get("city_district")
|
|
|
|
|
or a.get("borough"))
|
|
|
|
|
city = (a.get("city") or a.get("town") or a.get("village")
|
|
|
|
|
or a.get("hamlet") or a.get("county"))
|
|
|
|
|
region = a.get("state") or a.get("province") or a.get("region")
|
|
|
|
|
# dict.fromkeys drops duplicates (e.g. neighbourhood == city) in order.
|
|
|
|
|
parts = dict.fromkeys(p for p in (neighborhood, city, region) if p)
|
|
|
|
|
label = ", ".join(parts) or None
|
|
|
|
|
except Exception: # noqa: BLE001 - reverse geocoding is a nicety, never fatal
|
|
|
|
|
label = None
|
|
|
|
|
_revgeo_last = time.monotonic()
|
|
|
|
|
_REVGEO_CACHE[key] = label
|
|
|
|
|
store.put_revgeo(key, label) # survive restarts (a None label retries after its TTL)
|
|
|
|
|
return label
|
2026-07-11 00:29:47 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
def geocode(name: str, count: int = 5) -> list[dict]:
|
Worldwide coverage: grade any point on Earth (#32)
Remove the US+Canada bounding box so every endpoint accepts any lat/lon.
The grading pipeline was already global-ready (ERA5 archive, timezone=auto,
day-of-year climatology), so opening it up is mostly deleting the guard —
plus the edge cases that only exist once the whole globe is in play:
- grid.py: snap() wraps longitude into [-180, 180) and clamps latitude, and
cell centers are normalized so the polar row and the cells straddling the
antimeridian always report valid coordinates to the weather/geocoding APIs.
snap() and from_id() now share one _cell() builder, making id round-trips
exact by construction (verified with a 300k-point global sweep).
- nav.js: neighbor-cell prefetch skips rows past the poles and wraps
longitudes across the dateline instead of sending out-of-range queries.
- Nominatim reverse geocoding requests accept-language=en so place labels
render in one script worldwide (matching the forward geocoder).
- mappicker: search suggestions are no longer filtered to US/CA, the
placeholder and default map view are worldwide.
- calendar: season filter labels flip for southern-hemisphere locations
(Dec-Feb shows as Summer); the underlying month groups are unchanged, so
saved filter selections keep meaning the same months.
Verified end-to-end on a scratch server: Tokyo and Sydney grade with real
labels, a Fiji cell on the antimeridian's east edge builds and serves warm
hits from the derived store, and prefetch=1 on a cold cell still answers
204 without spending weather-API quota.
2026-07-11 15:02:28 +00:00
|
|
|
"""Look up places by name worldwide via Open-Meteo's geocoder."""
|
2026-07-11 00:29:47 +00:00
|
|
|
r = _request(
|
|
|
|
|
"https://geocoding-api.open-meteo.com/v1/search",
|
|
|
|
|
{"name": name, "count": count, "language": "en", "format": "json"},
|
|
|
|
|
30,
|
|
|
|
|
phase="geocode",
|
|
|
|
|
)
|
|
|
|
|
results = r.json().get("results", []) or []
|
|
|
|
|
return [
|
|
|
|
|
{
|
|
|
|
|
"name": g.get("name"),
|
|
|
|
|
"admin1": g.get("admin1"),
|
|
|
|
|
"country": g.get("country"),
|
|
|
|
|
"country_code": g.get("country_code"),
|
|
|
|
|
"lat": g.get("latitude"),
|
|
|
|
|
"lon": g.get("longitude"),
|
Typo-tolerant location search suggestions (#33)
Add /api/v2/suggest and wire the location picker's search box to it as a
debounced type-ahead: top-5 place suggestions that tolerate a single-letter
typo (substituted, missing, or extra letter, or two adjacent letters swapped)
anywhere in the query, including the first character.
- backend/places.py: local place index built from a GeoNames cities dump
(downloaded once into data/geonames/, loaded in a background thread; the
app boots and serves without it). Exact-prefix matches rank first by
population, then names one edit away; a token vocabulary respells one
mistyped word against known place-name tokens ("pest seattle" ->
"west seattle") for retry against the upstream geocoder, which covers
neighbourhood-level places the dump lacks. THERMOGRAPH_CITIES picks the
dump (default cities1000).
- /suggest blends local and upstream results by population with an exactness
boost, so "Seatle" (a Cumbrian hamlet) can't outrank Seattle when the
query is one edit from the city, while "munchen" still surfaces Munich via
upstream (the index only knows English names). Upstream lookups are
memoized and skipped entirely when the index answers convincingly.
- mappicker.js: debounced (250ms) live suggestions with abort + sequence
guards against stale responses, arrow-key navigation, Enter-picks-highlight,
Escape dismissing the list before closing the overlay. Submit goes through
the same typo-tolerant endpoint.
- climate.geocode results now carry population (used for ranking).
2026-07-11 15:36:14 +00:00
|
|
|
"population": g.get("population"),
|
2026-07-11 00:29:47 +00:00
|
|
|
}
|
|
|
|
|
for g in results
|
|
|
|
|
]
|