backend/drift_check.py quantifies the migration's data impact by fetching both
live history sources (NASA POWER primary + the Open-Meteo/ERA5 fallback) for a
sample of cells and reporting, per variable: mean-absolute-difference / bias / max
(raw drift) and grade-band divergence (the share of days whose graded band actually
changes). Since Open-Meteo is ERA5, this doubles as a fidelity check for the ERA5
seed.
Because grades are percentiles within each source's own distribution, a uniform
bias moves MAD but not grades — so the two numbers are read together. Open-Meteo is
kept as a dormant fallback (not deleted), so both sources stay fetchable for this
comparison. The comparison logic is unit-tested; the per-cell fetch runs on a
networked box (python drift_check.py [--limit N] [--days N]).
The recent-observations + forward-forecast bundle is now built without Open-Meteo:
the recent observed window comes from a NASA POWER range (measured, via the range
fetch added for the history flip) and the forward days from MET Norway, merged by
date. Meteostat fills the gusts neither source carries — measured for the observed
days, estimated from wind for the forecast days.
Open-Meteo's forecast API is demoted to the fallback, still gated by its existing
_forecast_cooldown_until rate-limit cooldown; then a stale cache. MET Norway
switched to /compact (same fields, smaller payload) and the bundle TTL relaxed
1h -> 4h. NASA's near-real-time lag leaves a 1-2 day recent-edge gap that grades
as missing, bracketed by history behind and forecast ahead.
Tests updated to the new source order (NASA+MET merge, Open-Meteo fallback, and
the forecast cooldown now gating that fallback); deletion of Open-Meteo is not
done — it stays as the dormant fallback.
Conflict resolved (UNVERIFIED -- tests not runnable locally, docker blocked):
tests/data/test_climate.py keeps BOTH appended test blocks (dev's
forecast-cooldown tests + this branch's NASA-primary tests). Same conflict
exists merging this branch into the split repo's dev.
# Conflicts:
# backend/tests/data/test_climate.py
seed_era5.py backfills the curated-city cells with true ERA5 data from the
Earthmover Icechunk ERA5 archive on AWS Open Data (anonymous, keyless), so
high-traffic pages keep ERA5 fidelity now that NASA POWER is the live primary.
It aggregates the hourly ERA5 point series to the daily store schema in polars
(no pandas), derives feels-like via the existing heat-index/wind-chill path, and
writes straight to the history store via climate._write_history_backed, bypassing
the live fetch. ERA5 carries real gusts (i10fg), so seeded cells keep measured
gusts rather than the Meteostat fallback.
The icechunk/xarray/zarr stack is isolated in requirements-seed.txt (not the app
image or CI) and lazy-imported, so the module and its unit-tested hourly->daily
transform load without those deps. The S3 access layer targets a young API and is
verified via `--dry-run` on the seed box, not in tests. Idempotent by default
(skips already-cached cells; --overwrite to reseed).
The live history path now tries NASA POWER first (keyless, independent of
Open-Meteo) and falls back to the Open-Meteo archive only when NASA is
unavailable — the reverse of before. Gusts NASA lacks are filled from Meteostat
inside _fetch_history_nasa (added in the prior change). Both sources must still
return a plausibly-full span (MIN_ARCHIVE_DAYS) before being cached as complete;
a short/partial response is rejected and the other source is tried.
_fetch_history_nasa now accepts a start/end range so it also serves the recent
tail top-up: _topup_tail fetches NASA-first via the new _fetch_history_tail
helper (Open-Meteo range as fallback, skipped during its rate-limit cooldown),
removing the last per-active-cell Open-Meteo call from the history path. The
Open-Meteo cooldown no longer gates the top-up, since NASA is not subject to it.
Open-Meteo remains wired as the fallback (deletion is a later step). Tests
updated to the new source order plus tail-fetch coverage.
NASA POWER (history) and MET Norway (forecast) carry no wind gusts, but gust is
a graded metric. This adds data/meteostat.py, which finds the nearest Meteostat
station to a cell and reads its daily peak gust (wpgt) from the keyless gzipped
bulk endpoints, filling the gust column on the NASA POWER history frame. Where no
station is within ~100 km it estimates from sustained wind (wind * GUST_FACTOR).
Bulk files (station list + per-station daily) are cached on disk so steady-state
network IO is near zero, and any lookup/fetch failure degrades to pure estimation
so a history fetch never fails. A constant estimate factor makes an estimated
gust redundant with wind, so real signal comes only where a station backs it.
Activates once NASA POWER becomes the primary history source; Open-Meteo already
carries its own gusts. New history_gust / meteostat_stations metrics phases map
to the meteostat source.
/suggest is now served purely from the local GeoNames index — no external
geocoder call per keystroke. /geocode answers from the local index first and
falls back to OSM Nominatim /search only on a miss (or while the index is still
loading), covering the neighbourhood/postcode/tiny-village/native-name long tail
the cities dump lacks.
Nominatim shares the reverse geocoder's lock and ~1/sec pacing since both hit
the same host, and only the low-volume /geocode miss path reaches it. Removes
the Open-Meteo geocoding dependency entirely; the "geocode" metrics phase now
maps to the nominatim source.
