Ship the data-pipeline chain: local geocoding, Meteostat gusts, NASA POWER primary, ERA5 seed

admin_emi commented

2026-07-23 16:07:52 +00:00

Owner

The four stacked feature branches as one chain (each contains the previous): forward geocoding moved off Open-Meteo to the local index + Nominatim; Meteostat gust supplier for gust-less backup sources; history primary flipped to NASA POWER with Open-Meteo as fallback; one-time ERA5 seed for curated-city cells. Merged with dev (CI tag-keying fixes), full backend suite green locally: 368 passed / 7 skipped.

admin_emi added 9 commits 2026-07-23 16:07:53 +00:00

Move forward geocoding off Open-Meteo to local index + Nominatim db86277103

/suggest is now served purely from the local GeoNames index — no external
geocoder call per keystroke. /geocode answers from the local index first and
falls back to OSM Nominatim /search only on a miss (or while the index is still
loading), covering the neighbourhood/postcode/tiny-village/native-name long tail
the cities dump lacks.

Nominatim shares the reverse geocoder's lock and ~1/sec pacing since both hit
the same host, and only the low-volume /geocode miss path reaches it. Removes
the Open-Meteo geocoding dependency entirely; the "geocode" metrics phase now
maps to the nominatim source.

Add Meteostat gust supplier for the gust-less backup sources b4d8d67825

NASA POWER (history) and MET Norway (forecast) carry no wind gusts, but gust is
a graded metric. This adds data/meteostat.py, which finds the nearest Meteostat
station to a cell and reads its daily peak gust (wpgt) from the keyless gzipped
bulk endpoints, filling the gust column on the NASA POWER history frame. Where no
station is within ~100 km it estimates from sustained wind (wind * GUST_FACTOR).

Bulk files (station list + per-station daily) are cached on disk so steady-state
network IO is near zero, and any lookup/fetch failure degrades to pure estimation
so a history fetch never fails. A constant estimate factor makes an estimated
gust redundant with wind, so real signal comes only where a station backs it.

Activates once NASA POWER becomes the primary history source; Open-Meteo already
carries its own gusts. New history_gust / meteostat_stations metrics phases map
to the meteostat source.

Flip history primary to NASA POWER, Open-Meteo to fallback 30153dbc64

The live history path now tries NASA POWER first (keyless, independent of
Open-Meteo) and falls back to the Open-Meteo archive only when NASA is
unavailable — the reverse of before. Gusts NASA lacks are filled from Meteostat
inside _fetch_history_nasa (added in the prior change). Both sources must still
return a plausibly-full span (MIN_ARCHIVE_DAYS) before being cached as complete;
a short/partial response is rejected and the other source is tried.

_fetch_history_nasa now accepts a start/end range so it also serves the recent
tail top-up: _topup_tail fetches NASA-first via the new _fetch_history_tail
helper (Open-Meteo range as fallback, skipped during its rate-limit cooldown),
removing the last per-active-cell Open-Meteo call from the history path. The
Open-Meteo cooldown no longer gates the top-up, since NASA is not subject to it.

Open-Meteo remains wired as the fallback (deletion is a later step). Tests
updated to the new source order plus tail-fetch coverage.

Add one-time ERA5 seed for curated-city cells (keyless Icechunk) 24d8cd9ba1

seed_era5.py backfills the curated-city cells with true ERA5 data from the
Earthmover Icechunk ERA5 archive on AWS Open Data (anonymous, keyless), so
high-traffic pages keep ERA5 fidelity now that NASA POWER is the live primary.
It aggregates the hourly ERA5 point series to the daily store schema in polars
(no pandas), derives feels-like via the existing heat-index/wind-chill path, and
writes straight to the history store via climate._write_history_backed, bypassing
the live fetch. ERA5 carries real gusts (i10fg), so seeded cells keep measured
gusts rather than the Meteostat fallback.

The icechunk/xarray/zarr stack is isolated in requirements-seed.txt (not the app
image or CI) and lazy-imported, so the module and its unit-tested hourly->daily
transform load without those deps. The S3 access layer targets a young API and is
verified via `--dry-run` on the seed box, not in tests. Idempotent by default
(skips already-cached cells; --overwrite to reseed).

Subtree-merge thermograph-backend geocode-local into backend/ 6eff3ba3a7

Subtree-merge thermograph-backend history-meteostat-gusts into backend/ 8301d5c7d9

Conflict resolved (UNVERIFIED -- tests not runnable locally, docker blocked):
data/climate.py combines dev's hardened _request signature
(ARCHIVE_FETCH_TIMEOUTS + phase) with this branch's Meteostat gust-fill wrap.
Same conflict exists merging this branch into the split repo's dev.

# Conflicts:
#	backend/data/climate.py

Subtree-merge thermograph-backend nasa-primary-flip into backend/ ffcb25885f

Conflict resolved (UNVERIFIED -- tests not runnable locally, docker blocked):
tests/data/test_climate.py keeps BOTH appended test blocks (dev's
forecast-cooldown tests + this branch's NASA-primary tests). Same conflict
exists merging this branch into the split repo's dev.

# Conflicts:
#	backend/tests/data/test_climate.py

Subtree-merge thermograph-backend era5-seed into backend/ 9567c51783

Merge dev into the data-pipeline chain (CI tag-keying fixes)

PR build (required check) / changes (pull_request) Successful in 7s

Details

secrets-guard / encrypted (pull_request) Successful in 6s

Details

PR build (required check) / build-backend (pull_request) Has been skipped

Details

PR build (required check) / build-frontend (pull_request) Has been skipped

Details

PR build (required check) / validate-observability (pull_request) Has been skipped

Details