Steady-state usage of the Open-Meteo API is removed; it remains only as a dormant fallback. Geocoding -> local GeoNames + Nominatim; wind gusts -> Meteostat; history -> NASA POWER (curated cells seeded with keyless ERA5); recent+forecast -> NASA range + MET Norway. Plus a drift-check tool comparing NASA vs Open-Meteo. All keyless, no new infra. Backend suite green in-image (gate).
Conflict resolved (UNVERIFIED -- tests not runnable locally, docker blocked):
tests/data/test_climate.py keeps BOTH appended test blocks (dev's
forecast-cooldown tests + this branch's NASA-primary tests). Same conflict
exists merging this branch into the split repo's dev.
# Conflicts:
# backend/tests/data/test_climate.py
Backend runs its full hermetic suite, frontend its unit tier, inside the
just-built image -- the dev-only feature both app repos carried in their own
build.yml (deleted here in favor of the root workflow).
/opt/thermograph becomes a monorepo checkout: deploy.sh gains INFRA_DIR (git
ops at the root, compose work cd'd into infra/), lock/tags/render paths move
under infra/deploy/, image-path defaults become emi/thermograph/backend|
frontend across compose, stack, and the tag-prune. docker-compose.yml pins
name: thermograph -- without it compose run from .../infra derives project
"infra" and recreates the whole stack beside the running one;
deploy-dev.sh pins COMPOSE_PROJECT_NAME=thermograph-dev (env wins over the
file key) to keep LAN dev's separate project.
One root workflow set replaces the four repos' copies (deleted -- root-only
is where Forgejo reads them, and dead copies are a trap): per-domain
build-push with explicit image paths (emi/thermograph/backend|frontend; the
old github.repository-derived path collides in a monorepo), path-filtered
per-domain beta/prod/dev deploys, a domain-input reusable build check, a
single always-reporting PR gate (path-filtered required checks deadlock
auto-merge), a new infra-sync pipeline (host checkout + secrets render on
infra/** pushes), and ports of secrets-guard / ops-cron /
observability-validate to monorepo paths.
seed_era5.py backfills the curated-city cells with true ERA5 data from the
Earthmover Icechunk ERA5 archive on AWS Open Data (anonymous, keyless), so
high-traffic pages keep ERA5 fidelity now that NASA POWER is the live primary.
It aggregates the hourly ERA5 point series to the daily store schema in polars
(no pandas), derives feels-like via the existing heat-index/wind-chill path, and
writes straight to the history store via climate._write_history_backed, bypassing
the live fetch. ERA5 carries real gusts (i10fg), so seeded cells keep measured
gusts rather than the Meteostat fallback.
The icechunk/xarray/zarr stack is isolated in requirements-seed.txt (not the app
image or CI) and lazy-imported, so the module and its unit-tested hourly->daily
transform load without those deps. The S3 access layer targets a young API and is
verified via `--dry-run` on the seed box, not in tests. Idempotent by default
(skips already-cached cells; --overwrite to reseed).
The live history path now tries NASA POWER first (keyless, independent of
Open-Meteo) and falls back to the Open-Meteo archive only when NASA is
unavailable — the reverse of before. Gusts NASA lacks are filled from Meteostat
inside _fetch_history_nasa (added in the prior change). Both sources must still
return a plausibly-full span (MIN_ARCHIVE_DAYS) before being cached as complete;
a short/partial response is rejected and the other source is tried.
_fetch_history_nasa now accepts a start/end range so it also serves the recent
tail top-up: _topup_tail fetches NASA-first via the new _fetch_history_tail
helper (Open-Meteo range as fallback, skipped during its rate-limit cooldown),
removing the last per-active-cell Open-Meteo call from the history path. The
Open-Meteo cooldown no longer gates the top-up, since NASA is not subject to it.
Open-Meteo remains wired as the fallback (deletion is a later step). Tests
updated to the new source order plus tail-fetch coverage.
main received the perf hardening line (#5, #8) and its own landing of the
Discord gateway bot (#7) while dev carried a parallel port of the same bot
(#3). The two implementations are near-identical ports of the same archived
source; main's includes one extra hardening (grading calls moved off the
gateway event loop via asyncio.to_thread) and broader tests, so bot files
resolve wholesale to main's side. dev keeps its test-runner/smoke tooling and
the run-tests-in-image CI, which main lacks.
# Conflicts:
# notifications/discord_bot.py
# tests/notifications/test_discord_bot.py
# web/app.py
NASA POWER (history) and MET Norway (forecast) carry no wind gusts, but gust is
a graded metric. This adds data/meteostat.py, which finds the nearest Meteostat
station to a cell and reads its daily peak gust (wpgt) from the keyless gzipped
bulk endpoints, filling the gust column on the NASA POWER history frame. Where no
station is within ~100 km it estimates from sustained wind (wind * GUST_FACTOR).
Bulk files (station list + per-station daily) are cached on disk so steady-state
network IO is near zero, and any lookup/fetch failure degrades to pure estimation
so a history fetch never fails. A constant estimate factor makes an estimated
gust redundant with wind, so real signal comes only where a station backs it.
Activates once NASA POWER becomes the primary history source; Open-Meteo already
carries its own gusts. New history_gust / meteostat_stations metrics phases map
to the meteostat source.
/suggest is now served purely from the local GeoNames index — no external
geocoder call per keystroke. /geocode answers from the local index first and
falls back to OSM Nominatim /search only on a miss (or while the index is still
loading), covering the neighbourhood/postcode/tiny-village/native-name long tail
the cities dump lacks.
Nominatim shares the reverse geocoder's lock and ~1/sec pacing since both hit
the same host, and only the low-volume /geocode miss path reaches it. Removes
the Open-Meteo geocoding dependency entirely; the "geocode" metrics phase now
maps to the nominatim source.
Two bugs surfaced wiring dev auto-deploy:
1. Sourcing .image-tags.env clobbered the caller's incoming *_IMAGE_TAG. The
first backend-only deploy persists FRONTEND_IMAGE_TAG=local (sibling unknown);
the next frontend deploy then sourced that and pulled :local -> 'manifest
unknown'. Now the incoming env is captured before sourcing and re-applied.
2. Health-checked services via a host-port curl, but the dev overlay leaves the
frontend port unpublished (reached via the backend proxy) -> false failure.
Now polls each container's own HEALTHCHECK status via docker inspect.
