backend/drift_check.py quantifies the migration's data impact by fetching both
live history sources (NASA POWER primary + the Open-Meteo/ERA5 fallback) for a
sample of cells and reporting, per variable: mean-absolute-difference / bias / max
(raw drift) and grade-band divergence (the share of days whose graded band actually
changes). Since Open-Meteo is ERA5, this doubles as a fidelity check for the ERA5
seed.
Because grades are percentiles within each source's own distribution, a uniform
bias moves MAD but not grades — so the two numbers are read together. Open-Meteo is
kept as a dormant fallback (not deleted), so both sources stay fetchable for this
comparison. The comparison logic is unit-tested; the per-cell fetch runs on a
networked box (python drift_check.py [--limit N] [--days N]).
The recent-observations + forward-forecast bundle is now built without Open-Meteo:
the recent observed window comes from a NASA POWER range (measured, via the range
fetch added for the history flip) and the forward days from MET Norway, merged by
date. Meteostat fills the gusts neither source carries — measured for the observed
days, estimated from wind for the forecast days.
Open-Meteo's forecast API is demoted to the fallback, still gated by its existing
_forecast_cooldown_until rate-limit cooldown; then a stale cache. MET Norway
switched to /compact (same fields, smaller payload) and the bundle TTL relaxed
1h -> 4h. NASA's near-real-time lag leaves a 1-2 day recent-edge gap that grades
as missing, bracketed by history behind and forecast ahead.
Tests updated to the new source order (NASA+MET merge, Open-Meteo fallback, and
the forecast cooldown now gating that fallback); deletion of Open-Meteo is not
done — it stays as the dormant fallback.
Conflict resolved (UNVERIFIED -- tests not runnable locally, docker blocked):
tests/data/test_climate.py keeps BOTH appended test blocks (dev's
forecast-cooldown tests + this branch's NASA-primary tests). Same conflict
exists merging this branch into the split repo's dev.
# Conflicts:
# backend/tests/data/test_climate.py
Backend runs its full hermetic suite, frontend its unit tier, inside the
just-built image -- the dev-only feature both app repos carried in their own
build.yml (deleted here in favor of the root workflow).
/opt/thermograph becomes a monorepo checkout: deploy.sh gains INFRA_DIR (git
ops at the root, compose work cd'd into infra/), lock/tags/render paths move
under infra/deploy/, image-path defaults become emi/thermograph/backend|
frontend across compose, stack, and the tag-prune. docker-compose.yml pins
name: thermograph -- without it compose run from .../infra derives project
"infra" and recreates the whole stack beside the running one;
deploy-dev.sh pins COMPOSE_PROJECT_NAME=thermograph-dev (env wins over the
file key) to keep LAN dev's separate project.
One root workflow set replaces the four repos' copies (deleted -- root-only
is where Forgejo reads them, and dead copies are a trap): per-domain
build-push with explicit image paths (emi/thermograph/backend|frontend; the
old github.repository-derived path collides in a monorepo), path-filtered
per-domain beta/prod/dev deploys, a domain-input reusable build check, a
single always-reporting PR gate (path-filtered required checks deadlock
auto-merge), a new infra-sync pipeline (host checkout + secrets render on
infra/** pushes), and ports of secrets-guard / ops-cron /
observability-validate to monorepo paths.
seed_era5.py backfills the curated-city cells with true ERA5 data from the
Earthmover Icechunk ERA5 archive on AWS Open Data (anonymous, keyless), so
high-traffic pages keep ERA5 fidelity now that NASA POWER is the live primary.
It aggregates the hourly ERA5 point series to the daily store schema in polars
(no pandas), derives feels-like via the existing heat-index/wind-chill path, and
writes straight to the history store via climate._write_history_backed, bypassing
the live fetch. ERA5 carries real gusts (i10fg), so seeded cells keep measured
gusts rather than the Meteostat fallback.
The icechunk/xarray/zarr stack is isolated in requirements-seed.txt (not the app
image or CI) and lazy-imported, so the module and its unit-tested hourly->daily
transform load without those deps. The S3 access layer targets a young API and is
verified via `--dry-run` on the seed box, not in tests. Idempotent by default
(skips already-cached cells; --overwrite to reseed).
The live history path now tries NASA POWER first (keyless, independent of
Open-Meteo) and falls back to the Open-Meteo archive only when NASA is
unavailable — the reverse of before. Gusts NASA lacks are filled from Meteostat
inside _fetch_history_nasa (added in the prior change). Both sources must still
return a plausibly-full span (MIN_ARCHIVE_DAYS) before being cached as complete;
a short/partial response is rejected and the other source is tried.
_fetch_history_nasa now accepts a start/end range so it also serves the recent
tail top-up: _topup_tail fetches NASA-first via the new _fetch_history_tail
helper (Open-Meteo range as fallback, skipped during its rate-limit cooldown),
removing the last per-active-cell Open-Meteo call from the history path. The
Open-Meteo cooldown no longer gates the top-up, since NASA is not subject to it.
Open-Meteo remains wired as the fallback (deletion is a later step). Tests
updated to the new source order plus tail-fetch coverage.
main received the perf hardening line (#5, #8) and its own landing of the
Discord gateway bot (#7) while dev carried a parallel port of the same bot
(#3). The two implementations are near-identical ports of the same archived
source; main's includes one extra hardening (grading calls moved off the
gateway event loop via asyncio.to_thread) and broader tests, so bot files
resolve wholesale to main's side. dev keeps its test-runner/smoke tooling and
the run-tests-in-image CI, which main lacks.
# Conflicts:
# notifications/discord_bot.py
# tests/notifications/test_discord_bot.py
# web/app.py
NASA POWER (history) and MET Norway (forecast) carry no wind gusts, but gust is
a graded metric. This adds data/meteostat.py, which finds the nearest Meteostat
station to a cell and reads its daily peak gust (wpgt) from the keyless gzipped
bulk endpoints, filling the gust column on the NASA POWER history frame. Where no
station is within ~100 km it estimates from sustained wind (wind * GUST_FACTOR).
Bulk files (station list + per-station daily) are cached on disk so steady-state
network IO is near zero, and any lookup/fetch failure degrades to pure estimation
so a history fetch never fails. A constant estimate factor makes an estimated
gust redundant with wind, so real signal comes only where a station backs it.
Activates once NASA POWER becomes the primary history source; Open-Meteo already
carries its own gusts. New history_gust / meteostat_stations metrics phases map
to the meteostat source.
/suggest is now served purely from the local GeoNames index — no external
geocoder call per keystroke. /geocode answers from the local index first and
falls back to OSM Nominatim /search only on a miss (or while the index is still
loading), covering the neighbourhood/postcode/tiny-village/native-name long tail
the cities dump lacks.
Nominatim shares the reverse geocoder's lock and ~1/sec pacing since both hit
the same host, and only the low-volume /geocode miss path reaches it. Removes
the Open-Meteo geocoding dependency entirely; the "geocode" metrics phase now
maps to the nominatim source.
Two bugs surfaced wiring dev auto-deploy:
1. Sourcing .image-tags.env clobbered the caller's incoming *_IMAGE_TAG. The
first backend-only deploy persists FRONTEND_IMAGE_TAG=local (sibling unknown);
the next frontend deploy then sourced that and pulled :local -> 'manifest
unknown'. Now the incoming env is captured before sourcing and re-applied.
2. Health-checked services via a host-port curl, but the dev overlay leaves the
frontend port unpublished (reached via the backend proxy) -> false failure.
Now polls each container's own HEALTHCHECK status via docker inspect.
The CI/SSH deploy paths (deploy.yml/deploy-prod.yml over SSH, deploy-dev on the
LAN runner) don't pass REGISTRY_TOKEN and rely on the host already being
docker-logged-in; an unconditional login with an empty token aborts deploy.sh
under set -e. Skip the login when no token is set and trust the host cred;
pull still fails loudly on a real auth problem.
The split backend's Python package data/ lives at /app/data; mounting the
appdata runtime volume there erased data/*.py and broke import at boot. Point
THERMOGRAPH_DATA_DIR + the appdata mount + the singleton lock at /state,
outside the code tree. Pairs with thermograph-backend paths.py making
DATA_DIR/LOGS_DIR env-overridable.
After the repo split the code lives at the image root, so the Python package
data/ and the runtime data dir <root>/data are the same path (/app/data). The
deploy compose mounts the appdata volume there, shadowing data/*.py so
'import data.climate' fails at container boot. THERMOGRAPH_DATA_DIR (and
_LOGS_DIR) now let the container point runtime state at a path outside the code
tree; local dev is unchanged (defaults to <root>/data).
The split branch carried the #34-era render-secrets.sh, missing two fixes
required for prod/beta deploy: (1) sudo-read the root-owned 0400 age key into
SOPS_AGE_KEY when the deploy user can't read it directly; (2) in-place-write
/etc/thermograph.env when it's group-writable (beta's non-root 'deploy' user)
instead of only install(1). Without these, deploy.sh's render fails on both
hosts. Flagged by the terraform-layer workstream.
