Phase 4: value-drift check (Open-Meteo kept as dormant fallback) #2

Merged

admin_emi merged 10 commits from feature/drift-check into dev

2026-07-23 14:34:53 +00:00

admin_emi commented

2026-07-23 13:34:12 +00:00

Owner

Final phase of the Open-Meteo migration, ported into the monorepo. Stacked on feature/metno-recent-flip (Phase 3).

Open-Meteo is kept as a dormant last-resort fallback (no deletion — steady state is Open-Meteo-free but the free/keyless safety net stays for reliability). This adds the validation tool.

Adds backend/drift_check.py: fetches both live history sources (NASA POWER primary + Open-Meteo/ERA5 fallback) for a sample of cells and reports per variable raw drift (MAD/bias/max) and grade-band divergence (share of days whose graded band changes). Doubles as an ERA5-seed fidelity check.

Insight baked in: grades are percentiles within each source's own distribution, so a uniform bias moves MAD but not grades — the grade divergence is the number that matters; the tool reports both.

Verified: cd backend && make test -> 368 passed, 7 skipped. 3 new hermetic tests (per-variable stats + identical-frames-zero-divergence). Per-cell fetch runs on a networked box: python drift_check.py [--limit N] [--days N].

Monorepo port complete. The 6 stacked feature branches now cover all of Phases 0-4: geocode-local, history-meteostat-gusts, nasa-primary-flip, era5-seed (pre-existing, now test-verified: 364 passed), + metno-recent-flip (#1) and drift-check (this). Merge bottom-up from feature/geocode-local.

Final phase of the Open-Meteo migration, ported into the monorepo. **Stacked on `feature/metno-recent-flip`** (Phase 3). Open-Meteo is **kept as a dormant last-resort fallback** (no deletion — steady state is Open-Meteo-free but the free/keyless safety net stays for reliability). This adds the validation tool. **Adds** `backend/drift_check.py`: fetches both live history sources (NASA POWER primary + Open-Meteo/ERA5 fallback) for a sample of cells and reports per variable raw drift (MAD/bias/max) and grade-band divergence (share of days whose graded band changes). Doubles as an ERA5-seed fidelity check. Insight baked in: grades are percentiles within each source's own distribution, so a uniform bias moves MAD but not grades — the grade divergence is the number that matters; the tool reports both. **Verified**: `cd backend && make test` -> **368 passed, 7 skipped**. 3 new hermetic tests (per-variable stats + identical-frames-zero-divergence). Per-cell fetch runs on a networked box: `python drift_check.py [--limit N] [--days N]`. --- **Monorepo port complete.** The 6 stacked feature branches now cover all of Phases 0-4: geocode-local, history-meteostat-gusts, nasa-primary-flip, era5-seed (pre-existing, now test-verified: 364 passed), + metno-recent-flip (#1) and drift-check (this). Merge bottom-up from `feature/geocode-local`.

admin_emi added 1 commit 2026-07-23 13:34:14 +00:00

Add value-drift check comparing NASA vs Open-Meteo (ERA5)

PR build (required check) / changes (pull_request) Successful in 7s

Details

secrets-guard / encrypted (pull_request) Successful in 5s

Details

PR build (required check) / build-frontend (pull_request) Has been skipped

Details

PR build (required check) / validate-observability (pull_request) Has been skipped

Details

PR build (required check) / build-backend (pull_request) Successful in 44s

Details

PR build (required check) / gate (pull_request) Successful in 2s

Details

abf37e6376

backend/drift_check.py quantifies the migration's data impact by fetching both
live history sources (NASA POWER primary + the Open-Meteo/ERA5 fallback) for a
sample of cells and reporting, per variable: mean-absolute-difference / bias / max
(raw drift) and grade-band divergence (the share of days whose graded band actually
changes). Since Open-Meteo is ERA5, this doubles as a fidelity check for the ERA5
seed.

Because grades are percentiles within each source's own distribution, a uniform
bias moves MAD but not grades — so the two numbers are read together. Open-Meteo is
kept as a dormant fallback (not deleted), so both sources stay fetchable for this
comparison. The comparison logic is unit-tested; the per-cell fetch runs on a
networked box (python drift_check.py [--limit N] [--days N]).