Recent/forecast grade window has date gaps and boundary-row artifacts (missing 07-22/07-23, future-dated rows, spurious 100th-pct boundary lows) #66

New issue

Closed

opened 2026-07-24 22:53:56 +00:00 by admin_emi · 0 comments

admin_emi commented

2026-07-24 22:53:56 +00:00

Owner

Symptom (reproduced independently twice, across 3 unrelated cells)

A 14-day grade call "graded to 2026-07-24" returns rows spanning 07-10→08-03:

07-22 and 07-23 are missing.
08-01, 08-02, 08-03 are present despite being future-dated relative to "today" (07-24).
The 08-03 row has high == low == feels (74.84 at the Durham cell), the shape of a partial/single-observation day.
Reproduced independently at Durham (1241_-2203), West Seattle (1641_-2849), and Waterloo/Greater London (1776_-3) — same two calendar dates missing at all three, on two continents. This points at the ingest chain, not any one location.

At every cell graded, today's low (07-24) grades as exactly 100th percentile "Near Record", and the final forecast row (08-03, where high==low==feels) also grades 100th pct everywhere tested (Durham 74.84, West Seattle 80.42, London 71.96). Identical behavior at the first and last row of the window across three unrelated cells is not a climate signal — it's what a partial/single-observation day looks like when the day's minimum is computed before the night has actually happened. This is user-facing: a Discord bot surfaced "abnormally warm nights" / "100th pct every day" narratives built partly on this artifact.

Suspected root cause (from independent code review of `backend/data/climate.py`)

No truncation to FORECAST_DAYS: _metno_to_frame returns every day in MET Norway's timeseries (~10 days), not the intended FORECAST_DAYS = 8 — explains the future-dated 08-01→08-03 rows.
Partial trailing day: the last day in that untruncated series has only 1-2 sub-daily steps, so max(t) == min(t) → tmax == tmin == feels — explains the high==low==feels boundary-row artifact.
Gap plausibly from RECENT_END_LAG_DAYS = 2: this assumes NASA POWER NRT lags 2 days behind real-time, but NASA POWER's actual NRT latency commonly runs 3-5 days. NASA's range ends at today-2 while MET Norway's forward range starts at today, leaving a hole for whatever days fall in the real latency gap — matches the missing 07-22/07-23 window if lag is actually running ~3-5 days rather than 2.

Not yet verified against a live repro with fix in hand — flagging the hypothesis so whoever picks this up doesn't start from zero.

Suggested fix

Truncate _metno_to_frame's output to FORECAST_DAYS and drop the partial trailing day.
Measure NASA POWER's actual current NRT latency and raise RECENT_END_LAG_DAYS to match (or backfill the gap from the history record instead).

Why this matters

Thermograph's entire product claim is that a percentile means something. A spurious 100th-percentile boundary row, and forecast days presented with fabricated calendar gaps, both quietly corrode that claim in exactly the shape of a real weather story — which is how this surfaced (as an apparent "persistent warm-night pattern") before anyone traced it to a data artifact.

## Symptom (reproduced independently twice, across 3 unrelated cells) A 14-day grade call "graded to 2026-07-24" returns rows spanning 07-10→08-03: - **07-22 and 07-23 are missing.** - **08-01, 08-02, 08-03 are present despite being future-dated** relative to "today" (07-24). - The **08-03 row has `high == low == feels`** (74.84 at the Durham cell), the shape of a partial/single-observation day. - Reproduced independently at Durham (`1241_-2203`), West Seattle (`1641_-2849`), and Waterloo/Greater London (`1776_-3`) — same two calendar dates missing at all three, on two continents. This points at the ingest chain, not any one location. ## Related, likely same root cause: spurious 100th-percentile boundary lows At every cell graded, **today's low (07-24) grades as exactly 100th percentile "Near Record"**, and the final forecast row (08-03, where high==low==feels) also grades 100th pct everywhere tested (Durham 74.84, West Seattle 80.42, London 71.96). Identical behavior at the first and last row of the window across three unrelated cells is not a climate signal — it's what a partial/single-observation day looks like when the day's minimum is computed before the night has actually happened. This is user-facing: a Discord bot surfaced "abnormally warm nights" / "100th pct every day" narratives built partly on this artifact. ## Suspected root cause (from independent code review of `backend/data/climate.py`) - **No truncation to `FORECAST_DAYS`**: `_metno_to_frame` returns every day in MET Norway's timeseries (~10 days), not the intended `FORECAST_DAYS = 8` — explains the future-dated 08-01→08-03 rows. - **Partial trailing day**: the last day in that untruncated series has only 1-2 sub-daily steps, so `max(t) == min(t)` → `tmax == tmin == feels` — explains the `high==low==feels` boundary-row artifact. - **Gap plausibly from `RECENT_END_LAG_DAYS = 2`**: this assumes NASA POWER NRT lags 2 days behind real-time, but NASA POWER's actual NRT latency commonly runs 3-5 days. NASA's range ends at `today-2` while MET Norway's forward range starts at `today`, leaving a hole for whatever days fall in the real latency gap — matches the missing 07-22/07-23 window if lag is actually running ~3-5 days rather than 2. Not yet verified against a live repro with fix in hand — flagging the hypothesis so whoever picks this up doesn't start from zero. ## Suggested fix 1. Truncate `_metno_to_frame`'s output to `FORECAST_DAYS` and drop the partial trailing day. 2. Measure NASA POWER's actual current NRT latency and raise `RECENT_END_LAG_DAYS` to match (or backfill the gap from the history record instead). ## Why this matters Thermograph's entire product claim is that a percentile means something. A spurious 100th-percentile boundary row, and forecast days presented with fabricated calendar gaps, both quietly corrode that claim in exactly the shape of a real weather story — which is how this surfaced (as an apparent "persistent warm-night pattern") before anyone traced it to a data artifact.