thermograph/tests/test_grading.py

141 lines
5.5 KiB
Python
Raw Normal View History

Migrate backend dataframe layer from pandas to polars (#90) * Migrate backend dataframe layer from pandas to polars Replace pandas with polars across the backend, dropping both pandas and its pyarrow parquet engine from the dependency set. numpy stays (the grading percentile math is unchanged). - climate.py: parquet IO, source→frame mappings, cache read/topup on polars. New _normalize_read casts the cached `date` column to pl.Date (older files were written by pandas as datetime64[ns]); frames now unify missing values as null so the grading boundary drops them consistently across sources. - grading.py: keep the numpy percentile core; swap the frame→numpy bridge to .to_numpy()/.drop_nulls(), day-of-year/year to polars dt expressions, and the per-row loop to iter_rows(named=True). - views.py: filter/anti-join/concat replace boolean-mask, isin and pd.concat; scalar dates are stdlib datetime.date; a local _months_before helper replaces DateOffset(months=) for the calendar-range default. - app.py, migrate.py: request-date parsing uses datetime.date, removing pandas from the endpoint and migrate layers entirely. - The date column is pl.Date end to end, eliminating the pandas normalize() calls and comparing cleanly against stdlib dates. Payloads are unchanged: calendar, day, grade and forecast responses are byte-for-byte identical to the pandas implementation on the same cached record. Tests ported to polars fixtures, with added coverage for the combined feels-like fallback, calendar month-offset (month-end/leap), and the concat/dedup "fresher source wins" rule. * Port notify.py to polars after merging dev's account system Merge origin/dev (accounts + notification subscriptions) and carry the pandas→ polars migration into the newly added notify.py, which the merge brought in still using pandas — with pandas removed from requirements this broke its import. - notify.py: _candidate_rows filters/sorts the recent bundle with polars expressions and returns iter_rows dicts; date scalars are datetime.date; history/recent emptiness via is_empty(). - test_notify.py: synthetic history/rows built with polars + datetime.
2026-07-15 19:07:38 +00:00
import datetime
import numpy as np
Migrate backend dataframe layer from pandas to polars (#90) * Migrate backend dataframe layer from pandas to polars Replace pandas with polars across the backend, dropping both pandas and its pyarrow parquet engine from the dependency set. numpy stays (the grading percentile math is unchanged). - climate.py: parquet IO, source→frame mappings, cache read/topup on polars. New _normalize_read casts the cached `date` column to pl.Date (older files were written by pandas as datetime64[ns]); frames now unify missing values as null so the grading boundary drops them consistently across sources. - grading.py: keep the numpy percentile core; swap the frame→numpy bridge to .to_numpy()/.drop_nulls(), day-of-year/year to polars dt expressions, and the per-row loop to iter_rows(named=True). - views.py: filter/anti-join/concat replace boolean-mask, isin and pd.concat; scalar dates are stdlib datetime.date; a local _months_before helper replaces DateOffset(months=) for the calendar-range default. - app.py, migrate.py: request-date parsing uses datetime.date, removing pandas from the endpoint and migrate layers entirely. - The date column is pl.Date end to end, eliminating the pandas normalize() calls and comparing cleanly against stdlib dates. Payloads are unchanged: calendar, day, grade and forecast responses are byte-for-byte identical to the pandas implementation on the same cached record. Tests ported to polars fixtures, with added coverage for the combined feels-like fallback, calendar month-offset (month-end/leap), and the concat/dedup "fresher source wins" rule. * Port notify.py to polars after merging dev's account system Merge origin/dev (accounts + notification subscriptions) and carry the pandas→ polars migration into the newly added notify.py, which the merge brought in still using pandas — with pandas removed from requirements this broke its import. - notify.py: _candidate_rows filters/sorts the recent bundle with polars expressions and returns iter_rows dicts; date scalars are datetime.date; history/recent emptiness via is_empty(). - test_notify.py: synthetic history/rows built with polars + datetime.
2026-07-15 19:07:38 +00:00
import polars as pl
import pytest
import grading
# ---- empirical percentile ----------------------------------------------------
def test_percentile_mid_rank_handles_ties():
samples = np.array([1.0, 2.0, 2.0, 3.0])
# less=1, equal=2 -> (1 + 0.5*2) / 4 = 50%
assert grading.empirical_percentile(samples, 2.0) == 50.0
def test_percentile_extremes_and_empties():
samples = np.array([1.0, 2.0, 3.0])
assert grading.empirical_percentile(samples, 0.0) == 0.0
assert grading.empirical_percentile(samples, 4.0) == 100.0
assert grading.empirical_percentile(np.array([]), 1.0) is None
assert grading.empirical_percentile(samples, None) is None
assert grading.empirical_percentile(samples, float("nan")) is None
# ---- tier bands ---------------------------------------------------------------
@pytest.mark.parametrize("pct,label,css", [
(99.5, "Near Record", "rec-hot"), # top tier is strict: > 99 only
(99.0, "Very High", "very-hot"), # p99 exactly is NOT near-record
(90.0, "Very High", "very-hot"),
(75.0, "High", "hot"),
(60.0, "Above Normal", "warm"),
(59.9, "Normal", "normal"),
(40.0, "Normal", "normal"),
(39.9, "Below Normal", "cool"),
(10.0, "Low", "cold"),
(1.0, "Very Low", "very-cold"),
(0.5, "Near Record", "rec-cold"), # strictly below the 1st percentile
])
def test_temp_band_boundaries(pct, label, css):
assert grading._band(pct, grading.TEMP_BANDS) == (label, css)
def test_ladders_stay_aligned_with_bands():
"""The detail-view ladders re-encode the band tables by hand; catch drift."""
for bands, ladder in [(grading.TEMP_BANDS, grading._TEMP_LADDER),
(grading.RAIN_BANDS, grading._RAIN_LADDER)]:
assert len(bands) == len(ladder)
for (_, label, css), (lcss, llabel, *_rest) in zip(bands, ladder):
assert (label, css) == (llabel, lcss)
# ---- seasonal window ----------------------------------------------------------
def test_window_mask_wraps_across_year_end():
doys = np.array([1, 180, 360, 366])
mask = grading.window_mask(doys, target_doy=1, half=7)
assert mask.tolist() == [True, False, True, True]
# ---- precip grading -----------------------------------------------------------
def test_dry_day_gets_dry_class_without_percentile():
g = grading._grade_precip(np.array([0.0, 0.5, 1.0]), 0.005)
assert g["class"] == "dry" and g["percentile"] is None and g["grade"] == "Dry"
def test_rain_percentile_ranks_among_rain_days_only():
# 7 dry days + rain days [0.1, 0.2, 0.4]; 0.2 ranks among the 3 rain days:
# less=1, equal=1 -> (1 + 0.5) / 3 = 50% -> Moderate, unaffected by the dry mass.
samples = np.array([0.0] * 7 + [0.1, 0.2, 0.4])
g = grading._grade_precip(samples, 0.2)
assert g["percentile"] == 50.0
assert g["grade"] == "Moderate"
def test_rain_with_no_historical_rain_days_is_extreme():
g = grading._grade_precip(np.zeros(10), 0.3)
assert g["percentile"] == 100.0 and g["class"] == "wet-9"
# ---- dry streaks ---------------------------------------------------------------
def test_dry_streaks_walk():
Migrate backend dataframe layer from pandas to polars (#90) * Migrate backend dataframe layer from pandas to polars Replace pandas with polars across the backend, dropping both pandas and its pyarrow parquet engine from the dependency set. numpy stays (the grading percentile math is unchanged). - climate.py: parquet IO, source→frame mappings, cache read/topup on polars. New _normalize_read casts the cached `date` column to pl.Date (older files were written by pandas as datetime64[ns]); frames now unify missing values as null so the grading boundary drops them consistently across sources. - grading.py: keep the numpy percentile core; swap the frame→numpy bridge to .to_numpy()/.drop_nulls(), day-of-year/year to polars dt expressions, and the per-row loop to iter_rows(named=True). - views.py: filter/anti-join/concat replace boolean-mask, isin and pd.concat; scalar dates are stdlib datetime.date; a local _months_before helper replaces DateOffset(months=) for the calendar-range default. - app.py, migrate.py: request-date parsing uses datetime.date, removing pandas from the endpoint and migrate layers entirely. - The date column is pl.Date end to end, eliminating the pandas normalize() calls and comparing cleanly against stdlib dates. Payloads are unchanged: calendar, day, grade and forecast responses are byte-for-byte identical to the pandas implementation on the same cached record. Tests ported to polars fixtures, with added coverage for the combined feels-like fallback, calendar month-offset (month-end/leap), and the concat/dedup "fresher source wins" rule. * Port notify.py to polars after merging dev's account system Merge origin/dev (accounts + notification subscriptions) and carry the pandas→ polars migration into the newly added notify.py, which the merge brought in still using pandas — with pandas removed from requirements this broke its import. - notify.py: _candidate_rows filters/sorts the recent bundle with polars expressions and returns iter_rows dicts; date scalars are datetime.date; history/recent emptiness via is_empty(). - test_notify.py: synthetic history/rows built with polars + datetime.
2026-07-15 19:07:38 +00:00
dates = [datetime.date(2024, 1, 1) + datetime.timedelta(days=i) for i in range(5)]
precips = [0.5, 0.0, float("nan"), 0.02, 0.005]
Migrate backend dataframe layer from pandas to polars (#90) * Migrate backend dataframe layer from pandas to polars Replace pandas with polars across the backend, dropping both pandas and its pyarrow parquet engine from the dependency set. numpy stays (the grading percentile math is unchanged). - climate.py: parquet IO, source→frame mappings, cache read/topup on polars. New _normalize_read casts the cached `date` column to pl.Date (older files were written by pandas as datetime64[ns]); frames now unify missing values as null so the grading boundary drops them consistently across sources. - grading.py: keep the numpy percentile core; swap the frame→numpy bridge to .to_numpy()/.drop_nulls(), day-of-year/year to polars dt expressions, and the per-row loop to iter_rows(named=True). - views.py: filter/anti-join/concat replace boolean-mask, isin and pd.concat; scalar dates are stdlib datetime.date; a local _months_before helper replaces DateOffset(months=) for the calendar-range default. - app.py, migrate.py: request-date parsing uses datetime.date, removing pandas from the endpoint and migrate layers entirely. - The date column is pl.Date end to end, eliminating the pandas normalize() calls and comparing cleanly against stdlib dates. Payloads are unchanged: calendar, day, grade and forecast responses are byte-for-byte identical to the pandas implementation on the same cached record. Tests ported to polars fixtures, with added coverage for the combined feels-like fallback, calendar month-offset (month-end/leap), and the concat/dedup "fresher source wins" rule. * Port notify.py to polars after merging dev's account system Merge origin/dev (accounts + notification subscriptions) and carry the pandas→ polars migration into the newly added notify.py, which the merge brought in still using pandas — with pandas removed from requirements this broke its import. - notify.py: _candidate_rows filters/sorts the recent bundle with polars expressions and returns iter_rows dicts; date scalars are datetime.date; history/recent emptiness via is_empty(). - test_notify.py: synthetic history/rows built with polars + datetime.
2026-07-15 19:07:38 +00:00
out = grading.dry_streaks(dates, precips)
assert list(out.values()) == [0, 1, 2, 0, 1] # NaN counts as dry
# ---- range + day grading over a synthetic record --------------------------------
def test_grade_range_compact_shape(history):
Migrate backend dataframe layer from pandas to polars (#90) * Migrate backend dataframe layer from pandas to polars Replace pandas with polars across the backend, dropping both pandas and its pyarrow parquet engine from the dependency set. numpy stays (the grading percentile math is unchanged). - climate.py: parquet IO, source→frame mappings, cache read/topup on polars. New _normalize_read casts the cached `date` column to pl.Date (older files were written by pandas as datetime64[ns]); frames now unify missing values as null so the grading boundary drops them consistently across sources. - grading.py: keep the numpy percentile core; swap the frame→numpy bridge to .to_numpy()/.drop_nulls(), day-of-year/year to polars dt expressions, and the per-row loop to iter_rows(named=True). - views.py: filter/anti-join/concat replace boolean-mask, isin and pd.concat; scalar dates are stdlib datetime.date; a local _months_before helper replaces DateOffset(months=) for the calendar-range default. - app.py, migrate.py: request-date parsing uses datetime.date, removing pandas from the endpoint and migrate layers entirely. - The date column is pl.Date end to end, eliminating the pandas normalize() calls and comparing cleanly against stdlib dates. Payloads are unchanged: calendar, day, grade and forecast responses are byte-for-byte identical to the pandas implementation on the same cached record. Tests ported to polars fixtures, with added coverage for the combined feels-like fallback, calendar month-offset (month-end/leap), and the concat/dedup "fresher source wins" rule. * Port notify.py to polars after merging dev's account system Merge origin/dev (accounts + notification subscriptions) and carry the pandas→ polars migration into the newly added notify.py, which the merge brought in still using pandas — with pandas removed from requirements this broke its import. - notify.py: _candidate_rows filters/sorts the recent bundle with polars expressions and returns iter_rows dicts; date scalars are datetime.date; history/recent emptiness via is_empty(). - test_notify.py: synthetic history/rows built with polars + datetime.
2026-07-15 19:07:38 +00:00
end = history["date"].max()
start = end - datetime.timedelta(days=30)
days = grading.grade_range(history, start, end)
assert len(days) == 31
first = days[0]
assert set(first) == {"date", "dsr", *grading.TEMP_METRICS, "precip"}
assert first["dsr"] >= 0
g = first["tmax"]
assert set(g) == {"v", "pct", "c", "g"}
for rec in days: # dry days carry no rain percentile, wet days always do
if rec["precip"]["c"] == "dry":
assert rec["precip"]["pct"] is None
else:
assert rec["precip"]["pct"] is not None
def test_grade_day_normals_and_departure(history):
Migrate backend dataframe layer from pandas to polars (#90) * Migrate backend dataframe layer from pandas to polars Replace pandas with polars across the backend, dropping both pandas and its pyarrow parquet engine from the dependency set. numpy stays (the grading percentile math is unchanged). - climate.py: parquet IO, source→frame mappings, cache read/topup on polars. New _normalize_read casts the cached `date` column to pl.Date (older files were written by pandas as datetime64[ns]); frames now unify missing values as null so the grading boundary drops them consistently across sources. - grading.py: keep the numpy percentile core; swap the frame→numpy bridge to .to_numpy()/.drop_nulls(), day-of-year/year to polars dt expressions, and the per-row loop to iter_rows(named=True). - views.py: filter/anti-join/concat replace boolean-mask, isin and pd.concat; scalar dates are stdlib datetime.date; a local _months_before helper replaces DateOffset(months=) for the calendar-range default. - app.py, migrate.py: request-date parsing uses datetime.date, removing pandas from the endpoint and migrate layers entirely. - The date column is pl.Date end to end, eliminating the pandas normalize() calls and comparing cleanly against stdlib dates. Payloads are unchanged: calendar, day, grade and forecast responses are byte-for-byte identical to the pandas implementation on the same cached record. Tests ported to polars fixtures, with added coverage for the combined feels-like fallback, calendar month-offset (month-end/leap), and the concat/dedup "fresher source wins" rule. * Port notify.py to polars after merging dev's account system Merge origin/dev (accounts + notification subscriptions) and carry the pandas→ polars migration into the newly added notify.py, which the merge brought in still using pandas — with pandas removed from requirements this broke its import. - notify.py: _candidate_rows filters/sorts the recent bundle with polars expressions and returns iter_rows dicts; date scalars are datetime.date; history/recent emptiness via is_empty(). - test_notify.py: synthetic history/rows built with polars + datetime.
2026-07-15 19:07:38 +00:00
target = history["date"].max()
row = history.filter(pl.col("date") == target).row(0, named=True)
result = grading.grade_day(history, target, {"tmax": row["tmax"], "tmin": row["tmin"],
"precip": row["precip"]})
assert set(result["normals"]["tmax"]) == {"p1", "p10", "p25", "p40", "p50", "p60",
"p75", "p90", "p99"}
expected = max(abs(result["tmax"]["percentile"] - 50), abs(result["tmin"]["percentile"] - 50))
assert result["departure"] == round(expected, 1)
def test_day_detail_with_and_without_observation(history):
Migrate backend dataframe layer from pandas to polars (#90) * Migrate backend dataframe layer from pandas to polars Replace pandas with polars across the backend, dropping both pandas and its pyarrow parquet engine from the dependency set. numpy stays (the grading percentile math is unchanged). - climate.py: parquet IO, source→frame mappings, cache read/topup on polars. New _normalize_read casts the cached `date` column to pl.Date (older files were written by pandas as datetime64[ns]); frames now unify missing values as null so the grading boundary drops them consistently across sources. - grading.py: keep the numpy percentile core; swap the frame→numpy bridge to .to_numpy()/.drop_nulls(), day-of-year/year to polars dt expressions, and the per-row loop to iter_rows(named=True). - views.py: filter/anti-join/concat replace boolean-mask, isin and pd.concat; scalar dates are stdlib datetime.date; a local _months_before helper replaces DateOffset(months=) for the calendar-range default. - app.py, migrate.py: request-date parsing uses datetime.date, removing pandas from the endpoint and migrate layers entirely. - The date column is pl.Date end to end, eliminating the pandas normalize() calls and comparing cleanly against stdlib dates. Payloads are unchanged: calendar, day, grade and forecast responses are byte-for-byte identical to the pandas implementation on the same cached record. Tests ported to polars fixtures, with added coverage for the combined feels-like fallback, calendar month-offset (month-end/leap), and the concat/dedup "fresher source wins" rule. * Port notify.py to polars after merging dev's account system Merge origin/dev (accounts + notification subscriptions) and carry the pandas→ polars migration into the newly added notify.py, which the merge brought in still using pandas — with pandas removed from requirements this broke its import. - notify.py: _candidate_rows filters/sorts the recent bundle with polars expressions and returns iter_rows dicts; date scalars are datetime.date; history/recent emptiness via is_empty(). - test_notify.py: synthetic history/rows built with polars + datetime.
2026-07-15 19:07:38 +00:00
target = history["date"].max()
detail = grading.day_detail(history, target, {"tmax": 60.0, "precip": 0.0})
assert detail["metrics"]["tmax"]["obs"]["value"] == 60.0
assert detail["metrics"]["tmax"]["ladder"]["tiers"][0]["c"] == "rec-hot"
assert detail["metrics"]["precip"]["ladder"]["tiers"][-1]["c"] == "dry"
bare = grading.day_detail(history, target, None)
assert bare["metrics"]["tmax"]["obs"] is None
assert bare["metrics"]["tmax"]["ladder"] is not None
def test_climatology_summary(history):
climo = grading.climatology(history, 180)
# ±7-day window over ~20 years -> ~15 samples per year.
assert climo["n_samples"] >= 15 * 19
assert climo["tmax"]["p40"] <= climo["tmax"]["p50"] <= climo["tmax"]["p60"]
assert climo["feels"] is None # column absent from this record