Migrate backend dataframe layer from pandas to polars (#90)
* Migrate backend dataframe layer from pandas to polars
Replace pandas with polars across the backend, dropping both pandas and its
pyarrow parquet engine from the dependency set. numpy stays (the grading
percentile math is unchanged).
- climate.py: parquet IO, source→frame mappings, cache read/topup on polars.
New _normalize_read casts the cached `date` column to pl.Date (older files
were written by pandas as datetime64[ns]); frames now unify missing values as
null so the grading boundary drops them consistently across sources.
- grading.py: keep the numpy percentile core; swap the frame→numpy bridge to
.to_numpy()/.drop_nulls(), day-of-year/year to polars dt expressions, and the
per-row loop to iter_rows(named=True).
- views.py: filter/anti-join/concat replace boolean-mask, isin and pd.concat;
scalar dates are stdlib datetime.date; a local _months_before helper replaces
DateOffset(months=) for the calendar-range default.
- app.py, migrate.py: request-date parsing uses datetime.date, removing pandas
from the endpoint and migrate layers entirely.
- The date column is pl.Date end to end, eliminating the pandas normalize() calls
and comparing cleanly against stdlib dates.
Payloads are unchanged: calendar, day, grade and forecast responses are
byte-for-byte identical to the pandas implementation on the same cached record.
Tests ported to polars fixtures, with added coverage for the combined feels-like
fallback, calendar month-offset (month-end/leap), and the concat/dedup
"fresher source wins" rule.
* Port notify.py to polars after merging dev's account system
Merge origin/dev (accounts + notification subscriptions) and carry the pandas→
polars migration into the newly added notify.py, which the merge brought in still
using pandas — with pandas removed from requirements this broke its import.
- notify.py: _candidate_rows filters/sorts the recent bundle with polars
expressions and returns iter_rows dicts; date scalars are datetime.date;
history/recent emptiness via is_empty().
- test_notify.py: synthetic history/rows built with polars + datetime.
2026-07-15 19:07:38 +00:00
|
|
|
|
import datetime
|
|
|
|
|
|
|
Add backend test suite; gate direct pushes; serialize LAN deploys (#41)
- backend/tests: 74 hermetic tests (no network, no repo data//logs/ writes)
covering grid snapping/round-trips, grading percentiles/bands/windows/
dry streaks, the places index (norm, one-edit matchers, search,
corrections), the derived store (token validity, cache=False, degraded
mode), and route-level API tests over a faked climate layer — routing,
validation, ETag/304 revalidation, store replay, the /cell bundle, and
the v1/v2 aliases. The API tests would have caught the /place
AttributeError regression.
- requirements-dev.txt + make test (venv prefers uv-pinned 3.12, matching
deploy-dev.sh — pyarrow wheels stop at 3.12 and some pyenv builds lack
sqlite).
- CI: extract the build job into a reusable build.yml, add the test run
and an API health probe (page-only curl can't catch route wiring
faults); deploy-dev.yml now runs the same build gate before deploying
direct pushes, which previously deployed with no CI at all.
- Deploys serialize under one dev-lan-deploy concurrency group across
both workflows (previously per-PR groups could interleave two deploys
to the same checkout), and are never cancelled mid-restart.
- deploy-dev.sh health check also probes /api/v2/place — best-effort
externals mean a failure there is a genuine server bug.
2026-07-11 19:37:49 +00:00
|
|
|
|
import numpy as np
|
Migrate backend dataframe layer from pandas to polars (#90)
* Migrate backend dataframe layer from pandas to polars
Replace pandas with polars across the backend, dropping both pandas and its
pyarrow parquet engine from the dependency set. numpy stays (the grading
percentile math is unchanged).
- climate.py: parquet IO, source→frame mappings, cache read/topup on polars.
New _normalize_read casts the cached `date` column to pl.Date (older files
were written by pandas as datetime64[ns]); frames now unify missing values as
null so the grading boundary drops them consistently across sources.
- grading.py: keep the numpy percentile core; swap the frame→numpy bridge to
.to_numpy()/.drop_nulls(), day-of-year/year to polars dt expressions, and the
per-row loop to iter_rows(named=True).
- views.py: filter/anti-join/concat replace boolean-mask, isin and pd.concat;
scalar dates are stdlib datetime.date; a local _months_before helper replaces
DateOffset(months=) for the calendar-range default.
- app.py, migrate.py: request-date parsing uses datetime.date, removing pandas
from the endpoint and migrate layers entirely.
- The date column is pl.Date end to end, eliminating the pandas normalize() calls
and comparing cleanly against stdlib dates.
Payloads are unchanged: calendar, day, grade and forecast responses are
byte-for-byte identical to the pandas implementation on the same cached record.
Tests ported to polars fixtures, with added coverage for the combined feels-like
fallback, calendar month-offset (month-end/leap), and the concat/dedup
"fresher source wins" rule.
* Port notify.py to polars after merging dev's account system
Merge origin/dev (accounts + notification subscriptions) and carry the pandas→
polars migration into the newly added notify.py, which the merge brought in still
using pandas — with pandas removed from requirements this broke its import.
- notify.py: _candidate_rows filters/sorts the recent bundle with polars
expressions and returns iter_rows dicts; date scalars are datetime.date;
history/recent emptiness via is_empty().
- test_notify.py: synthetic history/rows built with polars + datetime.
2026-07-15 19:07:38 +00:00
|
|
|
|
import polars as pl
|
Add backend test suite; gate direct pushes; serialize LAN deploys (#41)
- backend/tests: 74 hermetic tests (no network, no repo data//logs/ writes)
covering grid snapping/round-trips, grading percentiles/bands/windows/
dry streaks, the places index (norm, one-edit matchers, search,
corrections), the derived store (token validity, cache=False, degraded
mode), and route-level API tests over a faked climate layer — routing,
validation, ETag/304 revalidation, store replay, the /cell bundle, and
the v1/v2 aliases. The API tests would have caught the /place
AttributeError regression.
- requirements-dev.txt + make test (venv prefers uv-pinned 3.12, matching
deploy-dev.sh — pyarrow wheels stop at 3.12 and some pyenv builds lack
sqlite).
- CI: extract the build job into a reusable build.yml, add the test run
and an API health probe (page-only curl can't catch route wiring
faults); deploy-dev.yml now runs the same build gate before deploying
direct pushes, which previously deployed with no CI at all.
- Deploys serialize under one dev-lan-deploy concurrency group across
both workflows (previously per-PR groups could interleave two deploys
to the same checkout), and are never cancelled mid-restart.
- deploy-dev.sh health check also probes /api/v2/place — best-effort
externals mean a failure there is a genuine server bug.
2026-07-11 19:37:49 +00:00
|
|
|
|
import pytest
|
|
|
|
|
|
|
Split the backend into domain packages (#217)
* Centralize filesystem paths in a single module
Add paths.py, which resolves the repo root once and derives the cache,
accounts DB, logs, templates, frontend and bundled-city-data locations
from it. Replace the 13 per-module `dirname(__file__)/..` anchors with
references to it, so a module's location no longer determines where the
app reads its data. Env overrides (accounts DB, VAPID, IndexNow) are
unchanged; every resolved path is byte-identical to before.
Groundwork for moving modules into packages without re-pointing paths.
Claude-Session: https://claude.ai/code/session_01XXxmNFy9cZ6Gh8Y9thZn62
* Split the backend into domain packages
Group the flat backend modules into packages that mirror their concerns:
data/ climate, grading, scoring, grid, places, cities,
city_events, store
web/ app, views, homepage, content, schemas
notifications/ notify, digest, push, mailer, discord,
discord_interactions, discord_link
accounts/ models, users, api_accounts, db
core/ metrics, singleton, audit
Intra-project imports are rewritten to the package-qualified form. The
entry scripts (indexnow, warm_cities, migrate, gen_cities, gen_flavor)
and paths.py stay at the backend/ root, and backend/app.py becomes a
shim re-exporting web.app:app so the launch target stays `app:app` —
run.sh, the systemd units, and CI need no change.
Verified: full suite (318) passes, `uvicorn app:app` boots and serves
the home/SEO/static/API surfaces, and every root script imports clean.
Claude-Session: https://claude.ai/code/session_01XXxmNFy9cZ6Gh8Y9thZn62
2026-07-20 05:31:03 +00:00
|
|
|
|
from data import grading
|
Add backend test suite; gate direct pushes; serialize LAN deploys (#41)
- backend/tests: 74 hermetic tests (no network, no repo data//logs/ writes)
covering grid snapping/round-trips, grading percentiles/bands/windows/
dry streaks, the places index (norm, one-edit matchers, search,
corrections), the derived store (token validity, cache=False, degraded
mode), and route-level API tests over a faked climate layer — routing,
validation, ETag/304 revalidation, store replay, the /cell bundle, and
the v1/v2 aliases. The API tests would have caught the /place
AttributeError regression.
- requirements-dev.txt + make test (venv prefers uv-pinned 3.12, matching
deploy-dev.sh — pyarrow wheels stop at 3.12 and some pyenv builds lack
sqlite).
- CI: extract the build job into a reusable build.yml, add the test run
and an API health probe (page-only curl can't catch route wiring
faults); deploy-dev.yml now runs the same build gate before deploying
direct pushes, which previously deployed with no CI at all.
- Deploys serialize under one dev-lan-deploy concurrency group across
both workflows (previously per-PR groups could interleave two deploys
to the same checkout), and are never cancelled mid-restart.
- deploy-dev.sh health check also probes /api/v2/place — best-effort
externals mean a failure there is a genuine server bug.
2026-07-11 19:37:49 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
# ---- empirical percentile ----------------------------------------------------
|
|
|
|
|
|
|
|
|
|
|
|
def test_percentile_mid_rank_handles_ties():
|
|
|
|
|
|
samples = np.array([1.0, 2.0, 2.0, 3.0])
|
|
|
|
|
|
# less=1, equal=2 -> (1 + 0.5*2) / 4 = 50%
|
|
|
|
|
|
assert grading.empirical_percentile(samples, 2.0) == 50.0
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def test_percentile_extremes_and_empties():
|
|
|
|
|
|
samples = np.array([1.0, 2.0, 3.0])
|
|
|
|
|
|
assert grading.empirical_percentile(samples, 0.0) == 0.0
|
|
|
|
|
|
assert grading.empirical_percentile(samples, 4.0) == 100.0
|
|
|
|
|
|
assert grading.empirical_percentile(np.array([]), 1.0) is None
|
|
|
|
|
|
assert grading.empirical_percentile(samples, None) is None
|
|
|
|
|
|
assert grading.empirical_percentile(samples, float("nan")) is None
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
# ---- tier bands ---------------------------------------------------------------
|
|
|
|
|
|
|
|
|
|
|
|
@pytest.mark.parametrize("pct,label,css", [
|
|
|
|
|
|
(99.5, "Near Record", "rec-hot"), # top tier is strict: > 99 only
|
|
|
|
|
|
(99.0, "Very High", "very-hot"), # p99 exactly is NOT near-record
|
|
|
|
|
|
(90.0, "Very High", "very-hot"),
|
|
|
|
|
|
(75.0, "High", "hot"),
|
|
|
|
|
|
(60.0, "Above Normal", "warm"),
|
|
|
|
|
|
(59.9, "Normal", "normal"),
|
|
|
|
|
|
(40.0, "Normal", "normal"),
|
|
|
|
|
|
(39.9, "Below Normal", "cool"),
|
|
|
|
|
|
(10.0, "Low", "cold"),
|
|
|
|
|
|
(1.0, "Very Low", "very-cold"),
|
|
|
|
|
|
(0.5, "Near Record", "rec-cold"), # strictly below the 1st percentile
|
|
|
|
|
|
])
|
|
|
|
|
|
def test_temp_band_boundaries(pct, label, css):
|
|
|
|
|
|
assert grading._band(pct, grading.TEMP_BANDS) == (label, css)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def test_ladders_stay_aligned_with_bands():
|
|
|
|
|
|
"""The detail-view ladders re-encode the band tables by hand; catch drift."""
|
|
|
|
|
|
for bands, ladder in [(grading.TEMP_BANDS, grading._TEMP_LADDER),
|
|
|
|
|
|
(grading.RAIN_BANDS, grading._RAIN_LADDER)]:
|
|
|
|
|
|
assert len(bands) == len(ladder)
|
|
|
|
|
|
for (_, label, css), (lcss, llabel, *_rest) in zip(bands, ladder):
|
|
|
|
|
|
assert (label, css) == (llabel, lcss)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
# ---- seasonal window ----------------------------------------------------------
|
|
|
|
|
|
|
|
|
|
|
|
def test_window_mask_wraps_across_year_end():
|
|
|
|
|
|
doys = np.array([1, 180, 360, 366])
|
|
|
|
|
|
mask = grading.window_mask(doys, target_doy=1, half=7)
|
|
|
|
|
|
assert mask.tolist() == [True, False, True, True]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
# ---- precip grading -----------------------------------------------------------
|
|
|
|
|
|
|
2026-07-20 05:09:22 +00:00
|
|
|
|
def test_only_zero_precip_is_dry():
|
|
|
|
|
|
# Dry means no rain at all; any measurable rain, however slight, is at least a
|
|
|
|
|
|
# Trace day (not Dry).
|
|
|
|
|
|
dry = grading._grade_precip(np.array([0.0, 0.5, 1.0]), 0.0)
|
|
|
|
|
|
assert dry["class"] == "dry" and dry["percentile"] is None and dry["grade"] == "Dry"
|
|
|
|
|
|
trace = grading._grade_precip(np.array([0.0, 0.5, 1.0]), 0.005)
|
|
|
|
|
|
assert trace["class"] != "dry" and trace["grade"] == "Trace"
|
Add backend test suite; gate direct pushes; serialize LAN deploys (#41)
- backend/tests: 74 hermetic tests (no network, no repo data//logs/ writes)
covering grid snapping/round-trips, grading percentiles/bands/windows/
dry streaks, the places index (norm, one-edit matchers, search,
corrections), the derived store (token validity, cache=False, degraded
mode), and route-level API tests over a faked climate layer — routing,
validation, ETag/304 revalidation, store replay, the /cell bundle, and
the v1/v2 aliases. The API tests would have caught the /place
AttributeError regression.
- requirements-dev.txt + make test (venv prefers uv-pinned 3.12, matching
deploy-dev.sh — pyarrow wheels stop at 3.12 and some pyenv builds lack
sqlite).
- CI: extract the build job into a reusable build.yml, add the test run
and an API health probe (page-only curl can't catch route wiring
faults); deploy-dev.yml now runs the same build gate before deploying
direct pushes, which previously deployed with no CI at all.
- Deploys serialize under one dev-lan-deploy concurrency group across
both workflows (previously per-PR groups could interleave two deploys
to the same checkout), and are never cancelled mid-restart.
- deploy-dev.sh health check also probes /api/v2/place — best-effort
externals mean a failure there is a genuine server bug.
2026-07-11 19:37:49 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def test_rain_percentile_ranks_among_rain_days_only():
|
|
|
|
|
|
# 7 dry days + rain days [0.1, 0.2, 0.4]; 0.2 ranks among the 3 rain days:
|
2026-07-20 04:38:30 +00:00
|
|
|
|
# less=1, equal=1 -> (1 + 0.5) / 3 = 50% -> Typical, unaffected by the dry mass.
|
Add backend test suite; gate direct pushes; serialize LAN deploys (#41)
- backend/tests: 74 hermetic tests (no network, no repo data//logs/ writes)
covering grid snapping/round-trips, grading percentiles/bands/windows/
dry streaks, the places index (norm, one-edit matchers, search,
corrections), the derived store (token validity, cache=False, degraded
mode), and route-level API tests over a faked climate layer — routing,
validation, ETag/304 revalidation, store replay, the /cell bundle, and
the v1/v2 aliases. The API tests would have caught the /place
AttributeError regression.
- requirements-dev.txt + make test (venv prefers uv-pinned 3.12, matching
deploy-dev.sh — pyarrow wheels stop at 3.12 and some pyenv builds lack
sqlite).
- CI: extract the build job into a reusable build.yml, add the test run
and an API health probe (page-only curl can't catch route wiring
faults); deploy-dev.yml now runs the same build gate before deploying
direct pushes, which previously deployed with no CI at all.
- Deploys serialize under one dev-lan-deploy concurrency group across
both workflows (previously per-PR groups could interleave two deploys
to the same checkout), and are never cancelled mid-restart.
- deploy-dev.sh health check also probes /api/v2/place — best-effort
externals mean a failure there is a genuine server bug.
2026-07-11 19:37:49 +00:00
|
|
|
|
samples = np.array([0.0] * 7 + [0.1, 0.2, 0.4])
|
|
|
|
|
|
g = grading._grade_precip(samples, 0.2)
|
|
|
|
|
|
assert g["percentile"] == 50.0
|
2026-07-20 04:38:30 +00:00
|
|
|
|
assert g["grade"] == "Typical"
|
Add backend test suite; gate direct pushes; serialize LAN deploys (#41)
- backend/tests: 74 hermetic tests (no network, no repo data//logs/ writes)
covering grid snapping/round-trips, grading percentiles/bands/windows/
dry streaks, the places index (norm, one-edit matchers, search,
corrections), the derived store (token validity, cache=False, degraded
mode), and route-level API tests over a faked climate layer — routing,
validation, ETag/304 revalidation, store replay, the /cell bundle, and
the v1/v2 aliases. The API tests would have caught the /place
AttributeError regression.
- requirements-dev.txt + make test (venv prefers uv-pinned 3.12, matching
deploy-dev.sh — pyarrow wheels stop at 3.12 and some pyenv builds lack
sqlite).
- CI: extract the build job into a reusable build.yml, add the test run
and an API health probe (page-only curl can't catch route wiring
faults); deploy-dev.yml now runs the same build gate before deploying
direct pushes, which previously deployed with no CI at all.
- Deploys serialize under one dev-lan-deploy concurrency group across
both workflows (previously per-PR groups could interleave two deploys
to the same checkout), and are never cancelled mid-restart.
- deploy-dev.sh health check also probes /api/v2/place — best-effort
externals mean a failure there is a genuine server bug.
2026-07-11 19:37:49 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def test_rain_with_no_historical_rain_days_is_extreme():
|
|
|
|
|
|
g = grading._grade_precip(np.zeros(10), 0.3)
|
|
|
|
|
|
assert g["percentile"] == 100.0 and g["class"] == "wet-9"
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-07-20 05:09:22 +00:00
|
|
|
|
def test_rain_scale_eight_tiers_with_severe():
|
|
|
|
|
|
# Eight tiers filling the wet-2..wet-9 ramp: Trace / Light / Brisk / Typical /
|
|
|
|
|
|
# Heavy / Very Heavy / Severe / Extreme. Very Heavy was split, its top half
|
|
|
|
|
|
# becoming Severe (95–99).
|
|
|
|
|
|
assert [b[1] for b in grading.RAIN_BANDS] == \
|
|
|
|
|
|
["Extreme", "Severe", "Very Heavy", "Heavy", "Typical", "Brisk", "Light", "Trace"]
|
|
|
|
|
|
assert [b[2] for b in grading.RAIN_BANDS] == \
|
|
|
|
|
|
["wet-9", "wet-8", "wet-7", "wet-6", "wet-5", "wet-4", "wet-3", "wet-2"]
|
|
|
|
|
|
|
|
|
|
|
|
rain = np.arange(1, 201, dtype=float) / 100.0 # 200 rain days: 0.01 .. 2.00
|
2026-07-20 04:25:13 +00:00
|
|
|
|
samples = np.concatenate([np.zeros(20), rain])
|
2026-07-20 05:09:22 +00:00
|
|
|
|
|
|
|
|
|
|
def grade(v):
|
|
|
|
|
|
g = grading._grade_precip(samples, v)
|
|
|
|
|
|
return g["grade"], g["class"]
|
|
|
|
|
|
assert grade(0.01) == ("Trace", "wet-2") # <1st pct -> the bottom tier
|
|
|
|
|
|
assert grade(1.35) == ("Heavy", "wet-6") # ~67th pct
|
|
|
|
|
|
assert grade(1.85) == ("Very Heavy", "wet-7") # ~92nd pct (lower half)
|
|
|
|
|
|
assert grade(1.96) == ("Severe", "wet-8") # ~98th pct (top half -> Severe)
|
|
|
|
|
|
assert grade(2.00) == ("Extreme", "wet-9") # >99th pct
|
2026-07-20 04:25:13 +00:00
|
|
|
|
|
|
|
|
|
|
|
Add backend test suite; gate direct pushes; serialize LAN deploys (#41)
- backend/tests: 74 hermetic tests (no network, no repo data//logs/ writes)
covering grid snapping/round-trips, grading percentiles/bands/windows/
dry streaks, the places index (norm, one-edit matchers, search,
corrections), the derived store (token validity, cache=False, degraded
mode), and route-level API tests over a faked climate layer — routing,
validation, ETag/304 revalidation, store replay, the /cell bundle, and
the v1/v2 aliases. The API tests would have caught the /place
AttributeError regression.
- requirements-dev.txt + make test (venv prefers uv-pinned 3.12, matching
deploy-dev.sh — pyarrow wheels stop at 3.12 and some pyenv builds lack
sqlite).
- CI: extract the build job into a reusable build.yml, add the test run
and an API health probe (page-only curl can't catch route wiring
faults); deploy-dev.yml now runs the same build gate before deploying
direct pushes, which previously deployed with no CI at all.
- Deploys serialize under one dev-lan-deploy concurrency group across
both workflows (previously per-PR groups could interleave two deploys
to the same checkout), and are never cancelled mid-restart.
- deploy-dev.sh health check also probes /api/v2/place — best-effort
externals mean a failure there is a genuine server bug.
2026-07-11 19:37:49 +00:00
|
|
|
|
# ---- dry streaks ---------------------------------------------------------------
|
|
|
|
|
|
|
|
|
|
|
|
def test_dry_streaks_walk():
|
Migrate backend dataframe layer from pandas to polars (#90)
* Migrate backend dataframe layer from pandas to polars
Replace pandas with polars across the backend, dropping both pandas and its
pyarrow parquet engine from the dependency set. numpy stays (the grading
percentile math is unchanged).
- climate.py: parquet IO, source→frame mappings, cache read/topup on polars.
New _normalize_read casts the cached `date` column to pl.Date (older files
were written by pandas as datetime64[ns]); frames now unify missing values as
null so the grading boundary drops them consistently across sources.
- grading.py: keep the numpy percentile core; swap the frame→numpy bridge to
.to_numpy()/.drop_nulls(), day-of-year/year to polars dt expressions, and the
per-row loop to iter_rows(named=True).
- views.py: filter/anti-join/concat replace boolean-mask, isin and pd.concat;
scalar dates are stdlib datetime.date; a local _months_before helper replaces
DateOffset(months=) for the calendar-range default.
- app.py, migrate.py: request-date parsing uses datetime.date, removing pandas
from the endpoint and migrate layers entirely.
- The date column is pl.Date end to end, eliminating the pandas normalize() calls
and comparing cleanly against stdlib dates.
Payloads are unchanged: calendar, day, grade and forecast responses are
byte-for-byte identical to the pandas implementation on the same cached record.
Tests ported to polars fixtures, with added coverage for the combined feels-like
fallback, calendar month-offset (month-end/leap), and the concat/dedup
"fresher source wins" rule.
* Port notify.py to polars after merging dev's account system
Merge origin/dev (accounts + notification subscriptions) and carry the pandas→
polars migration into the newly added notify.py, which the merge brought in still
using pandas — with pandas removed from requirements this broke its import.
- notify.py: _candidate_rows filters/sorts the recent bundle with polars
expressions and returns iter_rows dicts; date scalars are datetime.date;
history/recent emptiness via is_empty().
- test_notify.py: synthetic history/rows built with polars + datetime.
2026-07-15 19:07:38 +00:00
|
|
|
|
dates = [datetime.date(2024, 1, 1) + datetime.timedelta(days=i) for i in range(5)]
|
Add backend test suite; gate direct pushes; serialize LAN deploys (#41)
- backend/tests: 74 hermetic tests (no network, no repo data//logs/ writes)
covering grid snapping/round-trips, grading percentiles/bands/windows/
dry streaks, the places index (norm, one-edit matchers, search,
corrections), the derived store (token validity, cache=False, degraded
mode), and route-level API tests over a faked climate layer — routing,
validation, ETag/304 revalidation, store replay, the /cell bundle, and
the v1/v2 aliases. The API tests would have caught the /place
AttributeError regression.
- requirements-dev.txt + make test (venv prefers uv-pinned 3.12, matching
deploy-dev.sh — pyarrow wheels stop at 3.12 and some pyenv builds lack
sqlite).
- CI: extract the build job into a reusable build.yml, add the test run
and an API health probe (page-only curl can't catch route wiring
faults); deploy-dev.yml now runs the same build gate before deploying
direct pushes, which previously deployed with no CI at all.
- Deploys serialize under one dev-lan-deploy concurrency group across
both workflows (previously per-PR groups could interleave two deploys
to the same checkout), and are never cancelled mid-restart.
- deploy-dev.sh health check also probes /api/v2/place — best-effort
externals mean a failure there is a genuine server bug.
2026-07-11 19:37:49 +00:00
|
|
|
|
precips = [0.5, 0.0, float("nan"), 0.02, 0.005]
|
Migrate backend dataframe layer from pandas to polars (#90)
* Migrate backend dataframe layer from pandas to polars
Replace pandas with polars across the backend, dropping both pandas and its
pyarrow parquet engine from the dependency set. numpy stays (the grading
percentile math is unchanged).
- climate.py: parquet IO, source→frame mappings, cache read/topup on polars.
New _normalize_read casts the cached `date` column to pl.Date (older files
were written by pandas as datetime64[ns]); frames now unify missing values as
null so the grading boundary drops them consistently across sources.
- grading.py: keep the numpy percentile core; swap the frame→numpy bridge to
.to_numpy()/.drop_nulls(), day-of-year/year to polars dt expressions, and the
per-row loop to iter_rows(named=True).
- views.py: filter/anti-join/concat replace boolean-mask, isin and pd.concat;
scalar dates are stdlib datetime.date; a local _months_before helper replaces
DateOffset(months=) for the calendar-range default.
- app.py, migrate.py: request-date parsing uses datetime.date, removing pandas
from the endpoint and migrate layers entirely.
- The date column is pl.Date end to end, eliminating the pandas normalize() calls
and comparing cleanly against stdlib dates.
Payloads are unchanged: calendar, day, grade and forecast responses are
byte-for-byte identical to the pandas implementation on the same cached record.
Tests ported to polars fixtures, with added coverage for the combined feels-like
fallback, calendar month-offset (month-end/leap), and the concat/dedup
"fresher source wins" rule.
* Port notify.py to polars after merging dev's account system
Merge origin/dev (accounts + notification subscriptions) and carry the pandas→
polars migration into the newly added notify.py, which the merge brought in still
using pandas — with pandas removed from requirements this broke its import.
- notify.py: _candidate_rows filters/sorts the recent bundle with polars
expressions and returns iter_rows dicts; date scalars are datetime.date;
history/recent emptiness via is_empty().
- test_notify.py: synthetic history/rows built with polars + datetime.
2026-07-15 19:07:38 +00:00
|
|
|
|
out = grading.dry_streaks(dates, precips)
|
Add backend test suite; gate direct pushes; serialize LAN deploys (#41)
- backend/tests: 74 hermetic tests (no network, no repo data//logs/ writes)
covering grid snapping/round-trips, grading percentiles/bands/windows/
dry streaks, the places index (norm, one-edit matchers, search,
corrections), the derived store (token validity, cache=False, degraded
mode), and route-level API tests over a faked climate layer — routing,
validation, ETag/304 revalidation, store replay, the /cell bundle, and
the v1/v2 aliases. The API tests would have caught the /place
AttributeError regression.
- requirements-dev.txt + make test (venv prefers uv-pinned 3.12, matching
deploy-dev.sh — pyarrow wheels stop at 3.12 and some pyenv builds lack
sqlite).
- CI: extract the build job into a reusable build.yml, add the test run
and an API health probe (page-only curl can't catch route wiring
faults); deploy-dev.yml now runs the same build gate before deploying
direct pushes, which previously deployed with no CI at all.
- Deploys serialize under one dev-lan-deploy concurrency group across
both workflows (previously per-PR groups could interleave two deploys
to the same checkout), and are never cancelled mid-restart.
- deploy-dev.sh health check also probes /api/v2/place — best-effort
externals mean a failure there is a genuine server bug.
2026-07-11 19:37:49 +00:00
|
|
|
|
assert list(out.values()) == [0, 1, 2, 0, 1] # NaN counts as dry
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
# ---- range + day grading over a synthetic record --------------------------------
|
|
|
|
|
|
|
|
|
|
|
|
def test_grade_range_compact_shape(history):
|
Migrate backend dataframe layer from pandas to polars (#90)
* Migrate backend dataframe layer from pandas to polars
Replace pandas with polars across the backend, dropping both pandas and its
pyarrow parquet engine from the dependency set. numpy stays (the grading
percentile math is unchanged).
- climate.py: parquet IO, source→frame mappings, cache read/topup on polars.
New _normalize_read casts the cached `date` column to pl.Date (older files
were written by pandas as datetime64[ns]); frames now unify missing values as
null so the grading boundary drops them consistently across sources.
- grading.py: keep the numpy percentile core; swap the frame→numpy bridge to
.to_numpy()/.drop_nulls(), day-of-year/year to polars dt expressions, and the
per-row loop to iter_rows(named=True).
- views.py: filter/anti-join/concat replace boolean-mask, isin and pd.concat;
scalar dates are stdlib datetime.date; a local _months_before helper replaces
DateOffset(months=) for the calendar-range default.
- app.py, migrate.py: request-date parsing uses datetime.date, removing pandas
from the endpoint and migrate layers entirely.
- The date column is pl.Date end to end, eliminating the pandas normalize() calls
and comparing cleanly against stdlib dates.
Payloads are unchanged: calendar, day, grade and forecast responses are
byte-for-byte identical to the pandas implementation on the same cached record.
Tests ported to polars fixtures, with added coverage for the combined feels-like
fallback, calendar month-offset (month-end/leap), and the concat/dedup
"fresher source wins" rule.
* Port notify.py to polars after merging dev's account system
Merge origin/dev (accounts + notification subscriptions) and carry the pandas→
polars migration into the newly added notify.py, which the merge brought in still
using pandas — with pandas removed from requirements this broke its import.
- notify.py: _candidate_rows filters/sorts the recent bundle with polars
expressions and returns iter_rows dicts; date scalars are datetime.date;
history/recent emptiness via is_empty().
- test_notify.py: synthetic history/rows built with polars + datetime.
2026-07-15 19:07:38 +00:00
|
|
|
|
end = history["date"].max()
|
|
|
|
|
|
start = end - datetime.timedelta(days=30)
|
Add backend test suite; gate direct pushes; serialize LAN deploys (#41)
- backend/tests: 74 hermetic tests (no network, no repo data//logs/ writes)
covering grid snapping/round-trips, grading percentiles/bands/windows/
dry streaks, the places index (norm, one-edit matchers, search,
corrections), the derived store (token validity, cache=False, degraded
mode), and route-level API tests over a faked climate layer — routing,
validation, ETag/304 revalidation, store replay, the /cell bundle, and
the v1/v2 aliases. The API tests would have caught the /place
AttributeError regression.
- requirements-dev.txt + make test (venv prefers uv-pinned 3.12, matching
deploy-dev.sh — pyarrow wheels stop at 3.12 and some pyenv builds lack
sqlite).
- CI: extract the build job into a reusable build.yml, add the test run
and an API health probe (page-only curl can't catch route wiring
faults); deploy-dev.yml now runs the same build gate before deploying
direct pushes, which previously deployed with no CI at all.
- Deploys serialize under one dev-lan-deploy concurrency group across
both workflows (previously per-PR groups could interleave two deploys
to the same checkout), and are never cancelled mid-restart.
- deploy-dev.sh health check also probes /api/v2/place — best-effort
externals mean a failure there is a genuine server bug.
2026-07-11 19:37:49 +00:00
|
|
|
|
days = grading.grade_range(history, start, end)
|
|
|
|
|
|
assert len(days) == 31
|
|
|
|
|
|
first = days[0]
|
|
|
|
|
|
assert set(first) == {"date", "dsr", *grading.TEMP_METRICS, "precip"}
|
|
|
|
|
|
assert first["dsr"] >= 0
|
|
|
|
|
|
g = first["tmax"]
|
|
|
|
|
|
assert set(g) == {"v", "pct", "c", "g"}
|
|
|
|
|
|
for rec in days: # dry days carry no rain percentile, wet days always do
|
|
|
|
|
|
if rec["precip"]["c"] == "dry":
|
|
|
|
|
|
assert rec["precip"]["pct"] is None
|
|
|
|
|
|
else:
|
|
|
|
|
|
assert rec["precip"]["pct"] is not None
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def test_grade_day_normals_and_departure(history):
|
Migrate backend dataframe layer from pandas to polars (#90)
* Migrate backend dataframe layer from pandas to polars
Replace pandas with polars across the backend, dropping both pandas and its
pyarrow parquet engine from the dependency set. numpy stays (the grading
percentile math is unchanged).
- climate.py: parquet IO, source→frame mappings, cache read/topup on polars.
New _normalize_read casts the cached `date` column to pl.Date (older files
were written by pandas as datetime64[ns]); frames now unify missing values as
null so the grading boundary drops them consistently across sources.
- grading.py: keep the numpy percentile core; swap the frame→numpy bridge to
.to_numpy()/.drop_nulls(), day-of-year/year to polars dt expressions, and the
per-row loop to iter_rows(named=True).
- views.py: filter/anti-join/concat replace boolean-mask, isin and pd.concat;
scalar dates are stdlib datetime.date; a local _months_before helper replaces
DateOffset(months=) for the calendar-range default.
- app.py, migrate.py: request-date parsing uses datetime.date, removing pandas
from the endpoint and migrate layers entirely.
- The date column is pl.Date end to end, eliminating the pandas normalize() calls
and comparing cleanly against stdlib dates.
Payloads are unchanged: calendar, day, grade and forecast responses are
byte-for-byte identical to the pandas implementation on the same cached record.
Tests ported to polars fixtures, with added coverage for the combined feels-like
fallback, calendar month-offset (month-end/leap), and the concat/dedup
"fresher source wins" rule.
* Port notify.py to polars after merging dev's account system
Merge origin/dev (accounts + notification subscriptions) and carry the pandas→
polars migration into the newly added notify.py, which the merge brought in still
using pandas — with pandas removed from requirements this broke its import.
- notify.py: _candidate_rows filters/sorts the recent bundle with polars
expressions and returns iter_rows dicts; date scalars are datetime.date;
history/recent emptiness via is_empty().
- test_notify.py: synthetic history/rows built with polars + datetime.
2026-07-15 19:07:38 +00:00
|
|
|
|
target = history["date"].max()
|
|
|
|
|
|
row = history.filter(pl.col("date") == target).row(0, named=True)
|
Add backend test suite; gate direct pushes; serialize LAN deploys (#41)
- backend/tests: 74 hermetic tests (no network, no repo data//logs/ writes)
covering grid snapping/round-trips, grading percentiles/bands/windows/
dry streaks, the places index (norm, one-edit matchers, search,
corrections), the derived store (token validity, cache=False, degraded
mode), and route-level API tests over a faked climate layer — routing,
validation, ETag/304 revalidation, store replay, the /cell bundle, and
the v1/v2 aliases. The API tests would have caught the /place
AttributeError regression.
- requirements-dev.txt + make test (venv prefers uv-pinned 3.12, matching
deploy-dev.sh — pyarrow wheels stop at 3.12 and some pyenv builds lack
sqlite).
- CI: extract the build job into a reusable build.yml, add the test run
and an API health probe (page-only curl can't catch route wiring
faults); deploy-dev.yml now runs the same build gate before deploying
direct pushes, which previously deployed with no CI at all.
- Deploys serialize under one dev-lan-deploy concurrency group across
both workflows (previously per-PR groups could interleave two deploys
to the same checkout), and are never cancelled mid-restart.
- deploy-dev.sh health check also probes /api/v2/place — best-effort
externals mean a failure there is a genuine server bug.
2026-07-11 19:37:49 +00:00
|
|
|
|
result = grading.grade_day(history, target, {"tmax": row["tmax"], "tmin": row["tmin"],
|
|
|
|
|
|
"precip": row["precip"]})
|
|
|
|
|
|
assert set(result["normals"]["tmax"]) == {"p1", "p10", "p25", "p40", "p50", "p60",
|
|
|
|
|
|
"p75", "p90", "p99"}
|
|
|
|
|
|
expected = max(abs(result["tmax"]["percentile"] - 50), abs(result["tmin"]["percentile"] - 50))
|
|
|
|
|
|
assert result["departure"] == round(expected, 1)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def test_day_detail_with_and_without_observation(history):
|
Migrate backend dataframe layer from pandas to polars (#90)
* Migrate backend dataframe layer from pandas to polars
Replace pandas with polars across the backend, dropping both pandas and its
pyarrow parquet engine from the dependency set. numpy stays (the grading
percentile math is unchanged).
- climate.py: parquet IO, source→frame mappings, cache read/topup on polars.
New _normalize_read casts the cached `date` column to pl.Date (older files
were written by pandas as datetime64[ns]); frames now unify missing values as
null so the grading boundary drops them consistently across sources.
- grading.py: keep the numpy percentile core; swap the frame→numpy bridge to
.to_numpy()/.drop_nulls(), day-of-year/year to polars dt expressions, and the
per-row loop to iter_rows(named=True).
- views.py: filter/anti-join/concat replace boolean-mask, isin and pd.concat;
scalar dates are stdlib datetime.date; a local _months_before helper replaces
DateOffset(months=) for the calendar-range default.
- app.py, migrate.py: request-date parsing uses datetime.date, removing pandas
from the endpoint and migrate layers entirely.
- The date column is pl.Date end to end, eliminating the pandas normalize() calls
and comparing cleanly against stdlib dates.
Payloads are unchanged: calendar, day, grade and forecast responses are
byte-for-byte identical to the pandas implementation on the same cached record.
Tests ported to polars fixtures, with added coverage for the combined feels-like
fallback, calendar month-offset (month-end/leap), and the concat/dedup
"fresher source wins" rule.
* Port notify.py to polars after merging dev's account system
Merge origin/dev (accounts + notification subscriptions) and carry the pandas→
polars migration into the newly added notify.py, which the merge brought in still
using pandas — with pandas removed from requirements this broke its import.
- notify.py: _candidate_rows filters/sorts the recent bundle with polars
expressions and returns iter_rows dicts; date scalars are datetime.date;
history/recent emptiness via is_empty().
- test_notify.py: synthetic history/rows built with polars + datetime.
2026-07-15 19:07:38 +00:00
|
|
|
|
target = history["date"].max()
|
Add backend test suite; gate direct pushes; serialize LAN deploys (#41)
- backend/tests: 74 hermetic tests (no network, no repo data//logs/ writes)
covering grid snapping/round-trips, grading percentiles/bands/windows/
dry streaks, the places index (norm, one-edit matchers, search,
corrections), the derived store (token validity, cache=False, degraded
mode), and route-level API tests over a faked climate layer — routing,
validation, ETag/304 revalidation, store replay, the /cell bundle, and
the v1/v2 aliases. The API tests would have caught the /place
AttributeError regression.
- requirements-dev.txt + make test (venv prefers uv-pinned 3.12, matching
deploy-dev.sh — pyarrow wheels stop at 3.12 and some pyenv builds lack
sqlite).
- CI: extract the build job into a reusable build.yml, add the test run
and an API health probe (page-only curl can't catch route wiring
faults); deploy-dev.yml now runs the same build gate before deploying
direct pushes, which previously deployed with no CI at all.
- Deploys serialize under one dev-lan-deploy concurrency group across
both workflows (previously per-PR groups could interleave two deploys
to the same checkout), and are never cancelled mid-restart.
- deploy-dev.sh health check also probes /api/v2/place — best-effort
externals mean a failure there is a genuine server bug.
2026-07-11 19:37:49 +00:00
|
|
|
|
detail = grading.day_detail(history, target, {"tmax": 60.0, "precip": 0.0})
|
|
|
|
|
|
assert detail["metrics"]["tmax"]["obs"]["value"] == 60.0
|
|
|
|
|
|
assert detail["metrics"]["tmax"]["ladder"]["tiers"][0]["c"] == "rec-hot"
|
|
|
|
|
|
assert detail["metrics"]["precip"]["ladder"]["tiers"][-1]["c"] == "dry"
|
|
|
|
|
|
|
|
|
|
|
|
bare = grading.day_detail(history, target, None)
|
|
|
|
|
|
assert bare["metrics"]["tmax"]["obs"] is None
|
|
|
|
|
|
assert bare["metrics"]["tmax"]["ladder"] is not None
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def test_climatology_summary(history):
|
|
|
|
|
|
climo = grading.climatology(history, 180)
|
|
|
|
|
|
# ±7-day window over ~20 years -> ~15 samples per year.
|
|
|
|
|
|
assert climo["n_samples"] >= 15 * 19
|
|
|
|
|
|
assert climo["tmax"]["p40"] <= climo["tmax"]["p50"] <= climo["tmax"]["p60"]
|
|
|
|
|
|
assert climo["feels"] is None # column absent from this record
|