Commit graph

3 commits

Author SHA1 Message Date
Emi Griffith
c3bacdce0c Migrate backend dataframe layer from pandas to polars (#90)
* Migrate backend dataframe layer from pandas to polars

Replace pandas with polars across the backend, dropping both pandas and its
pyarrow parquet engine from the dependency set. numpy stays (the grading
percentile math is unchanged).

- climate.py: parquet IO, source→frame mappings, cache read/topup on polars.
  New _normalize_read casts the cached `date` column to pl.Date (older files
  were written by pandas as datetime64[ns]); frames now unify missing values as
  null so the grading boundary drops them consistently across sources.
- grading.py: keep the numpy percentile core; swap the frame→numpy bridge to
  .to_numpy()/.drop_nulls(), day-of-year/year to polars dt expressions, and the
  per-row loop to iter_rows(named=True).
- views.py: filter/anti-join/concat replace boolean-mask, isin and pd.concat;
  scalar dates are stdlib datetime.date; a local _months_before helper replaces
  DateOffset(months=) for the calendar-range default.
- app.py, migrate.py: request-date parsing uses datetime.date, removing pandas
  from the endpoint and migrate layers entirely.
- The date column is pl.Date end to end, eliminating the pandas normalize() calls
  and comparing cleanly against stdlib dates.

Payloads are unchanged: calendar, day, grade and forecast responses are
byte-for-byte identical to the pandas implementation on the same cached record.

Tests ported to polars fixtures, with added coverage for the combined feels-like
fallback, calendar month-offset (month-end/leap), and the concat/dedup
"fresher source wins" rule.

* Port notify.py to polars after merging dev's account system

Merge origin/dev (accounts + notification subscriptions) and carry the pandas→
polars migration into the newly added notify.py, which the merge brought in still
using pandas — with pandas removed from requirements this broke its import.

- notify.py: _candidate_rows filters/sorts the recent bundle with polars
  expressions and returns iter_rows dicts; date scalars are datetime.date;
  history/recent emptiness via is_empty().
- test_notify.py: synthetic history/rows built with polars + datetime.
2026-07-15 19:07:38 +00:00
7aaad17603 Deduplicate the data-layer plumbing in climate.py and grading.py (#46)
climate.py repeated the same four mechanical patterns:
- the Open-Meteo daily params dict (3x) -> _om_daily_params(cell, **window)
- the doy attach line (6x) -> _with_doy
- makedirs + drop-doy + zstd to_parquet (3x) -> _write_cache
- the identical _to_frame/_nasa_to_frame tail (feels-like, valid-day
  filter, doy) -> _finalize_frame

grading.py encoded the tier tables twice — TEMP_BANDS/RAIN_BANDS plus the
hand-aligned _TEMP_LADDER/_RAIN_LADDER ('kept aligned' by comment). The
ladders are now derived from the bands (_ladder_from; verified
byte-identical to the old tables before landing), so tier boundaries have
exactly one definition. grade_range's inline dry-streak walk is replaced
with the existing dry_streaks(); its per-(doy,var) sample cache now also
memoizes the window mask per doy instead of recomputing it once per
metric (9x per day-of-year).

New tests pin the refactor: _to_frame schema/day-filter/missing-series
tolerance, the combined feels-like side selection, NASA unit conversions
and fill-sentinel handling, _om_daily_params windows, and _write_cache
stripping the derived doy column.
2026-07-11 20:05:57 +00:00
Emi Griffith
8d90dcb456 Initial commit: app +
VPS deploy pipeline
2026-07-10 17:29:47 -07:00