* Centralize filesystem paths in a single module
Add paths.py, which resolves the repo root once and derives the cache,
accounts DB, logs, templates, frontend and bundled-city-data locations
from it. Replace the 13 per-module `dirname(__file__)/..` anchors with
references to it, so a module's location no longer determines where the
app reads its data. Env overrides (accounts DB, VAPID, IndexNow) are
unchanged; every resolved path is byte-identical to before.
Groundwork for moving modules into packages without re-pointing paths.
Claude-Session: https://claude.ai/code/session_01XXxmNFy9cZ6Gh8Y9thZn62
* Split the backend into domain packages
Group the flat backend modules into packages that mirror their concerns:
data/ climate, grading, scoring, grid, places, cities,
city_events, store
web/ app, views, homepage, content, schemas
notifications/ notify, digest, push, mailer, discord,
discord_interactions, discord_link
accounts/ models, users, api_accounts, db
core/ metrics, singleton, audit
Intra-project imports are rewritten to the package-qualified form. The
entry scripts (indexnow, warm_cities, migrate, gen_cities, gen_flavor)
and paths.py stay at the backend/ root, and backend/app.py becomes a
shim re-exporting web.app:app so the launch target stays `app:app` —
run.sh, the systemd units, and CI need no change.
Verified: full suite (318) passes, `uvicorn app:app` boots and serves
the home/SEO/static/API surfaces, and every root script imports clean.
Claude-Session: https://claude.ai/code/session_01XXxmNFy9cZ6Gh8Y9thZn62
Rain intensity gains a Severe tier and Dry becomes strictly no-rain:
- The top half of Very Heavy (95th–99th rain-day percentile) becomes a new
"Severe" tier; Very Heavy keeps the 90–95 band. Eight rain tiers now — Trace /
Light / Brisk / Typical / Heavy / Very Heavy / Severe / Extreme — which refill
the wet-2..wet-9 colour ramp contiguously (no gap), so Heavy/Very Heavy shift
one shade lighter and Severe takes the second-darkest.
- A day is Dry only when it didn't rain at all; any measurable rain, however
slight, is at least Trace. _grade_precip splits on > 0 rather than the 0.01"
threshold (which still governs the separate dry-streak metric).
- The distribution strip drops the range under the Dry column — every dry day is
zero, so a "0–0" span was noise.
_precip_ladder derives its percentile marks from RAIN_BANDS now, so adding or
splitting a tier can't leave a hard-coded list behind (that was the bug the 95th
mark would have hit). The detail-view ladder derives from the band table as before.
Claude-Session: https://claude.ai/code/session_013dRZmX9D3JEntfMKWMTWZ8
The rain-intensity scale drops its two hyphenated compound labels for single
words and loses a tier, going from eight to seven:
Trace / Light / Brisk / Typical / Heavy / Very Heavy / Extreme
- Light–Mod -> Brisk, Moderate -> Typical (renames only; classes unchanged).
- Mod–Heavy is merged up into Heavy, whose floor drops from the 75th to the 60th
rain-day percentile, so Heavy now spans 60–90.
- The lightest tier (already the merged Very Light) is renamed Trace.
grading.py RAIN_BANDS and the frontend SCALE_RAIN mirror stay in lockstep, and
the detail-view ladder derives from the table so it follows automatically. The
now-unreferenced wet-6 colour token is kept so any day still cached under that
class renders until the derived store recomputes; the chart's percentile fan also
keeps it for a smooth gradient.
Claude-Session: https://claude.ai/code/session_013dRZmX9D3JEntfMKWMTWZ8
The rain-intensity scale had a Trace tier (below the 1st percentile of a place's
rain days) sitting under Very Light — a sliver category that mostly showed 1% and
crowded the distribution strip. Fold it into Very Light, which now bottoms out the
scale at 0, so the lightest measurable rain reads as Very Light.
- grading.py: RAIN_BANDS drops the Trace band; Very Light's floor goes 1 -> 0. The
detail-view ladder derives from the table, so it follows automatically.
- shared.js: SCALE_RAIN drops the Trace row; Very Light's range becomes "<10".
Eight rain tiers now instead of nine; the strip shows one fewer column. The wet-1
colour token is kept (unreferenced by new gradings) so any day still cached with
the old class renders until the derived store recomputes.
Claude-Session: https://claude.ai/code/session_013dRZmX9D3JEntfMKWMTWZ8
The homepage strip floored percentiles into 1-99 while the Day page rendered
"100th pct" for the same reading. Rather than teach the Day page a second copy
of the rule, put it in grading.pct_ordinal() and have every surface defer to it:
the Day page, the calendar tooltip, the chart labels, the city pages and the
strip.
Two bugs fell out of unifying them:
- The city pages rendered `{{ c.percentile }}th pct` against a float, so all
~1000 of them showed "97.1th pct", "66.4th pct", "16.5th pct" — and would say
"1th"/"2th"/"3th" for the low tail. Live in prod.
- Python's round() is half-to-even and JavaScript's Math.round is half-up, so
16.5 gave "16th" server-side and "17th" client-side. The Python helper uses
floor(x + 0.5) to match the frontend exactly; a cross-language check over every
.5 boundary now agrees on all of them.
The frontend mirrors the rule as pctOrd() in shared.js, replacing the bare ord()
at all five percentile call sites (day.js, calendar.js, chart.js x2, app.js).
ord() stays as the general ordinal helper it always was.
Record-low rainfall is meaningless (always ~0), so the precip row's low side now
shows the longest run of consecutive days without measurable rain and the date that
dry spell began (grading.longest_dry_streak), keeping the wettest day on the high
side. A footnote explains the substitution.
* SEO: generate curated city set for crawlable climate pages
gen_cities.py reuses the GeoNames index places.py already parses to select the top
~500 metros by population, assigns each a stable URL-safe slug (dropping admin1 when
it repeats the city name), and writes committed backend/cities.json. cities.py loads
it lazily with slug lookup, all_slugs(), display_name(), and by_country() grouping
for the upcoming hub + sitemap.
* SEO: rendering core, robots.txt, sitemap.xml, and metadata hygiene
- content.py: Jinja2 environment + HTML responder (ETag/304), dynamic /robots.txt
(disallows /api and /alerts, points at the sitemap) and /sitemap.xml (enumerates
the home/static pages plus every city, month, and records URL from cities.py).
Registered on the app before the StaticFiles mount so the routes win.
- templates/base.html.j2: shared layout with unique title/description, self-
referential canonical, Open Graph, favicon/manifest, header nav (adds a Climate
link) and a footer link graph.
- Give each existing page a unique <meta description> (were 5x identical) and a
self-referential <link rel=canonical>; add WebApplication JSON-LD to the home page.
- Pin jinja2.
* SEO: server-rendered per-city climate page (/climate/{slug})
The keystone crawlable page: for a city it snaps to the grid cell, loads the
archive (fetching once if missing, self-healing), and renders as real HTML — a
'how today compares' block (grade + percentile per metric from grade_day, tinted
by tier), a monthly normals table (climatology at each month's 15th, shown in °F
and °C), all-time records (new grading.all_time_records helper), a breadcrumb,
Dataset+Place+BreadcrumbList JSON-LD, self-referential canonical, and links into
the interactive tool + month/records pages. Content-page CSS added to style.css
(renamed the table class to avoid colliding with the app's .normals flex row).
* SEO: month (/climate/{slug}/{month}) and records (/climate/{slug}/records) pages
Month pages render the exact-month long-tail ('average weather in {city} in
{month}') with that month's average high/low, typical p10-p90 range, month-specific
records, and prev/next month links. Records pages show all-time record highs/lows
per metric with dates (grading.all_time_records). Shared _resolve_city helper; the
literal /records route is registered before the {month} param and month names are
validated (unknown month -> 404).
* SEO: climate hub, weather glossary, and about/methodology pages
- /climate: crawlable directory of all ~500 cities grouped by country — the
internal-link graph that lets search engines discover every city page.
- /glossary + /glossary/{term}: plain-language definitions (climate normal,
percentile, temperature anomaly, feels-like, heat index, wind chill, humidity,
reanalysis) with DefinedTerm JSON-LD and cross-links into the tool.
- /about: methodology page (ERA5 data source, 45-year baseline, +/-7-day window,
percentile grading) for E-E-A-T. All linked from the shared footer.
* SEO: archive warmer, content-page tests, and deploy docs
- warm_cities.py: paced, idempotent offline warmer that pre-fetches each city
cell's archive so /climate pages serve from cache and a crawl can't burst the
archive quota (pages self-heal if hit before warming).
- tests/test_content.py: city-set slug uniqueness/lookup, robots.txt, sitemap
enumerating city/month/records URLs, and that a rendered city page carries the
stats + canonical + Dataset JSON-LD in the HTML; plus month/records/hub/glossary/
about routing and 404s.
- DEPLOY.md: document the content pages, the warm step, and submitting the sitemap.
* Migrate backend dataframe layer from pandas to polars
Replace pandas with polars across the backend, dropping both pandas and its
pyarrow parquet engine from the dependency set. numpy stays (the grading
percentile math is unchanged).
- climate.py: parquet IO, source→frame mappings, cache read/topup on polars.
New _normalize_read casts the cached `date` column to pl.Date (older files
were written by pandas as datetime64[ns]); frames now unify missing values as
null so the grading boundary drops them consistently across sources.
- grading.py: keep the numpy percentile core; swap the frame→numpy bridge to
.to_numpy()/.drop_nulls(), day-of-year/year to polars dt expressions, and the
per-row loop to iter_rows(named=True).
- views.py: filter/anti-join/concat replace boolean-mask, isin and pd.concat;
scalar dates are stdlib datetime.date; a local _months_before helper replaces
DateOffset(months=) for the calendar-range default.
- app.py, migrate.py: request-date parsing uses datetime.date, removing pandas
from the endpoint and migrate layers entirely.
- The date column is pl.Date end to end, eliminating the pandas normalize() calls
and comparing cleanly against stdlib dates.
Payloads are unchanged: calendar, day, grade and forecast responses are
byte-for-byte identical to the pandas implementation on the same cached record.
Tests ported to polars fixtures, with added coverage for the combined feels-like
fallback, calendar month-offset (month-end/leap), and the concat/dedup
"fresher source wins" rule.
* Port notify.py to polars after merging dev's account system
Merge origin/dev (accounts + notification subscriptions) and carry the pandas→
polars migration into the newly added notify.py, which the merge brought in still
using pandas — with pandas removed from requirements this broke its import.
- notify.py: _candidate_rows filters/sorts the recent bundle with polars
expressions and returns iter_rows dicts; date scalars are datetime.date;
history/recent emptiness via is_empty().
- test_notify.py: synthetic history/rows built with polars + datetime.
climate.py repeated the same four mechanical patterns:
- the Open-Meteo daily params dict (3x) -> _om_daily_params(cell, **window)
- the doy attach line (6x) -> _with_doy
- makedirs + drop-doy + zstd to_parquet (3x) -> _write_cache
- the identical _to_frame/_nasa_to_frame tail (feels-like, valid-day
filter, doy) -> _finalize_frame
grading.py encoded the tier tables twice — TEMP_BANDS/RAIN_BANDS plus the
hand-aligned _TEMP_LADDER/_RAIN_LADDER ('kept aligned' by comment). The
ladders are now derived from the bands (_ladder_from; verified
byte-identical to the old tables before landing), so tier boundaries have
exactly one definition. grade_range's inline dry-streak walk is replaced
with the existing dry_streaks(); its per-(doy,var) sample cache now also
memoizes the window mask per doy instead of recomputing it once per
metric (9x per day-of-year).
New tests pin the refactor: _to_frame schema/day-filter/missing-series
tolerance, the combined feels-like side selection, NASA unit conversions
and fill-sentinel handling, _om_daily_params windows, and _write_cache
stripping the derived doy column.
