The month and season records tables showed only the record high (of the daytime
high) and the record low (of the overnight low) — one extreme each. Show both
ends of both metrics: for the daytime high, its record hottest (▲) and the
coldest a day ever stayed (▼, its record-low high); for the overnight low, its
mildest night (▲, its record-high low) and its record low (▼). All four already
came out of all_time_records; they were just being dropped.
Restructure each table to Period · Daytime high · Overnight low, with each metric
cell carrying two tinted, dated chips (▲ warmest / ▼ coldest). Fits mobile in
three columns; the all-time cards already showed both extremes, so this brings
the per-period tables in line.
* SEO: monthly & seasonal records on the city records page
The /climate/<city>/records page showed only all-time records per metric.
Expand it into a full records page: record high/low for each of the 12
months (each linking its month page) and for each meteorological season,
alongside the existing all-time table.
Seasons are hemisphere-aware — Dec–Feb reads as winter for northern cities
and summer for southern ones (picked from the city's latitude). Records
reuse grading.all_time_records over a month-filtered archive, so no new
data fetching or warming is needed. Adds a Dataset + breadcrumb JSON-LD
block and richer title/description for the expanded coverage.
* SEO: colour-code climate & records pages (heat-map + range strip)
The city, records and month pages were plain muted tables — generic climate-site
styling. Bring the interactive grader's diverging cold→hot palette onto them so
they read as heat maps in the site's own visual language:
- Map absolute °F to the 9-tier palette (temp_class, a Jinja global) and tint the
temperature cells across the monthly-normals, monthly/seasonal/all-time records
tables. The value stays in the ink token; the tint is a wash behind it. Non-temp
rows (humidity/wind/precip) and date columns stay untinted.
- Add a monthly temperature-range strip on the city page: one gradient bar per
month spanning the average low→high on a shared −10..115°F axis, so a city's
whole-year rhythm (and hot-vs-cold character) reads at a glance.
- Tint the month page's hero high/low and its record values inline.
Palette, light/dark, and 390–1920px layouts verified by rendering hot (Phoenix)
and cold (Anchorage) cities.
Auto-sourcing a notable weather event from Wikipedia search proved unreliable
(wrong matches, low coverage), so city_events.py is a small, fact-checked list of
one iconic event per city (Katrina/New Orleans, Harvey/Houston, the 1952 Great
Smog/London, the 2021 heat dome/Seattle, ...) — 31 cities, each with a Wikipedia
link. The city page shows a 'Notable weather in {city}' callout when one exists and
otherwise just the flavor blurb. Tests check the curated slugs are all valid and the
callout renders (and is absent for uncurated cities).
Add unique editorial content so the programmatic pages don't read as templated:
- gen_flavor.py seeds backend/cities_flavor.json with a short descriptive blurb per
city from Wikipedia's free REST summary API (no key), validating each match by
comparing article coordinates to the city's lat/lon so the wrong 'Springfield'
never attaches. Retries + modest concurrency; ~700/750 cities get a blurb, the
rest render without one. Text is CC BY-SA, attributed with a 'via Wikipedia' link.
- City pages show the blurb under the intro and a travel callout that deep-links to
the compare page with the city pre-loaded (/compare#loc=lat,lon) — the visitor
just adds their own city. Month pages get the same seasonal 'visiting in {month}?'
compare link. Both add unique per-page text and internal links.
- cities.py gains flavor(slug); tests cover the blurb + attribution + compare CTA.
* SEO: generate curated city set for crawlable climate pages
gen_cities.py reuses the GeoNames index places.py already parses to select the top
~500 metros by population, assigns each a stable URL-safe slug (dropping admin1 when
it repeats the city name), and writes committed backend/cities.json. cities.py loads
it lazily with slug lookup, all_slugs(), display_name(), and by_country() grouping
for the upcoming hub + sitemap.
* SEO: rendering core, robots.txt, sitemap.xml, and metadata hygiene
- content.py: Jinja2 environment + HTML responder (ETag/304), dynamic /robots.txt
(disallows /api and /alerts, points at the sitemap) and /sitemap.xml (enumerates
the home/static pages plus every city, month, and records URL from cities.py).
Registered on the app before the StaticFiles mount so the routes win.
- templates/base.html.j2: shared layout with unique title/description, self-
referential canonical, Open Graph, favicon/manifest, header nav (adds a Climate
link) and a footer link graph.
- Give each existing page a unique <meta description> (were 5x identical) and a
self-referential <link rel=canonical>; add WebApplication JSON-LD to the home page.
- Pin jinja2.
* SEO: server-rendered per-city climate page (/climate/{slug})
The keystone crawlable page: for a city it snaps to the grid cell, loads the
archive (fetching once if missing, self-healing), and renders as real HTML — a
'how today compares' block (grade + percentile per metric from grade_day, tinted
by tier), a monthly normals table (climatology at each month's 15th, shown in °F
and °C), all-time records (new grading.all_time_records helper), a breadcrumb,
Dataset+Place+BreadcrumbList JSON-LD, self-referential canonical, and links into
the interactive tool + month/records pages. Content-page CSS added to style.css
(renamed the table class to avoid colliding with the app's .normals flex row).
* SEO: month (/climate/{slug}/{month}) and records (/climate/{slug}/records) pages
Month pages render the exact-month long-tail ('average weather in {city} in
{month}') with that month's average high/low, typical p10-p90 range, month-specific
records, and prev/next month links. Records pages show all-time record highs/lows
per metric with dates (grading.all_time_records). Shared _resolve_city helper; the
literal /records route is registered before the {month} param and month names are
validated (unknown month -> 404).
* SEO: climate hub, weather glossary, and about/methodology pages
- /climate: crawlable directory of all ~500 cities grouped by country — the
internal-link graph that lets search engines discover every city page.
- /glossary + /glossary/{term}: plain-language definitions (climate normal,
percentile, temperature anomaly, feels-like, heat index, wind chill, humidity,
reanalysis) with DefinedTerm JSON-LD and cross-links into the tool.
- /about: methodology page (ERA5 data source, 45-year baseline, +/-7-day window,
percentile grading) for E-E-A-T. All linked from the shared footer.
* SEO: archive warmer, content-page tests, and deploy docs
- warm_cities.py: paced, idempotent offline warmer that pre-fetches each city
cell's archive so /climate pages serve from cache and a crawl can't burst the
archive quota (pages self-heal if hit before warming).
- tests/test_content.py: city-set slug uniqueness/lookup, robots.txt, sitemap
enumerating city/month/records URLs, and that a rendered city page carries the
stats + canonical + Dataset JSON-LD in the HTML; plus month/records/hub/glossary/
about routing and 404s.
- DEPLOY.md: document the content pages, the warm step, and submitting the sitemap.
Make the app installable and deliver existing alert notifications as OS
push, alongside the in-app bell.
Backend:
- PushSubscription model (per-device endpoint + keys, owned by a user) and
register/unregister/test endpoints under /api/v2/push, cookie-auth scoped
to the user like the subscription routes.
- push.py: VAPID key management (env -> data/vapid.json -> generated) and a
pywebpush send helper that reports gone endpoints for pruning. No DB coupling.
- notify.py: after creating an in-app Notification, dispatch Web Push to the
user's devices (guarded — a push failure never affects the in-app write;
endpoints reported gone are pruned).
- Serve the .webmanifest with the correct media type.
Frontend:
- manifest.webmanifest + 192/maskable icons; <link rel="manifest"> on all pages.
- sw.js: push + notificationclick handlers (push-only; no fetch caching, so it
doesn't fight the existing IndexedDB cache). Registered globally in nav.js in
secure contexts.
- push-client.js + a "Notifications on this device" toggle and test-send on the
/alerts page, subscribing through the existing cookie-aware apiFetch.
Push and service workers require a secure context, so this is active over HTTPS
(or http://localhost) and cleanly no-ops on a plain-HTTP LAN origin.
A subscription to a city that had never been viewed had no cached ~45-year
archive, so the evaluator skipped it and it never fired. Now when a subscribed
cell has no cached archive, the pass fetches it once via get_history (which caches
it); every later pass reads it from cache and never re-fetches. A per-pass budget
(THERMOGRAPH_NOTIFY_ARCHIVE_FETCHES, default 4) caps how many missing archives one
pass will fetch so a burst of new subscriptions can't exhaust the archive quota,
and a rate-limited fetch just retries on the next pass. The recent/forecast bundle
keeps refreshing on its own hourly cadence.
Add integration tests covering fetch-when-missing and never-refetch-when-cached.
The notification dropdown is anchored to the bell button, which on mobile sits
left of the account button rather than at the screen edge, so the wide panel
overflowed off the left of the viewport (title/text cut off). On narrow screens,
drop .notif's positioning context so the dropdown anchors to the .acct cluster at
the header's right edge, keeping it fully on-screen.
Also add route-level tests for the accounts feature (auth flow, subscription CRUD,
duplicate/validation, ownership 404s, notifications), plus a
THERMOGRAPH_ACCOUNTS_DB override so the suite writes to a throwaway DB and stays
hermetic.
* Migrate backend dataframe layer from pandas to polars
Replace pandas with polars across the backend, dropping both pandas and its
pyarrow parquet engine from the dependency set. numpy stays (the grading
percentile math is unchanged).
- climate.py: parquet IO, source→frame mappings, cache read/topup on polars.
New _normalize_read casts the cached `date` column to pl.Date (older files
were written by pandas as datetime64[ns]); frames now unify missing values as
null so the grading boundary drops them consistently across sources.
- grading.py: keep the numpy percentile core; swap the frame→numpy bridge to
.to_numpy()/.drop_nulls(), day-of-year/year to polars dt expressions, and the
per-row loop to iter_rows(named=True).
- views.py: filter/anti-join/concat replace boolean-mask, isin and pd.concat;
scalar dates are stdlib datetime.date; a local _months_before helper replaces
DateOffset(months=) for the calendar-range default.
- app.py, migrate.py: request-date parsing uses datetime.date, removing pandas
from the endpoint and migrate layers entirely.
- The date column is pl.Date end to end, eliminating the pandas normalize() calls
and comparing cleanly against stdlib dates.
Payloads are unchanged: calendar, day, grade and forecast responses are
byte-for-byte identical to the pandas implementation on the same cached record.
Tests ported to polars fixtures, with added coverage for the combined feels-like
fallback, calendar month-offset (month-end/leap), and the concat/dedup
"fresher source wins" rule.
* Port notify.py to polars after merging dev's account system
Merge origin/dev (accounts + notification subscriptions) and carry the pandas→
polars migration into the newly added notify.py, which the merge brought in still
using pandas — with pandas removed from requirements this broke its import.
- notify.py: _candidate_rows filters/sorts the recent bundle with polars
expressions and returns iter_rows dicts; date scalars are datetime.date;
history/recent emptiness via is_empty().
- test_notify.py: synthetic history/rows built with polars + datetime.
* Add account system foundation: email/password auth with cookie sessions
Introduce the app's first authoritative, user-owned data in a separate
data/accounts.sqlite (SQLAlchemy), kept apart from the disposable derived-cache
DB. Wire fastapi-users for email/password signup, cookie-based login/logout, and
a session-check endpoint, backed by a database session strategy so logins survive
restarts and are revocable.
- db.py: async (aiosqlite) + sync SQLAlchemy engines over accounts.sqlite, WAL +
foreign keys, create_db_and_tables().
- models.py: User, AccessToken, Subscription, Notification tables.
- users.py: pwdlib hashing, HttpOnly cookie transport (path-scoped, SameSite=Lax,
Secure via env), DatabaseStrategy sessions, current-user dependencies.
- schemas.py: user + subscription + notification Pydantic models.
- app.py: mount auth/register/users routers on v2, create tables at startup.
- Pin fastapi-users[sqlalchemy]/aiosqlite; ignore data/accounts.sqlite*.
* Add account header entry and auth modal (frontend)
account.js self-injects a header entry (following the units.js pattern) that
shows a Sign in button when logged out and an account menu when logged in, plus
an auth modal reusing the existing .mp-overlay/.mp-modal chrome for email/password
sign-in and account creation. A shared apiFetch helper sends the same-origin
cookie for authed calls; exported getUser/openAuth/onAuthChange back later phases.
Imported by every page entry module. On narrow screens the entry collapses to an
icon-only button so it doesn't crowd the title.
Enforce an 8-character minimum password in the user manager.
* Add subscription CRUD API and the alerts management page
Backend api_accounts.py adds user-scoped, cookie-authenticated endpoints to
create/list/update/delete subscriptions (and the notification reads used next):
POST snaps lat/lon to a grid cell, resolves a label, and rejects a duplicate
location+kind with 409; PATCH/DELETE are ownership-checked (404 on mismatch).
Mounted on the v2 prefix.
Frontend subscriptions.js + subscriptions.html serve the /alerts page: a sign-in
gate when logged out, an add flow that reuses the shared map picker and an editor
modal (kind, watched metrics, 95-99 percentile, two-sided), and a card list with
inline threshold/active edits and remove. Reachable from the account menu.
* Add background subscription evaluation engine
notify.py runs a daemon thread that periodically evaluates every active
subscription: it groups them by grid cell, reads history from the parquet cache
only (never spends archive quota) plus the hourly recent/forecast bundle, and
grades candidate days with the existing grading.grade_day. A watched metric that
lands at or beyond the threshold percentile fires a 'high' alert; a two-sided
subscription also fires 'low' for the symmetric cold/calm/dry tail (precip stays
one-directional). Observed subscriptions look at the last few recorded days,
forecast subscriptions at the coming week.
Two guards keep it quiet: a UNIQUE(subscription, event_date, metric, direction,
kind) constraint dedups repeat events, and a per-subscription weekly cap
(last_notified_at) limits each alert to one notification per 7 days. The loop
tolerates a bad cell or an upstream rate limit without aborting the pass. Started
and stopped from the app lifespan; gated by THERMOGRAPH_ENABLE_NOTIFIER.
* Add in-app notification center (header bell)
Extend account.js with a notification bell beside the account menu: an unread
badge, a dropdown listing recent notifications (title, body, relative time), a
per-item mark-read on click, and a Mark all read action, all through the
cookie-authed notifications API. Unread state refreshes on open and polls every
two minutes while signed in; polling stops on sign-out. Styled to match the app,
responsive down to mobile.
* Harden accounts: expired-session cleanup, engine tests, ops docs
- notify.py sweeps expired login sessions (access tokens past their lifetime)
once per evaluation pass.
- Add hermetic unit tests for the evaluation engine's trigger detection (high/low
tails, precip one-directional, normal = no trigger) and notification wording.
- Document accounts.sqlite (authoritative, back it up), the single-worker
requirement for the in-process evaluator, and the new env vars in DEPLOY.md.
- Fix the mobile width blow-out: the distribution grid item (.cmp-dist-row) had no
min-width:0, so each row expanded to the 9–10-column connected bar chart's
min-content and widened the whole page (zoom-out, clipped filter sheet). Add
grid-template-columns: minmax(0,1fr) + min-width:0 so the strip scrolls inside its
own .ct-strip, and tighten .ct-col to 34px on phones.
- Add a Load/Refresh button inside the compare filter sheet's date-range block, wired
to the same refresh()/isDirty(); editing the range on mobile no longer needs the
sheet closed. The existing button stays in the controls (loads added places).
- Default date range on both pages is now January six years back → the present. On
the calendar this is an explicit chunked range (the months=24 path is server-capped
at ~2yr).
- Names now include country: reverse_geocode appends it to the place label; revgeo
cache key gets a v2 prefix and PAYLOAD_VER bumps to p2 so labels/payloads
repopulate.
- Location names read as a hierarchy: compact chips show just the lead segment (full
name in the title/aria), and each ranked card shows the lead segment bold over a
muted "city · region · country" line.
Frontend + a small backend name/cache change; no schema migration.
cache.js contained a JavaScript clone of backend/grid.py's snapping math
(its own comment said so) to compute the 8 surrounding cells and fire 8
staggered prefetch requests — grid geometry had two homes, one per
language, plus a client-side guess at Nominatim pacing.
The server now owns it: grid.neighbors(cell) steps one cell width from
the center and re-snaps (adjacent rows have different longitude steps;
poles and the antimeridian handled by snap), and /api/v2/cell grew a
neighbors=1 flag that enqueues those cells for a single background
worker. The warm-only guarantee matches prefetch=1 — a cell with no
cached archive is skipped, so no weather-API quota is ever spent — and
reverse_geocode's own lock paces the at-most-one Nominatim call per
never-labeled cell. Re-enqueues are TTL-deduped; the worker starts from
the lifespan hook, so tests and offline importers never spawn it.
The client now sends its one conditional bundle request with
neighbors=1 (a warm spot costs an empty 304) instead of skipping the
bundle and firing 8 extra requests; the grid-math clone and the
now-unused hasFreshCache are deleted.
Tests (114): grid.neighbors mid-latitude/pole/antimeridian, the
neighbors=1 enqueue + TTL dedupe, _warm_cell materializing the
history-derived store rows, and the never-fetch-upstream guarantee.
Verified with the headless-Chromium smoke across all five pages.
climate.py repeated the same four mechanical patterns:
- the Open-Meteo daily params dict (3x) -> _om_daily_params(cell, **window)
- the doy attach line (6x) -> _with_doy
- makedirs + drop-doy + zstd to_parquet (3x) -> _write_cache
- the identical _to_frame/_nasa_to_frame tail (feels-like, valid-day
filter, doy) -> _finalize_frame
grading.py encoded the tier tables twice — TEMP_BANDS/RAIN_BANDS plus the
hand-aligned _TEMP_LADDER/_RAIN_LADDER ('kept aligned' by comment). The
ladders are now derived from the bands (_ladder_from; verified
byte-identical to the old tables before landing), so tier boundaries have
exactly one definition. grade_range's inline dry-streak walk is replaced
with the existing dry_streaks(); its per-(doy,var) sample cache now also
memoizes the window mask per doy instead of recomputing it once per
metric (9x per day-of-year).
New tests pin the refactor: _to_frame schema/day-filter/missing-series
tolerance, the combined feels-like side selection, NASA unit conversions
and fill-sentinel handling, _om_daily_params windows, and _write_cache
stripping the derived doy column.
api_suggest carried ~80 lines of ranking policy — the exactness-boosted
scoring, local/upstream blending with dedupe, and the token-respelling
correction loop — all consuming the match: prefix/fuzzy vocabulary that
places.py produces. Producer and consumer now live together:
places.suggest(q, upstream, limit) implements the whole policy with the
upstream geocoder lookup injected as a callable, so places stays free of
the fetch layer and the policy is testable without FastAPI.
The endpoint is the HTTP shim: call places.suggest, map the
nothing-at-all-to-serve failure to a 502. Behavior unchanged.
Policy tests: upstream skipped when the local answer convinces, blend +
dedupe, the exactness boost (a hamlet spelled like the typo can't beat
Seattle), single-typo queries answered by the fuzzy matcher without a
correction, the correction path verified via the upstream probe,
upstream failure degrading to local results, and the terminal
error-propagation case.
Rate-limit handling crossed the climate→app boundary as prose: climate
raised RuntimeError(cooldown text), and app._weather_fetch_error re-parsed
it by keyword ('429'/'rate-limited'/'daily'/'tomorrow') while also calling
climate's private _is_rate_limit/_rate_limit_reason — and the user-facing
daily-quota copy existed verbatim in both modules.
climate now raises WeatherUnavailable (a RuntimeError subclass carrying
the user-facing text and a daily flag): from _load_history when both
sources fail with no stale cache, and from the forecast fetch on a 429.
limit_message() is the single home of the rate-limit copy; is_rate_limit
is public for the one remaining raw-429 fallback. app maps the typed
error to 503 by isinstance — no message parsing, no private imports.
The burst-limit copy is unified on 'The weather service is rate-limited…'
(the archive path previously said 'weather archive' internally but the
API always rewrote it; user-visible text is unchanged).
Tests: daily-quota 503 carries the 'tomorrow' copy, an unclassified raw
429 still maps to a clean 503, and a genuine fault stays a 502 with the
raw error.
The derived store's key/token formats existed in three places — each
endpoint, api_cell's slice assembly, and migrate.py — where any drift
would silently split the cache (endpoints missing rows the bundle wrote,
migrate materializing rows nobody reads). They are now defined once in
views.py (grade_key/calendar_key/day_key/forecast_key, history_token/
recent_token/day_token) and consumed everywhere, with a pinning test so
a format change is always deliberate.
The four data endpoints shared two copy-pasted sequences, now helpers:
- _fetch_history: history (+ optional recent bundle) fetch with upstream
failures mapped to clean HTTP errors and an empty record to 404.
- _cached_response: If-None-Match 304 / token-valid store replay /
build + persist + serve, with calendar's dont-persist-placeless rule
as an explicit flag.
Each endpoint is now its audit run + identity + a build callback (~10
lines); api_day's hourly-token special case moved into day_token. New
tests: identity format pins, rate-limit 503 parametrized across all five
data routes, prefetch=1 never touching upstream (cold 204, warm
history-only slices), and calendar's placeless-payload retry behavior.
app.py had grown three co-resident strata: HTTP/caching plumbing, payload
assembly, and search policy. This moves the payload layer — the four
build_* functions, cal_span clamping, hist_end, PAYLOAD_VER, NullRun and
their helpers — into a new views.py with no web dependencies (pure moves,
public names). app.py keeps routing, ETag/derived-store plumbing, and
page serving; migrate.py imports views instead of reaching into app's
privates.
That import previously constructed the whole FastAPI app and — because
places.start_loading() ran at import time — kicked off a background
GeoNames download from an offline batch script. The index load now hangs
off the app's lifespan hook, so it fires when a server starts, not when
the module is imported (tests, migrate).
New tests: cal_span clamping (months-back default, record bounds, 2-year
cap, inversion), builder shapes (grade window ordering, day obs from the
recent bundle, forecast future-only), a regression test that build_day
degrades to climatology when the recent fetch fails, and a subprocess
layering guard that importing views/migrate pulls in neither FastAPI nor
the app module and starts no index download.
- backend/tests: 74 hermetic tests (no network, no repo data//logs/ writes)
covering grid snapping/round-trips, grading percentiles/bands/windows/
dry streaks, the places index (norm, one-edit matchers, search,
corrections), the derived store (token validity, cache=False, degraded
mode), and route-level API tests over a faked climate layer — routing,
validation, ETag/304 revalidation, store replay, the /cell bundle, and
the v1/v2 aliases. The API tests would have caught the /place
AttributeError regression.
- requirements-dev.txt + make test (venv prefers uv-pinned 3.12, matching
deploy-dev.sh — pyarrow wheels stop at 3.12 and some pyenv builds lack
sqlite).
- CI: extract the build job into a reusable build.yml, add the test run
and an API health probe (page-only curl can't catch route wiring
faults); deploy-dev.yml now runs the same build gate before deploying
direct pushes, which previously deployed with no CI at all.
- Deploys serialize under one dev-lan-deploy concurrency group across
both workflows (previously per-PR groups could interleave two deploys
to the same checkout), and are never cancelled mid-restart.
- deploy-dev.sh health check also probes /api/v2/place — best-effort
externals mean a failure there is a genuine server bug.
