2026-07-11 00:29:47 +00:00
|
|
|
|
"""Thermograph API — grade recent local weather against ~45 years of climatology."""
|
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21)
* Compress API responses and revalidate static assets instead of re-downloading
- Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x.
- Serve pages/assets with Cache-Control: no-cache instead of no-store, so
browsers revalidate via the ETag/Last-Modified that FileResponse and
StaticFiles already emit. Unchanged assets now cost an empty 304 rather
than a full transfer on every page navigation, while deploys still show
up immediately.
* Persist derived responses in SQLite so grading is computed once per cell, not per request
New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet
records — finished grade/calendar/day/forecast payloads and reverse-geocode
labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms)
becomes a ~5ms database read that survives restarts and is shared across views.
Raw parquet stays the source of truth; the store is a pure accelerator (every
reader falls back to recomputing on a miss, and deleting the db is a safe reset).
Freshness is token-driven, not clock-driven: each cached payload is validated by
a token encoding what it was computed from (payload schema version, the archive
record's end date, the recent-fetch stamp). The existing freshness drivers are
untouched — get_history still tops up the tail hourly and get_recent_forecast
still refetches hourly — and tokens are derived from what they return, so cached
payloads expire exactly when their inputs change. The same tokens double as weak
ETags: If-None-Match answers with an empty 304 without touching the payload.
- backend/store.py: derived-payload + revgeo tables, thread-local WAL conns,
every helper fail-soft.
- app.py: endpoints split into pure payload builders + HTTP/caching shells; the
in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store).
- climate.py: revgeo persisted through the store; recent_stamp() and
load_cached_history() (no-network read) helpers.
- backend/migrate.py + make migrate: idempotent, resumable backfill of the store
from existing parquet caches (default calendar span + latest-day detail +
revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather.
- grid.from_id(): rebuild a cell from its cache filename (migrate tooling).
* Add /api/v2/cell: one bundle carrying every view's payload
GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months)
and day (today) payloads in one response, each the exact payload its per-view
endpoint returns — built by the same builders and cached under the same
derived-store keys/tokens — paired with the etag that endpoint would emit. The
frontend can warm all views with a single request, seed its per-view cache from
the slices, and later revalidate each view individually with If-None-Match. The
bundle's own etag combines the slices', so an unchanged bundle is an empty 304.
prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard
guarantee: it never spends weather-API quota. A cell with no cached archive
answers 204, and only the history-derived slices (calendar + latest-day detail)
are built. At most one Nominatim lookup for a never-labeled cell.
* Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch
The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota,
which multi-year calendar payloads regularly blew through) to IndexedDB, with an
in-memory map in front. Entries carry the server's ETag, so anything stale
revalidates conditionally — unchanged data costs an empty 304 and a re-stamp,
never a re-transfer. On network failure the stale copy is served over an error.
getJSON gains an optional onUpdate callback opting into stale-while-revalidate:
the three views (weekly, day, calendar) now render a cached copy immediately —
spinners are delayed 150ms so warm loads never flash-blank — and repaint only if
background revalidation finds changed data. New data shows up the moment it
exists instead of waiting out a TTL.
Cross-view prefetch collapses from one request per view to a single /api/v2/cell
bundle, whose slices (exact per-view payloads + their etags) are seeded under the
URLs each view actually requests; the bundle call itself is conditional via a
remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side
with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one
possible Nominatim lookup each), so tapping nearby lands on already-graded data.
Legacy tg:* storage entries are cleared once; cache entries untouched for two
weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
|
|
|
|
import contextlib
|
2026-07-11 00:29:47 +00:00
|
|
|
|
import datetime
|
Typo-tolerant location search suggestions (#33)
Add /api/v2/suggest and wire the location picker's search box to it as a
debounced type-ahead: top-5 place suggestions that tolerate a single-letter
typo (substituted, missing, or extra letter, or two adjacent letters swapped)
anywhere in the query, including the first character.
- backend/places.py: local place index built from a GeoNames cities dump
(downloaded once into data/geonames/, loaded in a background thread; the
app boots and serves without it). Exact-prefix matches rank first by
population, then names one edit away; a token vocabulary respells one
mistyped word against known place-name tokens ("pest seattle" ->
"west seattle") for retry against the upstream geocoder, which covers
neighbourhood-level places the dump lacks. THERMOGRAPH_CITIES picks the
dump (default cities1000).
- /suggest blends local and upstream results by population with an exactness
boost, so "Seatle" (a Cumbrian hamlet) can't outrank Seattle when the
query is one edit from the city, while "munchen" still surfaces Munich via
upstream (the index only knows English names). Upstream lookups are
memoized and skipped entirely when the index answers convincingly.
- mappicker.js: debounced (250ms) live suggestions with abort + sequence
guards against stale responses, arrow-key navigation, Enter-picks-highlight,
Escape dismissing the list before closing the overlay. Submit goes through
the same typo-tolerant endpoint.
- climate.geocode results now carry population (used for ranking).
2026-07-11 15:36:14 +00:00
|
|
|
|
import functools
|
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21)
* Compress API responses and revalidate static assets instead of re-downloading
- Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x.
- Serve pages/assets with Cache-Control: no-cache instead of no-store, so
browsers revalidate via the ETag/Last-Modified that FileResponse and
StaticFiles already emit. Unchanged assets now cost an empty 304 rather
than a full transfer on every page navigation, while deploys still show
up immediately.
* Persist derived responses in SQLite so grading is computed once per cell, not per request
New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet
records — finished grade/calendar/day/forecast payloads and reverse-geocode
labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms)
becomes a ~5ms database read that survives restarts and is shared across views.
Raw parquet stays the source of truth; the store is a pure accelerator (every
reader falls back to recomputing on a miss, and deleting the db is a safe reset).
Freshness is token-driven, not clock-driven: each cached payload is validated by
a token encoding what it was computed from (payload schema version, the archive
record's end date, the recent-fetch stamp). The existing freshness drivers are
untouched — get_history still tops up the tail hourly and get_recent_forecast
still refetches hourly — and tokens are derived from what they return, so cached
payloads expire exactly when their inputs change. The same tokens double as weak
ETags: If-None-Match answers with an empty 304 without touching the payload.
- backend/store.py: derived-payload + revgeo tables, thread-local WAL conns,
every helper fail-soft.
- app.py: endpoints split into pure payload builders + HTTP/caching shells; the
in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store).
- climate.py: revgeo persisted through the store; recent_stamp() and
load_cached_history() (no-network read) helpers.
- backend/migrate.py + make migrate: idempotent, resumable backfill of the store
from existing parquet caches (default calendar span + latest-day detail +
revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather.
- grid.from_id(): rebuild a cell from its cache filename (migrate tooling).
* Add /api/v2/cell: one bundle carrying every view's payload
GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months)
and day (today) payloads in one response, each the exact payload its per-view
endpoint returns — built by the same builders and cached under the same
derived-store keys/tokens — paired with the etag that endpoint would emit. The
frontend can warm all views with a single request, seed its per-view cache from
the slices, and later revalidate each view individually with If-None-Match. The
bundle's own etag combines the slices', so an unchanged bundle is an empty 304.
prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard
guarantee: it never spends weather-API quota. A cell with no cached archive
answers 204, and only the history-derived slices (calendar + latest-day detail)
are built. At most one Nominatim lookup for a never-labeled cell.
* Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch
The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota,
which multi-year calendar payloads regularly blew through) to IndexedDB, with an
in-memory map in front. Entries carry the server's ETag, so anything stale
revalidates conditionally — unchanged data costs an empty 304 and a re-stamp,
never a re-transfer. On network failure the stale copy is served over an error.
getJSON gains an optional onUpdate callback opting into stale-while-revalidate:
the three views (weekly, day, calendar) now render a cached copy immediately —
spinners are delayed 150ms so warm loads never flash-blank — and repaint only if
background revalidation finds changed data. New data shows up the moment it
exists instead of waiting out a TTL.
Cross-view prefetch collapses from one request per view to a single /api/v2/cell
bundle, whose slices (exact per-view payloads + their etags) are seeded under the
URLs each view actually requests; the bundle call itself is conditional via a
remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side
with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one
possible Nominatim lookup each), so tapping nearby lands on already-graded data.
Legacy tg:* storage entries are cleared once; cache entries untouched for two
weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
|
|
|
|
import hashlib
|
Add ops metrics endpoint + SSH/Termux dashboard (#131)
Adds observability for the running app, viewable over SSH (e.g. Termux):
- backend/metrics.py: thread-safe, best-effort in-process counters for inbound
requests (per category) and outbound calls (per external source), plus a
phase->source map and snapshot(). Hooked into climate._request (outbound, all
six weather/geocode sources) and the existing revalidate_static middleware
(inbound). Never raises into the request path.
- GET /api/v2/metrics: token-gated JSON of the counters + warm-queue depth +
thread names/uptime/pid. Returns 404 unless a valid THERMOGRAPH_METRICS_TOKEN
is presented or the request is direct loopback with no proxy X-Forwarded-*
header, so ops data is never exposed through Caddy on the public domain.
- scripts/dashboard.py (+ scripts/dash, make dashboard): stdlib-only terminal
dashboard. Reads cache (parquet + SQLite), accounts (users + active
subscriptions), and system resources (/proc + systemd) directly, and pulls
traffic from the metrics endpoint over loopback. Live refresh, --once, --json;
mobile-terminal width. Paths resolve relative to the script so the same tool
works on the LAN dev and prod VPS checkouts.
- Monitoring sections in DEPLOY.md / DEPLOY-DEV.md; unit tests for the counters.
2026-07-16 20:46:59 +00:00
|
|
|
|
import hmac
|
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21)
* Compress API responses and revalidate static assets instead of re-downloading
- Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x.
- Serve pages/assets with Cache-Control: no-cache instead of no-store, so
browsers revalidate via the ETag/Last-Modified that FileResponse and
StaticFiles already emit. Unchanged assets now cost an empty 304 rather
than a full transfer on every page navigation, while deploys still show
up immediately.
* Persist derived responses in SQLite so grading is computed once per cell, not per request
New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet
records — finished grade/calendar/day/forecast payloads and reverse-geocode
labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms)
becomes a ~5ms database read that survives restarts and is shared across views.
Raw parquet stays the source of truth; the store is a pure accelerator (every
reader falls back to recomputing on a miss, and deleting the db is a safe reset).
Freshness is token-driven, not clock-driven: each cached payload is validated by
a token encoding what it was computed from (payload schema version, the archive
record's end date, the recent-fetch stamp). The existing freshness drivers are
untouched — get_history still tops up the tail hourly and get_recent_forecast
still refetches hourly — and tokens are derived from what they return, so cached
payloads expire exactly when their inputs change. The same tokens double as weak
ETags: If-None-Match answers with an empty 304 without touching the payload.
- backend/store.py: derived-payload + revgeo tables, thread-local WAL conns,
every helper fail-soft.
- app.py: endpoints split into pure payload builders + HTTP/caching shells; the
in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store).
- climate.py: revgeo persisted through the store; recent_stamp() and
load_cached_history() (no-network read) helpers.
- backend/migrate.py + make migrate: idempotent, resumable backfill of the store
from existing parquet caches (default calendar span + latest-day detail +
revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather.
- grid.from_id(): rebuild a cell from its cache filename (migrate tooling).
* Add /api/v2/cell: one bundle carrying every view's payload
GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months)
and day (today) payloads in one response, each the exact payload its per-view
endpoint returns — built by the same builders and cached under the same
derived-store keys/tokens — paired with the etag that endpoint would emit. The
frontend can warm all views with a single request, seed its per-view cache from
the slices, and later revalidate each view individually with If-None-Match. The
bundle's own etag combines the slices', so an unchanged bundle is an empty 304.
prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard
guarantee: it never spends weather-API quota. A cell with no cached archive
answers 204, and only the history-derived slices (calendar + latest-day detail)
are built. At most one Nominatim lookup for a never-labeled cell.
* Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch
The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota,
which multi-year calendar payloads regularly blew through) to IndexedDB, with an
in-memory map in front. Entries carry the server's ETag, so anything stale
revalidates conditionally — unchanged data costs an empty 304 and a re-stamp,
never a re-transfer. On network failure the stale copy is served over an error.
getJSON gains an optional onUpdate callback opting into stale-while-revalidate:
the three views (weekly, day, calendar) now render a cached copy immediately —
spinners are delayed 150ms so warm loads never flash-blank — and repaint only if
background revalidation finds changed data. New data shows up the moment it
exists instead of waiting out a TTL.
Cross-view prefetch collapses from one request per view to a single /api/v2/cell
bundle, whose slices (exact per-view payloads + their etags) are seeded under the
URLs each view actually requests; the bundle call itself is conditional via a
remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side
with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one
possible Nominatim lookup each), so tapping nearby lands on already-graded data.
Legacy tg:* storage entries are cleared once; cache entries untouched for two
weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
|
|
|
|
import json
|
2026-07-15 23:21:06 +00:00
|
|
|
|
import mimetypes
|
2026-07-11 00:29:47 +00:00
|
|
|
|
import os
|
2026-07-11 20:47:31 +00:00
|
|
|
|
import queue
|
|
|
|
|
|
import threading
|
|
|
|
|
|
import time
|
2026-07-11 00:29:47 +00:00
|
|
|
|
|
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21)
* Compress API responses and revalidate static assets instead of re-downloading
- Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x.
- Serve pages/assets with Cache-Control: no-cache instead of no-store, so
browsers revalidate via the ETag/Last-Modified that FileResponse and
StaticFiles already emit. Unchanged assets now cost an empty 304 rather
than a full transfer on every page navigation, while deploys still show
up immediately.
* Persist derived responses in SQLite so grading is computed once per cell, not per request
New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet
records — finished grade/calendar/day/forecast payloads and reverse-geocode
labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms)
becomes a ~5ms database read that survives restarts and is shared across views.
Raw parquet stays the source of truth; the store is a pure accelerator (every
reader falls back to recomputing on a miss, and deleting the db is a safe reset).
Freshness is token-driven, not clock-driven: each cached payload is validated by
a token encoding what it was computed from (payload schema version, the archive
record's end date, the recent-fetch stamp). The existing freshness drivers are
untouched — get_history still tops up the tail hourly and get_recent_forecast
still refetches hourly — and tokens are derived from what they return, so cached
payloads expire exactly when their inputs change. The same tokens double as weak
ETags: If-None-Match answers with an empty 304 without touching the payload.
- backend/store.py: derived-payload + revgeo tables, thread-local WAL conns,
every helper fail-soft.
- app.py: endpoints split into pure payload builders + HTTP/caching shells; the
in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store).
- climate.py: revgeo persisted through the store; recent_stamp() and
load_cached_history() (no-network read) helpers.
- backend/migrate.py + make migrate: idempotent, resumable backfill of the store
from existing parquet caches (default calendar span + latest-day detail +
revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather.
- grid.from_id(): rebuild a cell from its cache filename (migrate tooling).
* Add /api/v2/cell: one bundle carrying every view's payload
GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months)
and day (today) payloads in one response, each the exact payload its per-view
endpoint returns — built by the same builders and cached under the same
derived-store keys/tokens — paired with the etag that endpoint would emit. The
frontend can warm all views with a single request, seed its per-view cache from
the slices, and later revalidate each view individually with If-None-Match. The
bundle's own etag combines the slices', so an unchanged bundle is an empty 304.
prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard
guarantee: it never spends weather-API quota. A cell with no cached archive
answers 204, and only the history-derived slices (calendar + latest-day detail)
are built. At most one Nominatim lookup for a never-labeled cell.
* Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch
The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota,
which multi-year calendar payloads regularly blew through) to IndexedDB, with an
in-memory map in front. Entries carry the server's ETag, so anything stale
revalidates conditionally — unchanged data costs an empty 304 and a re-stamp,
never a re-transfer. On network failure the stale copy is served over an error.
getJSON gains an optional onUpdate callback opting into stale-while-revalidate:
the three views (weekly, day, calendar) now render a cached copy immediately —
spinners are delayed 150ms so warm loads never flash-blank — and repaint only if
background revalidation finds changed data. New data shows up the moment it
exists instead of waiting out a TTL.
Cross-view prefetch collapses from one request per view to a single /api/v2/cell
bundle, whose slices (exact per-view payloads + their etags) are seeded under the
URLs each view actually requests; the bundle call itself is conditional via a
remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side
with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one
possible Nominatim lookup each), so tapping nearby lands on already-graded data.
Legacy tg:* storage entries are cleared once; cache entries untouched for two
weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
|
|
|
|
from fastapi import APIRouter, FastAPI, HTTPException, Query, Request, Response
|
|
|
|
|
|
from fastapi.middleware.gzip import GZipMiddleware
|
2026-07-11 15:59:14 +00:00
|
|
|
|
from fastapi.responses import RedirectResponse
|
2026-07-11 00:29:47 +00:00
|
|
|
|
from fastapi.staticfiles import StaticFiles
|
|
|
|
|
|
|
Account system with weather-notification subscriptions (#89)
* Add account system foundation: email/password auth with cookie sessions
Introduce the app's first authoritative, user-owned data in a separate
data/accounts.sqlite (SQLAlchemy), kept apart from the disposable derived-cache
DB. Wire fastapi-users for email/password signup, cookie-based login/logout, and
a session-check endpoint, backed by a database session strategy so logins survive
restarts and are revocable.
- db.py: async (aiosqlite) + sync SQLAlchemy engines over accounts.sqlite, WAL +
foreign keys, create_db_and_tables().
- models.py: User, AccessToken, Subscription, Notification tables.
- users.py: pwdlib hashing, HttpOnly cookie transport (path-scoped, SameSite=Lax,
Secure via env), DatabaseStrategy sessions, current-user dependencies.
- schemas.py: user + subscription + notification Pydantic models.
- app.py: mount auth/register/users routers on v2, create tables at startup.
- Pin fastapi-users[sqlalchemy]/aiosqlite; ignore data/accounts.sqlite*.
* Add account header entry and auth modal (frontend)
account.js self-injects a header entry (following the units.js pattern) that
shows a Sign in button when logged out and an account menu when logged in, plus
an auth modal reusing the existing .mp-overlay/.mp-modal chrome for email/password
sign-in and account creation. A shared apiFetch helper sends the same-origin
cookie for authed calls; exported getUser/openAuth/onAuthChange back later phases.
Imported by every page entry module. On narrow screens the entry collapses to an
icon-only button so it doesn't crowd the title.
Enforce an 8-character minimum password in the user manager.
* Add subscription CRUD API and the alerts management page
Backend api_accounts.py adds user-scoped, cookie-authenticated endpoints to
create/list/update/delete subscriptions (and the notification reads used next):
POST snaps lat/lon to a grid cell, resolves a label, and rejects a duplicate
location+kind with 409; PATCH/DELETE are ownership-checked (404 on mismatch).
Mounted on the v2 prefix.
Frontend subscriptions.js + subscriptions.html serve the /alerts page: a sign-in
gate when logged out, an add flow that reuses the shared map picker and an editor
modal (kind, watched metrics, 95-99 percentile, two-sided), and a card list with
inline threshold/active edits and remove. Reachable from the account menu.
* Add background subscription evaluation engine
notify.py runs a daemon thread that periodically evaluates every active
subscription: it groups them by grid cell, reads history from the parquet cache
only (never spends archive quota) plus the hourly recent/forecast bundle, and
grades candidate days with the existing grading.grade_day. A watched metric that
lands at or beyond the threshold percentile fires a 'high' alert; a two-sided
subscription also fires 'low' for the symmetric cold/calm/dry tail (precip stays
one-directional). Observed subscriptions look at the last few recorded days,
forecast subscriptions at the coming week.
Two guards keep it quiet: a UNIQUE(subscription, event_date, metric, direction,
kind) constraint dedups repeat events, and a per-subscription weekly cap
(last_notified_at) limits each alert to one notification per 7 days. The loop
tolerates a bad cell or an upstream rate limit without aborting the pass. Started
and stopped from the app lifespan; gated by THERMOGRAPH_ENABLE_NOTIFIER.
* Add in-app notification center (header bell)
Extend account.js with a notification bell beside the account menu: an unread
badge, a dropdown listing recent notifications (title, body, relative time), a
per-item mark-read on click, and a Mark all read action, all through the
cookie-authed notifications API. Unread state refreshes on open and polls every
two minutes while signed in; polling stops on sign-out. Styled to match the app,
responsive down to mobile.
* Harden accounts: expired-session cleanup, engine tests, ops docs
- notify.py sweeps expired login sessions (access tokens past their lifetime)
once per evaluation pass.
- Add hermetic unit tests for the evaluation engine's trigger detection (high/low
tails, precip one-directional, normal = no trigger) and notification wording.
- Document accounts.sqlite (authoritative, back it up), the single-worker
requirement for the in-process evaluator, and the new env vars in DEPLOY.md.
2026-07-15 18:46:46 +00:00
|
|
|
|
import api_accounts
|
2026-07-11 00:29:47 +00:00
|
|
|
|
import audit
|
|
|
|
|
|
import climate
|
SEO: crawlable programmatic climate pages + technical hygiene (#96)
* SEO: generate curated city set for crawlable climate pages
gen_cities.py reuses the GeoNames index places.py already parses to select the top
~500 metros by population, assigns each a stable URL-safe slug (dropping admin1 when
it repeats the city name), and writes committed backend/cities.json. cities.py loads
it lazily with slug lookup, all_slugs(), display_name(), and by_country() grouping
for the upcoming hub + sitemap.
* SEO: rendering core, robots.txt, sitemap.xml, and metadata hygiene
- content.py: Jinja2 environment + HTML responder (ETag/304), dynamic /robots.txt
(disallows /api and /alerts, points at the sitemap) and /sitemap.xml (enumerates
the home/static pages plus every city, month, and records URL from cities.py).
Registered on the app before the StaticFiles mount so the routes win.
- templates/base.html.j2: shared layout with unique title/description, self-
referential canonical, Open Graph, favicon/manifest, header nav (adds a Climate
link) and a footer link graph.
- Give each existing page a unique <meta description> (were 5x identical) and a
self-referential <link rel=canonical>; add WebApplication JSON-LD to the home page.
- Pin jinja2.
* SEO: server-rendered per-city climate page (/climate/{slug})
The keystone crawlable page: for a city it snaps to the grid cell, loads the
archive (fetching once if missing, self-healing), and renders as real HTML — a
'how today compares' block (grade + percentile per metric from grade_day, tinted
by tier), a monthly normals table (climatology at each month's 15th, shown in °F
and °C), all-time records (new grading.all_time_records helper), a breadcrumb,
Dataset+Place+BreadcrumbList JSON-LD, self-referential canonical, and links into
the interactive tool + month/records pages. Content-page CSS added to style.css
(renamed the table class to avoid colliding with the app's .normals flex row).
* SEO: month (/climate/{slug}/{month}) and records (/climate/{slug}/records) pages
Month pages render the exact-month long-tail ('average weather in {city} in
{month}') with that month's average high/low, typical p10-p90 range, month-specific
records, and prev/next month links. Records pages show all-time record highs/lows
per metric with dates (grading.all_time_records). Shared _resolve_city helper; the
literal /records route is registered before the {month} param and month names are
validated (unknown month -> 404).
* SEO: climate hub, weather glossary, and about/methodology pages
- /climate: crawlable directory of all ~500 cities grouped by country — the
internal-link graph that lets search engines discover every city page.
- /glossary + /glossary/{term}: plain-language definitions (climate normal,
percentile, temperature anomaly, feels-like, heat index, wind chill, humidity,
reanalysis) with DefinedTerm JSON-LD and cross-links into the tool.
- /about: methodology page (ERA5 data source, 45-year baseline, +/-7-day window,
percentile grading) for E-E-A-T. All linked from the shared footer.
* SEO: archive warmer, content-page tests, and deploy docs
- warm_cities.py: paced, idempotent offline warmer that pre-fetches each city
cell's archive so /climate pages serve from cache and a crawl can't burst the
archive quota (pages self-heal if hit before warming).
- tests/test_content.py: city-set slug uniqueness/lookup, robots.txt, sitemap
enumerating city/month/records URLs, and that a rendered city page carries the
stats + canonical + Dataset JSON-LD in the HTML; plus month/records/hub/glossary/
about routing and 404s.
- DEPLOY.md: document the content pages, the warm step, and submitting the sitemap.
2026-07-15 23:53:11 +00:00
|
|
|
|
import content
|
Rebuild the homepage as a distribution landing; add an SMTP seam (#178)
The homepage was a bare tool: a find-bar and an empty panel reading "Find a
location to begin." A visitor arriving from a search result or a shared link
learned nothing about what the product does before deciding to leave.
Rebuild it around the Weekly view, which is untouched:
- Hero with the question as the h1, and a grade card showing a real graded
example — the most unusual city we're currently tracking, either tail. The
card's frame and text slots are server-rendered with reserved heights, so
app.js re-pointing it at the visitor's own place shifts nothing.
- "Unusual right now" strip, CSS scroll-snap, no JS carousel. A cold-tail city
is force-included whenever one qualifies.
- Stance line, how-it-works, explore cards, and 12 city chips linking into the
~1000-page /climate surface.
- Monthly digest form, in the footer of every page.
Serve / from Jinja instead of a static file with placeholder substitution, so
crawlers and no-JS readers get the whole page as real HTML. home.html.j2
extends base.html.j2 and carries app.js's DOM contract over verbatim; the brand
degrades to a <p> so the hero owns the sole h1. frontend/index.html is deleted
rather than left behind the static mount, where it would keep serving indexable
duplicate content.
"Where is it most unusual right now" has no cheap answer at request time —
percentiles live inside zlib-compressed payload blobs with no column to sort on.
So homepage.py sweeps the warm cache and writes data/homepage.json, read by the
template. The sweep is strictly cache-only (climate.load_cached_recent_forecast
is new, the sibling of load_cached_history), so grading ~1000 cities costs zero
upstream requests. It rides the notifier's timer behind an hourly guard rather
than starting a second daemon, and also runs at the tail of warm_cities so a
fresh deploy has a populated feed.
Instrumentation: metrics gains a product-event counter keyed by (event,
referrer domain, UTC day), behind an allowlist and a per-IP rate limit, fed by
POST /api/v2/event. The referrer is taken from the request's own header, never
from the client. The beacon gets its own inbound category that record_inbound
ignores, so reporting an interaction doesn't also count as traffic. The
dashboard grows an events block with per-referrer attribution.
Email is scaffolded but sends nothing yet. mailer.py talks stdlib smtplib to a
local Postfix null client on 127.0.0.1:25 (deploy/provision-mail.sh), so the
choice between direct-to-MX and relaying through a provider stays a Postfix
config change with no code change. The backend defaults to "console", which
logs and sends nothing, so dev and tests exercise the whole signup path safely.
Signups land in pending_digest unconfirmed; collecting the list shouldn't wait
on delivery.
Also adds /privacy, linked from the footer and kept out of the sitemap.
The strip's classes are named unusual-* rather than record-*: .record-card is
already the SEO records page's, and reusing it leaked layout rules onto
/climate/<slug>/records.
2026-07-18 07:39:47 +00:00
|
|
|
|
import digest
|
2026-07-20 02:19:29 +00:00
|
|
|
|
import discord_interactions as discord_interactions_mod
|
2026-07-20 02:26:33 +00:00
|
|
|
|
import discord_link
|
Account system with weather-notification subscriptions (#89)
* Add account system foundation: email/password auth with cookie sessions
Introduce the app's first authoritative, user-owned data in a separate
data/accounts.sqlite (SQLAlchemy), kept apart from the disposable derived-cache
DB. Wire fastapi-users for email/password signup, cookie-based login/logout, and
a session-check endpoint, backed by a database session strategy so logins survive
restarts and are revocable.
- db.py: async (aiosqlite) + sync SQLAlchemy engines over accounts.sqlite, WAL +
foreign keys, create_db_and_tables().
- models.py: User, AccessToken, Subscription, Notification tables.
- users.py: pwdlib hashing, HttpOnly cookie transport (path-scoped, SameSite=Lax,
Secure via env), DatabaseStrategy sessions, current-user dependencies.
- schemas.py: user + subscription + notification Pydantic models.
- app.py: mount auth/register/users routers on v2, create tables at startup.
- Pin fastapi-users[sqlalchemy]/aiosqlite; ignore data/accounts.sqlite*.
* Add account header entry and auth modal (frontend)
account.js self-injects a header entry (following the units.js pattern) that
shows a Sign in button when logged out and an account menu when logged in, plus
an auth modal reusing the existing .mp-overlay/.mp-modal chrome for email/password
sign-in and account creation. A shared apiFetch helper sends the same-origin
cookie for authed calls; exported getUser/openAuth/onAuthChange back later phases.
Imported by every page entry module. On narrow screens the entry collapses to an
icon-only button so it doesn't crowd the title.
Enforce an 8-character minimum password in the user manager.
* Add subscription CRUD API and the alerts management page
Backend api_accounts.py adds user-scoped, cookie-authenticated endpoints to
create/list/update/delete subscriptions (and the notification reads used next):
POST snaps lat/lon to a grid cell, resolves a label, and rejects a duplicate
location+kind with 409; PATCH/DELETE are ownership-checked (404 on mismatch).
Mounted on the v2 prefix.
Frontend subscriptions.js + subscriptions.html serve the /alerts page: a sign-in
gate when logged out, an add flow that reuses the shared map picker and an editor
modal (kind, watched metrics, 95-99 percentile, two-sided), and a card list with
inline threshold/active edits and remove. Reachable from the account menu.
* Add background subscription evaluation engine
notify.py runs a daemon thread that periodically evaluates every active
subscription: it groups them by grid cell, reads history from the parquet cache
only (never spends archive quota) plus the hourly recent/forecast bundle, and
grades candidate days with the existing grading.grade_day. A watched metric that
lands at or beyond the threshold percentile fires a 'high' alert; a two-sided
subscription also fires 'low' for the symmetric cold/calm/dry tail (precip stays
one-directional). Observed subscriptions look at the last few recorded days,
forecast subscriptions at the coming week.
Two guards keep it quiet: a UNIQUE(subscription, event_date, metric, direction,
kind) constraint dedups repeat events, and a per-subscription weekly cap
(last_notified_at) limits each alert to one notification per 7 days. The loop
tolerates a bad cell or an upstream rate limit without aborting the pass. Started
and stopped from the app lifespan; gated by THERMOGRAPH_ENABLE_NOTIFIER.
* Add in-app notification center (header bell)
Extend account.js with a notification bell beside the account menu: an unread
badge, a dropdown listing recent notifications (title, body, relative time), a
per-item mark-read on click, and a Mark all read action, all through the
cookie-authed notifications API. Unread state refreshes on open and polls every
two minutes while signed in; polling stops on sign-out. Styled to match the app,
responsive down to mobile.
* Harden accounts: expired-session cleanup, engine tests, ops docs
- notify.py sweeps expired login sessions (access tokens past their lifetime)
once per evaluation pass.
- Add hermetic unit tests for the evaluation engine's trigger detection (high/low
tails, precip one-directional, normal = no trigger) and notification wording.
- Document accounts.sqlite (authoritative, back it up), the single-worker
requirement for the in-process evaluator, and the new env vars in DEPLOY.md.
2026-07-15 18:46:46 +00:00
|
|
|
|
import db
|
2026-07-11 00:29:47 +00:00
|
|
|
|
import grid
|
Add ops metrics endpoint + SSH/Termux dashboard (#131)
Adds observability for the running app, viewable over SSH (e.g. Termux):
- backend/metrics.py: thread-safe, best-effort in-process counters for inbound
requests (per category) and outbound calls (per external source), plus a
phase->source map and snapshot(). Hooked into climate._request (outbound, all
six weather/geocode sources) and the existing revalidate_static middleware
(inbound). Never raises into the request path.
- GET /api/v2/metrics: token-gated JSON of the counters + warm-queue depth +
thread names/uptime/pid. Returns 404 unless a valid THERMOGRAPH_METRICS_TOKEN
is presented or the request is direct loopback with no proxy X-Forwarded-*
header, so ops data is never exposed through Caddy on the public domain.
- scripts/dashboard.py (+ scripts/dash, make dashboard): stdlib-only terminal
dashboard. Reads cache (parquet + SQLite), accounts (users + active
subscriptions), and system resources (/proc + systemd) directly, and pulls
traffic from the metrics endpoint over loopback. Live refresh, --once, --json;
mobile-terminal width. Paths resolve relative to the script so the same tool
works on the LAN dev and prod VPS checkouts.
- Monitoring sections in DEPLOY.md / DEPLOY-DEV.md; unit tests for the counters.
2026-07-16 20:46:59 +00:00
|
|
|
|
import metrics
|
Account system with weather-notification subscriptions (#89)
* Add account system foundation: email/password auth with cookie sessions
Introduce the app's first authoritative, user-owned data in a separate
data/accounts.sqlite (SQLAlchemy), kept apart from the disposable derived-cache
DB. Wire fastapi-users for email/password signup, cookie-based login/logout, and
a session-check endpoint, backed by a database session strategy so logins survive
restarts and are revocable.
- db.py: async (aiosqlite) + sync SQLAlchemy engines over accounts.sqlite, WAL +
foreign keys, create_db_and_tables().
- models.py: User, AccessToken, Subscription, Notification tables.
- users.py: pwdlib hashing, HttpOnly cookie transport (path-scoped, SameSite=Lax,
Secure via env), DatabaseStrategy sessions, current-user dependencies.
- schemas.py: user + subscription + notification Pydantic models.
- app.py: mount auth/register/users routers on v2, create tables at startup.
- Pin fastapi-users[sqlalchemy]/aiosqlite; ignore data/accounts.sqlite*.
* Add account header entry and auth modal (frontend)
account.js self-injects a header entry (following the units.js pattern) that
shows a Sign in button when logged out and an account menu when logged in, plus
an auth modal reusing the existing .mp-overlay/.mp-modal chrome for email/password
sign-in and account creation. A shared apiFetch helper sends the same-origin
cookie for authed calls; exported getUser/openAuth/onAuthChange back later phases.
Imported by every page entry module. On narrow screens the entry collapses to an
icon-only button so it doesn't crowd the title.
Enforce an 8-character minimum password in the user manager.
* Add subscription CRUD API and the alerts management page
Backend api_accounts.py adds user-scoped, cookie-authenticated endpoints to
create/list/update/delete subscriptions (and the notification reads used next):
POST snaps lat/lon to a grid cell, resolves a label, and rejects a duplicate
location+kind with 409; PATCH/DELETE are ownership-checked (404 on mismatch).
Mounted on the v2 prefix.
Frontend subscriptions.js + subscriptions.html serve the /alerts page: a sign-in
gate when logged out, an add flow that reuses the shared map picker and an editor
modal (kind, watched metrics, 95-99 percentile, two-sided), and a card list with
inline threshold/active edits and remove. Reachable from the account menu.
* Add background subscription evaluation engine
notify.py runs a daemon thread that periodically evaluates every active
subscription: it groups them by grid cell, reads history from the parquet cache
only (never spends archive quota) plus the hourly recent/forecast bundle, and
grades candidate days with the existing grading.grade_day. A watched metric that
lands at or beyond the threshold percentile fires a 'high' alert; a two-sided
subscription also fires 'low' for the symmetric cold/calm/dry tail (precip stays
one-directional). Observed subscriptions look at the last few recorded days,
forecast subscriptions at the coming week.
Two guards keep it quiet: a UNIQUE(subscription, event_date, metric, direction,
kind) constraint dedups repeat events, and a per-subscription weekly cap
(last_notified_at) limits each alert to one notification per 7 days. The loop
tolerates a bad cell or an upstream rate limit without aborting the pass. Started
and stopped from the app lifespan; gated by THERMOGRAPH_ENABLE_NOTIFIER.
* Add in-app notification center (header bell)
Extend account.js with a notification bell beside the account menu: an unread
badge, a dropdown listing recent notifications (title, body, relative time), a
per-item mark-read on click, and a Mark all read action, all through the
cookie-authed notifications API. Unread state refreshes on open and polls every
two minutes while signed in; polling stops on sign-out. Styled to match the app,
responsive down to mobile.
* Harden accounts: expired-session cleanup, engine tests, ops docs
- notify.py sweeps expired login sessions (access tokens past their lifetime)
once per evaluation pass.
- Add hermetic unit tests for the evaluation engine's trigger detection (high/low
tails, precip one-directional, normal = no trigger) and notification wording.
- Document accounts.sqlite (authoritative, back it up), the single-worker
requirement for the in-process evaluator, and the new env vars in DEPLOY.md.
2026-07-15 18:46:46 +00:00
|
|
|
|
import notify
|
Typo-tolerant location search suggestions (#33)
Add /api/v2/suggest and wire the location picker's search box to it as a
debounced type-ahead: top-5 place suggestions that tolerate a single-letter
typo (substituted, missing, or extra letter, or two adjacent letters swapped)
anywhere in the query, including the first character.
- backend/places.py: local place index built from a GeoNames cities dump
(downloaded once into data/geonames/, loaded in a background thread; the
app boots and serves without it). Exact-prefix matches rank first by
population, then names one edit away; a token vocabulary respells one
mistyped word against known place-name tokens ("pest seattle" ->
"west seattle") for retry against the upstream geocoder, which covers
neighbourhood-level places the dump lacks. THERMOGRAPH_CITIES picks the
dump (default cities1000).
- /suggest blends local and upstream results by population with an exactness
boost, so "Seatle" (a Cumbrian hamlet) can't outrank Seattle when the
query is one edit from the city, while "munchen" still surfaces Munich via
upstream (the index only knows English names). Upstream lookups are
memoized and skipped entirely when the index answers convincingly.
- mappicker.js: debounced (250ms) live suggestions with abort + sequence
guards against stale responses, arrow-key navigation, Enter-picks-highlight,
Escape dismissing the list before closing the overlay. Submit goes through
the same typo-tolerant endpoint.
- climate.geocode results now carry population (used for ranking).
2026-07-11 15:36:14 +00:00
|
|
|
|
import places
|
Gate the subscription notifier to one worker (#161)
Prod runs 3 uvicorn workers, but the in-process subscription notifier was
started in every worker's lifespan (it was written assuming a single-worker
deploy). Its sweep fetches recent-forecast/archive data from Open-Meteo on a
15-min timer independent of any request, so running it 3× tripled the
background upstream load against a shared quota — the source of the overnight
429/503 rate-limit errors in logs/errors, and 3× redundant subscription scans.
Elect one worker to own the notifier via a non-blocking exclusive flock on a
lockfile (THERMOGRAPH_SINGLETON_LOCK): the first worker wins and holds the fd
for its lifetime; others stand down; the OS releases the lock if the leader
dies, so a restart re-elects cleanly. Unset (single worker / dev / tests)
always wins, so behavior there is unchanged. Mirrors the shared-metrics store's
env-selected cross-worker coordination.
The neighbor warmer stays per-worker on purpose: each worker drains its own
request-fed queue, so gating it would leave non-leader queues undrained.
- backend/singleton.py: flock-based leader election (claim()).
- backend/app.py: gate notify.start() on singleton.claim().
- deploy/thermograph.service: default THERMOGRAPH_SINGLETON_LOCK to
data/notifier.lock (needs the unit reinstalled on prod to take effect).
- deploy/thermograph.env.example: document it alongside WORKERS.
- Tests: leader election — single-worker default, first-wins, idempotent
re-claim, second-holder stands down, re-election after release. 183 passed.
Co-authored-by: root <root@vmi3417050.contaboserver.net>
2026-07-17 12:54:55 +00:00
|
|
|
|
import singleton
|
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21)
* Compress API responses and revalidate static assets instead of re-downloading
- Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x.
- Serve pages/assets with Cache-Control: no-cache instead of no-store, so
browsers revalidate via the ETag/Last-Modified that FileResponse and
StaticFiles already emit. Unchanged assets now cost an empty 304 rather
than a full transfer on every page navigation, while deploys still show
up immediately.
* Persist derived responses in SQLite so grading is computed once per cell, not per request
New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet
records — finished grade/calendar/day/forecast payloads and reverse-geocode
labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms)
becomes a ~5ms database read that survives restarts and is shared across views.
Raw parquet stays the source of truth; the store is a pure accelerator (every
reader falls back to recomputing on a miss, and deleting the db is a safe reset).
Freshness is token-driven, not clock-driven: each cached payload is validated by
a token encoding what it was computed from (payload schema version, the archive
record's end date, the recent-fetch stamp). The existing freshness drivers are
untouched — get_history still tops up the tail hourly and get_recent_forecast
still refetches hourly — and tokens are derived from what they return, so cached
payloads expire exactly when their inputs change. The same tokens double as weak
ETags: If-None-Match answers with an empty 304 without touching the payload.
- backend/store.py: derived-payload + revgeo tables, thread-local WAL conns,
every helper fail-soft.
- app.py: endpoints split into pure payload builders + HTTP/caching shells; the
in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store).
- climate.py: revgeo persisted through the store; recent_stamp() and
load_cached_history() (no-network read) helpers.
- backend/migrate.py + make migrate: idempotent, resumable backfill of the store
from existing parquet caches (default calendar span + latest-day detail +
revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather.
- grid.from_id(): rebuild a cell from its cache filename (migrate tooling).
* Add /api/v2/cell: one bundle carrying every view's payload
GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months)
and day (today) payloads in one response, each the exact payload its per-view
endpoint returns — built by the same builders and cached under the same
derived-store keys/tokens — paired with the etag that endpoint would emit. The
frontend can warm all views with a single request, seed its per-view cache from
the slices, and later revalidate each view individually with If-None-Match. The
bundle's own etag combines the slices', so an unchanged bundle is an empty 304.
prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard
guarantee: it never spends weather-API quota. A cell with no cached archive
answers 204, and only the history-derived slices (calendar + latest-day detail)
are built. At most one Nominatim lookup for a never-labeled cell.
* Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch
The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota,
which multi-year calendar payloads regularly blew through) to IndexedDB, with an
in-memory map in front. Entries carry the server's ETag, so anything stale
revalidates conditionally — unchanged data costs an empty 304 and a re-stamp,
never a re-transfer. On network failure the stale copy is served over an error.
getJSON gains an optional onUpdate callback opting into stale-while-revalidate:
the three views (weekly, day, calendar) now render a cached copy immediately —
spinners are delayed 150ms so warm loads never flash-blank — and repaint only if
background revalidation finds changed data. New data shows up the moment it
exists instead of waiting out a TTL.
Cross-view prefetch collapses from one request per view to a single /api/v2/cell
bundle, whose slices (exact per-view payloads + their etags) are seeded under the
URLs each view actually requests; the bundle call itself is conditional via a
remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side
with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one
possible Nominatim lookup each), so tapping nearby lands on already-graded data.
Legacy tg:* storage entries are cleared once; cache entries untouched for two
weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
|
|
|
|
import store
|
Account system with weather-notification subscriptions (#89)
* Add account system foundation: email/password auth with cookie sessions
Introduce the app's first authoritative, user-owned data in a separate
data/accounts.sqlite (SQLAlchemy), kept apart from the disposable derived-cache
DB. Wire fastapi-users for email/password signup, cookie-based login/logout, and
a session-check endpoint, backed by a database session strategy so logins survive
restarts and are revocable.
- db.py: async (aiosqlite) + sync SQLAlchemy engines over accounts.sqlite, WAL +
foreign keys, create_db_and_tables().
- models.py: User, AccessToken, Subscription, Notification tables.
- users.py: pwdlib hashing, HttpOnly cookie transport (path-scoped, SameSite=Lax,
Secure via env), DatabaseStrategy sessions, current-user dependencies.
- schemas.py: user + subscription + notification Pydantic models.
- app.py: mount auth/register/users routers on v2, create tables at startup.
- Pin fastapi-users[sqlalchemy]/aiosqlite; ignore data/accounts.sqlite*.
* Add account header entry and auth modal (frontend)
account.js self-injects a header entry (following the units.js pattern) that
shows a Sign in button when logged out and an account menu when logged in, plus
an auth modal reusing the existing .mp-overlay/.mp-modal chrome for email/password
sign-in and account creation. A shared apiFetch helper sends the same-origin
cookie for authed calls; exported getUser/openAuth/onAuthChange back later phases.
Imported by every page entry module. On narrow screens the entry collapses to an
icon-only button so it doesn't crowd the title.
Enforce an 8-character minimum password in the user manager.
* Add subscription CRUD API and the alerts management page
Backend api_accounts.py adds user-scoped, cookie-authenticated endpoints to
create/list/update/delete subscriptions (and the notification reads used next):
POST snaps lat/lon to a grid cell, resolves a label, and rejects a duplicate
location+kind with 409; PATCH/DELETE are ownership-checked (404 on mismatch).
Mounted on the v2 prefix.
Frontend subscriptions.js + subscriptions.html serve the /alerts page: a sign-in
gate when logged out, an add flow that reuses the shared map picker and an editor
modal (kind, watched metrics, 95-99 percentile, two-sided), and a card list with
inline threshold/active edits and remove. Reachable from the account menu.
* Add background subscription evaluation engine
notify.py runs a daemon thread that periodically evaluates every active
subscription: it groups them by grid cell, reads history from the parquet cache
only (never spends archive quota) plus the hourly recent/forecast bundle, and
grades candidate days with the existing grading.grade_day. A watched metric that
lands at or beyond the threshold percentile fires a 'high' alert; a two-sided
subscription also fires 'low' for the symmetric cold/calm/dry tail (precip stays
one-directional). Observed subscriptions look at the last few recorded days,
forecast subscriptions at the coming week.
Two guards keep it quiet: a UNIQUE(subscription, event_date, metric, direction,
kind) constraint dedups repeat events, and a per-subscription weekly cap
(last_notified_at) limits each alert to one notification per 7 days. The loop
tolerates a bad cell or an upstream rate limit without aborting the pass. Started
and stopped from the app lifespan; gated by THERMOGRAPH_ENABLE_NOTIFIER.
* Add in-app notification center (header bell)
Extend account.js with a notification bell beside the account menu: an unread
badge, a dropdown listing recent notifications (title, body, relative time), a
per-item mark-read on click, and a Mark all read action, all through the
cookie-authed notifications API. Unread state refreshes on open and polls every
two minutes while signed in; polling stops on sign-out. Styled to match the app,
responsive down to mobile.
* Harden accounts: expired-session cleanup, engine tests, ops docs
- notify.py sweeps expired login sessions (access tokens past their lifetime)
once per evaluation pass.
- Add hermetic unit tests for the evaluation engine's trigger detection (high/low
tails, precip one-directional, normal = no trigger) and notification wording.
- Document accounts.sqlite (authoritative, back it up), the single-worker
requirement for the in-process evaluator, and the new env vars in DEPLOY.md.
2026-07-15 18:46:46 +00:00
|
|
|
|
import users
|
Extract payload builders into views.py; load places index at startup (#42)
app.py had grown three co-resident strata: HTTP/caching plumbing, payload
assembly, and search policy. This moves the payload layer — the four
build_* functions, cal_span clamping, hist_end, PAYLOAD_VER, NullRun and
their helpers — into a new views.py with no web dependencies (pure moves,
public names). app.py keeps routing, ETag/derived-store plumbing, and
page serving; migrate.py imports views instead of reaching into app's
privates.
That import previously constructed the whole FastAPI app and — because
places.start_loading() ran at import time — kicked off a background
GeoNames download from an offline batch script. The index load now hangs
off the app's lifespan hook, so it fires when a server starts, not when
the module is imported (tests, migrate).
New tests: cal_span clamping (months-back default, record bounds, 2-year
cap, inversion), builder shapes (grade window ordering, day obs from the
recent bundle, forecast future-only), a regression test that build_day
degrades to climatology when the recent fetch fails, and a subprocess
layering guard that importing views/migrate pulls in neither FastAPI nor
the app module and starts no index download.
2026-07-11 19:43:41 +00:00
|
|
|
|
import views
|
Account system with weather-notification subscriptions (#89)
* Add account system foundation: email/password auth with cookie sessions
Introduce the app's first authoritative, user-owned data in a separate
data/accounts.sqlite (SQLAlchemy), kept apart from the disposable derived-cache
DB. Wire fastapi-users for email/password signup, cookie-based login/logout, and
a session-check endpoint, backed by a database session strategy so logins survive
restarts and are revocable.
- db.py: async (aiosqlite) + sync SQLAlchemy engines over accounts.sqlite, WAL +
foreign keys, create_db_and_tables().
- models.py: User, AccessToken, Subscription, Notification tables.
- users.py: pwdlib hashing, HttpOnly cookie transport (path-scoped, SameSite=Lax,
Secure via env), DatabaseStrategy sessions, current-user dependencies.
- schemas.py: user + subscription + notification Pydantic models.
- app.py: mount auth/register/users routers on v2, create tables at startup.
- Pin fastapi-users[sqlalchemy]/aiosqlite; ignore data/accounts.sqlite*.
* Add account header entry and auth modal (frontend)
account.js self-injects a header entry (following the units.js pattern) that
shows a Sign in button when logged out and an account menu when logged in, plus
an auth modal reusing the existing .mp-overlay/.mp-modal chrome for email/password
sign-in and account creation. A shared apiFetch helper sends the same-origin
cookie for authed calls; exported getUser/openAuth/onAuthChange back later phases.
Imported by every page entry module. On narrow screens the entry collapses to an
icon-only button so it doesn't crowd the title.
Enforce an 8-character minimum password in the user manager.
* Add subscription CRUD API and the alerts management page
Backend api_accounts.py adds user-scoped, cookie-authenticated endpoints to
create/list/update/delete subscriptions (and the notification reads used next):
POST snaps lat/lon to a grid cell, resolves a label, and rejects a duplicate
location+kind with 409; PATCH/DELETE are ownership-checked (404 on mismatch).
Mounted on the v2 prefix.
Frontend subscriptions.js + subscriptions.html serve the /alerts page: a sign-in
gate when logged out, an add flow that reuses the shared map picker and an editor
modal (kind, watched metrics, 95-99 percentile, two-sided), and a card list with
inline threshold/active edits and remove. Reachable from the account menu.
* Add background subscription evaluation engine
notify.py runs a daemon thread that periodically evaluates every active
subscription: it groups them by grid cell, reads history from the parquet cache
only (never spends archive quota) plus the hourly recent/forecast bundle, and
grades candidate days with the existing grading.grade_day. A watched metric that
lands at or beyond the threshold percentile fires a 'high' alert; a two-sided
subscription also fires 'low' for the symmetric cold/calm/dry tail (precip stays
one-directional). Observed subscriptions look at the last few recorded days,
forecast subscriptions at the coming week.
Two guards keep it quiet: a UNIQUE(subscription, event_date, metric, direction,
kind) constraint dedups repeat events, and a per-subscription weekly cap
(last_notified_at) limits each alert to one notification per 7 days. The loop
tolerates a bad cell or an upstream rate limit without aborting the pass. Started
and stopped from the app lifespan; gated by THERMOGRAPH_ENABLE_NOTIFIER.
* Add in-app notification center (header bell)
Extend account.js with a notification bell beside the account menu: an unread
badge, a dropdown listing recent notifications (title, body, relative time), a
per-item mark-read on click, and a Mark all read action, all through the
cookie-authed notifications API. Unread state refreshes on open and polls every
two minutes while signed in; polling stops on sign-out. Styled to match the app,
responsive down to mobile.
* Harden accounts: expired-session cleanup, engine tests, ops docs
- notify.py sweeps expired login sessions (access tokens past their lifetime)
once per evaluation pass.
- Add hermetic unit tests for the evaluation engine's trigger detection (high/low
tails, precip one-directional, normal = no trigger) and notification wording.
- Document accounts.sqlite (authoritative, back it up), the single-worker
requirement for the in-process evaluator, and the new env vars in DEPLOY.md.
2026-07-15 18:46:46 +00:00
|
|
|
|
from schemas import UserCreate, UserRead, UserUpdate
|
2026-07-11 00:29:47 +00:00
|
|
|
|
|
|
|
|
|
|
FRONTEND_DIR = os.path.join(os.path.dirname(__file__), "..", "frontend")
|
|
|
|
|
|
|
|
|
|
|
|
# Everything (pages, assets, API) is served under this base path so the app can
|
|
|
|
|
|
# live at https://<host>/thermograph/ behind a shared domain. Override with the
|
|
|
|
|
|
# THERMOGRAPH_BASE env var. The frontend uses base-relative URLs, so it follows
|
|
|
|
|
|
# this automatically without hardcoding the prefix.
|
2026-07-15 19:58:32 +00:00
|
|
|
|
#
|
|
|
|
|
|
# THERMOGRAPH_BASE=/ (or empty) means the app owns the whole domain: BASE becomes
|
|
|
|
|
|
# "" so every route sits at the domain root ("/", "/calendar", "/api/v2/…") with
|
|
|
|
|
|
# no prefix and no leading double slash. A non-empty value keeps the sub-path form
|
|
|
|
|
|
# ("/thermograph").
|
|
|
|
|
|
_BASE = os.environ.get("THERMOGRAPH_BASE", "/thermograph").strip("/")
|
|
|
|
|
|
BASE = f"/{_BASE}" if _BASE else ""
|
2026-07-11 00:29:47 +00:00
|
|
|
|
|
2026-07-15 23:21:06 +00:00
|
|
|
|
# The PWA manifest is served as a static file; register its media type so
|
|
|
|
|
|
# StaticFiles labels it correctly (not in Python's default mimetypes map).
|
|
|
|
|
|
mimetypes.add_type("application/manifest+json", ".webmanifest")
|
|
|
|
|
|
|
2026-07-11 00:29:47 +00:00
|
|
|
|
|
|
|
|
|
|
def _weather_fetch_error(e) -> HTTPException:
|
2026-07-11 19:53:46 +00:00
|
|
|
|
"""A clean, retryable 503 when weather data is temporarily unavailable; the
|
|
|
|
|
|
raw error as a 502 otherwise. The fetch layer classifies its own failures
|
|
|
|
|
|
(climate.WeatherUnavailable carries the user-facing text) so no message
|
|
|
|
|
|
parsing happens here; the is_rate_limit fallback covers a raw upstream 429
|
|
|
|
|
|
from a fetcher that didn't classify it."""
|
|
|
|
|
|
if isinstance(e, climate.WeatherUnavailable):
|
|
|
|
|
|
return HTTPException(status_code=503, detail=str(e))
|
|
|
|
|
|
if climate.is_rate_limit(e):
|
|
|
|
|
|
return HTTPException(status_code=503, detail=climate.limit_message(False))
|
2026-07-11 00:29:47 +00:00
|
|
|
|
return HTTPException(status_code=502, detail=f"weather data fetch failed: {e}")
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-07-11 20:47:31 +00:00
|
|
|
|
# --- background neighbor warming ---------------------------------------------
|
|
|
|
|
|
# /cell?neighbors=1 asks the server to warm the 8 grid cells around the request
|
|
|
|
|
|
# (the frontend used to fire 8 staggered prefetch requests, mirroring grid.py's
|
|
|
|
|
|
# snapping math in JS). One worker drains the queue so warms never stack; the
|
|
|
|
|
|
# hard guarantee matches prefetch=1 — a cell with no cached archive is skipped,
|
|
|
|
|
|
# so no weather-API quota is ever spent, and reverse_geocode itself paces the
|
|
|
|
|
|
# at-most-one Nominatim call per never-labeled cell (~1/s policy).
|
|
|
|
|
|
|
|
|
|
|
|
_WARM_TTL = 3600.0 # don't re-enqueue a cell within the hour
|
|
|
|
|
|
_WARM_SEEN: dict[str, float] = {} # cell_id -> last enqueue time
|
|
|
|
|
|
_warm_queue: "queue.Queue[dict]" = queue.Queue()
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def _warm_cell(cell: dict) -> None:
|
|
|
|
|
|
"""Materialize the history-derived slices for one cell — the same rows the
|
|
|
|
|
|
prefetch=1 bundle serves. Never fetches weather upstream."""
|
|
|
|
|
|
history = climate.load_cached_history(cell)
|
Migrate backend dataframe layer from pandas to polars (#90)
* Migrate backend dataframe layer from pandas to polars
Replace pandas with polars across the backend, dropping both pandas and its
pyarrow parquet engine from the dependency set. numpy stays (the grading
percentile math is unchanged).
- climate.py: parquet IO, source→frame mappings, cache read/topup on polars.
New _normalize_read casts the cached `date` column to pl.Date (older files
were written by pandas as datetime64[ns]); frames now unify missing values as
null so the grading boundary drops them consistently across sources.
- grading.py: keep the numpy percentile core; swap the frame→numpy bridge to
.to_numpy()/.drop_nulls(), day-of-year/year to polars dt expressions, and the
per-row loop to iter_rows(named=True).
- views.py: filter/anti-join/concat replace boolean-mask, isin and pd.concat;
scalar dates are stdlib datetime.date; a local _months_before helper replaces
DateOffset(months=) for the calendar-range default.
- app.py, migrate.py: request-date parsing uses datetime.date, removing pandas
from the endpoint and migrate layers entirely.
- The date column is pl.Date end to end, eliminating the pandas normalize() calls
and comparing cleanly against stdlib dates.
Payloads are unchanged: calendar, day, grade and forecast responses are
byte-for-byte identical to the pandas implementation on the same cached record.
Tests ported to polars fixtures, with added coverage for the combined feels-like
fallback, calendar month-offset (month-end/leap), and the concat/dedup
"fresher source wins" rule.
* Port notify.py to polars after merging dev's account system
Merge origin/dev (accounts + notification subscriptions) and carry the pandas→
polars migration into the newly added notify.py, which the merge brought in still
using pandas — with pandas removed from requirements this broke its import.
- notify.py: _candidate_rows filters/sorts the recent bundle with polars
expressions and returns iter_rows dicts; date scalars are datetime.date;
history/recent emptiness via is_empty().
- test_notify.py: synthetic history/rows built with polars + datetime.
2026-07-15 19:07:38 +00:00
|
|
|
|
if history is None or history.is_empty():
|
2026-07-11 20:47:31 +00:00
|
|
|
|
return
|
|
|
|
|
|
place = climate.reverse_geocode(cell["center_lat"], cell["center_lon"])
|
|
|
|
|
|
token = views.history_token(history)
|
|
|
|
|
|
start_ts, end_ts = views.cal_span(history, None, None, 24)
|
|
|
|
|
|
cal_key = views.calendar_key(start_ts, end_ts, 24)
|
|
|
|
|
|
if store.get_payload("calendar", cell["id"], cal_key, token) is None:
|
|
|
|
|
|
store.put_payload("calendar", cell["id"], cal_key, token,
|
|
|
|
|
|
views.build_calendar(cell, history, start_ts, end_ts, 24, place))
|
Migrate backend dataframe layer from pandas to polars (#90)
* Migrate backend dataframe layer from pandas to polars
Replace pandas with polars across the backend, dropping both pandas and its
pyarrow parquet engine from the dependency set. numpy stays (the grading
percentile math is unchanged).
- climate.py: parquet IO, source→frame mappings, cache read/topup on polars.
New _normalize_read casts the cached `date` column to pl.Date (older files
were written by pandas as datetime64[ns]); frames now unify missing values as
null so the grading boundary drops them consistently across sources.
- grading.py: keep the numpy percentile core; swap the frame→numpy bridge to
.to_numpy()/.drop_nulls(), day-of-year/year to polars dt expressions, and the
per-row loop to iter_rows(named=True).
- views.py: filter/anti-join/concat replace boolean-mask, isin and pd.concat;
scalar dates are stdlib datetime.date; a local _months_before helper replaces
DateOffset(months=) for the calendar-range default.
- app.py, migrate.py: request-date parsing uses datetime.date, removing pandas
from the endpoint and migrate layers entirely.
- The date column is pl.Date end to end, eliminating the pandas normalize() calls
and comparing cleanly against stdlib dates.
Payloads are unchanged: calendar, day, grade and forecast responses are
byte-for-byte identical to the pandas implementation on the same cached record.
Tests ported to polars fixtures, with added coverage for the combined feels-like
fallback, calendar month-offset (month-end/leap), and the concat/dedup
"fresher source wins" rule.
* Port notify.py to polars after merging dev's account system
Merge origin/dev (accounts + notification subscriptions) and carry the pandas→
polars migration into the newly added notify.py, which the merge brought in still
using pandas — with pandas removed from requirements this broke its import.
- notify.py: _candidate_rows filters/sorts the recent bundle with polars
expressions and returns iter_rows dicts; date scalars are datetime.date;
history/recent emptiness via is_empty().
- test_notify.py: synthetic history/rows built with polars + datetime.
2026-07-15 19:07:38 +00:00
|
|
|
|
last = history["date"].max()
|
2026-07-11 20:47:31 +00:00
|
|
|
|
day_key = views.day_key(last)
|
|
|
|
|
|
if store.get_payload("day", cell["id"], day_key, token) is None:
|
|
|
|
|
|
store.put_payload("day", cell["id"], day_key, token,
|
|
|
|
|
|
views.build_day(cell, history, last, place))
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def _enqueue_neighbor_warming(cell: dict) -> None:
|
|
|
|
|
|
now = time.time()
|
|
|
|
|
|
for n in grid.neighbors(cell):
|
|
|
|
|
|
if now - _WARM_SEEN.get(n["id"], 0.0) < _WARM_TTL:
|
|
|
|
|
|
continue
|
|
|
|
|
|
_WARM_SEEN[n["id"]] = now
|
|
|
|
|
|
_warm_queue.put(n)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def _neighbor_warmer() -> None:
|
|
|
|
|
|
while True:
|
|
|
|
|
|
cell = _warm_queue.get()
|
|
|
|
|
|
try:
|
|
|
|
|
|
_warm_cell(cell)
|
|
|
|
|
|
except Exception: # noqa: BLE001 - warming is best-effort, never fatal
|
|
|
|
|
|
pass
|
|
|
|
|
|
finally:
|
|
|
|
|
|
_warm_queue.task_done()
|
|
|
|
|
|
|
|
|
|
|
|
|
Extract payload builders into views.py; load places index at startup (#42)
app.py had grown three co-resident strata: HTTP/caching plumbing, payload
assembly, and search policy. This moves the payload layer — the four
build_* functions, cal_span clamping, hist_end, PAYLOAD_VER, NullRun and
their helpers — into a new views.py with no web dependencies (pure moves,
public names). app.py keeps routing, ETag/derived-store plumbing, and
page serving; migrate.py imports views instead of reaching into app's
privates.
That import previously constructed the whole FastAPI app and — because
places.start_loading() ran at import time — kicked off a background
GeoNames download from an offline batch script. The index load now hangs
off the app's lifespan hook, so it fires when a server starts, not when
the module is imported (tests, migrate).
New tests: cal_span clamping (months-back default, record bounds, 2-year
cap, inversion), builder shapes (grade window ordering, day obs from the
recent bundle, forecast future-only), a regression test that build_day
degrades to climatology when the recent fetch fails, and a subprocess
layering guard that importing views/migrate pulls in neither FastAPI nor
the app module and starts no index download.
2026-07-11 19:43:41 +00:00
|
|
|
|
@contextlib.asynccontextmanager
|
|
|
|
|
|
async def _lifespan(app):
|
|
|
|
|
|
# Warm the local place-name index (/suggest's typo tolerance) in the
|
|
|
|
|
|
# background; the app boots and serves fine without it — suggestions just
|
|
|
|
|
|
# fall back to the upstream geocoder until it's ready. A server-startup
|
|
|
|
|
|
# hook (not import time) so offline importers (tests, the migrate script's
|
2026-07-11 20:47:31 +00:00
|
|
|
|
# dependencies) don't kick off a GeoNames download. The neighbor warmer is
|
|
|
|
|
|
# also a server concern: enqueues before startup just wait in the queue.
|
Account system with weather-notification subscriptions (#89)
* Add account system foundation: email/password auth with cookie sessions
Introduce the app's first authoritative, user-owned data in a separate
data/accounts.sqlite (SQLAlchemy), kept apart from the disposable derived-cache
DB. Wire fastapi-users for email/password signup, cookie-based login/logout, and
a session-check endpoint, backed by a database session strategy so logins survive
restarts and are revocable.
- db.py: async (aiosqlite) + sync SQLAlchemy engines over accounts.sqlite, WAL +
foreign keys, create_db_and_tables().
- models.py: User, AccessToken, Subscription, Notification tables.
- users.py: pwdlib hashing, HttpOnly cookie transport (path-scoped, SameSite=Lax,
Secure via env), DatabaseStrategy sessions, current-user dependencies.
- schemas.py: user + subscription + notification Pydantic models.
- app.py: mount auth/register/users routers on v2, create tables at startup.
- Pin fastapi-users[sqlalchemy]/aiosqlite; ignore data/accounts.sqlite*.
* Add account header entry and auth modal (frontend)
account.js self-injects a header entry (following the units.js pattern) that
shows a Sign in button when logged out and an account menu when logged in, plus
an auth modal reusing the existing .mp-overlay/.mp-modal chrome for email/password
sign-in and account creation. A shared apiFetch helper sends the same-origin
cookie for authed calls; exported getUser/openAuth/onAuthChange back later phases.
Imported by every page entry module. On narrow screens the entry collapses to an
icon-only button so it doesn't crowd the title.
Enforce an 8-character minimum password in the user manager.
* Add subscription CRUD API and the alerts management page
Backend api_accounts.py adds user-scoped, cookie-authenticated endpoints to
create/list/update/delete subscriptions (and the notification reads used next):
POST snaps lat/lon to a grid cell, resolves a label, and rejects a duplicate
location+kind with 409; PATCH/DELETE are ownership-checked (404 on mismatch).
Mounted on the v2 prefix.
Frontend subscriptions.js + subscriptions.html serve the /alerts page: a sign-in
gate when logged out, an add flow that reuses the shared map picker and an editor
modal (kind, watched metrics, 95-99 percentile, two-sided), and a card list with
inline threshold/active edits and remove. Reachable from the account menu.
* Add background subscription evaluation engine
notify.py runs a daemon thread that periodically evaluates every active
subscription: it groups them by grid cell, reads history from the parquet cache
only (never spends archive quota) plus the hourly recent/forecast bundle, and
grades candidate days with the existing grading.grade_day. A watched metric that
lands at or beyond the threshold percentile fires a 'high' alert; a two-sided
subscription also fires 'low' for the symmetric cold/calm/dry tail (precip stays
one-directional). Observed subscriptions look at the last few recorded days,
forecast subscriptions at the coming week.
Two guards keep it quiet: a UNIQUE(subscription, event_date, metric, direction,
kind) constraint dedups repeat events, and a per-subscription weekly cap
(last_notified_at) limits each alert to one notification per 7 days. The loop
tolerates a bad cell or an upstream rate limit without aborting the pass. Started
and stopped from the app lifespan; gated by THERMOGRAPH_ENABLE_NOTIFIER.
* Add in-app notification center (header bell)
Extend account.js with a notification bell beside the account menu: an unread
badge, a dropdown listing recent notifications (title, body, relative time), a
per-item mark-read on click, and a Mark all read action, all through the
cookie-authed notifications API. Unread state refreshes on open and polls every
two minutes while signed in; polling stops on sign-out. Styled to match the app,
responsive down to mobile.
* Harden accounts: expired-session cleanup, engine tests, ops docs
- notify.py sweeps expired login sessions (access tokens past their lifetime)
once per evaluation pass.
- Add hermetic unit tests for the evaluation engine's trigger detection (high/low
tails, precip one-directional, normal = no trigger) and notification wording.
- Document accounts.sqlite (authoritative, back it up), the single-worker
requirement for the in-process evaluator, and the new env vars in DEPLOY.md.
2026-07-15 18:46:46 +00:00
|
|
|
|
# Create the account DB tables (users/sessions/subscriptions/notifications)
|
|
|
|
|
|
# before serving — a fast no-op once they exist.
|
|
|
|
|
|
await db.create_db_and_tables()
|
Extract payload builders into views.py; load places index at startup (#42)
app.py had grown three co-resident strata: HTTP/caching plumbing, payload
assembly, and search policy. This moves the payload layer — the four
build_* functions, cal_span clamping, hist_end, PAYLOAD_VER, NullRun and
their helpers — into a new views.py with no web dependencies (pure moves,
public names). app.py keeps routing, ETag/derived-store plumbing, and
page serving; migrate.py imports views instead of reaching into app's
privates.
That import previously constructed the whole FastAPI app and — because
places.start_loading() ran at import time — kicked off a background
GeoNames download from an offline batch script. The index load now hangs
off the app's lifespan hook, so it fires when a server starts, not when
the module is imported (tests, migrate).
New tests: cal_span clamping (months-back default, record bounds, 2-year
cap, inversion), builder shapes (grade window ordering, day obs from the
recent bundle, forecast future-only), a regression test that build_day
degrades to climatology when the recent fetch fails, and a subprocess
layering guard that importing views/migrate pulls in neither FastAPI nor
the app module and starts no index download.
2026-07-11 19:43:41 +00:00
|
|
|
|
places.start_loading()
|
2026-07-11 20:47:31 +00:00
|
|
|
|
threading.Thread(target=_neighbor_warmer, name="neighbor-warmer", daemon=True).start()
|
Gate the subscription notifier to one worker (#161)
Prod runs 3 uvicorn workers, but the in-process subscription notifier was
started in every worker's lifespan (it was written assuming a single-worker
deploy). Its sweep fetches recent-forecast/archive data from Open-Meteo on a
15-min timer independent of any request, so running it 3× tripled the
background upstream load against a shared quota — the source of the overnight
429/503 rate-limit errors in logs/errors, and 3× redundant subscription scans.
Elect one worker to own the notifier via a non-blocking exclusive flock on a
lockfile (THERMOGRAPH_SINGLETON_LOCK): the first worker wins and holds the fd
for its lifetime; others stand down; the OS releases the lock if the leader
dies, so a restart re-elects cleanly. Unset (single worker / dev / tests)
always wins, so behavior there is unchanged. Mirrors the shared-metrics store's
env-selected cross-worker coordination.
The neighbor warmer stays per-worker on purpose: each worker drains its own
request-fed queue, so gating it would leave non-leader queues undrained.
- backend/singleton.py: flock-based leader election (claim()).
- backend/app.py: gate notify.start() on singleton.claim().
- deploy/thermograph.service: default THERMOGRAPH_SINGLETON_LOCK to
data/notifier.lock (needs the unit reinstalled on prod to take effect).
- deploy/thermograph.env.example: document it alongside WORKERS.
- Tests: leader election — single-worker default, first-wins, idempotent
re-claim, second-holder stands down, re-election after release. 183 passed.
Co-authored-by: root <root@vmi3417050.contaboserver.net>
2026-07-17 12:54:55 +00:00
|
|
|
|
# Background subscription evaluator. Its periodic sweep fetches upstream on a timer
|
|
|
|
|
|
# (not per request), so with multiple workers it must run in exactly one — elect a
|
|
|
|
|
|
# leader via a lockfile (THERMOGRAPH_SINGLETON_LOCK). Unset (single worker / dev /
|
|
|
|
|
|
# tests) => always the leader. The neighbor warmer above stays per-worker: each
|
|
|
|
|
|
# drains its own request-fed queue.
|
|
|
|
|
|
if singleton.claim(os.environ.get("THERMOGRAPH_SINGLETON_LOCK")):
|
|
|
|
|
|
notify.start()
|
Extract payload builders into views.py; load places index at startup (#42)
app.py had grown three co-resident strata: HTTP/caching plumbing, payload
assembly, and search policy. This moves the payload layer — the four
build_* functions, cal_span clamping, hist_end, PAYLOAD_VER, NullRun and
their helpers — into a new views.py with no web dependencies (pure moves,
public names). app.py keeps routing, ETag/derived-store plumbing, and
page serving; migrate.py imports views instead of reaching into app's
privates.
That import previously constructed the whole FastAPI app and — because
places.start_loading() ran at import time — kicked off a background
GeoNames download from an offline batch script. The index load now hangs
off the app's lifespan hook, so it fires when a server starts, not when
the module is imported (tests, migrate).
New tests: cal_span clamping (months-back default, record bounds, 2-year
cap, inversion), builder shapes (grade window ordering, day obs from the
recent bundle, forecast future-only), a regression test that build_day
degrades to climatology when the recent fetch fails, and a subprocess
layering guard that importing views/migrate pulls in neither FastAPI nor
the app module and starts no index download.
2026-07-11 19:43:41 +00:00
|
|
|
|
yield
|
Account system with weather-notification subscriptions (#89)
* Add account system foundation: email/password auth with cookie sessions
Introduce the app's first authoritative, user-owned data in a separate
data/accounts.sqlite (SQLAlchemy), kept apart from the disposable derived-cache
DB. Wire fastapi-users for email/password signup, cookie-based login/logout, and
a session-check endpoint, backed by a database session strategy so logins survive
restarts and are revocable.
- db.py: async (aiosqlite) + sync SQLAlchemy engines over accounts.sqlite, WAL +
foreign keys, create_db_and_tables().
- models.py: User, AccessToken, Subscription, Notification tables.
- users.py: pwdlib hashing, HttpOnly cookie transport (path-scoped, SameSite=Lax,
Secure via env), DatabaseStrategy sessions, current-user dependencies.
- schemas.py: user + subscription + notification Pydantic models.
- app.py: mount auth/register/users routers on v2, create tables at startup.
- Pin fastapi-users[sqlalchemy]/aiosqlite; ignore data/accounts.sqlite*.
* Add account header entry and auth modal (frontend)
account.js self-injects a header entry (following the units.js pattern) that
shows a Sign in button when logged out and an account menu when logged in, plus
an auth modal reusing the existing .mp-overlay/.mp-modal chrome for email/password
sign-in and account creation. A shared apiFetch helper sends the same-origin
cookie for authed calls; exported getUser/openAuth/onAuthChange back later phases.
Imported by every page entry module. On narrow screens the entry collapses to an
icon-only button so it doesn't crowd the title.
Enforce an 8-character minimum password in the user manager.
* Add subscription CRUD API and the alerts management page
Backend api_accounts.py adds user-scoped, cookie-authenticated endpoints to
create/list/update/delete subscriptions (and the notification reads used next):
POST snaps lat/lon to a grid cell, resolves a label, and rejects a duplicate
location+kind with 409; PATCH/DELETE are ownership-checked (404 on mismatch).
Mounted on the v2 prefix.
Frontend subscriptions.js + subscriptions.html serve the /alerts page: a sign-in
gate when logged out, an add flow that reuses the shared map picker and an editor
modal (kind, watched metrics, 95-99 percentile, two-sided), and a card list with
inline threshold/active edits and remove. Reachable from the account menu.
* Add background subscription evaluation engine
notify.py runs a daemon thread that periodically evaluates every active
subscription: it groups them by grid cell, reads history from the parquet cache
only (never spends archive quota) plus the hourly recent/forecast bundle, and
grades candidate days with the existing grading.grade_day. A watched metric that
lands at or beyond the threshold percentile fires a 'high' alert; a two-sided
subscription also fires 'low' for the symmetric cold/calm/dry tail (precip stays
one-directional). Observed subscriptions look at the last few recorded days,
forecast subscriptions at the coming week.
Two guards keep it quiet: a UNIQUE(subscription, event_date, metric, direction,
kind) constraint dedups repeat events, and a per-subscription weekly cap
(last_notified_at) limits each alert to one notification per 7 days. The loop
tolerates a bad cell or an upstream rate limit without aborting the pass. Started
and stopped from the app lifespan; gated by THERMOGRAPH_ENABLE_NOTIFIER.
* Add in-app notification center (header bell)
Extend account.js with a notification bell beside the account menu: an unread
badge, a dropdown listing recent notifications (title, body, relative time), a
per-item mark-read on click, and a Mark all read action, all through the
cookie-authed notifications API. Unread state refreshes on open and polls every
two minutes while signed in; polling stops on sign-out. Styled to match the app,
responsive down to mobile.
* Harden accounts: expired-session cleanup, engine tests, ops docs
- notify.py sweeps expired login sessions (access tokens past their lifetime)
once per evaluation pass.
- Add hermetic unit tests for the evaluation engine's trigger detection (high/low
tails, precip one-directional, normal = no trigger) and notification wording.
- Document accounts.sqlite (authoritative, back it up), the single-worker
requirement for the in-process evaluator, and the new env vars in DEPLOY.md.
2026-07-15 18:46:46 +00:00
|
|
|
|
notify.stop()
|
2026-07-11 00:29:47 +00:00
|
|
|
|
|
|
|
|
|
|
|
Extract payload builders into views.py; load places index at startup (#42)
app.py had grown three co-resident strata: HTTP/caching plumbing, payload
assembly, and search policy. This moves the payload layer — the four
build_* functions, cal_span clamping, hist_end, PAYLOAD_VER, NullRun and
their helpers — into a new views.py with no web dependencies (pure moves,
public names). app.py keeps routing, ETag/derived-store plumbing, and
page serving; migrate.py imports views instead of reaching into app's
privates.
That import previously constructed the whole FastAPI app and — because
places.start_loading() ran at import time — kicked off a background
GeoNames download from an offline batch script. The index load now hangs
off the app's lifespan hook, so it fires when a server starts, not when
the module is imported (tests, migrate).
New tests: cal_span clamping (months-back default, record bounds, 2-year
cap, inversion), builder shapes (grade window ordering, day obs from the
recent bundle, forecast future-only), a regression test that build_day
degrades to climatology when the recent fetch fails, and a subprocess
layering guard that importing views/migrate pulls in neither FastAPI nor
the app module and starts no index download.
2026-07-11 19:43:41 +00:00
|
|
|
|
app = FastAPI(title="Thermograph", version="0.2.0", lifespan=_lifespan)
|
2026-07-11 00:29:47 +00:00
|
|
|
|
|
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21)
* Compress API responses and revalidate static assets instead of re-downloading
- Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x.
- Serve pages/assets with Cache-Control: no-cache instead of no-store, so
browsers revalidate via the ETag/Last-Modified that FileResponse and
StaticFiles already emit. Unchanged assets now cost an empty 304 rather
than a full transfer on every page navigation, while deploys still show
up immediately.
* Persist derived responses in SQLite so grading is computed once per cell, not per request
New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet
records — finished grade/calendar/day/forecast payloads and reverse-geocode
labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms)
becomes a ~5ms database read that survives restarts and is shared across views.
Raw parquet stays the source of truth; the store is a pure accelerator (every
reader falls back to recomputing on a miss, and deleting the db is a safe reset).
Freshness is token-driven, not clock-driven: each cached payload is validated by
a token encoding what it was computed from (payload schema version, the archive
record's end date, the recent-fetch stamp). The existing freshness drivers are
untouched — get_history still tops up the tail hourly and get_recent_forecast
still refetches hourly — and tokens are derived from what they return, so cached
payloads expire exactly when their inputs change. The same tokens double as weak
ETags: If-None-Match answers with an empty 304 without touching the payload.
- backend/store.py: derived-payload + revgeo tables, thread-local WAL conns,
every helper fail-soft.
- app.py: endpoints split into pure payload builders + HTTP/caching shells; the
in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store).
- climate.py: revgeo persisted through the store; recent_stamp() and
load_cached_history() (no-network read) helpers.
- backend/migrate.py + make migrate: idempotent, resumable backfill of the store
from existing parquet caches (default calendar span + latest-day detail +
revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather.
- grid.from_id(): rebuild a cell from its cache filename (migrate tooling).
* Add /api/v2/cell: one bundle carrying every view's payload
GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months)
and day (today) payloads in one response, each the exact payload its per-view
endpoint returns — built by the same builders and cached under the same
derived-store keys/tokens — paired with the etag that endpoint would emit. The
frontend can warm all views with a single request, seed its per-view cache from
the slices, and later revalidate each view individually with If-None-Match. The
bundle's own etag combines the slices', so an unchanged bundle is an empty 304.
prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard
guarantee: it never spends weather-API quota. A cell with no cached archive
answers 204, and only the history-derived slices (calendar + latest-day detail)
are built. At most one Nominatim lookup for a never-labeled cell.
* Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch
The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota,
which multi-year calendar payloads regularly blew through) to IndexedDB, with an
in-memory map in front. Entries carry the server's ETag, so anything stale
revalidates conditionally — unchanged data costs an empty 304 and a re-stamp,
never a re-transfer. On network failure the stale copy is served over an error.
getJSON gains an optional onUpdate callback opting into stale-while-revalidate:
the three views (weekly, day, calendar) now render a cached copy immediately —
spinners are delayed 150ms so warm loads never flash-blank — and repaint only if
background revalidation finds changed data. New data shows up the moment it
exists instead of waiting out a TTL.
Cross-view prefetch collapses from one request per view to a single /api/v2/cell
bundle, whose slices (exact per-view payloads + their etags) are seeded under the
URLs each view actually requests; the bundle call itself is conditional via a
remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side
with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one
possible Nominatim lookup each), so tapping nearby lands on already-graded data.
Legacy tg:* storage entries are cleared once; cache entries untouched for two
weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
|
|
|
|
# Compress every sizeable response (the 2-year calendar JSON shrinks ~6-8×).
|
|
|
|
|
|
# Applies to API JSON and static assets alike; tiny responses are left alone.
|
|
|
|
|
|
app.add_middleware(GZipMiddleware, minimum_size=1024)
|
|
|
|
|
|
|
2026-07-11 00:29:47 +00:00
|
|
|
|
|
Dashboard: accounts totals only; log client IPs per request (#140)
Dashboard ACCOUNTS section now shows only totals (users, active/total subscriptions,
notifications, push subs) — the per-user and per-subscription lists are gone, and the
dashboard no longer queries that PII at all.
Backend: add a per-request access log (logs/access/access-<date>.jsonl) recording the
client IP (real IP via the left-most X-Forwarded-For hop when proxied behind Caddy,
else the peer address), method, path, status, and category. Static assets and the
dashboard's own metrics polling are skipped. Best-effort, never raises into the request
path — retained for later traffic/IP analysis.
2026-07-16 21:58:40 +00:00
|
|
|
|
def _client_ip(request) -> "str | None":
|
|
|
|
|
|
"""The real client IP: the left-most X-Forwarded-For hop when a proxy fronts us
|
|
|
|
|
|
(Caddy on prod sets it), otherwise the direct peer address (LAN dev)."""
|
|
|
|
|
|
xff = request.headers.get("x-forwarded-for")
|
|
|
|
|
|
if xff:
|
|
|
|
|
|
return xff.split(",")[0].strip()
|
|
|
|
|
|
return request.client.host if request.client else None
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-07-11 00:29:47 +00:00
|
|
|
|
@app.middleware("http")
|
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21)
* Compress API responses and revalidate static assets instead of re-downloading
- Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x.
- Serve pages/assets with Cache-Control: no-cache instead of no-store, so
browsers revalidate via the ETag/Last-Modified that FileResponse and
StaticFiles already emit. Unchanged assets now cost an empty 304 rather
than a full transfer on every page navigation, while deploys still show
up immediately.
* Persist derived responses in SQLite so grading is computed once per cell, not per request
New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet
records — finished grade/calendar/day/forecast payloads and reverse-geocode
labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms)
becomes a ~5ms database read that survives restarts and is shared across views.
Raw parquet stays the source of truth; the store is a pure accelerator (every
reader falls back to recomputing on a miss, and deleting the db is a safe reset).
Freshness is token-driven, not clock-driven: each cached payload is validated by
a token encoding what it was computed from (payload schema version, the archive
record's end date, the recent-fetch stamp). The existing freshness drivers are
untouched — get_history still tops up the tail hourly and get_recent_forecast
still refetches hourly — and tokens are derived from what they return, so cached
payloads expire exactly when their inputs change. The same tokens double as weak
ETags: If-None-Match answers with an empty 304 without touching the payload.
- backend/store.py: derived-payload + revgeo tables, thread-local WAL conns,
every helper fail-soft.
- app.py: endpoints split into pure payload builders + HTTP/caching shells; the
in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store).
- climate.py: revgeo persisted through the store; recent_stamp() and
load_cached_history() (no-network read) helpers.
- backend/migrate.py + make migrate: idempotent, resumable backfill of the store
from existing parquet caches (default calendar span + latest-day detail +
revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather.
- grid.from_id(): rebuild a cell from its cache filename (migrate tooling).
* Add /api/v2/cell: one bundle carrying every view's payload
GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months)
and day (today) payloads in one response, each the exact payload its per-view
endpoint returns — built by the same builders and cached under the same
derived-store keys/tokens — paired with the etag that endpoint would emit. The
frontend can warm all views with a single request, seed its per-view cache from
the slices, and later revalidate each view individually with If-None-Match. The
bundle's own etag combines the slices', so an unchanged bundle is an empty 304.
prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard
guarantee: it never spends weather-API quota. A cell with no cached archive
answers 204, and only the history-derived slices (calendar + latest-day detail)
are built. At most one Nominatim lookup for a never-labeled cell.
* Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch
The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota,
which multi-year calendar payloads regularly blew through) to IndexedDB, with an
in-memory map in front. Entries carry the server's ETag, so anything stale
revalidates conditionally — unchanged data costs an empty 304 and a re-stamp,
never a re-transfer. On network failure the stale copy is served over an error.
getJSON gains an optional onUpdate callback opting into stale-while-revalidate:
the three views (weekly, day, calendar) now render a cached copy immediately —
spinners are delayed 150ms so warm loads never flash-blank — and repaint only if
background revalidation finds changed data. New data shows up the moment it
exists instead of waiting out a TTL.
Cross-view prefetch collapses from one request per view to a single /api/v2/cell
bundle, whose slices (exact per-view payloads + their etags) are seeded under the
URLs each view actually requests; the bundle call itself is conditional via a
remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side
with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one
possible Nominatim lookup each), so tapping nearby lands on already-graded data.
Legacy tg:* storage entries are cleared once; cache entries untouched for two
weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
|
|
|
|
async def revalidate_static(request, call_next):
|
|
|
|
|
|
"""Serve the frontend with no-cache (NOT no-store): browsers may keep a copy
|
|
|
|
|
|
but must revalidate it on every use. Starlette's FileResponse/StaticFiles
|
|
|
|
|
|
already emit ETag + Last-Modified, so an unchanged asset costs one conditional
|
|
|
|
|
|
request answered with an empty 304 instead of a full re-download — page-to-page
|
|
|
|
|
|
navigation stops re-transferring the JS/CSS/HTML while still picking up every
|
|
|
|
|
|
deploy immediately (no "my change isn't showing" bugs)."""
|
2026-07-11 00:29:47 +00:00
|
|
|
|
response = await call_next(request)
|
|
|
|
|
|
path = request.url.path
|
Add ops metrics endpoint + SSH/Termux dashboard (#131)
Adds observability for the running app, viewable over SSH (e.g. Termux):
- backend/metrics.py: thread-safe, best-effort in-process counters for inbound
requests (per category) and outbound calls (per external source), plus a
phase->source map and snapshot(). Hooked into climate._request (outbound, all
six weather/geocode sources) and the existing revalidate_static middleware
(inbound). Never raises into the request path.
- GET /api/v2/metrics: token-gated JSON of the counters + warm-queue depth +
thread names/uptime/pid. Returns 404 unless a valid THERMOGRAPH_METRICS_TOKEN
is presented or the request is direct loopback with no proxy X-Forwarded-*
header, so ops data is never exposed through Caddy on the public domain.
- scripts/dashboard.py (+ scripts/dash, make dashboard): stdlib-only terminal
dashboard. Reads cache (parquet + SQLite), accounts (users + active
subscriptions), and system resources (/proc + systemd) directly, and pulls
traffic from the metrics endpoint over loopback. Live refresh, --once, --json;
mobile-terminal width. Paths resolve relative to the script so the same tool
works on the LAN dev and prod VPS checkouts.
- Monitoring sections in DEPLOY.md / DEPLOY-DEV.md; unit tests for the counters.
2026-07-16 20:46:59 +00:00
|
|
|
|
try:
|
Dashboard: accounts totals only; log client IPs per request (#140)
Dashboard ACCOUNTS section now shows only totals (users, active/total subscriptions,
notifications, push subs) — the per-user and per-subscription lists are gone, and the
dashboard no longer queries that PII at all.
Backend: add a per-request access log (logs/access/access-<date>.jsonl) recording the
client IP (real IP via the left-most X-Forwarded-For hop when proxied behind Caddy,
else the peer address), method, path, status, and category. Static assets and the
dashboard's own metrics polling are skipped. Best-effort, never raises into the request
path — retained for later traffic/IP analysis.
2026-07-16 21:58:40 +00:00
|
|
|
|
cat = metrics.classify_inbound(path, BASE)
|
|
|
|
|
|
metrics.record_inbound(cat, response.status_code)
|
|
|
|
|
|
# Retain per-request client IPs for later analysis; skip static assets and the
|
|
|
|
|
|
# dashboard's own metrics polling to keep the log to real, meaningful traffic.
|
Rebuild the homepage as a distribution landing; add an SMTP seam (#178)
The homepage was a bare tool: a find-bar and an empty panel reading "Find a
location to begin." A visitor arriving from a search result or a shared link
learned nothing about what the product does before deciding to leave.
Rebuild it around the Weekly view, which is untouched:
- Hero with the question as the h1, and a grade card showing a real graded
example — the most unusual city we're currently tracking, either tail. The
card's frame and text slots are server-rendered with reserved heights, so
app.js re-pointing it at the visitor's own place shifts nothing.
- "Unusual right now" strip, CSS scroll-snap, no JS carousel. A cold-tail city
is force-included whenever one qualifies.
- Stance line, how-it-works, explore cards, and 12 city chips linking into the
~1000-page /climate surface.
- Monthly digest form, in the footer of every page.
Serve / from Jinja instead of a static file with placeholder substitution, so
crawlers and no-JS readers get the whole page as real HTML. home.html.j2
extends base.html.j2 and carries app.js's DOM contract over verbatim; the brand
degrades to a <p> so the hero owns the sole h1. frontend/index.html is deleted
rather than left behind the static mount, where it would keep serving indexable
duplicate content.
"Where is it most unusual right now" has no cheap answer at request time —
percentiles live inside zlib-compressed payload blobs with no column to sort on.
So homepage.py sweeps the warm cache and writes data/homepage.json, read by the
template. The sweep is strictly cache-only (climate.load_cached_recent_forecast
is new, the sibling of load_cached_history), so grading ~1000 cities costs zero
upstream requests. It rides the notifier's timer behind an hourly guard rather
than starting a second daemon, and also runs at the tail of warm_cities so a
fresh deploy has a populated feed.
Instrumentation: metrics gains a product-event counter keyed by (event,
referrer domain, UTC day), behind an allowlist and a per-IP rate limit, fed by
POST /api/v2/event. The referrer is taken from the request's own header, never
from the client. The beacon gets its own inbound category that record_inbound
ignores, so reporting an interaction doesn't also count as traffic. The
dashboard grows an events block with per-referrer attribution.
Email is scaffolded but sends nothing yet. mailer.py talks stdlib smtplib to a
local Postfix null client on 127.0.0.1:25 (deploy/provision-mail.sh), so the
choice between direct-to-MX and relaying through a provider stays a Postfix
config change with no code change. The backend defaults to "console", which
logs and sends nothing, so dev and tests exercise the whole signup path safely.
Signups land in pending_digest unconfirmed; collecting the list shouldn't wait
on delivery.
Also adds /privacy, linked from the footer and kept out of the sitemap.
The strip's classes are named unusual-* rather than record-*: .record-card is
already the SEO records page's, and reusing it leaked layout rules onto
/climate/<slug>/records.
2026-07-18 07:39:47 +00:00
|
|
|
|
if cat not in ("static", "metrics", "event"):
|
Dashboard: accounts totals only; log client IPs per request (#140)
Dashboard ACCOUNTS section now shows only totals (users, active/total subscriptions,
notifications, push subs) — the per-user and per-subscription lists are gone, and the
dashboard no longer queries that PII at all.
Backend: add a per-request access log (logs/access/access-<date>.jsonl) recording the
client IP (real IP via the left-most X-Forwarded-For hop when proxied behind Caddy,
else the peer address), method, path, status, and category. Static assets and the
dashboard's own metrics polling are skipped. Best-effort, never raises into the request
path — retained for later traffic/IP analysis.
2026-07-16 21:58:40 +00:00
|
|
|
|
audit.log_access({"ip": _client_ip(request), "method": request.method,
|
|
|
|
|
|
"path": path, "status": response.status_code, "cat": cat})
|
Add ops metrics endpoint + SSH/Termux dashboard (#131)
Adds observability for the running app, viewable over SSH (e.g. Termux):
- backend/metrics.py: thread-safe, best-effort in-process counters for inbound
requests (per category) and outbound calls (per external source), plus a
phase->source map and snapshot(). Hooked into climate._request (outbound, all
six weather/geocode sources) and the existing revalidate_static middleware
(inbound). Never raises into the request path.
- GET /api/v2/metrics: token-gated JSON of the counters + warm-queue depth +
thread names/uptime/pid. Returns 404 unless a valid THERMOGRAPH_METRICS_TOKEN
is presented or the request is direct loopback with no proxy X-Forwarded-*
header, so ops data is never exposed through Caddy on the public domain.
- scripts/dashboard.py (+ scripts/dash, make dashboard): stdlib-only terminal
dashboard. Reads cache (parquet + SQLite), accounts (users + active
subscriptions), and system resources (/proc + systemd) directly, and pulls
traffic from the metrics endpoint over loopback. Live refresh, --once, --json;
mobile-terminal width. Paths resolve relative to the script so the same tool
works on the LAN dev and prod VPS checkouts.
- Monitoring sections in DEPLOY.md / DEPLOY-DEV.md; unit tests for the counters.
2026-07-16 20:46:59 +00:00
|
|
|
|
except Exception: # noqa: BLE001 - never let instrumentation break a response
|
|
|
|
|
|
pass
|
Add a climate-score page from recent-vs-baseline percentile divergence (#196)
Score how far a location's last 6 years have drifted from its full 45-year
record. For each metric and percentile category (p10/p25/p50/p75/p90), the
recent-years value is placed on the baseline distribution and the gap from the
expected percentile is the divergence — unit-free, so metrics compare directly.
Scored per meteorological season plus annual, weighted into per-metric and
overall scores (temps, humidity and feels-like weighted heaviest).
- backend/scoring.py: divergence math, seasonal slicing, precip zero-inflation
split (wet-day frequency + amount), tier mapping onto the existing temp scale.
- climate.py: derive a wet-bulb column (Stull 2011) at the read boundary, before
the humidity column is converted to absolute — via a shared _derive_metrics
wrapper at all four read sites.
- api/v2/score endpoint + build_score payload, cached on the history token with
a scoring-version key.
- frontend score page: overall hero, per-metric cards, by-season chips, and a
button-revealed summary (sentences + metrics×season table + per-percentile
detail). Score nav link across all headers.
- Tests for the scoring math, wet-bulb formula, payload shape and route.
2026-07-19 23:02:33 +00:00
|
|
|
|
pages = (BASE, f"{BASE}/", f"{BASE}/calendar", f"{BASE}/day", f"{BASE}/score",
|
|
|
|
|
|
f"{BASE}/compare", f"{BASE}/legend", f"{BASE}/alerts", f"{BASE}/privacy")
|
2026-07-11 00:29:47 +00:00
|
|
|
|
if path.endswith((".js", ".css", ".html")) or path in pages:
|
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21)
* Compress API responses and revalidate static assets instead of re-downloading
- Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x.
- Serve pages/assets with Cache-Control: no-cache instead of no-store, so
browsers revalidate via the ETag/Last-Modified that FileResponse and
StaticFiles already emit. Unchanged assets now cost an empty 304 rather
than a full transfer on every page navigation, while deploys still show
up immediately.
* Persist derived responses in SQLite so grading is computed once per cell, not per request
New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet
records — finished grade/calendar/day/forecast payloads and reverse-geocode
labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms)
becomes a ~5ms database read that survives restarts and is shared across views.
Raw parquet stays the source of truth; the store is a pure accelerator (every
reader falls back to recomputing on a miss, and deleting the db is a safe reset).
Freshness is token-driven, not clock-driven: each cached payload is validated by
a token encoding what it was computed from (payload schema version, the archive
record's end date, the recent-fetch stamp). The existing freshness drivers are
untouched — get_history still tops up the tail hourly and get_recent_forecast
still refetches hourly — and tokens are derived from what they return, so cached
payloads expire exactly when their inputs change. The same tokens double as weak
ETags: If-None-Match answers with an empty 304 without touching the payload.
- backend/store.py: derived-payload + revgeo tables, thread-local WAL conns,
every helper fail-soft.
- app.py: endpoints split into pure payload builders + HTTP/caching shells; the
in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store).
- climate.py: revgeo persisted through the store; recent_stamp() and
load_cached_history() (no-network read) helpers.
- backend/migrate.py + make migrate: idempotent, resumable backfill of the store
from existing parquet caches (default calendar span + latest-day detail +
revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather.
- grid.from_id(): rebuild a cell from its cache filename (migrate tooling).
* Add /api/v2/cell: one bundle carrying every view's payload
GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months)
and day (today) payloads in one response, each the exact payload its per-view
endpoint returns — built by the same builders and cached under the same
derived-store keys/tokens — paired with the etag that endpoint would emit. The
frontend can warm all views with a single request, seed its per-view cache from
the slices, and later revalidate each view individually with If-None-Match. The
bundle's own etag combines the slices', so an unchanged bundle is an empty 304.
prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard
guarantee: it never spends weather-API quota. A cell with no cached archive
answers 204, and only the history-derived slices (calendar + latest-day detail)
are built. At most one Nominatim lookup for a never-labeled cell.
* Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch
The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota,
which multi-year calendar payloads regularly blew through) to IndexedDB, with an
in-memory map in front. Entries carry the server's ETag, so anything stale
revalidates conditionally — unchanged data costs an empty 304 and a re-stamp,
never a re-transfer. On network failure the stale copy is served over an error.
getJSON gains an optional onUpdate callback opting into stale-while-revalidate:
the three views (weekly, day, calendar) now render a cached copy immediately —
spinners are delayed 150ms so warm loads never flash-blank — and repaint only if
background revalidation finds changed data. New data shows up the moment it
exists instead of waiting out a TTL.
Cross-view prefetch collapses from one request per view to a single /api/v2/cell
bundle, whose slices (exact per-view payloads + their etags) are seeded under the
URLs each view actually requests; the bundle call itself is conditional via a
remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side
with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one
possible Nominatim lookup each), so tapping nearby lands on already-graded data.
Legacy tg:* storage entries are cleared once; cache entries untouched for two
weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
|
|
|
|
response.headers["Cache-Control"] = "no-cache"
|
2026-07-11 00:29:47 +00:00
|
|
|
|
return response
|
|
|
|
|
|
|
|
|
|
|
|
|
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21)
* Compress API responses and revalidate static assets instead of re-downloading
- Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x.
- Serve pages/assets with Cache-Control: no-cache instead of no-store, so
browsers revalidate via the ETag/Last-Modified that FileResponse and
StaticFiles already emit. Unchanged assets now cost an empty 304 rather
than a full transfer on every page navigation, while deploys still show
up immediately.
* Persist derived responses in SQLite so grading is computed once per cell, not per request
New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet
records — finished grade/calendar/day/forecast payloads and reverse-geocode
labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms)
becomes a ~5ms database read that survives restarts and is shared across views.
Raw parquet stays the source of truth; the store is a pure accelerator (every
reader falls back to recomputing on a miss, and deleting the db is a safe reset).
Freshness is token-driven, not clock-driven: each cached payload is validated by
a token encoding what it was computed from (payload schema version, the archive
record's end date, the recent-fetch stamp). The existing freshness drivers are
untouched — get_history still tops up the tail hourly and get_recent_forecast
still refetches hourly — and tokens are derived from what they return, so cached
payloads expire exactly when their inputs change. The same tokens double as weak
ETags: If-None-Match answers with an empty 304 without touching the payload.
- backend/store.py: derived-payload + revgeo tables, thread-local WAL conns,
every helper fail-soft.
- app.py: endpoints split into pure payload builders + HTTP/caching shells; the
in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store).
- climate.py: revgeo persisted through the store; recent_stamp() and
load_cached_history() (no-network read) helpers.
- backend/migrate.py + make migrate: idempotent, resumable backfill of the store
from existing parquet caches (default calendar span + latest-day detail +
revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather.
- grid.from_id(): rebuild a cell from its cache filename (migrate tooling).
* Add /api/v2/cell: one bundle carrying every view's payload
GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months)
and day (today) payloads in one response, each the exact payload its per-view
endpoint returns — built by the same builders and cached under the same
derived-store keys/tokens — paired with the etag that endpoint would emit. The
frontend can warm all views with a single request, seed its per-view cache from
the slices, and later revalidate each view individually with If-None-Match. The
bundle's own etag combines the slices', so an unchanged bundle is an empty 304.
prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard
guarantee: it never spends weather-API quota. A cell with no cached archive
answers 204, and only the history-derived slices (calendar + latest-day detail)
are built. At most one Nominatim lookup for a never-labeled cell.
* Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch
The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota,
which multi-year calendar payloads regularly blew through) to IndexedDB, with an
in-memory map in front. Entries carry the server's ETag, so anything stale
revalidates conditionally — unchanged data costs an empty 304 and a re-stamp,
never a re-transfer. On network failure the stale copy is served over an error.
getJSON gains an optional onUpdate callback opting into stale-while-revalidate:
the three views (weekly, day, calendar) now render a cached copy immediately —
spinners are delayed 150ms so warm loads never flash-blank — and repaint only if
background revalidation finds changed data. New data shows up the moment it
exists instead of waiting out a TTL.
Cross-view prefetch collapses from one request per view to a single /api/v2/cell
bundle, whose slices (exact per-view payloads + their etags) are seeded under the
URLs each view actually requests; the bundle call itself is conditional via a
remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side
with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one
possible Nominatim lookup each), so tapping nearby lands on already-graded data.
Legacy tg:* storage entries are cleared once; cache entries untouched for two
weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
|
|
|
|
# --- derived-store plumbing --------------------------------------------------
|
|
|
|
|
|
# Every graded payload is cached in SQLite under (kind, cell, key) and validated
|
|
|
|
|
|
# by a token that encodes what it was computed from (see store.py). The freshness
|
|
|
|
|
|
# drivers stay where they were — climate.get_history tops up the archive tail
|
|
|
|
|
|
# hourly and get_recent_forecast refetches hourly — and the tokens are derived
|
|
|
|
|
|
# from what those return, so a cached payload expires exactly when its inputs
|
|
|
|
|
|
# change and never before. The same tokens double as ETags: a client sending
|
|
|
|
|
|
# If-None-Match gets an empty 304 without the payload even being loaded.
|
Extract payload builders into views.py; load places index at startup (#42)
app.py had grown three co-resident strata: HTTP/caching plumbing, payload
assembly, and search policy. This moves the payload layer — the four
build_* functions, cal_span clamping, hist_end, PAYLOAD_VER, NullRun and
their helpers — into a new views.py with no web dependencies (pure moves,
public names). app.py keeps routing, ETag/derived-store plumbing, and
page serving; migrate.py imports views instead of reaching into app's
privates.
That import previously constructed the whole FastAPI app and — because
places.start_loading() ran at import time — kicked off a background
GeoNames download from an offline batch script. The index load now hangs
off the app's lifespan hook, so it fires when a server starts, not when
the module is imported (tests, migrate).
New tests: cal_span clamping (months-back default, record bounds, 2-year
cap, inversion), builder shapes (grade window ordering, day obs from the
recent bundle, forecast future-only), a regression test that build_day
degrades to climatology when the recent fetch fails, and a subprocess
layering guard that importing views/migrate pulls in neither FastAPI nor
the app module and starts no index download.
2026-07-11 19:43:41 +00:00
|
|
|
|
# The payloads themselves are assembled in views.py, shared with the offline
|
|
|
|
|
|
# migrate script.
|
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21)
* Compress API responses and revalidate static assets instead of re-downloading
- Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x.
- Serve pages/assets with Cache-Control: no-cache instead of no-store, so
browsers revalidate via the ETag/Last-Modified that FileResponse and
StaticFiles already emit. Unchanged assets now cost an empty 304 rather
than a full transfer on every page navigation, while deploys still show
up immediately.
* Persist derived responses in SQLite so grading is computed once per cell, not per request
New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet
records — finished grade/calendar/day/forecast payloads and reverse-geocode
labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms)
becomes a ~5ms database read that survives restarts and is shared across views.
Raw parquet stays the source of truth; the store is a pure accelerator (every
reader falls back to recomputing on a miss, and deleting the db is a safe reset).
Freshness is token-driven, not clock-driven: each cached payload is validated by
a token encoding what it was computed from (payload schema version, the archive
record's end date, the recent-fetch stamp). The existing freshness drivers are
untouched — get_history still tops up the tail hourly and get_recent_forecast
still refetches hourly — and tokens are derived from what they return, so cached
payloads expire exactly when their inputs change. The same tokens double as weak
ETags: If-None-Match answers with an empty 304 without touching the payload.
- backend/store.py: derived-payload + revgeo tables, thread-local WAL conns,
every helper fail-soft.
- app.py: endpoints split into pure payload builders + HTTP/caching shells; the
in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store).
- climate.py: revgeo persisted through the store; recent_stamp() and
load_cached_history() (no-network read) helpers.
- backend/migrate.py + make migrate: idempotent, resumable backfill of the store
from existing parquet caches (default calendar span + latest-day detail +
revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather.
- grid.from_id(): rebuild a cell from its cache filename (migrate tooling).
* Add /api/v2/cell: one bundle carrying every view's payload
GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months)
and day (today) payloads in one response, each the exact payload its per-view
endpoint returns — built by the same builders and cached under the same
derived-store keys/tokens — paired with the etag that endpoint would emit. The
frontend can warm all views with a single request, seed its per-view cache from
the slices, and later revalidate each view individually with If-None-Match. The
bundle's own etag combines the slices', so an unchanged bundle is an empty 304.
prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard
guarantee: it never spends weather-API quota. A cell with no cached archive
answers 204, and only the history-derived slices (calendar + latest-day detail)
are built. At most one Nominatim lookup for a never-labeled cell.
* Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch
The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota,
which multi-year calendar payloads regularly blew through) to IndexedDB, with an
in-memory map in front. Entries carry the server's ETag, so anything stale
revalidates conditionally — unchanged data costs an empty 304 and a re-stamp,
never a re-transfer. On network failure the stale copy is served over an error.
getJSON gains an optional onUpdate callback opting into stale-while-revalidate:
the three views (weekly, day, calendar) now render a cached copy immediately —
spinners are delayed 150ms so warm loads never flash-blank — and repaint only if
background revalidation finds changed data. New data shows up the moment it
exists instead of waiting out a TTL.
Cross-view prefetch collapses from one request per view to a single /api/v2/cell
bundle, whose slices (exact per-view payloads + their etags) are seeded under the
URLs each view actually requests; the bundle call itself is conditional via a
remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side
with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one
possible Nominatim lookup each), so tapping nearby lands on already-graded data.
Legacy tg:* storage entries are cleared once; cache entries untouched for two
weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
|
|
|
|
|
|
|
|
|
|
def _etag_for(kind: str, cell_id: str, key: str, token: str) -> str:
|
|
|
|
|
|
"""Deterministic weak ETag from a payload's identity + validity token. Weak
|
|
|
|
|
|
because the same content may be served under different encodings (gzip)."""
|
|
|
|
|
|
h = hashlib.sha1(f"{kind}:{cell_id}:{key}:{token}".encode()).hexdigest()[:20]
|
|
|
|
|
|
return f'W/"{h}"'
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def _not_modified(request: Request, etag: str) -> bool:
|
|
|
|
|
|
"""Does the client already hold this exact payload? Answerable from the token
|
|
|
|
|
|
alone — no payload load needed."""
|
|
|
|
|
|
inm = request.headers.get("if-none-match")
|
|
|
|
|
|
if not inm:
|
|
|
|
|
|
return False
|
|
|
|
|
|
tags = {t.strip() for t in inm.split(",")}
|
|
|
|
|
|
return "*" in tags or etag in tags or etag.removeprefix("W/") in tags
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def _encode(payload: dict) -> bytes:
|
|
|
|
|
|
"""Compact JSON bytes, matching store.put_payload's encoding (allow_nan=False
|
|
|
|
|
|
mirrors Starlette — a NaN from grading should fail loudly, not reach a client)."""
|
|
|
|
|
|
return json.dumps(payload, default=str, allow_nan=False, separators=(",", ":")).encode()
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def _json_response(body: bytes, etag: str | None = None) -> Response:
|
|
|
|
|
|
headers = {"ETag": etag} if etag else {}
|
|
|
|
|
|
return Response(content=body, media_type="application/json", headers=headers)
|
|
|
|
|
|
|
|
|
|
|
|
|
Single-source cache identity; shared fetch preamble and cache flow (#43)
The derived store's key/token formats existed in three places — each
endpoint, api_cell's slice assembly, and migrate.py — where any drift
would silently split the cache (endpoints missing rows the bundle wrote,
migrate materializing rows nobody reads). They are now defined once in
views.py (grade_key/calendar_key/day_key/forecast_key, history_token/
recent_token/day_token) and consumed everywhere, with a pinning test so
a format change is always deliberate.
The four data endpoints shared two copy-pasted sequences, now helpers:
- _fetch_history: history (+ optional recent bundle) fetch with upstream
failures mapped to clean HTTP errors and an empty record to 404.
- _cached_response: If-None-Match 304 / token-valid store replay /
build + persist + serve, with calendar's dont-persist-placeless rule
as an explicit flag.
Each endpoint is now its audit run + identity + a build callback (~10
lines); api_day's hourly-token special case moved into day_token. New
tests: identity format pins, rate-limit 503 parametrized across all five
data routes, prefetch=1 never touching upstream (cold 204, warm
history-only slices), and calendar's placeless-payload retry behavior.
2026-07-11 19:49:15 +00:00
|
|
|
|
def _fetch_history(run, cell, recent_too=False, recent_phase="recent"):
|
|
|
|
|
|
"""The shared fetch preamble for every data route: the archive record (and
|
|
|
|
|
|
optionally the hourly recent/forecast bundle) with upstream failures mapped
|
|
|
|
|
|
to clean HTTP errors and an empty record to a 404."""
|
|
|
|
|
|
try:
|
|
|
|
|
|
with run.phase("history"):
|
|
|
|
|
|
history, cache_meta = climate.get_history(cell)
|
|
|
|
|
|
recent = None
|
|
|
|
|
|
if recent_too:
|
|
|
|
|
|
with run.phase(recent_phase):
|
|
|
|
|
|
recent = climate.get_recent_forecast(cell)
|
|
|
|
|
|
except Exception as e: # noqa: BLE001
|
|
|
|
|
|
raise _weather_fetch_error(e)
|
Migrate backend dataframe layer from pandas to polars (#90)
* Migrate backend dataframe layer from pandas to polars
Replace pandas with polars across the backend, dropping both pandas and its
pyarrow parquet engine from the dependency set. numpy stays (the grading
percentile math is unchanged).
- climate.py: parquet IO, source→frame mappings, cache read/topup on polars.
New _normalize_read casts the cached `date` column to pl.Date (older files
were written by pandas as datetime64[ns]); frames now unify missing values as
null so the grading boundary drops them consistently across sources.
- grading.py: keep the numpy percentile core; swap the frame→numpy bridge to
.to_numpy()/.drop_nulls(), day-of-year/year to polars dt expressions, and the
per-row loop to iter_rows(named=True).
- views.py: filter/anti-join/concat replace boolean-mask, isin and pd.concat;
scalar dates are stdlib datetime.date; a local _months_before helper replaces
DateOffset(months=) for the calendar-range default.
- app.py, migrate.py: request-date parsing uses datetime.date, removing pandas
from the endpoint and migrate layers entirely.
- The date column is pl.Date end to end, eliminating the pandas normalize() calls
and comparing cleanly against stdlib dates.
Payloads are unchanged: calendar, day, grade and forecast responses are
byte-for-byte identical to the pandas implementation on the same cached record.
Tests ported to polars fixtures, with added coverage for the combined feels-like
fallback, calendar month-offset (month-end/leap), and the concat/dedup
"fresher source wins" rule.
* Port notify.py to polars after merging dev's account system
Merge origin/dev (accounts + notification subscriptions) and carry the pandas→
polars migration into the newly added notify.py, which the merge brought in still
using pandas — with pandas removed from requirements this broke its import.
- notify.py: _candidate_rows filters/sorts the recent bundle with polars
expressions and returns iter_rows dicts; date scalars are datetime.date;
history/recent emptiness via is_empty().
- test_notify.py: synthetic history/rows built with polars + datetime.
2026-07-15 19:07:38 +00:00
|
|
|
|
if history.is_empty():
|
Single-source cache identity; shared fetch preamble and cache flow (#43)
The derived store's key/token formats existed in three places — each
endpoint, api_cell's slice assembly, and migrate.py — where any drift
would silently split the cache (endpoints missing rows the bundle wrote,
migrate materializing rows nobody reads). They are now defined once in
views.py (grade_key/calendar_key/day_key/forecast_key, history_token/
recent_token/day_token) and consumed everywhere, with a pinning test so
a format change is always deliberate.
The four data endpoints shared two copy-pasted sequences, now helpers:
- _fetch_history: history (+ optional recent bundle) fetch with upstream
failures mapped to clean HTTP errors and an empty record to 404.
- _cached_response: If-None-Match 304 / token-valid store replay /
build + persist + serve, with calendar's dont-persist-placeless rule
as an explicit flag.
Each endpoint is now its audit run + identity + a build callback (~10
lines); api_day's hourly-token special case moved into day_token. New
tests: identity format pins, rate-limit 503 parametrized across all five
data routes, prefetch=1 never touching upstream (cold 204, warm
history-only slices), and calendar's placeless-payload retry behavior.
2026-07-11 19:49:15 +00:00
|
|
|
|
raise HTTPException(status_code=404, detail="No historical data for this cell.")
|
|
|
|
|
|
return history, cache_meta, recent
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def _cached_response(request, run, kind, cell, key, token, build, cache_placeless=True):
|
|
|
|
|
|
"""The shared derived-store flow behind every per-view endpoint:
|
|
|
|
|
|
If-None-Match -> empty 304; a token-valid store row -> replay its exact
|
|
|
|
|
|
bytes; otherwise resolve the place label, build the payload, persist, serve.
|
|
|
|
|
|
|
|
|
|
|
|
``build(place) -> payload`` does any endpoint-specific audit tagging itself.
|
|
|
|
|
|
``cache_placeless=False`` skips persisting a payload whose place label
|
|
|
|
|
|
failed to resolve (a transient reverse-geocode miss) — otherwise the
|
|
|
|
|
|
bare-coordinates fallback would stick for the life of the token."""
|
|
|
|
|
|
cid = cell["id"]
|
|
|
|
|
|
etag = _etag_for(kind, cid, key, token)
|
|
|
|
|
|
if _not_modified(request, etag):
|
|
|
|
|
|
run.set(run_type="cache", not_modified=True)
|
|
|
|
|
|
return Response(status_code=304, headers={"ETag": etag})
|
|
|
|
|
|
body = store.get_payload(kind, cid, key, token)
|
|
|
|
|
|
if body is not None:
|
|
|
|
|
|
run.set(run_type="cache", history_source="store")
|
|
|
|
|
|
return _json_response(body, etag)
|
|
|
|
|
|
with run.phase("reverse_geocode"):
|
|
|
|
|
|
place = climate.reverse_geocode(cell["center_lat"], cell["center_lon"])
|
|
|
|
|
|
payload = build(place)
|
|
|
|
|
|
return _json_response(
|
|
|
|
|
|
store.put_payload(kind, cid, key, token, payload,
|
|
|
|
|
|
cache=cache_placeless or place is not None), etag)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21)
* Compress API responses and revalidate static assets instead of re-downloading
- Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x.
- Serve pages/assets with Cache-Control: no-cache instead of no-store, so
browsers revalidate via the ETag/Last-Modified that FileResponse and
StaticFiles already emit. Unchanged assets now cost an empty 304 rather
than a full transfer on every page navigation, while deploys still show
up immediately.
* Persist derived responses in SQLite so grading is computed once per cell, not per request
New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet
records — finished grade/calendar/day/forecast payloads and reverse-geocode
labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms)
becomes a ~5ms database read that survives restarts and is shared across views.
Raw parquet stays the source of truth; the store is a pure accelerator (every
reader falls back to recomputing on a miss, and deleting the db is a safe reset).
Freshness is token-driven, not clock-driven: each cached payload is validated by
a token encoding what it was computed from (payload schema version, the archive
record's end date, the recent-fetch stamp). The existing freshness drivers are
untouched — get_history still tops up the tail hourly and get_recent_forecast
still refetches hourly — and tokens are derived from what they return, so cached
payloads expire exactly when their inputs change. The same tokens double as weak
ETags: If-None-Match answers with an empty 304 without touching the payload.
- backend/store.py: derived-payload + revgeo tables, thread-local WAL conns,
every helper fail-soft.
- app.py: endpoints split into pure payload builders + HTTP/caching shells; the
in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store).
- climate.py: revgeo persisted through the store; recent_stamp() and
load_cached_history() (no-network read) helpers.
- backend/migrate.py + make migrate: idempotent, resumable backfill of the store
from existing parquet caches (default calendar span + latest-day detail +
revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather.
- grid.from_id(): rebuild a cell from its cache filename (migrate tooling).
* Add /api/v2/cell: one bundle carrying every view's payload
GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months)
and day (today) payloads in one response, each the exact payload its per-view
endpoint returns — built by the same builders and cached under the same
derived-store keys/tokens — paired with the etag that endpoint would emit. The
frontend can warm all views with a single request, seed its per-view cache from
the slices, and later revalidate each view individually with If-None-Match. The
bundle's own etag combines the slices', so an unchanged bundle is an empty 304.
prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard
guarantee: it never spends weather-API quota. A cell with no cached archive
answers 204, and only the history-derived slices (calendar + latest-day detail)
are built. At most one Nominatim lookup for a never-labeled cell.
* Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch
The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota,
which multi-year calendar payloads regularly blew through) to IndexedDB, with an
in-memory map in front. Entries carry the server's ETag, so anything stale
revalidates conditionally — unchanged data costs an empty 304 and a re-stamp,
never a re-transfer. On network failure the stale copy is served over an error.
getJSON gains an optional onUpdate callback opting into stale-while-revalidate:
the three views (weekly, day, calendar) now render a cached copy immediately —
spinners are delayed 150ms so warm loads never flash-blank — and repaint only if
background revalidation finds changed data. New data shows up the moment it
exists instead of waiting out a TTL.
Cross-view prefetch collapses from one request per view to a single /api/v2/cell
bundle, whose slices (exact per-view payloads + their etags) are seeded under the
URLs each view actually requests; the bundle call itself is conditional via a
remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side
with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one
possible Nominatim lookup each), so tapping nearby lands on already-graded data.
Legacy tg:* storage entries are cleared once; cache entries untouched for two
weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
|
|
|
|
# --- endpoints ----------------------------------------------------------------
|
|
|
|
|
|
|
2026-07-11 00:29:47 +00:00
|
|
|
|
def api_geocode(q: str = Query(..., min_length=1)):
|
|
|
|
|
|
try:
|
|
|
|
|
|
return {"results": climate.geocode(q)}
|
|
|
|
|
|
except Exception as e: # noqa: BLE001 - surface upstream failures to the client
|
|
|
|
|
|
raise HTTPException(status_code=502, detail=f"geocoding failed: {e}")
|
|
|
|
|
|
|
|
|
|
|
|
|
Typo-tolerant location search suggestions (#33)
Add /api/v2/suggest and wire the location picker's search box to it as a
debounced type-ahead: top-5 place suggestions that tolerate a single-letter
typo (substituted, missing, or extra letter, or two adjacent letters swapped)
anywhere in the query, including the first character.
- backend/places.py: local place index built from a GeoNames cities dump
(downloaded once into data/geonames/, loaded in a background thread; the
app boots and serves without it). Exact-prefix matches rank first by
population, then names one edit away; a token vocabulary respells one
mistyped word against known place-name tokens ("pest seattle" ->
"west seattle") for retry against the upstream geocoder, which covers
neighbourhood-level places the dump lacks. THERMOGRAPH_CITIES picks the
dump (default cities1000).
- /suggest blends local and upstream results by population with an exactness
boost, so "Seatle" (a Cumbrian hamlet) can't outrank Seattle when the
query is one edit from the city, while "munchen" still surfaces Munich via
upstream (the index only knows English names). Upstream lookups are
memoized and skipped entirely when the index answers convincingly.
- mappicker.js: debounced (250ms) live suggestions with abort + sequence
guards against stale responses, arrow-key navigation, Enter-picks-highlight,
Escape dismissing the list before closing the overlay. Submit goes through
the same typo-tolerant endpoint.
- climate.geocode results now carry population (used for ranking).
2026-07-11 15:36:14 +00:00
|
|
|
|
_SUGGEST_LIMIT = 5
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@functools.lru_cache(maxsize=1024)
|
|
|
|
|
|
def _suggest_upstream(q: str) -> tuple:
|
|
|
|
|
|
"""Open-Meteo lookup for /suggest, memoized per query string — type-ahead
|
|
|
|
|
|
re-asks the same prefixes constantly (backspacing, retyping). Failures
|
|
|
|
|
|
raise and are not cached, so a transient upstream error doesn't stick."""
|
|
|
|
|
|
return tuple(climate.geocode(q, count=_SUGGEST_LIMIT))
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def api_suggest(q: str = Query(..., min_length=1)):
|
|
|
|
|
|
"""Type-ahead location suggestions: the top 5 places for a (possibly
|
Move the suggestion policy into places.py (#45)
api_suggest carried ~80 lines of ranking policy — the exactness-boosted
scoring, local/upstream blending with dedupe, and the token-respelling
correction loop — all consuming the match: prefix/fuzzy vocabulary that
places.py produces. Producer and consumer now live together:
places.suggest(q, upstream, limit) implements the whole policy with the
upstream geocoder lookup injected as a callable, so places stays free of
the fetch layer and the policy is testable without FastAPI.
The endpoint is the HTTP shim: call places.suggest, map the
nothing-at-all-to-serve failure to a 502. Behavior unchanged.
Policy tests: upstream skipped when the local answer convinces, blend +
dedupe, the exactness boost (a hamlet spelled like the typo can't beat
Seattle), single-typo queries answered by the fuzzy matcher without a
correction, the correction path verified via the upstream probe,
upstream failure degrading to local results, and the terminal
error-propagation case.
2026-07-11 20:00:24 +00:00
|
|
|
|
typo'd) query prefix; `corrected` reports the respelling that produced the
|
|
|
|
|
|
results ("pest seattle" → "west seattle"), or null. The ranking/typo policy
|
|
|
|
|
|
lives in places.suggest."""
|
|
|
|
|
|
try:
|
|
|
|
|
|
results, corrected = places.suggest(q, _suggest_upstream, _SUGGEST_LIMIT)
|
|
|
|
|
|
except Exception as e: # noqa: BLE001 - nothing at all to serve
|
|
|
|
|
|
raise HTTPException(status_code=502, detail=f"geocoding failed: {e}")
|
|
|
|
|
|
return {"results": results, "corrected": corrected}
|
Typo-tolerant location search suggestions (#33)
Add /api/v2/suggest and wire the location picker's search box to it as a
debounced type-ahead: top-5 place suggestions that tolerate a single-letter
typo (substituted, missing, or extra letter, or two adjacent letters swapped)
anywhere in the query, including the first character.
- backend/places.py: local place index built from a GeoNames cities dump
(downloaded once into data/geonames/, loaded in a background thread; the
app boots and serves without it). Exact-prefix matches rank first by
population, then names one edit away; a token vocabulary respells one
mistyped word against known place-name tokens ("pest seattle" ->
"west seattle") for retry against the upstream geocoder, which covers
neighbourhood-level places the dump lacks. THERMOGRAPH_CITIES picks the
dump (default cities1000).
- /suggest blends local and upstream results by population with an exactness
boost, so "Seatle" (a Cumbrian hamlet) can't outrank Seattle when the
query is one edit from the city, while "munchen" still surfaces Munich via
upstream (the index only knows English names). Upstream lookups are
memoized and skipped entirely when the index answers convincingly.
- mappicker.js: debounced (250ms) live suggestions with abort + sequence
guards against stale responses, arrow-key navigation, Enter-picks-highlight,
Escape dismissing the list before closing the overlay. Submit goes through
the same typo-tolerant endpoint.
- climate.geocode results now carry population (used for ranking).
2026-07-11 15:36:14 +00:00
|
|
|
|
|
|
|
|
|
|
|
2026-07-11 14:50:48 +00:00
|
|
|
|
def api_place(
|
|
|
|
|
|
lat: float = Query(..., ge=-90, le=90),
|
|
|
|
|
|
lon: float = Query(..., ge=-180, le=180),
|
|
|
|
|
|
):
|
|
|
|
|
|
"""Best-effort neighbourhood/city label for a point — the same string the view
|
|
|
|
|
|
endpoints expose as ``place`` (snapped to the grid cell, reverse-geocoded and
|
|
|
|
|
|
cached). Resolved on its own so the compare page can show a location's name as
|
|
|
|
|
|
soon as it's added, before its full series loads."""
|
|
|
|
|
|
cell = grid.snap(lat, lon)
|
|
|
|
|
|
with audit.RunAudit(endpoint="place", lat=round(lat, 4), lon=round(lon, 4),
|
|
|
|
|
|
cell_id=cell["id"]) as run:
|
|
|
|
|
|
with run.phase("reverse_geocode"):
|
|
|
|
|
|
place = climate.reverse_geocode(cell["center_lat"], cell["center_lon"])
|
|
|
|
|
|
return {"place": place,
|
|
|
|
|
|
"cell": {"center_lat": cell["center_lat"], "center_lon": cell["center_lon"]}}
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-07-11 00:29:47 +00:00
|
|
|
|
def api_grade(
|
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21)
* Compress API responses and revalidate static assets instead of re-downloading
- Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x.
- Serve pages/assets with Cache-Control: no-cache instead of no-store, so
browsers revalidate via the ETag/Last-Modified that FileResponse and
StaticFiles already emit. Unchanged assets now cost an empty 304 rather
than a full transfer on every page navigation, while deploys still show
up immediately.
* Persist derived responses in SQLite so grading is computed once per cell, not per request
New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet
records — finished grade/calendar/day/forecast payloads and reverse-geocode
labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms)
becomes a ~5ms database read that survives restarts and is shared across views.
Raw parquet stays the source of truth; the store is a pure accelerator (every
reader falls back to recomputing on a miss, and deleting the db is a safe reset).
Freshness is token-driven, not clock-driven: each cached payload is validated by
a token encoding what it was computed from (payload schema version, the archive
record's end date, the recent-fetch stamp). The existing freshness drivers are
untouched — get_history still tops up the tail hourly and get_recent_forecast
still refetches hourly — and tokens are derived from what they return, so cached
payloads expire exactly when their inputs change. The same tokens double as weak
ETags: If-None-Match answers with an empty 304 without touching the payload.
- backend/store.py: derived-payload + revgeo tables, thread-local WAL conns,
every helper fail-soft.
- app.py: endpoints split into pure payload builders + HTTP/caching shells; the
in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store).
- climate.py: revgeo persisted through the store; recent_stamp() and
load_cached_history() (no-network read) helpers.
- backend/migrate.py + make migrate: idempotent, resumable backfill of the store
from existing parquet caches (default calendar span + latest-day detail +
revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather.
- grid.from_id(): rebuild a cell from its cache filename (migrate tooling).
* Add /api/v2/cell: one bundle carrying every view's payload
GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months)
and day (today) payloads in one response, each the exact payload its per-view
endpoint returns — built by the same builders and cached under the same
derived-store keys/tokens — paired with the etag that endpoint would emit. The
frontend can warm all views with a single request, seed its per-view cache from
the slices, and later revalidate each view individually with If-None-Match. The
bundle's own etag combines the slices', so an unchanged bundle is an empty 304.
prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard
guarantee: it never spends weather-API quota. A cell with no cached archive
answers 204, and only the history-derived slices (calendar + latest-day detail)
are built. At most one Nominatim lookup for a never-labeled cell.
* Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch
The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota,
which multi-year calendar payloads regularly blew through) to IndexedDB, with an
in-memory map in front. Entries carry the server's ETag, so anything stale
revalidates conditionally — unchanged data costs an empty 304 and a re-stamp,
never a re-transfer. On network failure the stale copy is served over an error.
getJSON gains an optional onUpdate callback opting into stale-while-revalidate:
the three views (weekly, day, calendar) now render a cached copy immediately —
spinners are delayed 150ms so warm loads never flash-blank — and repaint only if
background revalidation finds changed data. New data shows up the moment it
exists instead of waiting out a TTL.
Cross-view prefetch collapses from one request per view to a single /api/v2/cell
bundle, whose slices (exact per-view payloads + their etags) are seeded under the
URLs each view actually requests; the bundle call itself is conditional via a
remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side
with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one
possible Nominatim lookup each), so tapping nearby lands on already-graded data.
Legacy tg:* storage entries are cleared once; cache entries untouched for two
weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
|
|
|
|
request: Request,
|
2026-07-11 00:29:47 +00:00
|
|
|
|
lat: float = Query(..., ge=-90, le=90),
|
|
|
|
|
|
lon: float = Query(..., ge=-180, le=180),
|
|
|
|
|
|
date: str | None = Query(None, description="target date YYYY-MM-DD (default today)"),
|
Weekly view: center the daily-graded window on the target day (#28)
* Pin the °F/°C toggle to the header's top-right on all widths
The unit toggle sat inside .brand after a flex-basis:auto title block, so on
narrow (phone) widths the long tagline claimed the first row and bumped the
toggle onto its own line mid-header. Give the title block flex:1 1 0 with
min-width:0 so it contributes ~nothing to the wrap decision and shrinks
instead — keeping the toggle on the logo's row, top-right, with the tagline
wrapping beneath and the view tabs on their own row below.
* Weekly view: center the daily-graded window on the target day
Reframe the "Daily, graded" chart + table on the weekly view around the
selected target day instead of ending at it:
- Show two weeks of history before the target, the target itself, then up
to 7 days after it. Days after the target are real observations when they
are already in the past and the forward forecast when they run into the
future.
- Mark the target day with a solid orange line on the chart (and an accent
edge on its table column); draw the dashed "forecast →" divider only where
the window actually crosses into the future and it is separated from the
target marker.
Backend: _build_grade now grades a [target-days, target+after] window built
from both the archive and the recent+forecast bundle (the bundle wins per
date; the archive fills older gaps), so any past target renders its
surrounding week. api/grade gains an `after` param (default 7) and the cache
key includes it; the /cell prefetch bundle builds the matching slice.
Frontend: the chart consumes the server-built window directly — the separate
forecast fetch and gap-merge are gone. The target is looked up by date for
the comparison cards (the newest row is now a forecast day), and the graded
table opens centered on the target column.
* Weekly view: extend the after-target window to 14 days
Bump the daily-graded window's after-target span from 7 to 14 days. A target
far enough in the past now shows 14 observed days after it; a recent target
shows its observed days plus the 1-7 available forecast days, and the forecast
naturally caps at ~7 days out. The after cap stays at 14.
2026-07-11 14:58:42 +00:00
|
|
|
|
days: int = Query(14, ge=1, le=60, description="days of history to grade before the target"),
|
|
|
|
|
|
after: int = Query(14, ge=0, le=14,
|
|
|
|
|
|
description="days to grade after the target (observed, or forecast when future; "
|
|
|
|
|
|
"the forecast reaches ~7 days out so future days cap there)"),
|
2026-07-11 00:29:47 +00:00
|
|
|
|
):
|
Migrate backend dataframe layer from pandas to polars (#90)
* Migrate backend dataframe layer from pandas to polars
Replace pandas with polars across the backend, dropping both pandas and its
pyarrow parquet engine from the dependency set. numpy stays (the grading
percentile math is unchanged).
- climate.py: parquet IO, source→frame mappings, cache read/topup on polars.
New _normalize_read casts the cached `date` column to pl.Date (older files
were written by pandas as datetime64[ns]); frames now unify missing values as
null so the grading boundary drops them consistently across sources.
- grading.py: keep the numpy percentile core; swap the frame→numpy bridge to
.to_numpy()/.drop_nulls(), day-of-year/year to polars dt expressions, and the
per-row loop to iter_rows(named=True).
- views.py: filter/anti-join/concat replace boolean-mask, isin and pd.concat;
scalar dates are stdlib datetime.date; a local _months_before helper replaces
DateOffset(months=) for the calendar-range default.
- app.py, migrate.py: request-date parsing uses datetime.date, removing pandas
from the endpoint and migrate layers entirely.
- The date column is pl.Date end to end, eliminating the pandas normalize() calls
and comparing cleanly against stdlib dates.
Payloads are unchanged: calendar, day, grade and forecast responses are
byte-for-byte identical to the pandas implementation on the same cached record.
Tests ported to polars fixtures, with added coverage for the combined feels-like
fallback, calendar month-offset (month-end/leap), and the concat/dedup
"fresher source wins" rule.
* Port notify.py to polars after merging dev's account system
Merge origin/dev (accounts + notification subscriptions) and carry the pandas→
polars migration into the newly added notify.py, which the merge brought in still
using pandas — with pandas removed from requirements this broke its import.
- notify.py: _candidate_rows filters/sorts the recent bundle with polars
expressions and returns iter_rows dicts; date scalars are datetime.date;
history/recent emptiness via is_empty().
- test_notify.py: synthetic history/rows built with polars + datetime.
2026-07-15 19:07:38 +00:00
|
|
|
|
target = datetime.date.fromisoformat(date[:10]) if date else datetime.date.today()
|
2026-07-11 00:29:47 +00:00
|
|
|
|
cell = grid.snap(lat, lon)
|
|
|
|
|
|
|
|
|
|
|
|
with audit.RunAudit(
|
|
|
|
|
|
endpoint="grade",
|
|
|
|
|
|
lat=round(lat, 4),
|
|
|
|
|
|
lon=round(lon, 4),
|
Migrate backend dataframe layer from pandas to polars (#90)
* Migrate backend dataframe layer from pandas to polars
Replace pandas with polars across the backend, dropping both pandas and its
pyarrow parquet engine from the dependency set. numpy stays (the grading
percentile math is unchanged).
- climate.py: parquet IO, source→frame mappings, cache read/topup on polars.
New _normalize_read casts the cached `date` column to pl.Date (older files
were written by pandas as datetime64[ns]); frames now unify missing values as
null so the grading boundary drops them consistently across sources.
- grading.py: keep the numpy percentile core; swap the frame→numpy bridge to
.to_numpy()/.drop_nulls(), day-of-year/year to polars dt expressions, and the
per-row loop to iter_rows(named=True).
- views.py: filter/anti-join/concat replace boolean-mask, isin and pd.concat;
scalar dates are stdlib datetime.date; a local _months_before helper replaces
DateOffset(months=) for the calendar-range default.
- app.py, migrate.py: request-date parsing uses datetime.date, removing pandas
from the endpoint and migrate layers entirely.
- The date column is pl.Date end to end, eliminating the pandas normalize() calls
and comparing cleanly against stdlib dates.
Payloads are unchanged: calendar, day, grade and forecast responses are
byte-for-byte identical to the pandas implementation on the same cached record.
Tests ported to polars fixtures, with added coverage for the combined feels-like
fallback, calendar month-offset (month-end/leap), and the concat/dedup
"fresher source wins" rule.
* Port notify.py to polars after merging dev's account system
Merge origin/dev (accounts + notification subscriptions) and carry the pandas→
polars migration into the newly added notify.py, which the merge brought in still
using pandas — with pandas removed from requirements this broke its import.
- notify.py: _candidate_rows filters/sorts the recent bundle with polars
expressions and returns iter_rows dicts; date scalars are datetime.date;
history/recent emptiness via is_empty().
- test_notify.py: synthetic history/rows built with polars + datetime.
2026-07-15 19:07:38 +00:00
|
|
|
|
target_date=target.isoformat(),
|
2026-07-11 00:29:47 +00:00
|
|
|
|
recent_days=days,
|
|
|
|
|
|
cell_id=cell["id"],
|
|
|
|
|
|
) as run:
|
Single-source cache identity; shared fetch preamble and cache flow (#43)
The derived store's key/token formats existed in three places — each
endpoint, api_cell's slice assembly, and migrate.py — where any drift
would silently split the cache (endpoints missing rows the bundle wrote,
migrate materializing rows nobody reads). They are now defined once in
views.py (grade_key/calendar_key/day_key/forecast_key, history_token/
recent_token/day_token) and consumed everywhere, with a pinning test so
a format change is always deliberate.
The four data endpoints shared two copy-pasted sequences, now helpers:
- _fetch_history: history (+ optional recent bundle) fetch with upstream
failures mapped to clean HTTP errors and an empty record to 404.
- _cached_response: If-None-Match 304 / token-valid store replay /
build + persist + serve, with calendar's dont-persist-placeless rule
as an explicit flag.
Each endpoint is now its audit run + identity + a build callback (~10
lines); api_day's hourly-token special case moved into day_token. New
tests: identity format pins, rate-limit 503 parametrized across all five
data routes, prefetch=1 never touching upstream (cold 204, warm
history-only slices), and calendar's placeless-payload retry behavior.
2026-07-11 19:49:15 +00:00
|
|
|
|
history, cache_meta, recent = _fetch_history(run, cell, recent_too=True)
|
|
|
|
|
|
return _cached_response(
|
|
|
|
|
|
request, run, "grade", cell,
|
|
|
|
|
|
views.grade_key(target, days, after), views.recent_token(history, cell["id"]),
|
|
|
|
|
|
lambda place: views.build_grade(cell, target, days, history, recent,
|
|
|
|
|
|
cache_meta, place, run, after=after))
|
2026-07-11 00:29:47 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def api_calendar(
|
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21)
* Compress API responses and revalidate static assets instead of re-downloading
- Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x.
- Serve pages/assets with Cache-Control: no-cache instead of no-store, so
browsers revalidate via the ETag/Last-Modified that FileResponse and
StaticFiles already emit. Unchanged assets now cost an empty 304 rather
than a full transfer on every page navigation, while deploys still show
up immediately.
* Persist derived responses in SQLite so grading is computed once per cell, not per request
New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet
records — finished grade/calendar/day/forecast payloads and reverse-geocode
labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms)
becomes a ~5ms database read that survives restarts and is shared across views.
Raw parquet stays the source of truth; the store is a pure accelerator (every
reader falls back to recomputing on a miss, and deleting the db is a safe reset).
Freshness is token-driven, not clock-driven: each cached payload is validated by
a token encoding what it was computed from (payload schema version, the archive
record's end date, the recent-fetch stamp). The existing freshness drivers are
untouched — get_history still tops up the tail hourly and get_recent_forecast
still refetches hourly — and tokens are derived from what they return, so cached
payloads expire exactly when their inputs change. The same tokens double as weak
ETags: If-None-Match answers with an empty 304 without touching the payload.
- backend/store.py: derived-payload + revgeo tables, thread-local WAL conns,
every helper fail-soft.
- app.py: endpoints split into pure payload builders + HTTP/caching shells; the
in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store).
- climate.py: revgeo persisted through the store; recent_stamp() and
load_cached_history() (no-network read) helpers.
- backend/migrate.py + make migrate: idempotent, resumable backfill of the store
from existing parquet caches (default calendar span + latest-day detail +
revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather.
- grid.from_id(): rebuild a cell from its cache filename (migrate tooling).
* Add /api/v2/cell: one bundle carrying every view's payload
GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months)
and day (today) payloads in one response, each the exact payload its per-view
endpoint returns — built by the same builders and cached under the same
derived-store keys/tokens — paired with the etag that endpoint would emit. The
frontend can warm all views with a single request, seed its per-view cache from
the slices, and later revalidate each view individually with If-None-Match. The
bundle's own etag combines the slices', so an unchanged bundle is an empty 304.
prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard
guarantee: it never spends weather-API quota. A cell with no cached archive
answers 204, and only the history-derived slices (calendar + latest-day detail)
are built. At most one Nominatim lookup for a never-labeled cell.
* Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch
The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota,
which multi-year calendar payloads regularly blew through) to IndexedDB, with an
in-memory map in front. Entries carry the server's ETag, so anything stale
revalidates conditionally — unchanged data costs an empty 304 and a re-stamp,
never a re-transfer. On network failure the stale copy is served over an error.
getJSON gains an optional onUpdate callback opting into stale-while-revalidate:
the three views (weekly, day, calendar) now render a cached copy immediately —
spinners are delayed 150ms so warm loads never flash-blank — and repaint only if
background revalidation finds changed data. New data shows up the moment it
exists instead of waiting out a TTL.
Cross-view prefetch collapses from one request per view to a single /api/v2/cell
bundle, whose slices (exact per-view payloads + their etags) are seeded under the
URLs each view actually requests; the bundle call itself is conditional via a
remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side
with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one
possible Nominatim lookup each), so tapping nearby lands on already-graded data.
Legacy tg:* storage entries are cleared once; cache entries untouched for two
weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
|
|
|
|
request: Request,
|
2026-07-11 00:29:47 +00:00
|
|
|
|
lat: float = Query(..., ge=-90, le=90),
|
|
|
|
|
|
lon: float = Query(..., ge=-180, le=180),
|
|
|
|
|
|
start: str | None = Query(None, description="first date YYYY-MM-DD (overrides months)"),
|
|
|
|
|
|
end: str | None = Query(None, description="last date YYYY-MM-DD (default = latest available)"),
|
|
|
|
|
|
months: int = Query(24, ge=1, le=24, description="months back to grade when no start given"),
|
|
|
|
|
|
):
|
|
|
|
|
|
"""Grade every day over a date range (default: last 2 years) for the calendar view.
|
|
|
|
|
|
|
|
|
|
|
|
A custom [start, end] range is supported and capped at ~2 years per request;
|
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21)
* Compress API responses and revalidate static assets instead of re-downloading
- Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x.
- Serve pages/assets with Cache-Control: no-cache instead of no-store, so
browsers revalidate via the ETag/Last-Modified that FileResponse and
StaticFiles already emit. Unchanged assets now cost an empty 304 rather
than a full transfer on every page navigation, while deploys still show
up immediately.
* Persist derived responses in SQLite so grading is computed once per cell, not per request
New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet
records — finished grade/calendar/day/forecast payloads and reverse-geocode
labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms)
becomes a ~5ms database read that survives restarts and is shared across views.
Raw parquet stays the source of truth; the store is a pure accelerator (every
reader falls back to recomputing on a miss, and deleting the db is a safe reset).
Freshness is token-driven, not clock-driven: each cached payload is validated by
a token encoding what it was computed from (payload schema version, the archive
record's end date, the recent-fetch stamp). The existing freshness drivers are
untouched — get_history still tops up the tail hourly and get_recent_forecast
still refetches hourly — and tokens are derived from what they return, so cached
payloads expire exactly when their inputs change. The same tokens double as weak
ETags: If-None-Match answers with an empty 304 without touching the payload.
- backend/store.py: derived-payload + revgeo tables, thread-local WAL conns,
every helper fail-soft.
- app.py: endpoints split into pure payload builders + HTTP/caching shells; the
in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store).
- climate.py: revgeo persisted through the store; recent_stamp() and
load_cached_history() (no-network read) helpers.
- backend/migrate.py + make migrate: idempotent, resumable backfill of the store
from existing parquet caches (default calendar span + latest-day detail +
revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather.
- grid.from_id(): rebuild a cell from its cache filename (migrate tooling).
* Add /api/v2/cell: one bundle carrying every view's payload
GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months)
and day (today) payloads in one response, each the exact payload its per-view
endpoint returns — built by the same builders and cached under the same
derived-store keys/tokens — paired with the etag that endpoint would emit. The
frontend can warm all views with a single request, seed its per-view cache from
the slices, and later revalidate each view individually with If-None-Match. The
bundle's own etag combines the slices', so an unchanged bundle is an empty 304.
prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard
guarantee: it never spends weather-API quota. A cell with no cached archive
answers 204, and only the history-derived slices (calendar + latest-day detail)
are built. At most one Nominatim lookup for a never-labeled cell.
* Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch
The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota,
which multi-year calendar payloads regularly blew through) to IndexedDB, with an
in-memory map in front. Entries carry the server's ETag, so anything stale
revalidates conditionally — unchanged data costs an empty 304 and a re-stamp,
never a re-transfer. On network failure the stale copy is served over an error.
getJSON gains an optional onUpdate callback opting into stale-while-revalidate:
the three views (weekly, day, calendar) now render a cached copy immediately —
spinners are delayed 150ms so warm loads never flash-blank — and repaint only if
background revalidation finds changed data. New data shows up the moment it
exists instead of waiting out a TTL.
Cross-view prefetch collapses from one request per view to a single /api/v2/cell
bundle, whose slices (exact per-view payloads + their etags) are seeded under the
URLs each view actually requests; the bundle call itself is conditional via a
remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side
with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one
possible Nominatim lookup each), so tapping nearby lands on already-graded data.
Legacy tg:* storage entries are cleared once; cache entries untouched for two
weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
|
|
|
|
the frontend splits longer spans into successive 2-year chunks. Grades against
|
|
|
|
|
|
the already-cached history record (which reaches to ~6 days ago), so no extra
|
|
|
|
|
|
fetch is needed. The graded payload is cached in the derived store keyed by
|
|
|
|
|
|
the clamped span and validated by the record's end date, so a repeat span is
|
|
|
|
|
|
a database read — across restarts — until new archive days arrive. Returns a
|
|
|
|
|
|
compact per-day shape (see grading.grade_range). New in API v2.
|
2026-07-11 00:29:47 +00:00
|
|
|
|
"""
|
|
|
|
|
|
cell = grid.snap(lat, lon)
|
|
|
|
|
|
|
|
|
|
|
|
with audit.RunAudit(
|
|
|
|
|
|
endpoint="calendar",
|
|
|
|
|
|
lat=round(lat, 4),
|
|
|
|
|
|
lon=round(lon, 4),
|
|
|
|
|
|
recent_days=months * 31, # approximate span graded
|
|
|
|
|
|
cell_id=cell["id"],
|
|
|
|
|
|
) as run:
|
Single-source cache identity; shared fetch preamble and cache flow (#43)
The derived store's key/token formats existed in three places — each
endpoint, api_cell's slice assembly, and migrate.py — where any drift
would silently split the cache (endpoints missing rows the bundle wrote,
migrate materializing rows nobody reads). They are now defined once in
views.py (grade_key/calendar_key/day_key/forecast_key, history_token/
recent_token/day_token) and consumed everywhere, with a pinning test so
a format change is always deliberate.
The four data endpoints shared two copy-pasted sequences, now helpers:
- _fetch_history: history (+ optional recent bundle) fetch with upstream
failures mapped to clean HTTP errors and an empty record to 404.
- _cached_response: If-None-Match 304 / token-valid store replay /
build + persist + serve, with calendar's dont-persist-placeless rule
as an explicit flag.
Each endpoint is now its audit run + identity + a build callback (~10
lines); api_day's hourly-token special case moved into day_token. New
tests: identity format pins, rate-limit 503 parametrized across all five
data routes, prefetch=1 never touching upstream (cold 204, warm
history-only slices), and calendar's placeless-payload retry behavior.
2026-07-11 19:49:15 +00:00
|
|
|
|
history, cache_meta, _ = _fetch_history(run, cell)
|
Extract payload builders into views.py; load places index at startup (#42)
app.py had grown three co-resident strata: HTTP/caching plumbing, payload
assembly, and search policy. This moves the payload layer — the four
build_* functions, cal_span clamping, hist_end, PAYLOAD_VER, NullRun and
their helpers — into a new views.py with no web dependencies (pure moves,
public names). app.py keeps routing, ETag/derived-store plumbing, and
page serving; migrate.py imports views instead of reaching into app's
privates.
That import previously constructed the whole FastAPI app and — because
places.start_loading() ran at import time — kicked off a background
GeoNames download from an offline batch script. The index load now hangs
off the app's lifespan hook, so it fires when a server starts, not when
the module is imported (tests, migrate).
New tests: cal_span clamping (months-back default, record bounds, 2-year
cap, inversion), builder shapes (grade window ordering, day obs from the
recent bundle, forecast future-only), a regression test that build_day
degrades to climatology when the recent fetch fails, and a subprocess
layering guard that importing views/migrate pulls in neither FastAPI nor
the app module and starts no index download.
2026-07-11 19:43:41 +00:00
|
|
|
|
start_ts, end_ts = views.cal_span(history, start, end, months)
|
2026-07-11 00:29:47 +00:00
|
|
|
|
|
Single-source cache identity; shared fetch preamble and cache flow (#43)
The derived store's key/token formats existed in three places — each
endpoint, api_cell's slice assembly, and migrate.py — where any drift
would silently split the cache (endpoints missing rows the bundle wrote,
migrate materializing rows nobody reads). They are now defined once in
views.py (grade_key/calendar_key/day_key/forecast_key, history_token/
recent_token/day_token) and consumed everywhere, with a pinning test so
a format change is always deliberate.
The four data endpoints shared two copy-pasted sequences, now helpers:
- _fetch_history: history (+ optional recent bundle) fetch with upstream
failures mapped to clean HTTP errors and an empty record to 404.
- _cached_response: If-None-Match 304 / token-valid store replay /
build + persist + serve, with calendar's dont-persist-placeless rule
as an explicit flag.
Each endpoint is now its audit run + identity + a build callback (~10
lines); api_day's hourly-token special case moved into day_token. New
tests: identity format pins, rate-limit 503 parametrized across all five
data routes, prefetch=1 never touching upstream (cold 204, warm
history-only slices), and calendar's placeless-payload retry behavior.
2026-07-11 19:49:15 +00:00
|
|
|
|
def build(place):
|
|
|
|
|
|
payload = views.build_calendar(cell, history, start_ts, end_ts, months, place, run)
|
|
|
|
|
|
full = not cache_meta.get("cached", False)
|
|
|
|
|
|
run.set(run_type="full" if full else "partial",
|
|
|
|
|
|
history_source="fetch" if full else "cache")
|
|
|
|
|
|
return payload
|
|
|
|
|
|
|
|
|
|
|
|
# Compare loads several cells at once, so a transient reverse-geocode miss
|
|
|
|
|
|
# is most likely here; cache_placeless=False keeps that miss un-persisted.
|
|
|
|
|
|
return _cached_response(request, run, "calendar", cell,
|
|
|
|
|
|
views.calendar_key(start_ts, end_ts, months),
|
|
|
|
|
|
views.history_token(history), build,
|
|
|
|
|
|
cache_placeless=False)
|
2026-07-11 00:29:47 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def api_day(
|
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21)
* Compress API responses and revalidate static assets instead of re-downloading
- Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x.
- Serve pages/assets with Cache-Control: no-cache instead of no-store, so
browsers revalidate via the ETag/Last-Modified that FileResponse and
StaticFiles already emit. Unchanged assets now cost an empty 304 rather
than a full transfer on every page navigation, while deploys still show
up immediately.
* Persist derived responses in SQLite so grading is computed once per cell, not per request
New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet
records — finished grade/calendar/day/forecast payloads and reverse-geocode
labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms)
becomes a ~5ms database read that survives restarts and is shared across views.
Raw parquet stays the source of truth; the store is a pure accelerator (every
reader falls back to recomputing on a miss, and deleting the db is a safe reset).
Freshness is token-driven, not clock-driven: each cached payload is validated by
a token encoding what it was computed from (payload schema version, the archive
record's end date, the recent-fetch stamp). The existing freshness drivers are
untouched — get_history still tops up the tail hourly and get_recent_forecast
still refetches hourly — and tokens are derived from what they return, so cached
payloads expire exactly when their inputs change. The same tokens double as weak
ETags: If-None-Match answers with an empty 304 without touching the payload.
- backend/store.py: derived-payload + revgeo tables, thread-local WAL conns,
every helper fail-soft.
- app.py: endpoints split into pure payload builders + HTTP/caching shells; the
in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store).
- climate.py: revgeo persisted through the store; recent_stamp() and
load_cached_history() (no-network read) helpers.
- backend/migrate.py + make migrate: idempotent, resumable backfill of the store
from existing parquet caches (default calendar span + latest-day detail +
revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather.
- grid.from_id(): rebuild a cell from its cache filename (migrate tooling).
* Add /api/v2/cell: one bundle carrying every view's payload
GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months)
and day (today) payloads in one response, each the exact payload its per-view
endpoint returns — built by the same builders and cached under the same
derived-store keys/tokens — paired with the etag that endpoint would emit. The
frontend can warm all views with a single request, seed its per-view cache from
the slices, and later revalidate each view individually with If-None-Match. The
bundle's own etag combines the slices', so an unchanged bundle is an empty 304.
prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard
guarantee: it never spends weather-API quota. A cell with no cached archive
answers 204, and only the history-derived slices (calendar + latest-day detail)
are built. At most one Nominatim lookup for a never-labeled cell.
* Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch
The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota,
which multi-year calendar payloads regularly blew through) to IndexedDB, with an
in-memory map in front. Entries carry the server's ETag, so anything stale
revalidates conditionally — unchanged data costs an empty 304 and a re-stamp,
never a re-transfer. On network failure the stale copy is served over an error.
getJSON gains an optional onUpdate callback opting into stale-while-revalidate:
the three views (weekly, day, calendar) now render a cached copy immediately —
spinners are delayed 150ms so warm loads never flash-blank — and repaint only if
background revalidation finds changed data. New data shows up the moment it
exists instead of waiting out a TTL.
Cross-view prefetch collapses from one request per view to a single /api/v2/cell
bundle, whose slices (exact per-view payloads + their etags) are seeded under the
URLs each view actually requests; the bundle call itself is conditional via a
remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side
with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one
possible Nominatim lookup each), so tapping nearby lands on already-graded data.
Legacy tg:* storage entries are cleared once; cache entries untouched for two
weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
|
|
|
|
request: Request,
|
2026-07-11 00:29:47 +00:00
|
|
|
|
lat: float = Query(..., ge=-90, le=90),
|
|
|
|
|
|
lon: float = Query(..., ge=-180, le=180),
|
|
|
|
|
|
date: str | None = Query(None, description="target date YYYY-MM-DD (default = latest available)"),
|
|
|
|
|
|
):
|
|
|
|
|
|
"""Full percentile breakdown for a single day: the value at every tier boundary
|
|
|
|
|
|
in that day-of-year's ±7-day window, plus where the observed values land.
|
|
|
|
|
|
|
|
|
|
|
|
Powers the single-day detail page. Grades against the cached history record;
|
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21)
* Compress API responses and revalidate static assets instead of re-downloading
- Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x.
- Serve pages/assets with Cache-Control: no-cache instead of no-store, so
browsers revalidate via the ETag/Last-Modified that FileResponse and
StaticFiles already emit. Unchanged assets now cost an empty 304 rather
than a full transfer on every page navigation, while deploys still show
up immediately.
* Persist derived responses in SQLite so grading is computed once per cell, not per request
New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet
records — finished grade/calendar/day/forecast payloads and reverse-geocode
labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms)
becomes a ~5ms database read that survives restarts and is shared across views.
Raw parquet stays the source of truth; the store is a pure accelerator (every
reader falls back to recomputing on a miss, and deleting the db is a safe reset).
Freshness is token-driven, not clock-driven: each cached payload is validated by
a token encoding what it was computed from (payload schema version, the archive
record's end date, the recent-fetch stamp). The existing freshness drivers are
untouched — get_history still tops up the tail hourly and get_recent_forecast
still refetches hourly — and tokens are derived from what they return, so cached
payloads expire exactly when their inputs change. The same tokens double as weak
ETags: If-None-Match answers with an empty 304 without touching the payload.
- backend/store.py: derived-payload + revgeo tables, thread-local WAL conns,
every helper fail-soft.
- app.py: endpoints split into pure payload builders + HTTP/caching shells; the
in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store).
- climate.py: revgeo persisted through the store; recent_stamp() and
load_cached_history() (no-network read) helpers.
- backend/migrate.py + make migrate: idempotent, resumable backfill of the store
from existing parquet caches (default calendar span + latest-day detail +
revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather.
- grid.from_id(): rebuild a cell from its cache filename (migrate tooling).
* Add /api/v2/cell: one bundle carrying every view's payload
GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months)
and day (today) payloads in one response, each the exact payload its per-view
endpoint returns — built by the same builders and cached under the same
derived-store keys/tokens — paired with the etag that endpoint would emit. The
frontend can warm all views with a single request, seed its per-view cache from
the slices, and later revalidate each view individually with If-None-Match. The
bundle's own etag combines the slices', so an unchanged bundle is an empty 304.
prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard
guarantee: it never spends weather-API quota. A cell with no cached archive
answers 204, and only the history-derived slices (calendar + latest-day detail)
are built. At most one Nominatim lookup for a never-labeled cell.
* Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch
The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota,
which multi-year calendar payloads regularly blew through) to IndexedDB, with an
in-memory map in front. Entries carry the server's ETag, so anything stale
revalidates conditionally — unchanged data costs an empty 304 and a re-stamp,
never a re-transfer. On network failure the stale copy is served over an error.
getJSON gains an optional onUpdate callback opting into stale-while-revalidate:
the three views (weekly, day, calendar) now render a cached copy immediately —
spinners are delayed 150ms so warm loads never flash-blank — and repaint only if
background revalidation finds changed data. New data shows up the moment it
exists instead of waiting out a TTL.
Cross-view prefetch collapses from one request per view to a single /api/v2/cell
bundle, whose slices (exact per-view payloads + their etags) are seeded under the
URLs each view actually requests; the bundle call itself is conditional via a
remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side
with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one
possible Nominatim lookup each), so tapping nearby lands on already-graded data.
Legacy tg:* storage entries are cleared once; cache entries untouched for two
weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
|
|
|
|
for a date newer than the cache it pulls the recent window for the observation
|
|
|
|
|
|
(those payloads expire hourly — the recent bundle's own cadence — while fully
|
|
|
|
|
|
archived days stay valid until the record itself advances). New in API v2.
|
2026-07-11 00:29:47 +00:00
|
|
|
|
"""
|
|
|
|
|
|
cell = grid.snap(lat, lon)
|
|
|
|
|
|
|
|
|
|
|
|
with audit.RunAudit(
|
|
|
|
|
|
endpoint="day",
|
|
|
|
|
|
lat=round(lat, 4),
|
|
|
|
|
|
lon=round(lon, 4),
|
|
|
|
|
|
cell_id=cell["id"],
|
|
|
|
|
|
) as run:
|
Single-source cache identity; shared fetch preamble and cache flow (#43)
The derived store's key/token formats existed in three places — each
endpoint, api_cell's slice assembly, and migrate.py — where any drift
would silently split the cache (endpoints missing rows the bundle wrote,
migrate materializing rows nobody reads). They are now defined once in
views.py (grade_key/calendar_key/day_key/forecast_key, history_token/
recent_token/day_token) and consumed everywhere, with a pinning test so
a format change is always deliberate.
The four data endpoints shared two copy-pasted sequences, now helpers:
- _fetch_history: history (+ optional recent bundle) fetch with upstream
failures mapped to clean HTTP errors and an empty record to 404.
- _cached_response: If-None-Match 304 / token-valid store replay /
build + persist + serve, with calendar's dont-persist-placeless rule
as an explicit flag.
Each endpoint is now its audit run + identity + a build callback (~10
lines); api_day's hourly-token special case moved into day_token. New
tests: identity format pins, rate-limit 503 parametrized across all five
data routes, prefetch=1 never touching upstream (cold 204, warm
history-only slices), and calendar's placeless-payload retry behavior.
2026-07-11 19:49:15 +00:00
|
|
|
|
history, cache_meta, _ = _fetch_history(run, cell)
|
Migrate backend dataframe layer from pandas to polars (#90)
* Migrate backend dataframe layer from pandas to polars
Replace pandas with polars across the backend, dropping both pandas and its
pyarrow parquet engine from the dependency set. numpy stays (the grading
percentile math is unchanged).
- climate.py: parquet IO, source→frame mappings, cache read/topup on polars.
New _normalize_read casts the cached `date` column to pl.Date (older files
were written by pandas as datetime64[ns]); frames now unify missing values as
null so the grading boundary drops them consistently across sources.
- grading.py: keep the numpy percentile core; swap the frame→numpy bridge to
.to_numpy()/.drop_nulls(), day-of-year/year to polars dt expressions, and the
per-row loop to iter_rows(named=True).
- views.py: filter/anti-join/concat replace boolean-mask, isin and pd.concat;
scalar dates are stdlib datetime.date; a local _months_before helper replaces
DateOffset(months=) for the calendar-range default.
- app.py, migrate.py: request-date parsing uses datetime.date, removing pandas
from the endpoint and migrate layers entirely.
- The date column is pl.Date end to end, eliminating the pandas normalize() calls
and comparing cleanly against stdlib dates.
Payloads are unchanged: calendar, day, grade and forecast responses are
byte-for-byte identical to the pandas implementation on the same cached record.
Tests ported to polars fixtures, with added coverage for the combined feels-like
fallback, calendar month-offset (month-end/leap), and the concat/dedup
"fresher source wins" rule.
* Port notify.py to polars after merging dev's account system
Merge origin/dev (accounts + notification subscriptions) and carry the pandas→
polars migration into the newly added notify.py, which the merge brought in still
using pandas — with pandas removed from requirements this broke its import.
- notify.py: _candidate_rows filters/sorts the recent bundle with polars
expressions and returns iter_rows dicts; date scalars are datetime.date;
history/recent emptiness via is_empty().
- test_notify.py: synthetic history/rows built with polars + datetime.
2026-07-15 19:07:38 +00:00
|
|
|
|
last = history["date"].max()
|
|
|
|
|
|
target = datetime.date.fromisoformat(date[:10]) if date else last
|
|
|
|
|
|
run.set(target_date=target.isoformat())
|
2026-07-11 00:29:47 +00:00
|
|
|
|
|
Single-source cache identity; shared fetch preamble and cache flow (#43)
The derived store's key/token formats existed in three places — each
endpoint, api_cell's slice assembly, and migrate.py — where any drift
would silently split the cache (endpoints missing rows the bundle wrote,
migrate materializing rows nobody reads). They are now defined once in
views.py (grade_key/calendar_key/day_key/forecast_key, history_token/
recent_token/day_token) and consumed everywhere, with a pinning test so
a format change is always deliberate.
The four data endpoints shared two copy-pasted sequences, now helpers:
- _fetch_history: history (+ optional recent bundle) fetch with upstream
failures mapped to clean HTTP errors and an empty record to 404.
- _cached_response: If-None-Match 304 / token-valid store replay /
build + persist + serve, with calendar's dont-persist-placeless rule
as an explicit flag.
Each endpoint is now its audit run + identity + a build callback (~10
lines); api_day's hourly-token special case moved into day_token. New
tests: identity format pins, rate-limit 503 parametrized across all five
data routes, prefetch=1 never touching upstream (cold 204, warm
history-only slices), and calendar's placeless-payload retry behavior.
2026-07-11 19:49:15 +00:00
|
|
|
|
def build(place):
|
|
|
|
|
|
payload = views.build_day(cell, history, target, place, run)
|
|
|
|
|
|
full = not cache_meta.get("cached", False)
|
|
|
|
|
|
run.set(run_type="full" if full else "partial",
|
|
|
|
|
|
history_source="fetch" if full else "cache")
|
|
|
|
|
|
return payload
|
2026-07-11 00:29:47 +00:00
|
|
|
|
|
Single-source cache identity; shared fetch preamble and cache flow (#43)
The derived store's key/token formats existed in three places — each
endpoint, api_cell's slice assembly, and migrate.py — where any drift
would silently split the cache (endpoints missing rows the bundle wrote,
migrate materializing rows nobody reads). They are now defined once in
views.py (grade_key/calendar_key/day_key/forecast_key, history_token/
recent_token/day_token) and consumed everywhere, with a pinning test so
a format change is always deliberate.
The four data endpoints shared two copy-pasted sequences, now helpers:
- _fetch_history: history (+ optional recent bundle) fetch with upstream
failures mapped to clean HTTP errors and an empty record to 404.
- _cached_response: If-None-Match 304 / token-valid store replay /
build + persist + serve, with calendar's dont-persist-placeless rule
as an explicit flag.
Each endpoint is now its audit run + identity + a build callback (~10
lines); api_day's hourly-token special case moved into day_token. New
tests: identity format pins, rate-limit 503 parametrized across all five
data routes, prefetch=1 never touching upstream (cold 204, warm
history-only slices), and calendar's placeless-payload retry behavior.
2026-07-11 19:49:15 +00:00
|
|
|
|
return _cached_response(request, run, "day", cell,
|
|
|
|
|
|
views.day_key(target), views.day_token(history, target),
|
|
|
|
|
|
build)
|
2026-07-11 00:29:47 +00:00
|
|
|
|
|
|
|
|
|
|
|
Add a climate-score page from recent-vs-baseline percentile divergence (#196)
Score how far a location's last 6 years have drifted from its full 45-year
record. For each metric and percentile category (p10/p25/p50/p75/p90), the
recent-years value is placed on the baseline distribution and the gap from the
expected percentile is the divergence — unit-free, so metrics compare directly.
Scored per meteorological season plus annual, weighted into per-metric and
overall scores (temps, humidity and feels-like weighted heaviest).
- backend/scoring.py: divergence math, seasonal slicing, precip zero-inflation
split (wet-day frequency + amount), tier mapping onto the existing temp scale.
- climate.py: derive a wet-bulb column (Stull 2011) at the read boundary, before
the humidity column is converted to absolute — via a shared _derive_metrics
wrapper at all four read sites.
- api/v2/score endpoint + build_score payload, cached on the history token with
a scoring-version key.
- frontend score page: overall hero, per-metric cards, by-season chips, and a
button-revealed summary (sentences + metrics×season table + per-percentile
detail). Score nav link across all headers.
- Tests for the scoring math, wet-bulb formula, payload shape and route.
2026-07-19 23:02:33 +00:00
|
|
|
|
def api_score(
|
|
|
|
|
|
request: Request,
|
|
|
|
|
|
lat: float = Query(..., ge=-90, le=90),
|
|
|
|
|
|
lon: float = Query(..., ge=-180, le=180),
|
|
|
|
|
|
):
|
|
|
|
|
|
"""Climate-drift score for a location: how far the last 6 years have moved
|
|
|
|
|
|
from the full 45-year baseline, per metric, percentile category and season,
|
|
|
|
|
|
rolled into per-metric and overall scores.
|
|
|
|
|
|
|
|
|
|
|
|
Derived purely from the archive record, so it is cached until the record's
|
|
|
|
|
|
tail advances (the hourly top-up); the scoring-math version is baked into the
|
|
|
|
|
|
cache key so a math change invalidates only score rows. New in API v2.
|
|
|
|
|
|
"""
|
|
|
|
|
|
cell = grid.snap(lat, lon)
|
|
|
|
|
|
|
|
|
|
|
|
with audit.RunAudit(endpoint="score", lat=round(lat, 4), lon=round(lon, 4),
|
|
|
|
|
|
cell_id=cell["id"]) as run:
|
|
|
|
|
|
history, cache_meta, _ = _fetch_history(run, cell)
|
|
|
|
|
|
|
|
|
|
|
|
def build(place):
|
|
|
|
|
|
payload = views.build_score(cell, history, place, run)
|
|
|
|
|
|
full = not cache_meta.get("cached", False)
|
|
|
|
|
|
run.set(run_type="full" if full else "partial",
|
|
|
|
|
|
history_source="fetch" if full else "cache")
|
|
|
|
|
|
return payload
|
|
|
|
|
|
|
|
|
|
|
|
return _cached_response(request, run, "score", cell,
|
|
|
|
|
|
views.score_key(), views.history_token(history), build)
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-07-11 00:29:47 +00:00
|
|
|
|
def api_forecast(
|
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21)
* Compress API responses and revalidate static assets instead of re-downloading
- Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x.
- Serve pages/assets with Cache-Control: no-cache instead of no-store, so
browsers revalidate via the ETag/Last-Modified that FileResponse and
StaticFiles already emit. Unchanged assets now cost an empty 304 rather
than a full transfer on every page navigation, while deploys still show
up immediately.
* Persist derived responses in SQLite so grading is computed once per cell, not per request
New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet
records — finished grade/calendar/day/forecast payloads and reverse-geocode
labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms)
becomes a ~5ms database read that survives restarts and is shared across views.
Raw parquet stays the source of truth; the store is a pure accelerator (every
reader falls back to recomputing on a miss, and deleting the db is a safe reset).
Freshness is token-driven, not clock-driven: each cached payload is validated by
a token encoding what it was computed from (payload schema version, the archive
record's end date, the recent-fetch stamp). The existing freshness drivers are
untouched — get_history still tops up the tail hourly and get_recent_forecast
still refetches hourly — and tokens are derived from what they return, so cached
payloads expire exactly when their inputs change. The same tokens double as weak
ETags: If-None-Match answers with an empty 304 without touching the payload.
- backend/store.py: derived-payload + revgeo tables, thread-local WAL conns,
every helper fail-soft.
- app.py: endpoints split into pure payload builders + HTTP/caching shells; the
in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store).
- climate.py: revgeo persisted through the store; recent_stamp() and
load_cached_history() (no-network read) helpers.
- backend/migrate.py + make migrate: idempotent, resumable backfill of the store
from existing parquet caches (default calendar span + latest-day detail +
revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather.
- grid.from_id(): rebuild a cell from its cache filename (migrate tooling).
* Add /api/v2/cell: one bundle carrying every view's payload
GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months)
and day (today) payloads in one response, each the exact payload its per-view
endpoint returns — built by the same builders and cached under the same
derived-store keys/tokens — paired with the etag that endpoint would emit. The
frontend can warm all views with a single request, seed its per-view cache from
the slices, and later revalidate each view individually with If-None-Match. The
bundle's own etag combines the slices', so an unchanged bundle is an empty 304.
prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard
guarantee: it never spends weather-API quota. A cell with no cached archive
answers 204, and only the history-derived slices (calendar + latest-day detail)
are built. At most one Nominatim lookup for a never-labeled cell.
* Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch
The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota,
which multi-year calendar payloads regularly blew through) to IndexedDB, with an
in-memory map in front. Entries carry the server's ETag, so anything stale
revalidates conditionally — unchanged data costs an empty 304 and a re-stamp,
never a re-transfer. On network failure the stale copy is served over an error.
getJSON gains an optional onUpdate callback opting into stale-while-revalidate:
the three views (weekly, day, calendar) now render a cached copy immediately —
spinners are delayed 150ms so warm loads never flash-blank — and repaint only if
background revalidation finds changed data. New data shows up the moment it
exists instead of waiting out a TTL.
Cross-view prefetch collapses from one request per view to a single /api/v2/cell
bundle, whose slices (exact per-view payloads + their etags) are seeded under the
URLs each view actually requests; the bundle call itself is conditional via a
remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side
with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one
possible Nominatim lookup each), so tapping nearby lands on already-graded data.
Legacy tg:* storage entries are cleared once; cache entries untouched for two
weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
|
|
|
|
request: Request,
|
2026-07-11 00:29:47 +00:00
|
|
|
|
lat: float = Query(..., ge=-90, le=90),
|
|
|
|
|
|
lon: float = Query(..., ge=-180, le=180),
|
|
|
|
|
|
days: int = Query(7, ge=1, le=14, description="how many forecast days ahead to grade"),
|
|
|
|
|
|
):
|
|
|
|
|
|
"""Grade the next `days` of forecast weather against local climatology.
|
|
|
|
|
|
|
|
|
|
|
|
Same shape as /grade (so the frontend renders it identically), but `recent`
|
|
|
|
|
|
holds the forward forecast — furthest-out day first. The forecast is fetched
|
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21)
* Compress API responses and revalidate static assets instead of re-downloading
- Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x.
- Serve pages/assets with Cache-Control: no-cache instead of no-store, so
browsers revalidate via the ETag/Last-Modified that FileResponse and
StaticFiles already emit. Unchanged assets now cost an empty 304 rather
than a full transfer on every page navigation, while deploys still show
up immediately.
* Persist derived responses in SQLite so grading is computed once per cell, not per request
New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet
records — finished grade/calendar/day/forecast payloads and reverse-geocode
labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms)
becomes a ~5ms database read that survives restarts and is shared across views.
Raw parquet stays the source of truth; the store is a pure accelerator (every
reader falls back to recomputing on a miss, and deleting the db is a safe reset).
Freshness is token-driven, not clock-driven: each cached payload is validated by
a token encoding what it was computed from (payload schema version, the archive
record's end date, the recent-fetch stamp). The existing freshness drivers are
untouched — get_history still tops up the tail hourly and get_recent_forecast
still refetches hourly — and tokens are derived from what they return, so cached
payloads expire exactly when their inputs change. The same tokens double as weak
ETags: If-None-Match answers with an empty 304 without touching the payload.
- backend/store.py: derived-payload + revgeo tables, thread-local WAL conns,
every helper fail-soft.
- app.py: endpoints split into pure payload builders + HTTP/caching shells; the
in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store).
- climate.py: revgeo persisted through the store; recent_stamp() and
load_cached_history() (no-network read) helpers.
- backend/migrate.py + make migrate: idempotent, resumable backfill of the store
from existing parquet caches (default calendar span + latest-day detail +
revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather.
- grid.from_id(): rebuild a cell from its cache filename (migrate tooling).
* Add /api/v2/cell: one bundle carrying every view's payload
GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months)
and day (today) payloads in one response, each the exact payload its per-view
endpoint returns — built by the same builders and cached under the same
derived-store keys/tokens — paired with the etag that endpoint would emit. The
frontend can warm all views with a single request, seed its per-view cache from
the slices, and later revalidate each view individually with If-None-Match. The
bundle's own etag combines the slices', so an unchanged bundle is an empty 304.
prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard
guarantee: it never spends weather-API quota. A cell with no cached archive
answers 204, and only the history-derived slices (calendar + latest-day detail)
are built. At most one Nominatim lookup for a never-labeled cell.
* Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch
The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota,
which multi-year calendar payloads regularly blew through) to IndexedDB, with an
in-memory map in front. Entries carry the server's ETag, so anything stale
revalidates conditionally — unchanged data costs an empty 304 and a re-stamp,
never a re-transfer. On network failure the stale copy is served over an error.
getJSON gains an optional onUpdate callback opting into stale-while-revalidate:
the three views (weekly, day, calendar) now render a cached copy immediately —
spinners are delayed 150ms so warm loads never flash-blank — and repaint only if
background revalidation finds changed data. New data shows up the moment it
exists instead of waiting out a TTL.
Cross-view prefetch collapses from one request per view to a single /api/v2/cell
bundle, whose slices (exact per-view payloads + their etags) are seeded under the
URLs each view actually requests; the bundle call itself is conditional via a
remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side
with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one
possible Nominatim lookup each), so tapping nearby lands on already-graded data.
Legacy tg:* storage entries are cleared once; cache entries untouched for two
weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
|
|
|
|
fresh and cached only ~1 hour (see climate.get_recent_forecast) to track
|
|
|
|
|
|
updates; the graded payload's validity is tied to that fetch stamp, so it
|
|
|
|
|
|
expires exactly when a new forecast lands. New in API v2.
|
2026-07-11 00:29:47 +00:00
|
|
|
|
"""
|
|
|
|
|
|
cell = grid.snap(lat, lon)
|
|
|
|
|
|
|
|
|
|
|
|
with audit.RunAudit(endpoint="forecast", lat=round(lat, 4), lon=round(lon, 4),
|
|
|
|
|
|
recent_days=days, cell_id=cell["id"]) as run:
|
Single-source cache identity; shared fetch preamble and cache flow (#43)
The derived store's key/token formats existed in three places — each
endpoint, api_cell's slice assembly, and migrate.py — where any drift
would silently split the cache (endpoints missing rows the bundle wrote,
migrate materializing rows nobody reads). They are now defined once in
views.py (grade_key/calendar_key/day_key/forecast_key, history_token/
recent_token/day_token) and consumed everywhere, with a pinning test so
a format change is always deliberate.
The four data endpoints shared two copy-pasted sequences, now helpers:
- _fetch_history: history (+ optional recent bundle) fetch with upstream
failures mapped to clean HTTP errors and an empty record to 404.
- _cached_response: If-None-Match 304 / token-valid store replay /
build + persist + serve, with calendar's dont-persist-placeless rule
as an explicit flag.
Each endpoint is now its audit run + identity + a build callback (~10
lines); api_day's hourly-token special case moved into day_token. New
tests: identity format pins, rate-limit 503 parametrized across all five
data routes, prefetch=1 never touching upstream (cold 204, warm
history-only slices), and calendar's placeless-payload retry behavior.
2026-07-11 19:49:15 +00:00
|
|
|
|
history, _, fc = _fetch_history(run, cell, recent_too=True, recent_phase="forecast")
|
Migrate backend dataframe layer from pandas to polars (#90)
* Migrate backend dataframe layer from pandas to polars
Replace pandas with polars across the backend, dropping both pandas and its
pyarrow parquet engine from the dependency set. numpy stays (the grading
percentile math is unchanged).
- climate.py: parquet IO, source→frame mappings, cache read/topup on polars.
New _normalize_read casts the cached `date` column to pl.Date (older files
were written by pandas as datetime64[ns]); frames now unify missing values as
null so the grading boundary drops them consistently across sources.
- grading.py: keep the numpy percentile core; swap the frame→numpy bridge to
.to_numpy()/.drop_nulls(), day-of-year/year to polars dt expressions, and the
per-row loop to iter_rows(named=True).
- views.py: filter/anti-join/concat replace boolean-mask, isin and pd.concat;
scalar dates are stdlib datetime.date; a local _months_before helper replaces
DateOffset(months=) for the calendar-range default.
- app.py, migrate.py: request-date parsing uses datetime.date, removing pandas
from the endpoint and migrate layers entirely.
- The date column is pl.Date end to end, eliminating the pandas normalize() calls
and comparing cleanly against stdlib dates.
Payloads are unchanged: calendar, day, grade and forecast responses are
byte-for-byte identical to the pandas implementation on the same cached record.
Tests ported to polars fixtures, with added coverage for the combined feels-like
fallback, calendar month-offset (month-end/leap), and the concat/dedup
"fresher source wins" rule.
* Port notify.py to polars after merging dev's account system
Merge origin/dev (accounts + notification subscriptions) and carry the pandas→
polars migration into the newly added notify.py, which the merge brought in still
using pandas — with pandas removed from requirements this broke its import.
- notify.py: _candidate_rows filters/sorts the recent bundle with polars
expressions and returns iter_rows dicts; date scalars are datetime.date;
history/recent emptiness via is_empty().
- test_notify.py: synthetic history/rows built with polars + datetime.
2026-07-15 19:07:38 +00:00
|
|
|
|
today = datetime.date.today()
|
2026-07-11 00:29:47 +00:00
|
|
|
|
|
Single-source cache identity; shared fetch preamble and cache flow (#43)
The derived store's key/token formats existed in three places — each
endpoint, api_cell's slice assembly, and migrate.py — where any drift
would silently split the cache (endpoints missing rows the bundle wrote,
migrate materializing rows nobody reads). They are now defined once in
views.py (grade_key/calendar_key/day_key/forecast_key, history_token/
recent_token/day_token) and consumed everywhere, with a pinning test so
a format change is always deliberate.
The four data endpoints shared two copy-pasted sequences, now helpers:
- _fetch_history: history (+ optional recent bundle) fetch with upstream
failures mapped to clean HTTP errors and an empty record to 404.
- _cached_response: If-None-Match 304 / token-valid store replay /
build + persist + serve, with calendar's dont-persist-placeless rule
as an explicit flag.
Each endpoint is now its audit run + identity + a build callback (~10
lines); api_day's hourly-token special case moved into day_token. New
tests: identity format pins, rate-limit 503 parametrized across all five
data routes, prefetch=1 never touching upstream (cold 204, warm
history-only slices), and calendar's placeless-payload retry behavior.
2026-07-11 19:49:15 +00:00
|
|
|
|
def build(place):
|
|
|
|
|
|
payload = views.build_forecast(cell, days, history, fc, today, place, run)
|
|
|
|
|
|
run.set(run_type="partial")
|
|
|
|
|
|
return payload
|
|
|
|
|
|
|
|
|
|
|
|
return _cached_response(request, run, "forecast", cell,
|
|
|
|
|
|
views.forecast_key(today, days),
|
|
|
|
|
|
views.recent_token(history, cell["id"]), build)
|
2026-07-11 00:29:47 +00:00
|
|
|
|
|
|
|
|
|
|
|
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21)
* Compress API responses and revalidate static assets instead of re-downloading
- Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x.
- Serve pages/assets with Cache-Control: no-cache instead of no-store, so
browsers revalidate via the ETag/Last-Modified that FileResponse and
StaticFiles already emit. Unchanged assets now cost an empty 304 rather
than a full transfer on every page navigation, while deploys still show
up immediately.
* Persist derived responses in SQLite so grading is computed once per cell, not per request
New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet
records — finished grade/calendar/day/forecast payloads and reverse-geocode
labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms)
becomes a ~5ms database read that survives restarts and is shared across views.
Raw parquet stays the source of truth; the store is a pure accelerator (every
reader falls back to recomputing on a miss, and deleting the db is a safe reset).
Freshness is token-driven, not clock-driven: each cached payload is validated by
a token encoding what it was computed from (payload schema version, the archive
record's end date, the recent-fetch stamp). The existing freshness drivers are
untouched — get_history still tops up the tail hourly and get_recent_forecast
still refetches hourly — and tokens are derived from what they return, so cached
payloads expire exactly when their inputs change. The same tokens double as weak
ETags: If-None-Match answers with an empty 304 without touching the payload.
- backend/store.py: derived-payload + revgeo tables, thread-local WAL conns,
every helper fail-soft.
- app.py: endpoints split into pure payload builders + HTTP/caching shells; the
in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store).
- climate.py: revgeo persisted through the store; recent_stamp() and
load_cached_history() (no-network read) helpers.
- backend/migrate.py + make migrate: idempotent, resumable backfill of the store
from existing parquet caches (default calendar span + latest-day detail +
revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather.
- grid.from_id(): rebuild a cell from its cache filename (migrate tooling).
* Add /api/v2/cell: one bundle carrying every view's payload
GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months)
and day (today) payloads in one response, each the exact payload its per-view
endpoint returns — built by the same builders and cached under the same
derived-store keys/tokens — paired with the etag that endpoint would emit. The
frontend can warm all views with a single request, seed its per-view cache from
the slices, and later revalidate each view individually with If-None-Match. The
bundle's own etag combines the slices', so an unchanged bundle is an empty 304.
prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard
guarantee: it never spends weather-API quota. A cell with no cached archive
answers 204, and only the history-derived slices (calendar + latest-day detail)
are built. At most one Nominatim lookup for a never-labeled cell.
* Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch
The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota,
which multi-year calendar payloads regularly blew through) to IndexedDB, with an
in-memory map in front. Entries carry the server's ETag, so anything stale
revalidates conditionally — unchanged data costs an empty 304 and a re-stamp,
never a re-transfer. On network failure the stale copy is served over an error.
getJSON gains an optional onUpdate callback opting into stale-while-revalidate:
the three views (weekly, day, calendar) now render a cached copy immediately —
spinners are delayed 150ms so warm loads never flash-blank — and repaint only if
background revalidation finds changed data. New data shows up the moment it
exists instead of waiting out a TTL.
Cross-view prefetch collapses from one request per view to a single /api/v2/cell
bundle, whose slices (exact per-view payloads + their etags) are seeded under the
URLs each view actually requests; the bundle call itself is conditional via a
remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side
with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one
possible Nominatim lookup each), so tapping nearby lands on already-graded data.
Legacy tg:* storage entries are cleared once; cache entries untouched for two
weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
|
|
|
|
def api_cell(
|
|
|
|
|
|
request: Request,
|
|
|
|
|
|
lat: float = Query(..., ge=-90, le=90),
|
|
|
|
|
|
lon: float = Query(..., ge=-180, le=180),
|
|
|
|
|
|
prefetch: int = Query(0, ge=0, le=1,
|
|
|
|
|
|
description="1 = warm-only: never fetch weather upstream; 204 for a cold cell"),
|
2026-07-11 20:47:31 +00:00
|
|
|
|
neighbors: int = Query(0, ge=0, le=1,
|
|
|
|
|
|
description="1 = also warm the 8 surrounding cells in the background "
|
|
|
|
|
|
"(warm-only: cold neighbors are skipped, no upstream quota)"),
|
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21)
* Compress API responses and revalidate static assets instead of re-downloading
- Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x.
- Serve pages/assets with Cache-Control: no-cache instead of no-store, so
browsers revalidate via the ETag/Last-Modified that FileResponse and
StaticFiles already emit. Unchanged assets now cost an empty 304 rather
than a full transfer on every page navigation, while deploys still show
up immediately.
* Persist derived responses in SQLite so grading is computed once per cell, not per request
New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet
records — finished grade/calendar/day/forecast payloads and reverse-geocode
labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms)
becomes a ~5ms database read that survives restarts and is shared across views.
Raw parquet stays the source of truth; the store is a pure accelerator (every
reader falls back to recomputing on a miss, and deleting the db is a safe reset).
Freshness is token-driven, not clock-driven: each cached payload is validated by
a token encoding what it was computed from (payload schema version, the archive
record's end date, the recent-fetch stamp). The existing freshness drivers are
untouched — get_history still tops up the tail hourly and get_recent_forecast
still refetches hourly — and tokens are derived from what they return, so cached
payloads expire exactly when their inputs change. The same tokens double as weak
ETags: If-None-Match answers with an empty 304 without touching the payload.
- backend/store.py: derived-payload + revgeo tables, thread-local WAL conns,
every helper fail-soft.
- app.py: endpoints split into pure payload builders + HTTP/caching shells; the
in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store).
- climate.py: revgeo persisted through the store; recent_stamp() and
load_cached_history() (no-network read) helpers.
- backend/migrate.py + make migrate: idempotent, resumable backfill of the store
from existing parquet caches (default calendar span + latest-day detail +
revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather.
- grid.from_id(): rebuild a cell from its cache filename (migrate tooling).
* Add /api/v2/cell: one bundle carrying every view's payload
GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months)
and day (today) payloads in one response, each the exact payload its per-view
endpoint returns — built by the same builders and cached under the same
derived-store keys/tokens — paired with the etag that endpoint would emit. The
frontend can warm all views with a single request, seed its per-view cache from
the slices, and later revalidate each view individually with If-None-Match. The
bundle's own etag combines the slices', so an unchanged bundle is an empty 304.
prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard
guarantee: it never spends weather-API quota. A cell with no cached archive
answers 204, and only the history-derived slices (calendar + latest-day detail)
are built. At most one Nominatim lookup for a never-labeled cell.
* Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch
The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota,
which multi-year calendar payloads regularly blew through) to IndexedDB, with an
in-memory map in front. Entries carry the server's ETag, so anything stale
revalidates conditionally — unchanged data costs an empty 304 and a re-stamp,
never a re-transfer. On network failure the stale copy is served over an error.
getJSON gains an optional onUpdate callback opting into stale-while-revalidate:
the three views (weekly, day, calendar) now render a cached copy immediately —
spinners are delayed 150ms so warm loads never flash-blank — and repaint only if
background revalidation finds changed data. New data shows up the moment it
exists instead of waiting out a TTL.
Cross-view prefetch collapses from one request per view to a single /api/v2/cell
bundle, whose slices (exact per-view payloads + their etags) are seeded under the
URLs each view actually requests; the bundle call itself is conditional via a
remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side
with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one
possible Nominatim lookup each), so tapping nearby lands on already-graded data.
Legacy tg:* storage entries are cleared once; cache entries untouched for two
weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
|
|
|
|
):
|
|
|
|
|
|
"""One bundle carrying every view's payload for a cell, so the frontend warms
|
|
|
|
|
|
all views with a single request instead of four.
|
|
|
|
|
|
|
|
|
|
|
|
Each slice is the EXACT payload its per-view endpoint returns — built by the
|
|
|
|
|
|
same builders and cached under the same derived-store keys/tokens — paired
|
|
|
|
|
|
with the etag that endpoint would emit. The client seeds its per-view cache
|
|
|
|
|
|
from the slices and later revalidates each view individually with
|
|
|
|
|
|
If-None-Match, so the bundle and the per-view endpoints stay one cache.
|
|
|
|
|
|
|
|
|
|
|
|
prefetch=1 is the neighbor-warming mode with a hard guarantee: it never
|
|
|
|
|
|
spends weather-API quota. A cell with no cached archive answers 204 (no
|
|
|
|
|
|
body), and only the history-derived slices (calendar + latest-day detail)
|
|
|
|
|
|
are built — grading recent/forecast days needs the hourly upstream bundle.
|
|
|
|
|
|
Reverse geocoding makes at most one Nominatim call for a never-labeled cell;
|
|
|
|
|
|
the client staggers neighbor prefetches to respect that service. New in API v2.
|
|
|
|
|
|
"""
|
|
|
|
|
|
cell = grid.snap(lat, lon)
|
Migrate backend dataframe layer from pandas to polars (#90)
* Migrate backend dataframe layer from pandas to polars
Replace pandas with polars across the backend, dropping both pandas and its
pyarrow parquet engine from the dependency set. numpy stays (the grading
percentile math is unchanged).
- climate.py: parquet IO, source→frame mappings, cache read/topup on polars.
New _normalize_read casts the cached `date` column to pl.Date (older files
were written by pandas as datetime64[ns]); frames now unify missing values as
null so the grading boundary drops them consistently across sources.
- grading.py: keep the numpy percentile core; swap the frame→numpy bridge to
.to_numpy()/.drop_nulls(), day-of-year/year to polars dt expressions, and the
per-row loop to iter_rows(named=True).
- views.py: filter/anti-join/concat replace boolean-mask, isin and pd.concat;
scalar dates are stdlib datetime.date; a local _months_before helper replaces
DateOffset(months=) for the calendar-range default.
- app.py, migrate.py: request-date parsing uses datetime.date, removing pandas
from the endpoint and migrate layers entirely.
- The date column is pl.Date end to end, eliminating the pandas normalize() calls
and comparing cleanly against stdlib dates.
Payloads are unchanged: calendar, day, grade and forecast responses are
byte-for-byte identical to the pandas implementation on the same cached record.
Tests ported to polars fixtures, with added coverage for the combined feels-like
fallback, calendar month-offset (month-end/leap), and the concat/dedup
"fresher source wins" rule.
* Port notify.py to polars after merging dev's account system
Merge origin/dev (accounts + notification subscriptions) and carry the pandas→
polars migration into the newly added notify.py, which the merge brought in still
using pandas — with pandas removed from requirements this broke its import.
- notify.py: _candidate_rows filters/sorts the recent bundle with polars
expressions and returns iter_rows dicts; date scalars are datetime.date;
history/recent emptiness via is_empty().
- test_notify.py: synthetic history/rows built with polars + datetime.
2026-07-15 19:07:38 +00:00
|
|
|
|
today = datetime.date.today()
|
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21)
* Compress API responses and revalidate static assets instead of re-downloading
- Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x.
- Serve pages/assets with Cache-Control: no-cache instead of no-store, so
browsers revalidate via the ETag/Last-Modified that FileResponse and
StaticFiles already emit. Unchanged assets now cost an empty 304 rather
than a full transfer on every page navigation, while deploys still show
up immediately.
* Persist derived responses in SQLite so grading is computed once per cell, not per request
New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet
records — finished grade/calendar/day/forecast payloads and reverse-geocode
labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms)
becomes a ~5ms database read that survives restarts and is shared across views.
Raw parquet stays the source of truth; the store is a pure accelerator (every
reader falls back to recomputing on a miss, and deleting the db is a safe reset).
Freshness is token-driven, not clock-driven: each cached payload is validated by
a token encoding what it was computed from (payload schema version, the archive
record's end date, the recent-fetch stamp). The existing freshness drivers are
untouched — get_history still tops up the tail hourly and get_recent_forecast
still refetches hourly — and tokens are derived from what they return, so cached
payloads expire exactly when their inputs change. The same tokens double as weak
ETags: If-None-Match answers with an empty 304 without touching the payload.
- backend/store.py: derived-payload + revgeo tables, thread-local WAL conns,
every helper fail-soft.
- app.py: endpoints split into pure payload builders + HTTP/caching shells; the
in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store).
- climate.py: revgeo persisted through the store; recent_stamp() and
load_cached_history() (no-network read) helpers.
- backend/migrate.py + make migrate: idempotent, resumable backfill of the store
from existing parquet caches (default calendar span + latest-day detail +
revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather.
- grid.from_id(): rebuild a cell from its cache filename (migrate tooling).
* Add /api/v2/cell: one bundle carrying every view's payload
GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months)
and day (today) payloads in one response, each the exact payload its per-view
endpoint returns — built by the same builders and cached under the same
derived-store keys/tokens — paired with the etag that endpoint would emit. The
frontend can warm all views with a single request, seed its per-view cache from
the slices, and later revalidate each view individually with If-None-Match. The
bundle's own etag combines the slices', so an unchanged bundle is an empty 304.
prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard
guarantee: it never spends weather-API quota. A cell with no cached archive
answers 204, and only the history-derived slices (calendar + latest-day detail)
are built. At most one Nominatim lookup for a never-labeled cell.
* Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch
The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota,
which multi-year calendar payloads regularly blew through) to IndexedDB, with an
in-memory map in front. Entries carry the server's ETag, so anything stale
revalidates conditionally — unchanged data costs an empty 304 and a re-stamp,
never a re-transfer. On network failure the stale copy is served over an error.
getJSON gains an optional onUpdate callback opting into stale-while-revalidate:
the three views (weekly, day, calendar) now render a cached copy immediately —
spinners are delayed 150ms so warm loads never flash-blank — and repaint only if
background revalidation finds changed data. New data shows up the moment it
exists instead of waiting out a TTL.
Cross-view prefetch collapses from one request per view to a single /api/v2/cell
bundle, whose slices (exact per-view payloads + their etags) are seeded under the
URLs each view actually requests; the bundle call itself is conditional via a
remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side
with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one
possible Nominatim lookup each), so tapping nearby lands on already-graded data.
Legacy tg:* storage entries are cleared once; cache entries untouched for two
weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
|
|
|
|
|
|
|
|
|
|
with audit.RunAudit(endpoint="cell", lat=round(lat, 4), lon=round(lon, 4),
|
|
|
|
|
|
cell_id=cell["id"], prefetch=bool(prefetch)) as run:
|
|
|
|
|
|
recent = None
|
|
|
|
|
|
if prefetch:
|
|
|
|
|
|
history = climate.load_cached_history(cell)
|
Migrate backend dataframe layer from pandas to polars (#90)
* Migrate backend dataframe layer from pandas to polars
Replace pandas with polars across the backend, dropping both pandas and its
pyarrow parquet engine from the dependency set. numpy stays (the grading
percentile math is unchanged).
- climate.py: parquet IO, source→frame mappings, cache read/topup on polars.
New _normalize_read casts the cached `date` column to pl.Date (older files
were written by pandas as datetime64[ns]); frames now unify missing values as
null so the grading boundary drops them consistently across sources.
- grading.py: keep the numpy percentile core; swap the frame→numpy bridge to
.to_numpy()/.drop_nulls(), day-of-year/year to polars dt expressions, and the
per-row loop to iter_rows(named=True).
- views.py: filter/anti-join/concat replace boolean-mask, isin and pd.concat;
scalar dates are stdlib datetime.date; a local _months_before helper replaces
DateOffset(months=) for the calendar-range default.
- app.py, migrate.py: request-date parsing uses datetime.date, removing pandas
from the endpoint and migrate layers entirely.
- The date column is pl.Date end to end, eliminating the pandas normalize() calls
and comparing cleanly against stdlib dates.
Payloads are unchanged: calendar, day, grade and forecast responses are
byte-for-byte identical to the pandas implementation on the same cached record.
Tests ported to polars fixtures, with added coverage for the combined feels-like
fallback, calendar month-offset (month-end/leap), and the concat/dedup
"fresher source wins" rule.
* Port notify.py to polars after merging dev's account system
Merge origin/dev (accounts + notification subscriptions) and carry the pandas→
polars migration into the newly added notify.py, which the merge brought in still
using pandas — with pandas removed from requirements this broke its import.
- notify.py: _candidate_rows filters/sorts the recent bundle with polars
expressions and returns iter_rows dicts; date scalars are datetime.date;
history/recent emptiness via is_empty().
- test_notify.py: synthetic history/rows built with polars + datetime.
2026-07-15 19:07:38 +00:00
|
|
|
|
if history is None or history.is_empty():
|
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21)
* Compress API responses and revalidate static assets instead of re-downloading
- Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x.
- Serve pages/assets with Cache-Control: no-cache instead of no-store, so
browsers revalidate via the ETag/Last-Modified that FileResponse and
StaticFiles already emit. Unchanged assets now cost an empty 304 rather
than a full transfer on every page navigation, while deploys still show
up immediately.
* Persist derived responses in SQLite so grading is computed once per cell, not per request
New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet
records — finished grade/calendar/day/forecast payloads and reverse-geocode
labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms)
becomes a ~5ms database read that survives restarts and is shared across views.
Raw parquet stays the source of truth; the store is a pure accelerator (every
reader falls back to recomputing on a miss, and deleting the db is a safe reset).
Freshness is token-driven, not clock-driven: each cached payload is validated by
a token encoding what it was computed from (payload schema version, the archive
record's end date, the recent-fetch stamp). The existing freshness drivers are
untouched — get_history still tops up the tail hourly and get_recent_forecast
still refetches hourly — and tokens are derived from what they return, so cached
payloads expire exactly when their inputs change. The same tokens double as weak
ETags: If-None-Match answers with an empty 304 without touching the payload.
- backend/store.py: derived-payload + revgeo tables, thread-local WAL conns,
every helper fail-soft.
- app.py: endpoints split into pure payload builders + HTTP/caching shells; the
in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store).
- climate.py: revgeo persisted through the store; recent_stamp() and
load_cached_history() (no-network read) helpers.
- backend/migrate.py + make migrate: idempotent, resumable backfill of the store
from existing parquet caches (default calendar span + latest-day detail +
revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather.
- grid.from_id(): rebuild a cell from its cache filename (migrate tooling).
* Add /api/v2/cell: one bundle carrying every view's payload
GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months)
and day (today) payloads in one response, each the exact payload its per-view
endpoint returns — built by the same builders and cached under the same
derived-store keys/tokens — paired with the etag that endpoint would emit. The
frontend can warm all views with a single request, seed its per-view cache from
the slices, and later revalidate each view individually with If-None-Match. The
bundle's own etag combines the slices', so an unchanged bundle is an empty 304.
prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard
guarantee: it never spends weather-API quota. A cell with no cached archive
answers 204, and only the history-derived slices (calendar + latest-day detail)
are built. At most one Nominatim lookup for a never-labeled cell.
* Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch
The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota,
which multi-year calendar payloads regularly blew through) to IndexedDB, with an
in-memory map in front. Entries carry the server's ETag, so anything stale
revalidates conditionally — unchanged data costs an empty 304 and a re-stamp,
never a re-transfer. On network failure the stale copy is served over an error.
getJSON gains an optional onUpdate callback opting into stale-while-revalidate:
the three views (weekly, day, calendar) now render a cached copy immediately —
spinners are delayed 150ms so warm loads never flash-blank — and repaint only if
background revalidation finds changed data. New data shows up the moment it
exists instead of waiting out a TTL.
Cross-view prefetch collapses from one request per view to a single /api/v2/cell
bundle, whose slices (exact per-view payloads + their etags) are seeded under the
URLs each view actually requests; the bundle call itself is conditional via a
remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side
with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one
possible Nominatim lookup each), so tapping nearby lands on already-graded data.
Legacy tg:* storage entries are cleared once; cache entries untouched for two
weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
|
|
|
|
run.set(run_type="cold-skip")
|
|
|
|
|
|
return Response(status_code=204)
|
|
|
|
|
|
cache_meta = {"cached": True}
|
|
|
|
|
|
else:
|
Single-source cache identity; shared fetch preamble and cache flow (#43)
The derived store's key/token formats existed in three places — each
endpoint, api_cell's slice assembly, and migrate.py — where any drift
would silently split the cache (endpoints missing rows the bundle wrote,
migrate materializing rows nobody reads). They are now defined once in
views.py (grade_key/calendar_key/day_key/forecast_key, history_token/
recent_token/day_token) and consumed everywhere, with a pinning test so
a format change is always deliberate.
The four data endpoints shared two copy-pasted sequences, now helpers:
- _fetch_history: history (+ optional recent bundle) fetch with upstream
failures mapped to clean HTTP errors and an empty record to 404.
- _cached_response: If-None-Match 304 / token-valid store replay /
build + persist + serve, with calendar's dont-persist-placeless rule
as an explicit flag.
Each endpoint is now its audit run + identity + a build callback (~10
lines); api_day's hourly-token special case moved into day_token. New
tests: identity format pins, rate-limit 503 parametrized across all five
data routes, prefetch=1 never touching upstream (cold 204, warm
history-only slices), and calendar's placeless-payload retry behavior.
2026-07-11 19:49:15 +00:00
|
|
|
|
history, cache_meta, recent = _fetch_history(run, cell, recent_too=True)
|
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21)
* Compress API responses and revalidate static assets instead of re-downloading
- Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x.
- Serve pages/assets with Cache-Control: no-cache instead of no-store, so
browsers revalidate via the ETag/Last-Modified that FileResponse and
StaticFiles already emit. Unchanged assets now cost an empty 304 rather
than a full transfer on every page navigation, while deploys still show
up immediately.
* Persist derived responses in SQLite so grading is computed once per cell, not per request
New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet
records — finished grade/calendar/day/forecast payloads and reverse-geocode
labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms)
becomes a ~5ms database read that survives restarts and is shared across views.
Raw parquet stays the source of truth; the store is a pure accelerator (every
reader falls back to recomputing on a miss, and deleting the db is a safe reset).
Freshness is token-driven, not clock-driven: each cached payload is validated by
a token encoding what it was computed from (payload schema version, the archive
record's end date, the recent-fetch stamp). The existing freshness drivers are
untouched — get_history still tops up the tail hourly and get_recent_forecast
still refetches hourly — and tokens are derived from what they return, so cached
payloads expire exactly when their inputs change. The same tokens double as weak
ETags: If-None-Match answers with an empty 304 without touching the payload.
- backend/store.py: derived-payload + revgeo tables, thread-local WAL conns,
every helper fail-soft.
- app.py: endpoints split into pure payload builders + HTTP/caching shells; the
in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store).
- climate.py: revgeo persisted through the store; recent_stamp() and
load_cached_history() (no-network read) helpers.
- backend/migrate.py + make migrate: idempotent, resumable backfill of the store
from existing parquet caches (default calendar span + latest-day detail +
revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather.
- grid.from_id(): rebuild a cell from its cache filename (migrate tooling).
* Add /api/v2/cell: one bundle carrying every view's payload
GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months)
and day (today) payloads in one response, each the exact payload its per-view
endpoint returns — built by the same builders and cached under the same
derived-store keys/tokens — paired with the etag that endpoint would emit. The
frontend can warm all views with a single request, seed its per-view cache from
the slices, and later revalidate each view individually with If-None-Match. The
bundle's own etag combines the slices', so an unchanged bundle is an empty 304.
prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard
guarantee: it never spends weather-API quota. A cell with no cached archive
answers 204, and only the history-derived slices (calendar + latest-day detail)
are built. At most one Nominatim lookup for a never-labeled cell.
* Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch
The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota,
which multi-year calendar payloads regularly blew through) to IndexedDB, with an
in-memory map in front. Entries carry the server's ETag, so anything stale
revalidates conditionally — unchanged data costs an empty 304 and a re-stamp,
never a re-transfer. On network failure the stale copy is served over an error.
getJSON gains an optional onUpdate callback opting into stale-while-revalidate:
the three views (weekly, day, calendar) now render a cached copy immediately —
spinners are delayed 150ms so warm loads never flash-blank — and repaint only if
background revalidation finds changed data. New data shows up the moment it
exists instead of waiting out a TTL.
Cross-view prefetch collapses from one request per view to a single /api/v2/cell
bundle, whose slices (exact per-view payloads + their etags) are seeded under the
URLs each view actually requests; the bundle call itself is conditional via a
remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side
with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one
possible Nominatim lookup each), so tapping nearby lands on already-graded data.
Legacy tg:* storage entries are cleared once; cache entries untouched for two
weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
|
|
|
|
|
|
|
|
|
|
cid = cell["id"]
|
Migrate backend dataframe layer from pandas to polars (#90)
* Migrate backend dataframe layer from pandas to polars
Replace pandas with polars across the backend, dropping both pandas and its
pyarrow parquet engine from the dependency set. numpy stays (the grading
percentile math is unchanged).
- climate.py: parquet IO, source→frame mappings, cache read/topup on polars.
New _normalize_read casts the cached `date` column to pl.Date (older files
were written by pandas as datetime64[ns]); frames now unify missing values as
null so the grading boundary drops them consistently across sources.
- grading.py: keep the numpy percentile core; swap the frame→numpy bridge to
.to_numpy()/.drop_nulls(), day-of-year/year to polars dt expressions, and the
per-row loop to iter_rows(named=True).
- views.py: filter/anti-join/concat replace boolean-mask, isin and pd.concat;
scalar dates are stdlib datetime.date; a local _months_before helper replaces
DateOffset(months=) for the calendar-range default.
- app.py, migrate.py: request-date parsing uses datetime.date, removing pandas
from the endpoint and migrate layers entirely.
- The date column is pl.Date end to end, eliminating the pandas normalize() calls
and comparing cleanly against stdlib dates.
Payloads are unchanged: calendar, day, grade and forecast responses are
byte-for-byte identical to the pandas implementation on the same cached record.
Tests ported to polars fixtures, with added coverage for the combined feels-like
fallback, calendar month-offset (month-end/leap), and the concat/dedup
"fresher source wins" rule.
* Port notify.py to polars after merging dev's account system
Merge origin/dev (accounts + notification subscriptions) and carry the pandas→
polars migration into the newly added notify.py, which the merge brought in still
using pandas — with pandas removed from requirements this broke its import.
- notify.py: _candidate_rows filters/sorts the recent bundle with polars
expressions and returns iter_rows dicts; date scalars are datetime.date;
history/recent emptiness via is_empty().
- test_notify.py: synthetic history/rows built with polars + datetime.
2026-07-15 19:07:38 +00:00
|
|
|
|
last = history["date"].max()
|
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21)
* Compress API responses and revalidate static assets instead of re-downloading
- Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x.
- Serve pages/assets with Cache-Control: no-cache instead of no-store, so
browsers revalidate via the ETag/Last-Modified that FileResponse and
StaticFiles already emit. Unchanged assets now cost an empty 304 rather
than a full transfer on every page navigation, while deploys still show
up immediately.
* Persist derived responses in SQLite so grading is computed once per cell, not per request
New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet
records — finished grade/calendar/day/forecast payloads and reverse-geocode
labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms)
becomes a ~5ms database read that survives restarts and is shared across views.
Raw parquet stays the source of truth; the store is a pure accelerator (every
reader falls back to recomputing on a miss, and deleting the db is a safe reset).
Freshness is token-driven, not clock-driven: each cached payload is validated by
a token encoding what it was computed from (payload schema version, the archive
record's end date, the recent-fetch stamp). The existing freshness drivers are
untouched — get_history still tops up the tail hourly and get_recent_forecast
still refetches hourly — and tokens are derived from what they return, so cached
payloads expire exactly when their inputs change. The same tokens double as weak
ETags: If-None-Match answers with an empty 304 without touching the payload.
- backend/store.py: derived-payload + revgeo tables, thread-local WAL conns,
every helper fail-soft.
- app.py: endpoints split into pure payload builders + HTTP/caching shells; the
in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store).
- climate.py: revgeo persisted through the store; recent_stamp() and
load_cached_history() (no-network read) helpers.
- backend/migrate.py + make migrate: idempotent, resumable backfill of the store
from existing parquet caches (default calendar span + latest-day detail +
revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather.
- grid.from_id(): rebuild a cell from its cache filename (migrate tooling).
* Add /api/v2/cell: one bundle carrying every view's payload
GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months)
and day (today) payloads in one response, each the exact payload its per-view
endpoint returns — built by the same builders and cached under the same
derived-store keys/tokens — paired with the etag that endpoint would emit. The
frontend can warm all views with a single request, seed its per-view cache from
the slices, and later revalidate each view individually with If-None-Match. The
bundle's own etag combines the slices', so an unchanged bundle is an empty 304.
prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard
guarantee: it never spends weather-API quota. A cell with no cached archive
answers 204, and only the history-derived slices (calendar + latest-day detail)
are built. At most one Nominatim lookup for a never-labeled cell.
* Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch
The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota,
which multi-year calendar payloads regularly blew through) to IndexedDB, with an
in-memory map in front. Entries carry the server's ETag, so anything stale
revalidates conditionally — unchanged data costs an empty 304 and a re-stamp,
never a re-transfer. On network failure the stale copy is served over an error.
getJSON gains an optional onUpdate callback opting into stale-while-revalidate:
the three views (weekly, day, calendar) now render a cached copy immediately —
spinners are delayed 150ms so warm loads never flash-blank — and repaint only if
background revalidation finds changed data. New data shows up the moment it
exists instead of waiting out a TTL.
Cross-view prefetch collapses from one request per view to a single /api/v2/cell
bundle, whose slices (exact per-view payloads + their etags) are seeded under the
URLs each view actually requests; the bundle call itself is conditional via a
remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side
with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one
possible Nominatim lookup each), so tapping nearby lands on already-graded data.
Legacy tg:* storage entries are cleared once; cache entries untouched for two
weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
|
|
|
|
|
2026-07-11 20:47:31 +00:00
|
|
|
|
if neighbors:
|
|
|
|
|
|
_enqueue_neighbor_warming(cell)
|
|
|
|
|
|
|
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21)
* Compress API responses and revalidate static assets instead of re-downloading
- Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x.
- Serve pages/assets with Cache-Control: no-cache instead of no-store, so
browsers revalidate via the ETag/Last-Modified that FileResponse and
StaticFiles already emit. Unchanged assets now cost an empty 304 rather
than a full transfer on every page navigation, while deploys still show
up immediately.
* Persist derived responses in SQLite so grading is computed once per cell, not per request
New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet
records — finished grade/calendar/day/forecast payloads and reverse-geocode
labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms)
becomes a ~5ms database read that survives restarts and is shared across views.
Raw parquet stays the source of truth; the store is a pure accelerator (every
reader falls back to recomputing on a miss, and deleting the db is a safe reset).
Freshness is token-driven, not clock-driven: each cached payload is validated by
a token encoding what it was computed from (payload schema version, the archive
record's end date, the recent-fetch stamp). The existing freshness drivers are
untouched — get_history still tops up the tail hourly and get_recent_forecast
still refetches hourly — and tokens are derived from what they return, so cached
payloads expire exactly when their inputs change. The same tokens double as weak
ETags: If-None-Match answers with an empty 304 without touching the payload.
- backend/store.py: derived-payload + revgeo tables, thread-local WAL conns,
every helper fail-soft.
- app.py: endpoints split into pure payload builders + HTTP/caching shells; the
in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store).
- climate.py: revgeo persisted through the store; recent_stamp() and
load_cached_history() (no-network read) helpers.
- backend/migrate.py + make migrate: idempotent, resumable backfill of the store
from existing parquet caches (default calendar span + latest-day detail +
revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather.
- grid.from_id(): rebuild a cell from its cache filename (migrate tooling).
* Add /api/v2/cell: one bundle carrying every view's payload
GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months)
and day (today) payloads in one response, each the exact payload its per-view
endpoint returns — built by the same builders and cached under the same
derived-store keys/tokens — paired with the etag that endpoint would emit. The
frontend can warm all views with a single request, seed its per-view cache from
the slices, and later revalidate each view individually with If-None-Match. The
bundle's own etag combines the slices', so an unchanged bundle is an empty 304.
prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard
guarantee: it never spends weather-API quota. A cell with no cached archive
answers 204, and only the history-derived slices (calendar + latest-day detail)
are built. At most one Nominatim lookup for a never-labeled cell.
* Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch
The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota,
which multi-year calendar payloads regularly blew through) to IndexedDB, with an
in-memory map in front. Entries carry the server's ETag, so anything stale
revalidates conditionally — unchanged data costs an empty 304 and a re-stamp,
never a re-transfer. On network failure the stale copy is served over an error.
getJSON gains an optional onUpdate callback opting into stale-while-revalidate:
the three views (weekly, day, calendar) now render a cached copy immediately —
spinners are delayed 150ms so warm loads never flash-blank — and repaint only if
background revalidation finds changed data. New data shows up the moment it
exists instead of waiting out a TTL.
Cross-view prefetch collapses from one request per view to a single /api/v2/cell
bundle, whose slices (exact per-view payloads + their etags) are seeded under the
URLs each view actually requests; the bundle call itself is conditional via a
remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side
with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one
possible Nominatim lookup each), so tapping nearby lands on already-graded data.
Legacy tg:* storage entries are cleared once; cache entries untouched for two
weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
|
|
|
|
with run.phase("reverse_geocode"):
|
|
|
|
|
|
place = climate.reverse_geocode(cell["center_lat"], cell["center_lon"])
|
|
|
|
|
|
|
|
|
|
|
|
def slice_for(kind: str, key: str, token: str, build) -> dict:
|
|
|
|
|
|
"""The endpoint's derived row (or build + store it), paired with the
|
|
|
|
|
|
etag that endpoint would emit for the same request."""
|
|
|
|
|
|
etag = _etag_for(kind, cid, key, token)
|
|
|
|
|
|
data = store.get_json(kind, cid, key, token)
|
|
|
|
|
|
if data is None:
|
|
|
|
|
|
data = build()
|
|
|
|
|
|
store.put_payload(kind, cid, key, token, data)
|
|
|
|
|
|
return {"etag": etag, "data": data}
|
|
|
|
|
|
|
|
|
|
|
|
slices = {}
|
Single-source cache identity; shared fetch preamble and cache flow (#43)
The derived store's key/token formats existed in three places — each
endpoint, api_cell's slice assembly, and migrate.py — where any drift
would silently split the cache (endpoints missing rows the bundle wrote,
migrate materializing rows nobody reads). They are now defined once in
views.py (grade_key/calendar_key/day_key/forecast_key, history_token/
recent_token/day_token) and consumed everywhere, with a pinning test so
a format change is always deliberate.
The four data endpoints shared two copy-pasted sequences, now helpers:
- _fetch_history: history (+ optional recent bundle) fetch with upstream
failures mapped to clean HTTP errors and an empty record to 404.
- _cached_response: If-None-Match 304 / token-valid store replay /
build + persist + serve, with calendar's dont-persist-placeless rule
as an explicit flag.
Each endpoint is now its audit run + identity + a build callback (~10
lines); api_day's hourly-token special case moved into day_token. New
tests: identity format pins, rate-limit 503 parametrized across all five
data routes, prefetch=1 never touching upstream (cold 204, warm
history-only slices), and calendar's placeless-payload retry behavior.
2026-07-11 19:49:15 +00:00
|
|
|
|
hist_token = views.history_token(history)
|
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21)
* Compress API responses and revalidate static assets instead of re-downloading
- Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x.
- Serve pages/assets with Cache-Control: no-cache instead of no-store, so
browsers revalidate via the ETag/Last-Modified that FileResponse and
StaticFiles already emit. Unchanged assets now cost an empty 304 rather
than a full transfer on every page navigation, while deploys still show
up immediately.
* Persist derived responses in SQLite so grading is computed once per cell, not per request
New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet
records — finished grade/calendar/day/forecast payloads and reverse-geocode
labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms)
becomes a ~5ms database read that survives restarts and is shared across views.
Raw parquet stays the source of truth; the store is a pure accelerator (every
reader falls back to recomputing on a miss, and deleting the db is a safe reset).
Freshness is token-driven, not clock-driven: each cached payload is validated by
a token encoding what it was computed from (payload schema version, the archive
record's end date, the recent-fetch stamp). The existing freshness drivers are
untouched — get_history still tops up the tail hourly and get_recent_forecast
still refetches hourly — and tokens are derived from what they return, so cached
payloads expire exactly when their inputs change. The same tokens double as weak
ETags: If-None-Match answers with an empty 304 without touching the payload.
- backend/store.py: derived-payload + revgeo tables, thread-local WAL conns,
every helper fail-soft.
- app.py: endpoints split into pure payload builders + HTTP/caching shells; the
in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store).
- climate.py: revgeo persisted through the store; recent_stamp() and
load_cached_history() (no-network read) helpers.
- backend/migrate.py + make migrate: idempotent, resumable backfill of the store
from existing parquet caches (default calendar span + latest-day detail +
revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather.
- grid.from_id(): rebuild a cell from its cache filename (migrate tooling).
* Add /api/v2/cell: one bundle carrying every view's payload
GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months)
and day (today) payloads in one response, each the exact payload its per-view
endpoint returns — built by the same builders and cached under the same
derived-store keys/tokens — paired with the etag that endpoint would emit. The
frontend can warm all views with a single request, seed its per-view cache from
the slices, and later revalidate each view individually with If-None-Match. The
bundle's own etag combines the slices', so an unchanged bundle is an empty 304.
prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard
guarantee: it never spends weather-API quota. A cell with no cached archive
answers 204, and only the history-derived slices (calendar + latest-day detail)
are built. At most one Nominatim lookup for a never-labeled cell.
* Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch
The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota,
which multi-year calendar payloads regularly blew through) to IndexedDB, with an
in-memory map in front. Entries carry the server's ETag, so anything stale
revalidates conditionally — unchanged data costs an empty 304 and a re-stamp,
never a re-transfer. On network failure the stale copy is served over an error.
getJSON gains an optional onUpdate callback opting into stale-while-revalidate:
the three views (weekly, day, calendar) now render a cached copy immediately —
spinners are delayed 150ms so warm loads never flash-blank — and repaint only if
background revalidation finds changed data. New data shows up the moment it
exists instead of waiting out a TTL.
Cross-view prefetch collapses from one request per view to a single /api/v2/cell
bundle, whose slices (exact per-view payloads + their etags) are seeded under the
URLs each view actually requests; the bundle call itself is conditional via a
remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side
with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one
possible Nominatim lookup each), so tapping nearby lands on already-graded data.
Legacy tg:* storage entries are cleared once; cache entries untouched for two
weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
|
|
|
|
# Calendar: the default last-24-months span — what the calendar view (and
|
|
|
|
|
|
# the cross-view prefetch) requests first.
|
Extract payload builders into views.py; load places index at startup (#42)
app.py had grown three co-resident strata: HTTP/caching plumbing, payload
assembly, and search policy. This moves the payload layer — the four
build_* functions, cal_span clamping, hist_end, PAYLOAD_VER, NullRun and
their helpers — into a new views.py with no web dependencies (pure moves,
public names). app.py keeps routing, ETag/derived-store plumbing, and
page serving; migrate.py imports views instead of reaching into app's
privates.
That import previously constructed the whole FastAPI app and — because
places.start_loading() ran at import time — kicked off a background
GeoNames download from an offline batch script. The index load now hangs
off the app's lifespan hook, so it fires when a server starts, not when
the module is imported (tests, migrate).
New tests: cal_span clamping (months-back default, record bounds, 2-year
cap, inversion), builder shapes (grade window ordering, day obs from the
recent bundle, forecast future-only), a regression test that build_day
degrades to climatology when the recent fetch fails, and a subprocess
layering guard that importing views/migrate pulls in neither FastAPI nor
the app module and starts no index download.
2026-07-11 19:43:41 +00:00
|
|
|
|
start_ts, end_ts = views.cal_span(history, None, None, 24)
|
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21)
* Compress API responses and revalidate static assets instead of re-downloading
- Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x.
- Serve pages/assets with Cache-Control: no-cache instead of no-store, so
browsers revalidate via the ETag/Last-Modified that FileResponse and
StaticFiles already emit. Unchanged assets now cost an empty 304 rather
than a full transfer on every page navigation, while deploys still show
up immediately.
* Persist derived responses in SQLite so grading is computed once per cell, not per request
New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet
records — finished grade/calendar/day/forecast payloads and reverse-geocode
labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms)
becomes a ~5ms database read that survives restarts and is shared across views.
Raw parquet stays the source of truth; the store is a pure accelerator (every
reader falls back to recomputing on a miss, and deleting the db is a safe reset).
Freshness is token-driven, not clock-driven: each cached payload is validated by
a token encoding what it was computed from (payload schema version, the archive
record's end date, the recent-fetch stamp). The existing freshness drivers are
untouched — get_history still tops up the tail hourly and get_recent_forecast
still refetches hourly — and tokens are derived from what they return, so cached
payloads expire exactly when their inputs change. The same tokens double as weak
ETags: If-None-Match answers with an empty 304 without touching the payload.
- backend/store.py: derived-payload + revgeo tables, thread-local WAL conns,
every helper fail-soft.
- app.py: endpoints split into pure payload builders + HTTP/caching shells; the
in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store).
- climate.py: revgeo persisted through the store; recent_stamp() and
load_cached_history() (no-network read) helpers.
- backend/migrate.py + make migrate: idempotent, resumable backfill of the store
from existing parquet caches (default calendar span + latest-day detail +
revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather.
- grid.from_id(): rebuild a cell from its cache filename (migrate tooling).
* Add /api/v2/cell: one bundle carrying every view's payload
GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months)
and day (today) payloads in one response, each the exact payload its per-view
endpoint returns — built by the same builders and cached under the same
derived-store keys/tokens — paired with the etag that endpoint would emit. The
frontend can warm all views with a single request, seed its per-view cache from
the slices, and later revalidate each view individually with If-None-Match. The
bundle's own etag combines the slices', so an unchanged bundle is an empty 304.
prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard
guarantee: it never spends weather-API quota. A cell with no cached archive
answers 204, and only the history-derived slices (calendar + latest-day detail)
are built. At most one Nominatim lookup for a never-labeled cell.
* Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch
The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota,
which multi-year calendar payloads regularly blew through) to IndexedDB, with an
in-memory map in front. Entries carry the server's ETag, so anything stale
revalidates conditionally — unchanged data costs an empty 304 and a re-stamp,
never a re-transfer. On network failure the stale copy is served over an error.
getJSON gains an optional onUpdate callback opting into stale-while-revalidate:
the three views (weekly, day, calendar) now render a cached copy immediately —
spinners are delayed 150ms so warm loads never flash-blank — and repaint only if
background revalidation finds changed data. New data shows up the moment it
exists instead of waiting out a TTL.
Cross-view prefetch collapses from one request per view to a single /api/v2/cell
bundle, whose slices (exact per-view payloads + their etags) are seeded under the
URLs each view actually requests; the bundle call itself is conditional via a
remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side
with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one
possible Nominatim lookup each), so tapping nearby lands on already-graded data.
Legacy tg:* storage entries are cleared once; cache entries untouched for two
weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
|
|
|
|
slices["calendar"] = slice_for(
|
Single-source cache identity; shared fetch preamble and cache flow (#43)
The derived store's key/token formats existed in three places — each
endpoint, api_cell's slice assembly, and migrate.py — where any drift
would silently split the cache (endpoints missing rows the bundle wrote,
migrate materializing rows nobody reads). They are now defined once in
views.py (grade_key/calendar_key/day_key/forecast_key, history_token/
recent_token/day_token) and consumed everywhere, with a pinning test so
a format change is always deliberate.
The four data endpoints shared two copy-pasted sequences, now helpers:
- _fetch_history: history (+ optional recent bundle) fetch with upstream
failures mapped to clean HTTP errors and an empty record to 404.
- _cached_response: If-None-Match 304 / token-valid store replay /
build + persist + serve, with calendar's dont-persist-placeless rule
as an explicit flag.
Each endpoint is now its audit run + identity + a build callback (~10
lines); api_day's hourly-token special case moved into day_token. New
tests: identity format pins, rate-limit 503 parametrized across all five
data routes, prefetch=1 never touching upstream (cold 204, warm
history-only slices), and calendar's placeless-payload retry behavior.
2026-07-11 19:49:15 +00:00
|
|
|
|
"calendar", views.calendar_key(start_ts, end_ts, 24), hist_token,
|
Extract payload builders into views.py; load places index at startup (#42)
app.py had grown three co-resident strata: HTTP/caching plumbing, payload
assembly, and search policy. This moves the payload layer — the four
build_* functions, cal_span clamping, hist_end, PAYLOAD_VER, NullRun and
their helpers — into a new views.py with no web dependencies (pure moves,
public names). app.py keeps routing, ETag/derived-store plumbing, and
page serving; migrate.py imports views instead of reaching into app's
privates.
That import previously constructed the whole FastAPI app and — because
places.start_loading() ran at import time — kicked off a background
GeoNames download from an offline batch script. The index load now hangs
off the app's lifespan hook, so it fires when a server starts, not when
the module is imported (tests, migrate).
New tests: cal_span clamping (months-back default, record bounds, 2-year
cap, inversion), builder shapes (grade window ordering, day obs from the
recent bundle, forecast future-only), a regression test that build_day
degrades to climatology when the recent fetch fails, and a subprocess
layering guard that importing views/migrate pulls in neither FastAPI nor
the app module and starts no index download.
2026-07-11 19:43:41 +00:00
|
|
|
|
lambda: views.build_calendar(cell, history, start_ts, end_ts, 24, place, run))
|
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21)
* Compress API responses and revalidate static assets instead of re-downloading
- Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x.
- Serve pages/assets with Cache-Control: no-cache instead of no-store, so
browsers revalidate via the ETag/Last-Modified that FileResponse and
StaticFiles already emit. Unchanged assets now cost an empty 304 rather
than a full transfer on every page navigation, while deploys still show
up immediately.
* Persist derived responses in SQLite so grading is computed once per cell, not per request
New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet
records — finished grade/calendar/day/forecast payloads and reverse-geocode
labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms)
becomes a ~5ms database read that survives restarts and is shared across views.
Raw parquet stays the source of truth; the store is a pure accelerator (every
reader falls back to recomputing on a miss, and deleting the db is a safe reset).
Freshness is token-driven, not clock-driven: each cached payload is validated by
a token encoding what it was computed from (payload schema version, the archive
record's end date, the recent-fetch stamp). The existing freshness drivers are
untouched — get_history still tops up the tail hourly and get_recent_forecast
still refetches hourly — and tokens are derived from what they return, so cached
payloads expire exactly when their inputs change. The same tokens double as weak
ETags: If-None-Match answers with an empty 304 without touching the payload.
- backend/store.py: derived-payload + revgeo tables, thread-local WAL conns,
every helper fail-soft.
- app.py: endpoints split into pure payload builders + HTTP/caching shells; the
in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store).
- climate.py: revgeo persisted through the store; recent_stamp() and
load_cached_history() (no-network read) helpers.
- backend/migrate.py + make migrate: idempotent, resumable backfill of the store
from existing parquet caches (default calendar span + latest-day detail +
revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather.
- grid.from_id(): rebuild a cell from its cache filename (migrate tooling).
* Add /api/v2/cell: one bundle carrying every view's payload
GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months)
and day (today) payloads in one response, each the exact payload its per-view
endpoint returns — built by the same builders and cached under the same
derived-store keys/tokens — paired with the etag that endpoint would emit. The
frontend can warm all views with a single request, seed its per-view cache from
the slices, and later revalidate each view individually with If-None-Match. The
bundle's own etag combines the slices', so an unchanged bundle is an empty 304.
prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard
guarantee: it never spends weather-API quota. A cell with no cached archive
answers 204, and only the history-derived slices (calendar + latest-day detail)
are built. At most one Nominatim lookup for a never-labeled cell.
* Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch
The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota,
which multi-year calendar payloads regularly blew through) to IndexedDB, with an
in-memory map in front. Entries carry the server's ETag, so anything stale
revalidates conditionally — unchanged data costs an empty 304 and a re-stamp,
never a re-transfer. On network failure the stale copy is served over an error.
getJSON gains an optional onUpdate callback opting into stale-while-revalidate:
the three views (weekly, day, calendar) now render a cached copy immediately —
spinners are delayed 150ms so warm loads never flash-blank — and repaint only if
background revalidation finds changed data. New data shows up the moment it
exists instead of waiting out a TTL.
Cross-view prefetch collapses from one request per view to a single /api/v2/cell
bundle, whose slices (exact per-view payloads + their etags) are seeded under the
URLs each view actually requests; the bundle call itself is conditional via a
remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side
with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one
possible Nominatim lookup each), so tapping nearby lands on already-graded data.
Legacy tg:* storage entries are cleared once; cache entries untouched for two
weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
|
|
|
|
|
|
|
|
|
|
if prefetch:
|
|
|
|
|
|
# Latest archived day: its observation comes from history alone, so
|
|
|
|
|
|
# it's buildable without the (skipped) recent bundle.
|
|
|
|
|
|
slices["day"] = slice_for(
|
Single-source cache identity; shared fetch preamble and cache flow (#43)
The derived store's key/token formats existed in three places — each
endpoint, api_cell's slice assembly, and migrate.py — where any drift
would silently split the cache (endpoints missing rows the bundle wrote,
migrate materializing rows nobody reads). They are now defined once in
views.py (grade_key/calendar_key/day_key/forecast_key, history_token/
recent_token/day_token) and consumed everywhere, with a pinning test so
a format change is always deliberate.
The four data endpoints shared two copy-pasted sequences, now helpers:
- _fetch_history: history (+ optional recent bundle) fetch with upstream
failures mapped to clean HTTP errors and an empty record to 404.
- _cached_response: If-None-Match 304 / token-valid store replay /
build + persist + serve, with calendar's dont-persist-placeless rule
as an explicit flag.
Each endpoint is now its audit run + identity + a build callback (~10
lines); api_day's hourly-token special case moved into day_token. New
tests: identity format pins, rate-limit 503 parametrized across all five
data routes, prefetch=1 never touching upstream (cold 204, warm
history-only slices), and calendar's placeless-payload retry behavior.
2026-07-11 19:49:15 +00:00
|
|
|
|
"day", views.day_key(last), hist_token,
|
Extract payload builders into views.py; load places index at startup (#42)
app.py had grown three co-resident strata: HTTP/caching plumbing, payload
assembly, and search policy. This moves the payload layer — the four
build_* functions, cal_span clamping, hist_end, PAYLOAD_VER, NullRun and
their helpers — into a new views.py with no web dependencies (pure moves,
public names). app.py keeps routing, ETag/derived-store plumbing, and
page serving; migrate.py imports views instead of reaching into app's
privates.
That import previously constructed the whole FastAPI app and — because
places.start_loading() ran at import time — kicked off a background
GeoNames download from an offline batch script. The index load now hangs
off the app's lifespan hook, so it fires when a server starts, not when
the module is imported (tests, migrate).
New tests: cal_span clamping (months-back default, record bounds, 2-year
cap, inversion), builder shapes (grade window ordering, day obs from the
recent bundle, forecast future-only), a regression test that build_day
degrades to climatology when the recent fetch fails, and a subprocess
layering guard that importing views/migrate pulls in neither FastAPI nor
the app module and starts no index download.
2026-07-11 19:43:41 +00:00
|
|
|
|
lambda: views.build_day(cell, history, last, place, run))
|
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21)
* Compress API responses and revalidate static assets instead of re-downloading
- Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x.
- Serve pages/assets with Cache-Control: no-cache instead of no-store, so
browsers revalidate via the ETag/Last-Modified that FileResponse and
StaticFiles already emit. Unchanged assets now cost an empty 304 rather
than a full transfer on every page navigation, while deploys still show
up immediately.
* Persist derived responses in SQLite so grading is computed once per cell, not per request
New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet
records — finished grade/calendar/day/forecast payloads and reverse-geocode
labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms)
becomes a ~5ms database read that survives restarts and is shared across views.
Raw parquet stays the source of truth; the store is a pure accelerator (every
reader falls back to recomputing on a miss, and deleting the db is a safe reset).
Freshness is token-driven, not clock-driven: each cached payload is validated by
a token encoding what it was computed from (payload schema version, the archive
record's end date, the recent-fetch stamp). The existing freshness drivers are
untouched — get_history still tops up the tail hourly and get_recent_forecast
still refetches hourly — and tokens are derived from what they return, so cached
payloads expire exactly when their inputs change. The same tokens double as weak
ETags: If-None-Match answers with an empty 304 without touching the payload.
- backend/store.py: derived-payload + revgeo tables, thread-local WAL conns,
every helper fail-soft.
- app.py: endpoints split into pure payload builders + HTTP/caching shells; the
in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store).
- climate.py: revgeo persisted through the store; recent_stamp() and
load_cached_history() (no-network read) helpers.
- backend/migrate.py + make migrate: idempotent, resumable backfill of the store
from existing parquet caches (default calendar span + latest-day detail +
revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather.
- grid.from_id(): rebuild a cell from its cache filename (migrate tooling).
* Add /api/v2/cell: one bundle carrying every view's payload
GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months)
and day (today) payloads in one response, each the exact payload its per-view
endpoint returns — built by the same builders and cached under the same
derived-store keys/tokens — paired with the etag that endpoint would emit. The
frontend can warm all views with a single request, seed its per-view cache from
the slices, and later revalidate each view individually with If-None-Match. The
bundle's own etag combines the slices', so an unchanged bundle is an empty 304.
prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard
guarantee: it never spends weather-API quota. A cell with no cached archive
answers 204, and only the history-derived slices (calendar + latest-day detail)
are built. At most one Nominatim lookup for a never-labeled cell.
* Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch
The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota,
which multi-year calendar payloads regularly blew through) to IndexedDB, with an
in-memory map in front. Entries carry the server's ETag, so anything stale
revalidates conditionally — unchanged data costs an empty 304 and a re-stamp,
never a re-transfer. On network failure the stale copy is served over an error.
getJSON gains an optional onUpdate callback opting into stale-while-revalidate:
the three views (weekly, day, calendar) now render a cached copy immediately —
spinners are delayed 150ms so warm loads never flash-blank — and repaint only if
background revalidation finds changed data. New data shows up the moment it
exists instead of waiting out a TTL.
Cross-view prefetch collapses from one request per view to a single /api/v2/cell
bundle, whose slices (exact per-view payloads + their etags) are seeded under the
URLs each view actually requests; the bundle call itself is conditional via a
remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side
with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one
possible Nominatim lookup each), so tapping nearby lands on already-graded data.
Legacy tg:* storage entries are cleared once; cache entries untouched for two
weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
|
|
|
|
else:
|
Single-source cache identity; shared fetch preamble and cache flow (#43)
The derived store's key/token formats existed in three places — each
endpoint, api_cell's slice assembly, and migrate.py — where any drift
would silently split the cache (endpoints missing rows the bundle wrote,
migrate materializing rows nobody reads). They are now defined once in
views.py (grade_key/calendar_key/day_key/forecast_key, history_token/
recent_token/day_token) and consumed everywhere, with a pinning test so
a format change is always deliberate.
The four data endpoints shared two copy-pasted sequences, now helpers:
- _fetch_history: history (+ optional recent bundle) fetch with upstream
failures mapped to clean HTTP errors and an empty record to 404.
- _cached_response: If-None-Match 304 / token-valid store replay /
build + persist + serve, with calendar's dont-persist-placeless rule
as an explicit flag.
Each endpoint is now its audit run + identity + a build callback (~10
lines); api_day's hourly-token special case moved into day_token. New
tests: identity format pins, rate-limit 503 parametrized across all five
data routes, prefetch=1 never touching upstream (cold 204, warm
history-only slices), and calendar's placeless-payload retry behavior.
2026-07-11 19:49:15 +00:00
|
|
|
|
rf_token = views.recent_token(history, cid)
|
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21)
* Compress API responses and revalidate static assets instead of re-downloading
- Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x.
- Serve pages/assets with Cache-Control: no-cache instead of no-store, so
browsers revalidate via the ETag/Last-Modified that FileResponse and
StaticFiles already emit. Unchanged assets now cost an empty 304 rather
than a full transfer on every page navigation, while deploys still show
up immediately.
* Persist derived responses in SQLite so grading is computed once per cell, not per request
New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet
records — finished grade/calendar/day/forecast payloads and reverse-geocode
labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms)
becomes a ~5ms database read that survives restarts and is shared across views.
Raw parquet stays the source of truth; the store is a pure accelerator (every
reader falls back to recomputing on a miss, and deleting the db is a safe reset).
Freshness is token-driven, not clock-driven: each cached payload is validated by
a token encoding what it was computed from (payload schema version, the archive
record's end date, the recent-fetch stamp). The existing freshness drivers are
untouched — get_history still tops up the tail hourly and get_recent_forecast
still refetches hourly — and tokens are derived from what they return, so cached
payloads expire exactly when their inputs change. The same tokens double as weak
ETags: If-None-Match answers with an empty 304 without touching the payload.
- backend/store.py: derived-payload + revgeo tables, thread-local WAL conns,
every helper fail-soft.
- app.py: endpoints split into pure payload builders + HTTP/caching shells; the
in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store).
- climate.py: revgeo persisted through the store; recent_stamp() and
load_cached_history() (no-network read) helpers.
- backend/migrate.py + make migrate: idempotent, resumable backfill of the store
from existing parquet caches (default calendar span + latest-day detail +
revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather.
- grid.from_id(): rebuild a cell from its cache filename (migrate tooling).
* Add /api/v2/cell: one bundle carrying every view's payload
GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months)
and day (today) payloads in one response, each the exact payload its per-view
endpoint returns — built by the same builders and cached under the same
derived-store keys/tokens — paired with the etag that endpoint would emit. The
frontend can warm all views with a single request, seed its per-view cache from
the slices, and later revalidate each view individually with If-None-Match. The
bundle's own etag combines the slices', so an unchanged bundle is an empty 304.
prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard
guarantee: it never spends weather-API quota. A cell with no cached archive
answers 204, and only the history-derived slices (calendar + latest-day detail)
are built. At most one Nominatim lookup for a never-labeled cell.
* Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch
The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota,
which multi-year calendar payloads regularly blew through) to IndexedDB, with an
in-memory map in front. Entries carry the server's ETag, so anything stale
revalidates conditionally — unchanged data costs an empty 304 and a re-stamp,
never a re-transfer. On network failure the stale copy is served over an error.
getJSON gains an optional onUpdate callback opting into stale-while-revalidate:
the three views (weekly, day, calendar) now render a cached copy immediately —
spinners are delayed 150ms so warm loads never flash-blank — and repaint only if
background revalidation finds changed data. New data shows up the moment it
exists instead of waiting out a TTL.
Cross-view prefetch collapses from one request per view to a single /api/v2/cell
bundle, whose slices (exact per-view payloads + their etags) are seeded under the
URLs each view actually requests; the bundle call itself is conditional via a
remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side
with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one
possible Nominatim lookup each), so tapping nearby lands on already-graded data.
Legacy tg:* storage entries are cleared once; cache entries untouched for two
weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
|
|
|
|
slices["grade"] = slice_for(
|
Single-source cache identity; shared fetch preamble and cache flow (#43)
The derived store's key/token formats existed in three places — each
endpoint, api_cell's slice assembly, and migrate.py — where any drift
would silently split the cache (endpoints missing rows the bundle wrote,
migrate materializing rows nobody reads). They are now defined once in
views.py (grade_key/calendar_key/day_key/forecast_key, history_token/
recent_token/day_token) and consumed everywhere, with a pinning test so
a format change is always deliberate.
The four data endpoints shared two copy-pasted sequences, now helpers:
- _fetch_history: history (+ optional recent bundle) fetch with upstream
failures mapped to clean HTTP errors and an empty record to 404.
- _cached_response: If-None-Match 304 / token-valid store replay /
build + persist + serve, with calendar's dont-persist-placeless rule
as an explicit flag.
Each endpoint is now its audit run + identity + a build callback (~10
lines); api_day's hourly-token special case moved into day_token. New
tests: identity format pins, rate-limit 503 parametrized across all five
data routes, prefetch=1 never touching upstream (cold 204, warm
history-only slices), and calendar's placeless-payload retry behavior.
2026-07-11 19:49:15 +00:00
|
|
|
|
"grade", views.grade_key(today, 14, 14), rf_token,
|
Extract payload builders into views.py; load places index at startup (#42)
app.py had grown three co-resident strata: HTTP/caching plumbing, payload
assembly, and search policy. This moves the payload layer — the four
build_* functions, cal_span clamping, hist_end, PAYLOAD_VER, NullRun and
their helpers — into a new views.py with no web dependencies (pure moves,
public names). app.py keeps routing, ETag/derived-store plumbing, and
page serving; migrate.py imports views instead of reaching into app's
privates.
That import previously constructed the whole FastAPI app and — because
places.start_loading() ran at import time — kicked off a background
GeoNames download from an offline batch script. The index load now hangs
off the app's lifespan hook, so it fires when a server starts, not when
the module is imported (tests, migrate).
New tests: cal_span clamping (months-back default, record bounds, 2-year
cap, inversion), builder shapes (grade window ordering, day obs from the
recent bundle, forecast future-only), a regression test that build_day
degrades to climatology when the recent fetch fails, and a subprocess
layering guard that importing views/migrate pulls in neither FastAPI nor
the app module and starts no index download.
2026-07-11 19:43:41 +00:00
|
|
|
|
lambda: views.build_grade(cell, today, 14, history, recent, cache_meta, place, run, after=14))
|
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21)
* Compress API responses and revalidate static assets instead of re-downloading
- Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x.
- Serve pages/assets with Cache-Control: no-cache instead of no-store, so
browsers revalidate via the ETag/Last-Modified that FileResponse and
StaticFiles already emit. Unchanged assets now cost an empty 304 rather
than a full transfer on every page navigation, while deploys still show
up immediately.
* Persist derived responses in SQLite so grading is computed once per cell, not per request
New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet
records — finished grade/calendar/day/forecast payloads and reverse-geocode
labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms)
becomes a ~5ms database read that survives restarts and is shared across views.
Raw parquet stays the source of truth; the store is a pure accelerator (every
reader falls back to recomputing on a miss, and deleting the db is a safe reset).
Freshness is token-driven, not clock-driven: each cached payload is validated by
a token encoding what it was computed from (payload schema version, the archive
record's end date, the recent-fetch stamp). The existing freshness drivers are
untouched — get_history still tops up the tail hourly and get_recent_forecast
still refetches hourly — and tokens are derived from what they return, so cached
payloads expire exactly when their inputs change. The same tokens double as weak
ETags: If-None-Match answers with an empty 304 without touching the payload.
- backend/store.py: derived-payload + revgeo tables, thread-local WAL conns,
every helper fail-soft.
- app.py: endpoints split into pure payload builders + HTTP/caching shells; the
in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store).
- climate.py: revgeo persisted through the store; recent_stamp() and
load_cached_history() (no-network read) helpers.
- backend/migrate.py + make migrate: idempotent, resumable backfill of the store
from existing parquet caches (default calendar span + latest-day detail +
revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather.
- grid.from_id(): rebuild a cell from its cache filename (migrate tooling).
* Add /api/v2/cell: one bundle carrying every view's payload
GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months)
and day (today) payloads in one response, each the exact payload its per-view
endpoint returns — built by the same builders and cached under the same
derived-store keys/tokens — paired with the etag that endpoint would emit. The
frontend can warm all views with a single request, seed its per-view cache from
the slices, and later revalidate each view individually with If-None-Match. The
bundle's own etag combines the slices', so an unchanged bundle is an empty 304.
prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard
guarantee: it never spends weather-API quota. A cell with no cached archive
answers 204, and only the history-derived slices (calendar + latest-day detail)
are built. At most one Nominatim lookup for a never-labeled cell.
* Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch
The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota,
which multi-year calendar payloads regularly blew through) to IndexedDB, with an
in-memory map in front. Entries carry the server's ETag, so anything stale
revalidates conditionally — unchanged data costs an empty 304 and a re-stamp,
never a re-transfer. On network failure the stale copy is served over an error.
getJSON gains an optional onUpdate callback opting into stale-while-revalidate:
the three views (weekly, day, calendar) now render a cached copy immediately —
spinners are delayed 150ms so warm loads never flash-blank — and repaint only if
background revalidation finds changed data. New data shows up the moment it
exists instead of waiting out a TTL.
Cross-view prefetch collapses from one request per view to a single /api/v2/cell
bundle, whose slices (exact per-view payloads + their etags) are seeded under the
URLs each view actually requests; the bundle call itself is conditional via a
remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side
with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one
possible Nominatim lookup each), so tapping nearby lands on already-graded data.
Legacy tg:* storage entries are cleared once; cache entries untouched for two
weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
|
|
|
|
slices["forecast"] = slice_for(
|
Single-source cache identity; shared fetch preamble and cache flow (#43)
The derived store's key/token formats existed in three places — each
endpoint, api_cell's slice assembly, and migrate.py — where any drift
would silently split the cache (endpoints missing rows the bundle wrote,
migrate materializing rows nobody reads). They are now defined once in
views.py (grade_key/calendar_key/day_key/forecast_key, history_token/
recent_token/day_token) and consumed everywhere, with a pinning test so
a format change is always deliberate.
The four data endpoints shared two copy-pasted sequences, now helpers:
- _fetch_history: history (+ optional recent bundle) fetch with upstream
failures mapped to clean HTTP errors and an empty record to 404.
- _cached_response: If-None-Match 304 / token-valid store replay /
build + persist + serve, with calendar's dont-persist-placeless rule
as an explicit flag.
Each endpoint is now its audit run + identity + a build callback (~10
lines); api_day's hourly-token special case moved into day_token. New
tests: identity format pins, rate-limit 503 parametrized across all five
data routes, prefetch=1 never touching upstream (cold 204, warm
history-only slices), and calendar's placeless-payload retry behavior.
2026-07-11 19:49:15 +00:00
|
|
|
|
"forecast", views.forecast_key(today, 7), rf_token,
|
Extract payload builders into views.py; load places index at startup (#42)
app.py had grown three co-resident strata: HTTP/caching plumbing, payload
assembly, and search policy. This moves the payload layer — the four
build_* functions, cal_span clamping, hist_end, PAYLOAD_VER, NullRun and
their helpers — into a new views.py with no web dependencies (pure moves,
public names). app.py keeps routing, ETag/derived-store plumbing, and
page serving; migrate.py imports views instead of reaching into app's
privates.
That import previously constructed the whole FastAPI app and — because
places.start_loading() ran at import time — kicked off a background
GeoNames download from an offline batch script. The index load now hangs
off the app's lifespan hook, so it fires when a server starts, not when
the module is imported (tests, migrate).
New tests: cal_span clamping (months-back default, record bounds, 2-year
cap, inversion), builder shapes (grade window ordering, day obs from the
recent bundle, forecast future-only), a regression test that build_day
degrades to climatology when the recent fetch fails, and a subprocess
layering guard that importing views/migrate pulls in neither FastAPI nor
the app module and starts no index download.
2026-07-11 19:43:41 +00:00
|
|
|
|
lambda: views.build_forecast(cell, 7, history, recent, today, place, run))
|
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21)
* Compress API responses and revalidate static assets instead of re-downloading
- Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x.
- Serve pages/assets with Cache-Control: no-cache instead of no-store, so
browsers revalidate via the ETag/Last-Modified that FileResponse and
StaticFiles already emit. Unchanged assets now cost an empty 304 rather
than a full transfer on every page navigation, while deploys still show
up immediately.
* Persist derived responses in SQLite so grading is computed once per cell, not per request
New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet
records — finished grade/calendar/day/forecast payloads and reverse-geocode
labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms)
becomes a ~5ms database read that survives restarts and is shared across views.
Raw parquet stays the source of truth; the store is a pure accelerator (every
reader falls back to recomputing on a miss, and deleting the db is a safe reset).
Freshness is token-driven, not clock-driven: each cached payload is validated by
a token encoding what it was computed from (payload schema version, the archive
record's end date, the recent-fetch stamp). The existing freshness drivers are
untouched — get_history still tops up the tail hourly and get_recent_forecast
still refetches hourly — and tokens are derived from what they return, so cached
payloads expire exactly when their inputs change. The same tokens double as weak
ETags: If-None-Match answers with an empty 304 without touching the payload.
- backend/store.py: derived-payload + revgeo tables, thread-local WAL conns,
every helper fail-soft.
- app.py: endpoints split into pure payload builders + HTTP/caching shells; the
in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store).
- climate.py: revgeo persisted through the store; recent_stamp() and
load_cached_history() (no-network read) helpers.
- backend/migrate.py + make migrate: idempotent, resumable backfill of the store
from existing parquet caches (default calendar span + latest-day detail +
revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather.
- grid.from_id(): rebuild a cell from its cache filename (migrate tooling).
* Add /api/v2/cell: one bundle carrying every view's payload
GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months)
and day (today) payloads in one response, each the exact payload its per-view
endpoint returns — built by the same builders and cached under the same
derived-store keys/tokens — paired with the etag that endpoint would emit. The
frontend can warm all views with a single request, seed its per-view cache from
the slices, and later revalidate each view individually with If-None-Match. The
bundle's own etag combines the slices', so an unchanged bundle is an empty 304.
prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard
guarantee: it never spends weather-API quota. A cell with no cached archive
answers 204, and only the history-derived slices (calendar + latest-day detail)
are built. At most one Nominatim lookup for a never-labeled cell.
* Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch
The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota,
which multi-year calendar payloads regularly blew through) to IndexedDB, with an
in-memory map in front. Entries carry the server's ETag, so anything stale
revalidates conditionally — unchanged data costs an empty 304 and a re-stamp,
never a re-transfer. On network failure the stale copy is served over an error.
getJSON gains an optional onUpdate callback opting into stale-while-revalidate:
the three views (weekly, day, calendar) now render a cached copy immediately —
spinners are delayed 150ms so warm loads never flash-blank — and repaint only if
background revalidation finds changed data. New data shows up the moment it
exists instead of waiting out a TTL.
Cross-view prefetch collapses from one request per view to a single /api/v2/cell
bundle, whose slices (exact per-view payloads + their etags) are seeded under the
URLs each view actually requests; the bundle call itself is conditional via a
remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side
with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one
possible Nominatim lookup each), so tapping nearby lands on already-graded data.
Legacy tg:* storage entries are cleared once; cache entries untouched for two
weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
|
|
|
|
slices["day"] = slice_for(
|
Single-source cache identity; shared fetch preamble and cache flow (#43)
The derived store's key/token formats existed in three places — each
endpoint, api_cell's slice assembly, and migrate.py — where any drift
would silently split the cache (endpoints missing rows the bundle wrote,
migrate materializing rows nobody reads). They are now defined once in
views.py (grade_key/calendar_key/day_key/forecast_key, history_token/
recent_token/day_token) and consumed everywhere, with a pinning test so
a format change is always deliberate.
The four data endpoints shared two copy-pasted sequences, now helpers:
- _fetch_history: history (+ optional recent bundle) fetch with upstream
failures mapped to clean HTTP errors and an empty record to 404.
- _cached_response: If-None-Match 304 / token-valid store replay /
build + persist + serve, with calendar's dont-persist-placeless rule
as an explicit flag.
Each endpoint is now its audit run + identity + a build callback (~10
lines); api_day's hourly-token special case moved into day_token. New
tests: identity format pins, rate-limit 503 parametrized across all five
data routes, prefetch=1 never touching upstream (cold 204, warm
history-only slices), and calendar's placeless-payload retry behavior.
2026-07-11 19:49:15 +00:00
|
|
|
|
"day", views.day_key(today), views.day_token(history, today),
|
Extract payload builders into views.py; load places index at startup (#42)
app.py had grown three co-resident strata: HTTP/caching plumbing, payload
assembly, and search policy. This moves the payload layer — the four
build_* functions, cal_span clamping, hist_end, PAYLOAD_VER, NullRun and
their helpers — into a new views.py with no web dependencies (pure moves,
public names). app.py keeps routing, ETag/derived-store plumbing, and
page serving; migrate.py imports views instead of reaching into app's
privates.
That import previously constructed the whole FastAPI app and — because
places.start_loading() ran at import time — kicked off a background
GeoNames download from an offline batch script. The index load now hangs
off the app's lifespan hook, so it fires when a server starts, not when
the module is imported (tests, migrate).
New tests: cal_span clamping (months-back default, record bounds, 2-year
cap, inversion), builder shapes (grade window ordering, day obs from the
recent bundle, forecast future-only), a regression test that build_day
degrades to climatology when the recent fetch fails, and a subprocess
layering guard that importing views/migrate pulls in neither FastAPI nor
the app module and starts no index download.
2026-07-11 19:43:41 +00:00
|
|
|
|
lambda: views.build_day(cell, history, today, place, run, recent=recent))
|
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21)
* Compress API responses and revalidate static assets instead of re-downloading
- Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x.
- Serve pages/assets with Cache-Control: no-cache instead of no-store, so
browsers revalidate via the ETag/Last-Modified that FileResponse and
StaticFiles already emit. Unchanged assets now cost an empty 304 rather
than a full transfer on every page navigation, while deploys still show
up immediately.
* Persist derived responses in SQLite so grading is computed once per cell, not per request
New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet
records — finished grade/calendar/day/forecast payloads and reverse-geocode
labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms)
becomes a ~5ms database read that survives restarts and is shared across views.
Raw parquet stays the source of truth; the store is a pure accelerator (every
reader falls back to recomputing on a miss, and deleting the db is a safe reset).
Freshness is token-driven, not clock-driven: each cached payload is validated by
a token encoding what it was computed from (payload schema version, the archive
record's end date, the recent-fetch stamp). The existing freshness drivers are
untouched — get_history still tops up the tail hourly and get_recent_forecast
still refetches hourly — and tokens are derived from what they return, so cached
payloads expire exactly when their inputs change. The same tokens double as weak
ETags: If-None-Match answers with an empty 304 without touching the payload.
- backend/store.py: derived-payload + revgeo tables, thread-local WAL conns,
every helper fail-soft.
- app.py: endpoints split into pure payload builders + HTTP/caching shells; the
in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store).
- climate.py: revgeo persisted through the store; recent_stamp() and
load_cached_history() (no-network read) helpers.
- backend/migrate.py + make migrate: idempotent, resumable backfill of the store
from existing parquet caches (default calendar span + latest-day detail +
revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather.
- grid.from_id(): rebuild a cell from its cache filename (migrate tooling).
* Add /api/v2/cell: one bundle carrying every view's payload
GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months)
and day (today) payloads in one response, each the exact payload its per-view
endpoint returns — built by the same builders and cached under the same
derived-store keys/tokens — paired with the etag that endpoint would emit. The
frontend can warm all views with a single request, seed its per-view cache from
the slices, and later revalidate each view individually with If-None-Match. The
bundle's own etag combines the slices', so an unchanged bundle is an empty 304.
prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard
guarantee: it never spends weather-API quota. A cell with no cached archive
answers 204, and only the history-derived slices (calendar + latest-day detail)
are built. At most one Nominatim lookup for a never-labeled cell.
* Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch
The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota,
which multi-year calendar payloads regularly blew through) to IndexedDB, with an
in-memory map in front. Entries carry the server's ETag, so anything stale
revalidates conditionally — unchanged data costs an empty 304 and a re-stamp,
never a re-transfer. On network failure the stale copy is served over an error.
getJSON gains an optional onUpdate callback opting into stale-while-revalidate:
the three views (weekly, day, calendar) now render a cached copy immediately —
spinners are delayed 150ms so warm loads never flash-blank — and repaint only if
background revalidation finds changed data. New data shows up the moment it
exists instead of waiting out a TTL.
Cross-view prefetch collapses from one request per view to a single /api/v2/cell
bundle, whose slices (exact per-view payloads + their etags) are seeded under the
URLs each view actually requests; the bundle call itself is conditional via a
remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side
with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one
possible Nominatim lookup each), so tapping nearby lands on already-graded data.
Legacy tg:* storage entries are cleared once; cache entries untouched for two
weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
|
|
|
|
|
|
|
|
|
|
# The bundle's identity is the combination of its slices' identities.
|
|
|
|
|
|
etag = _etag_for("cell", cid, "bundle" + (":p" if prefetch else ""),
|
|
|
|
|
|
"|".join(s["etag"] for s in slices.values()))
|
|
|
|
|
|
if _not_modified(request, etag):
|
|
|
|
|
|
run.set(run_type="cache", not_modified=True)
|
|
|
|
|
|
return Response(status_code=304, headers={"ETag": etag})
|
|
|
|
|
|
run.set(run_type="bundle", slices=sorted(slices))
|
|
|
|
|
|
payload = {
|
2026-07-11 00:29:47 +00:00
|
|
|
|
"api_version": "v2",
|
|
|
|
|
|
"cell": cell,
|
|
|
|
|
|
"place": place,
|
Migrate backend dataframe layer from pandas to polars (#90)
* Migrate backend dataframe layer from pandas to polars
Replace pandas with polars across the backend, dropping both pandas and its
pyarrow parquet engine from the dependency set. numpy stays (the grading
percentile math is unchanged).
- climate.py: parquet IO, source→frame mappings, cache read/topup on polars.
New _normalize_read casts the cached `date` column to pl.Date (older files
were written by pandas as datetime64[ns]); frames now unify missing values as
null so the grading boundary drops them consistently across sources.
- grading.py: keep the numpy percentile core; swap the frame→numpy bridge to
.to_numpy()/.drop_nulls(), day-of-year/year to polars dt expressions, and the
per-row loop to iter_rows(named=True).
- views.py: filter/anti-join/concat replace boolean-mask, isin and pd.concat;
scalar dates are stdlib datetime.date; a local _months_before helper replaces
DateOffset(months=) for the calendar-range default.
- app.py, migrate.py: request-date parsing uses datetime.date, removing pandas
from the endpoint and migrate layers entirely.
- The date column is pl.Date end to end, eliminating the pandas normalize() calls
and comparing cleanly against stdlib dates.
Payloads are unchanged: calendar, day, grade and forecast responses are
byte-for-byte identical to the pandas implementation on the same cached record.
Tests ported to polars fixtures, with added coverage for the combined feels-like
fallback, calendar month-offset (month-end/leap), and the concat/dedup
"fresher source wins" rule.
* Port notify.py to polars after merging dev's account system
Merge origin/dev (accounts + notification subscriptions) and carry the pandas→
polars migration into the newly added notify.py, which the merge brought in still
using pandas — with pandas removed from requirements this broke its import.
- notify.py: _candidate_rows filters/sorts the recent bundle with polars
expressions and returns iter_rows dicts; date scalars are datetime.date;
history/recent emptiness via is_empty().
- test_notify.py: synthetic history/rows built with polars + datetime.
2026-07-15 19:07:38 +00:00
|
|
|
|
"today": today.isoformat(),
|
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21)
* Compress API responses and revalidate static assets instead of re-downloading
- Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x.
- Serve pages/assets with Cache-Control: no-cache instead of no-store, so
browsers revalidate via the ETag/Last-Modified that FileResponse and
StaticFiles already emit. Unchanged assets now cost an empty 304 rather
than a full transfer on every page navigation, while deploys still show
up immediately.
* Persist derived responses in SQLite so grading is computed once per cell, not per request
New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet
records — finished grade/calendar/day/forecast payloads and reverse-geocode
labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms)
becomes a ~5ms database read that survives restarts and is shared across views.
Raw parquet stays the source of truth; the store is a pure accelerator (every
reader falls back to recomputing on a miss, and deleting the db is a safe reset).
Freshness is token-driven, not clock-driven: each cached payload is validated by
a token encoding what it was computed from (payload schema version, the archive
record's end date, the recent-fetch stamp). The existing freshness drivers are
untouched — get_history still tops up the tail hourly and get_recent_forecast
still refetches hourly — and tokens are derived from what they return, so cached
payloads expire exactly when their inputs change. The same tokens double as weak
ETags: If-None-Match answers with an empty 304 without touching the payload.
- backend/store.py: derived-payload + revgeo tables, thread-local WAL conns,
every helper fail-soft.
- app.py: endpoints split into pure payload builders + HTTP/caching shells; the
in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store).
- climate.py: revgeo persisted through the store; recent_stamp() and
load_cached_history() (no-network read) helpers.
- backend/migrate.py + make migrate: idempotent, resumable backfill of the store
from existing parquet caches (default calendar span + latest-day detail +
revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather.
- grid.from_id(): rebuild a cell from its cache filename (migrate tooling).
* Add /api/v2/cell: one bundle carrying every view's payload
GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months)
and day (today) payloads in one response, each the exact payload its per-view
endpoint returns — built by the same builders and cached under the same
derived-store keys/tokens — paired with the etag that endpoint would emit. The
frontend can warm all views with a single request, seed its per-view cache from
the slices, and later revalidate each view individually with If-None-Match. The
bundle's own etag combines the slices', so an unchanged bundle is an empty 304.
prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard
guarantee: it never spends weather-API quota. A cell with no cached archive
answers 204, and only the history-derived slices (calendar + latest-day detail)
are built. At most one Nominatim lookup for a never-labeled cell.
* Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch
The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota,
which multi-year calendar payloads regularly blew through) to IndexedDB, with an
in-memory map in front. Entries carry the server's ETag, so anything stale
revalidates conditionally — unchanged data costs an empty 304 and a re-stamp,
never a re-transfer. On network failure the stale copy is served over an error.
getJSON gains an optional onUpdate callback opting into stale-while-revalidate:
the three views (weekly, day, calendar) now render a cached copy immediately —
spinners are delayed 150ms so warm loads never flash-blank — and repaint only if
background revalidation finds changed data. New data shows up the moment it
exists instead of waiting out a TTL.
Cross-view prefetch collapses from one request per view to a single /api/v2/cell
bundle, whose slices (exact per-view payloads + their etags) are seeded under the
URLs each view actually requests; the bundle call itself is conditional via a
remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side
with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one
possible Nominatim lookup each), so tapping nearby lands on already-graded data.
Legacy tg:* storage entries are cleared once; cache entries untouched for two
weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
|
|
|
|
"slices": slices,
|
2026-07-11 00:29:47 +00:00
|
|
|
|
}
|
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21)
* Compress API responses and revalidate static assets instead of re-downloading
- Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x.
- Serve pages/assets with Cache-Control: no-cache instead of no-store, so
browsers revalidate via the ETag/Last-Modified that FileResponse and
StaticFiles already emit. Unchanged assets now cost an empty 304 rather
than a full transfer on every page navigation, while deploys still show
up immediately.
* Persist derived responses in SQLite so grading is computed once per cell, not per request
New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet
records — finished grade/calendar/day/forecast payloads and reverse-geocode
labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms)
becomes a ~5ms database read that survives restarts and is shared across views.
Raw parquet stays the source of truth; the store is a pure accelerator (every
reader falls back to recomputing on a miss, and deleting the db is a safe reset).
Freshness is token-driven, not clock-driven: each cached payload is validated by
a token encoding what it was computed from (payload schema version, the archive
record's end date, the recent-fetch stamp). The existing freshness drivers are
untouched — get_history still tops up the tail hourly and get_recent_forecast
still refetches hourly — and tokens are derived from what they return, so cached
payloads expire exactly when their inputs change. The same tokens double as weak
ETags: If-None-Match answers with an empty 304 without touching the payload.
- backend/store.py: derived-payload + revgeo tables, thread-local WAL conns,
every helper fail-soft.
- app.py: endpoints split into pure payload builders + HTTP/caching shells; the
in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store).
- climate.py: revgeo persisted through the store; recent_stamp() and
load_cached_history() (no-network read) helpers.
- backend/migrate.py + make migrate: idempotent, resumable backfill of the store
from existing parquet caches (default calendar span + latest-day detail +
revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather.
- grid.from_id(): rebuild a cell from its cache filename (migrate tooling).
* Add /api/v2/cell: one bundle carrying every view's payload
GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months)
and day (today) payloads in one response, each the exact payload its per-view
endpoint returns — built by the same builders and cached under the same
derived-store keys/tokens — paired with the etag that endpoint would emit. The
frontend can warm all views with a single request, seed its per-view cache from
the slices, and later revalidate each view individually with If-None-Match. The
bundle's own etag combines the slices', so an unchanged bundle is an empty 304.
prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard
guarantee: it never spends weather-API quota. A cell with no cached archive
answers 204, and only the history-derived slices (calendar + latest-day detail)
are built. At most one Nominatim lookup for a never-labeled cell.
* Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch
The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota,
which multi-year calendar payloads regularly blew through) to IndexedDB, with an
in-memory map in front. Entries carry the server's ETag, so anything stale
revalidates conditionally — unchanged data costs an empty 304 and a re-stamp,
never a re-transfer. On network failure the stale copy is served over an error.
getJSON gains an optional onUpdate callback opting into stale-while-revalidate:
the three views (weekly, day, calendar) now render a cached copy immediately —
spinners are delayed 150ms so warm loads never flash-blank — and repaint only if
background revalidation finds changed data. New data shows up the moment it
exists instead of waiting out a TTL.
Cross-view prefetch collapses from one request per view to a single /api/v2/cell
bundle, whose slices (exact per-view payloads + their etags) are seeded under the
URLs each view actually requests; the bundle call itself is conditional via a
remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side
with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one
possible Nominatim lookup each), so tapping nearby lands on already-graded data.
Legacy tg:* storage entries are cleared once; cache entries untouched for two
weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
|
|
|
|
# Not stored as its own derived row — the slices already are.
|
|
|
|
|
|
return _json_response(_encode(payload), etag)
|
2026-07-11 00:29:47 +00:00
|
|
|
|
|
|
|
|
|
|
|
Add ops metrics endpoint + SSH/Termux dashboard (#131)
Adds observability for the running app, viewable over SSH (e.g. Termux):
- backend/metrics.py: thread-safe, best-effort in-process counters for inbound
requests (per category) and outbound calls (per external source), plus a
phase->source map and snapshot(). Hooked into climate._request (outbound, all
six weather/geocode sources) and the existing revalidate_static middleware
(inbound). Never raises into the request path.
- GET /api/v2/metrics: token-gated JSON of the counters + warm-queue depth +
thread names/uptime/pid. Returns 404 unless a valid THERMOGRAPH_METRICS_TOKEN
is presented or the request is direct loopback with no proxy X-Forwarded-*
header, so ops data is never exposed through Caddy on the public domain.
- scripts/dashboard.py (+ scripts/dash, make dashboard): stdlib-only terminal
dashboard. Reads cache (parquet + SQLite), accounts (users + active
subscriptions), and system resources (/proc + systemd) directly, and pulls
traffic from the metrics endpoint over loopback. Live refresh, --once, --json;
mobile-terminal width. Paths resolve relative to the script so the same tool
works on the LAN dev and prod VPS checkouts.
- Monitoring sections in DEPLOY.md / DEPLOY-DEV.md; unit tests for the counters.
2026-07-16 20:46:59 +00:00
|
|
|
|
# --- ops metrics (gated) ---------------------------------------------------
|
|
|
|
|
|
def _metrics_allowed(request: Request) -> bool:
|
|
|
|
|
|
"""Guard the ops metrics: a valid token allows from anywhere (for SSH tunnels);
|
|
|
|
|
|
with no token it's direct-loopback-only, and any forwarded header (i.e. proxied
|
|
|
|
|
|
via Caddy) is refused — so prod ops data never leaks through the public domain."""
|
|
|
|
|
|
token = os.environ.get("THERMOGRAPH_METRICS_TOKEN", "").strip()
|
|
|
|
|
|
if token:
|
|
|
|
|
|
supplied = request.headers.get("x-metrics-token") or request.query_params.get("token") or ""
|
|
|
|
|
|
if hmac.compare_digest(supplied, token):
|
|
|
|
|
|
return True
|
|
|
|
|
|
client = request.client.host if request.client else None
|
|
|
|
|
|
forwarded = any(h in request.headers for h in
|
|
|
|
|
|
("x-forwarded-for", "x-forwarded-host", "x-forwarded-proto"))
|
|
|
|
|
|
return client in ("127.0.0.1", "::1") and not forwarded
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def api_metrics(request: Request):
|
|
|
|
|
|
"""Live in-process traffic counters + worker facts for the ops dashboard."""
|
|
|
|
|
|
if not _metrics_allowed(request):
|
|
|
|
|
|
raise HTTPException(status_code=404) # 404, not 403 — don't reveal the route
|
|
|
|
|
|
snap = metrics.snapshot()
|
|
|
|
|
|
snap["warm_queue_depth"] = _warm_queue.qsize()
|
|
|
|
|
|
snap["warm_seen"] = len(_WARM_SEEN)
|
|
|
|
|
|
snap["threads"] = threading.active_count()
|
|
|
|
|
|
snap["thread_names"] = sorted(t.name for t in threading.enumerate())
|
|
|
|
|
|
return snap
|
|
|
|
|
|
|
|
|
|
|
|
|
Rebuild the homepage as a distribution landing; add an SMTP seam (#178)
The homepage was a bare tool: a find-bar and an empty panel reading "Find a
location to begin." A visitor arriving from a search result or a shared link
learned nothing about what the product does before deciding to leave.
Rebuild it around the Weekly view, which is untouched:
- Hero with the question as the h1, and a grade card showing a real graded
example — the most unusual city we're currently tracking, either tail. The
card's frame and text slots are server-rendered with reserved heights, so
app.js re-pointing it at the visitor's own place shifts nothing.
- "Unusual right now" strip, CSS scroll-snap, no JS carousel. A cold-tail city
is force-included whenever one qualifies.
- Stance line, how-it-works, explore cards, and 12 city chips linking into the
~1000-page /climate surface.
- Monthly digest form, in the footer of every page.
Serve / from Jinja instead of a static file with placeholder substitution, so
crawlers and no-JS readers get the whole page as real HTML. home.html.j2
extends base.html.j2 and carries app.js's DOM contract over verbatim; the brand
degrades to a <p> so the hero owns the sole h1. frontend/index.html is deleted
rather than left behind the static mount, where it would keep serving indexable
duplicate content.
"Where is it most unusual right now" has no cheap answer at request time —
percentiles live inside zlib-compressed payload blobs with no column to sort on.
So homepage.py sweeps the warm cache and writes data/homepage.json, read by the
template. The sweep is strictly cache-only (climate.load_cached_recent_forecast
is new, the sibling of load_cached_history), so grading ~1000 cities costs zero
upstream requests. It rides the notifier's timer behind an hourly guard rather
than starting a second daemon, and also runs at the tail of warm_cities so a
fresh deploy has a populated feed.
Instrumentation: metrics gains a product-event counter keyed by (event,
referrer domain, UTC day), behind an allowlist and a per-IP rate limit, fed by
POST /api/v2/event. The referrer is taken from the request's own header, never
from the client. The beacon gets its own inbound category that record_inbound
ignores, so reporting an interaction doesn't also count as traffic. The
dashboard grows an events block with per-referrer attribution.
Email is scaffolded but sends nothing yet. mailer.py talks stdlib smtplib to a
local Postfix null client on 127.0.0.1:25 (deploy/provision-mail.sh), so the
choice between direct-to-MX and relaying through a provider stays a Postfix
config change with no code change. The backend defaults to "console", which
logs and sends nothing, so dev and tests exercise the whole signup path safely.
Signups land in pending_digest unconfirmed; collecting the list shouldn't wait
on delivery.
Also adds /privacy, linked from the footer and kept out of the sitemap.
The strip's classes are named unusual-* rather than record-*: .record-card is
already the SEO records page's, and reusing it leaked layout rules onto
/climate/<slug>/records.
2026-07-18 07:39:47 +00:00
|
|
|
|
async def api_event(request: Request) -> Response:
|
|
|
|
|
|
"""Record one product event (a tap on "use my location", a digest signup, …).
|
|
|
|
|
|
|
|
|
|
|
|
Deliberately minimal: the body carries only an allowlisted event name. The
|
|
|
|
|
|
referrer is read from this request's own Referer header — a client-supplied
|
|
|
|
|
|
one would be forgeable and buys nothing — and is reduced to a bare domain.
|
|
|
|
|
|
No cookies, no per-visitor id, no full URLs.
|
|
|
|
|
|
|
|
|
|
|
|
Always answers 204, whether the event was counted, rejected by the allowlist,
|
|
|
|
|
|
or dropped by the rate limiter, so probing the endpoint reveals nothing. That
|
|
|
|
|
|
also suits navigator.sendBeacon, which ignores the response.
|
|
|
|
|
|
"""
|
|
|
|
|
|
name = ""
|
|
|
|
|
|
try:
|
|
|
|
|
|
body = await request.json()
|
|
|
|
|
|
if isinstance(body, dict):
|
|
|
|
|
|
name = str(body.get("event") or "")
|
|
|
|
|
|
except Exception: # noqa: BLE001 - a junk body is just a no-op event
|
|
|
|
|
|
pass
|
|
|
|
|
|
if name:
|
|
|
|
|
|
metrics.record_event(
|
|
|
|
|
|
name,
|
|
|
|
|
|
referer=request.headers.get("referer"),
|
|
|
|
|
|
host=request.headers.get("host"),
|
|
|
|
|
|
ip=_client_ip(request),
|
|
|
|
|
|
)
|
|
|
|
|
|
return Response(status_code=204)
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-07-11 00:29:47 +00:00
|
|
|
|
# --- API versioning --------------------------------------------------------
|
|
|
|
|
|
# Non-backward-compatible additions land in a new version; every version stays
|
|
|
|
|
|
# mounted and served simultaneously. v1 is the original contract (grade/geocode);
|
|
|
|
|
|
# v2 adds the calendar. The unversioned /api/* paths are kept as aliases of v1 so
|
|
|
|
|
|
# existing clients keep working.
|
|
|
|
|
|
v1 = APIRouter(tags=["v1"])
|
|
|
|
|
|
v1.add_api_route("/geocode", api_geocode, methods=["GET"])
|
|
|
|
|
|
v1.add_api_route("/grade", api_grade, methods=["GET"])
|
|
|
|
|
|
|
|
|
|
|
|
v2 = APIRouter(tags=["v2"])
|
|
|
|
|
|
v2.add_api_route("/geocode", api_geocode, methods=["GET"])
|
Typo-tolerant location search suggestions (#33)
Add /api/v2/suggest and wire the location picker's search box to it as a
debounced type-ahead: top-5 place suggestions that tolerate a single-letter
typo (substituted, missing, or extra letter, or two adjacent letters swapped)
anywhere in the query, including the first character.
- backend/places.py: local place index built from a GeoNames cities dump
(downloaded once into data/geonames/, loaded in a background thread; the
app boots and serves without it). Exact-prefix matches rank first by
population, then names one edit away; a token vocabulary respells one
mistyped word against known place-name tokens ("pest seattle" ->
"west seattle") for retry against the upstream geocoder, which covers
neighbourhood-level places the dump lacks. THERMOGRAPH_CITIES picks the
dump (default cities1000).
- /suggest blends local and upstream results by population with an exactness
boost, so "Seatle" (a Cumbrian hamlet) can't outrank Seattle when the
query is one edit from the city, while "munchen" still surfaces Munich via
upstream (the index only knows English names). Upstream lookups are
memoized and skipped entirely when the index answers convincingly.
- mappicker.js: debounced (250ms) live suggestions with abort + sequence
guards against stale responses, arrow-key navigation, Enter-picks-highlight,
Escape dismissing the list before closing the overlay. Submit goes through
the same typo-tolerant endpoint.
- climate.geocode results now carry population (used for ranking).
2026-07-11 15:36:14 +00:00
|
|
|
|
v2.add_api_route("/suggest", api_suggest, methods=["GET"])
|
2026-07-11 14:50:48 +00:00
|
|
|
|
v2.add_api_route("/place", api_place, methods=["GET"])
|
2026-07-11 00:29:47 +00:00
|
|
|
|
v2.add_api_route("/grade", api_grade, methods=["GET"])
|
|
|
|
|
|
v2.add_api_route("/calendar", api_calendar, methods=["GET"])
|
|
|
|
|
|
v2.add_api_route("/day", api_day, methods=["GET"])
|
Add a climate-score page from recent-vs-baseline percentile divergence (#196)
Score how far a location's last 6 years have drifted from its full 45-year
record. For each metric and percentile category (p10/p25/p50/p75/p90), the
recent-years value is placed on the baseline distribution and the gap from the
expected percentile is the divergence — unit-free, so metrics compare directly.
Scored per meteorological season plus annual, weighted into per-metric and
overall scores (temps, humidity and feels-like weighted heaviest).
- backend/scoring.py: divergence math, seasonal slicing, precip zero-inflation
split (wet-day frequency + amount), tier mapping onto the existing temp scale.
- climate.py: derive a wet-bulb column (Stull 2011) at the read boundary, before
the humidity column is converted to absolute — via a shared _derive_metrics
wrapper at all four read sites.
- api/v2/score endpoint + build_score payload, cached on the history token with
a scoring-version key.
- frontend score page: overall hero, per-metric cards, by-season chips, and a
button-revealed summary (sentences + metrics×season table + per-percentile
detail). Score nav link across all headers.
- Tests for the scoring math, wet-bulb formula, payload shape and route.
2026-07-19 23:02:33 +00:00
|
|
|
|
v2.add_api_route("/score", api_score, methods=["GET"])
|
2026-07-11 00:29:47 +00:00
|
|
|
|
v2.add_api_route("/forecast", api_forecast, methods=["GET"])
|
Persistent derived-data cache: SQLite store, ETag revalidation, view bundle, IndexedDB frontend (#21)
* Compress API responses and revalidate static assets instead of re-downloading
- Add GZipMiddleware (min 1 KB): the 2-year calendar JSON shrinks ~6-8x.
- Serve pages/assets with Cache-Control: no-cache instead of no-store, so
browsers revalidate via the ETag/Last-Modified that FileResponse and
StaticFiles already emit. Unchanged assets now cost an empty 304 rather
than a full transfer on every page navigation, while deploys still show
up immediately.
* Persist derived responses in SQLite so grading is computed once per cell, not per request
New data/thermograph.sqlite (WAL) holds what's derived from the raw parquet
records — finished grade/calendar/day/forecast payloads and reverse-geocode
labels — so the expensive work (notably the 2-year calendar grade_range, ~270ms)
becomes a ~5ms database read that survives restarts and is shared across views.
Raw parquet stays the source of truth; the store is a pure accelerator (every
reader falls back to recomputing on a miss, and deleting the db is a safe reset).
Freshness is token-driven, not clock-driven: each cached payload is validated by
a token encoding what it was computed from (payload schema version, the archive
record's end date, the recent-fetch stamp). The existing freshness drivers are
untouched — get_history still tops up the tail hourly and get_recent_forecast
still refetches hourly — and tokens are derived from what they return, so cached
payloads expire exactly when their inputs change. The same tokens double as weak
ETags: If-None-Match answers with an empty 304 without touching the payload.
- backend/store.py: derived-payload + revgeo tables, thread-local WAL conns,
every helper fail-soft.
- app.py: endpoints split into pure payload builders + HTTP/caching shells; the
in-memory 10-minute _CAL_CACHE is retired (superseded by the persistent store).
- climate.py: revgeo persisted through the store; recent_stamp() and
load_cached_history() (no-network read) helpers.
- backend/migrate.py + make migrate: idempotent, resumable backfill of the store
from existing parquet caches (default calendar span + latest-day detail +
revgeo, ≤1 throttled Nominatim call per unlabeled cell). Never fetches weather.
- grid.from_id(): rebuild a cell from its cache filename (migrate tooling).
* Add /api/v2/cell: one bundle carrying every view's payload
GET /api/v2/cell?lat&lon returns the grade, forecast, calendar (last 24 months)
and day (today) payloads in one response, each the exact payload its per-view
endpoint returns — built by the same builders and cached under the same
derived-store keys/tokens — paired with the etag that endpoint would emit. The
frontend can warm all views with a single request, seed its per-view cache from
the slices, and later revalidate each view individually with If-None-Match. The
bundle's own etag combines the slices', so an unchanged bundle is an empty 304.
prefetch=1 is a warm-only mode for neighbor-cell prefetching with a hard
guarantee: it never spends weather-API quota. A cell with no cached archive
answers 204, and only the history-derived slices (calendar + latest-day detail)
are built. At most one Nominatim lookup for a never-labeled cell.
* Frontend: IndexedDB response cache with stale-while-revalidate + bundle prefetch
The per-URL response cache moves from localStorage/sessionStorage (~5 MB quota,
which multi-year calendar payloads regularly blew through) to IndexedDB, with an
in-memory map in front. Entries carry the server's ETag, so anything stale
revalidates conditionally — unchanged data costs an empty 304 and a re-stamp,
never a re-transfer. On network failure the stale copy is served over an error.
getJSON gains an optional onUpdate callback opting into stale-while-revalidate:
the three views (weekly, day, calendar) now render a cached copy immediately —
spinners are delayed 150ms so warm loads never flash-blank — and repaint only if
background revalidation finds changed data. New data shows up the moment it
exists instead of waiting out a TTL.
Cross-view prefetch collapses from one request per view to a single /api/v2/cell
bundle, whose slices (exact per-view payloads + their etags) are seeded under the
URLs each view actually requests; the bundle call itself is conditional via a
remembered etag. Afterwards the 8 surrounding grid cells are warmed server-side
with prefetch=1 (never spends weather-API quota; staggered ≥1.1s for the one
possible Nominatim lookup each), so tapping nearby lands on already-graded data.
Legacy tg:* storage entries are cleared once; cache entries untouched for two
weeks are pruned on page load.
2026-07-11 07:31:28 +00:00
|
|
|
|
v2.add_api_route("/cell", api_cell, methods=["GET"])
|
Add ops metrics endpoint + SSH/Termux dashboard (#131)
Adds observability for the running app, viewable over SSH (e.g. Termux):
- backend/metrics.py: thread-safe, best-effort in-process counters for inbound
requests (per category) and outbound calls (per external source), plus a
phase->source map and snapshot(). Hooked into climate._request (outbound, all
six weather/geocode sources) and the existing revalidate_static middleware
(inbound). Never raises into the request path.
- GET /api/v2/metrics: token-gated JSON of the counters + warm-queue depth +
thread names/uptime/pid. Returns 404 unless a valid THERMOGRAPH_METRICS_TOKEN
is presented or the request is direct loopback with no proxy X-Forwarded-*
header, so ops data is never exposed through Caddy on the public domain.
- scripts/dashboard.py (+ scripts/dash, make dashboard): stdlib-only terminal
dashboard. Reads cache (parquet + SQLite), accounts (users + active
subscriptions), and system resources (/proc + systemd) directly, and pulls
traffic from the metrics endpoint over loopback. Live refresh, --once, --json;
mobile-terminal width. Paths resolve relative to the script so the same tool
works on the LAN dev and prod VPS checkouts.
- Monitoring sections in DEPLOY.md / DEPLOY-DEV.md; unit tests for the counters.
2026-07-16 20:46:59 +00:00
|
|
|
|
v2.add_api_route("/metrics", api_metrics, methods=["GET"], include_in_schema=False)
|
Rebuild the homepage as a distribution landing; add an SMTP seam (#178)
The homepage was a bare tool: a find-bar and an empty panel reading "Find a
location to begin." A visitor arriving from a search result or a shared link
learned nothing about what the product does before deciding to leave.
Rebuild it around the Weekly view, which is untouched:
- Hero with the question as the h1, and a grade card showing a real graded
example — the most unusual city we're currently tracking, either tail. The
card's frame and text slots are server-rendered with reserved heights, so
app.js re-pointing it at the visitor's own place shifts nothing.
- "Unusual right now" strip, CSS scroll-snap, no JS carousel. A cold-tail city
is force-included whenever one qualifies.
- Stance line, how-it-works, explore cards, and 12 city chips linking into the
~1000-page /climate surface.
- Monthly digest form, in the footer of every page.
Serve / from Jinja instead of a static file with placeholder substitution, so
crawlers and no-JS readers get the whole page as real HTML. home.html.j2
extends base.html.j2 and carries app.js's DOM contract over verbatim; the brand
degrades to a <p> so the hero owns the sole h1. frontend/index.html is deleted
rather than left behind the static mount, where it would keep serving indexable
duplicate content.
"Where is it most unusual right now" has no cheap answer at request time —
percentiles live inside zlib-compressed payload blobs with no column to sort on.
So homepage.py sweeps the warm cache and writes data/homepage.json, read by the
template. The sweep is strictly cache-only (climate.load_cached_recent_forecast
is new, the sibling of load_cached_history), so grading ~1000 cities costs zero
upstream requests. It rides the notifier's timer behind an hourly guard rather
than starting a second daemon, and also runs at the tail of warm_cities so a
fresh deploy has a populated feed.
Instrumentation: metrics gains a product-event counter keyed by (event,
referrer domain, UTC day), behind an allowlist and a per-IP rate limit, fed by
POST /api/v2/event. The referrer is taken from the request's own header, never
from the client. The beacon gets its own inbound category that record_inbound
ignores, so reporting an interaction doesn't also count as traffic. The
dashboard grows an events block with per-referrer attribution.
Email is scaffolded but sends nothing yet. mailer.py talks stdlib smtplib to a
local Postfix null client on 127.0.0.1:25 (deploy/provision-mail.sh), so the
choice between direct-to-MX and relaying through a provider stays a Postfix
config change with no code change. The backend defaults to "console", which
logs and sends nothing, so dev and tests exercise the whole signup path safely.
Signups land in pending_digest unconfirmed; collecting the list shouldn't wait
on delivery.
Also adds /privacy, linked from the footer and kept out of the sitemap.
The strip's classes are named unusual-* rather than record-*: .record-card is
already the SEO records page's, and reusing it leaked layout rules onto
/climate/<slug>/records.
2026-07-18 07:39:47 +00:00
|
|
|
|
v2.add_api_route("/event", api_event, methods=["POST"], include_in_schema=False)
|
2026-07-11 00:29:47 +00:00
|
|
|
|
|
|
|
|
|
|
app.include_router(v1, prefix=f"{BASE}/api") # legacy unversioned == v1 (backward compatible)
|
|
|
|
|
|
app.include_router(v1, prefix=f"{BASE}/api/v1")
|
|
|
|
|
|
app.include_router(v2, prefix=f"{BASE}/api/v2")
|
|
|
|
|
|
|
2026-07-20 02:19:29 +00:00
|
|
|
|
|
|
|
|
|
|
# --- Discord slash commands (HTTP interactions) ----------------------------
|
|
|
|
|
|
async def discord_interactions(request: Request) -> Response:
|
|
|
|
|
|
"""Discord posts each slash-command interaction here. Verification runs over the
|
|
|
|
|
|
RAW body, so this must read bytes, not request.json()."""
|
|
|
|
|
|
raw = await request.body()
|
|
|
|
|
|
status, payload = discord_interactions_mod.handle(
|
|
|
|
|
|
raw,
|
|
|
|
|
|
request.headers.get("x-signature-ed25519", ""),
|
|
|
|
|
|
request.headers.get("x-signature-timestamp", ""),
|
|
|
|
|
|
)
|
|
|
|
|
|
return Response(json.dumps(payload), status_code=status, media_type="application/json")
|
|
|
|
|
|
|
|
|
|
|
|
app.add_api_route(f"{BASE}/discord/interactions", discord_interactions,
|
|
|
|
|
|
methods=["POST"], include_in_schema=False)
|
|
|
|
|
|
|
Account system with weather-notification subscriptions (#89)
* Add account system foundation: email/password auth with cookie sessions
Introduce the app's first authoritative, user-owned data in a separate
data/accounts.sqlite (SQLAlchemy), kept apart from the disposable derived-cache
DB. Wire fastapi-users for email/password signup, cookie-based login/logout, and
a session-check endpoint, backed by a database session strategy so logins survive
restarts and are revocable.
- db.py: async (aiosqlite) + sync SQLAlchemy engines over accounts.sqlite, WAL +
foreign keys, create_db_and_tables().
- models.py: User, AccessToken, Subscription, Notification tables.
- users.py: pwdlib hashing, HttpOnly cookie transport (path-scoped, SameSite=Lax,
Secure via env), DatabaseStrategy sessions, current-user dependencies.
- schemas.py: user + subscription + notification Pydantic models.
- app.py: mount auth/register/users routers on v2, create tables at startup.
- Pin fastapi-users[sqlalchemy]/aiosqlite; ignore data/accounts.sqlite*.
* Add account header entry and auth modal (frontend)
account.js self-injects a header entry (following the units.js pattern) that
shows a Sign in button when logged out and an account menu when logged in, plus
an auth modal reusing the existing .mp-overlay/.mp-modal chrome for email/password
sign-in and account creation. A shared apiFetch helper sends the same-origin
cookie for authed calls; exported getUser/openAuth/onAuthChange back later phases.
Imported by every page entry module. On narrow screens the entry collapses to an
icon-only button so it doesn't crowd the title.
Enforce an 8-character minimum password in the user manager.
* Add subscription CRUD API and the alerts management page
Backend api_accounts.py adds user-scoped, cookie-authenticated endpoints to
create/list/update/delete subscriptions (and the notification reads used next):
POST snaps lat/lon to a grid cell, resolves a label, and rejects a duplicate
location+kind with 409; PATCH/DELETE are ownership-checked (404 on mismatch).
Mounted on the v2 prefix.
Frontend subscriptions.js + subscriptions.html serve the /alerts page: a sign-in
gate when logged out, an add flow that reuses the shared map picker and an editor
modal (kind, watched metrics, 95-99 percentile, two-sided), and a card list with
inline threshold/active edits and remove. Reachable from the account menu.
* Add background subscription evaluation engine
notify.py runs a daemon thread that periodically evaluates every active
subscription: it groups them by grid cell, reads history from the parquet cache
only (never spends archive quota) plus the hourly recent/forecast bundle, and
grades candidate days with the existing grading.grade_day. A watched metric that
lands at or beyond the threshold percentile fires a 'high' alert; a two-sided
subscription also fires 'low' for the symmetric cold/calm/dry tail (precip stays
one-directional). Observed subscriptions look at the last few recorded days,
forecast subscriptions at the coming week.
Two guards keep it quiet: a UNIQUE(subscription, event_date, metric, direction,
kind) constraint dedups repeat events, and a per-subscription weekly cap
(last_notified_at) limits each alert to one notification per 7 days. The loop
tolerates a bad cell or an upstream rate limit without aborting the pass. Started
and stopped from the app lifespan; gated by THERMOGRAPH_ENABLE_NOTIFIER.
* Add in-app notification center (header bell)
Extend account.js with a notification bell beside the account menu: an unread
badge, a dropdown listing recent notifications (title, body, relative time), a
per-item mark-read on click, and a Mark all read action, all through the
cookie-authed notifications API. Unread state refreshes on open and polls every
two minutes while signed in; polling stops on sign-out. Styled to match the app,
responsive down to mobile.
* Harden accounts: expired-session cleanup, engine tests, ops docs
- notify.py sweeps expired login sessions (access tokens past their lifetime)
once per evaluation pass.
- Add hermetic unit tests for the evaluation engine's trigger detection (high/low
tails, precip one-directional, normal = no trigger) and notification wording.
- Document accounts.sqlite (authoritative, back it up), the single-worker
requirement for the in-process evaluator, and the new env vars in DEPLOY.md.
2026-07-15 18:46:46 +00:00
|
|
|
|
# --- accounts (fastapi-users) ----------------------------------------------
|
|
|
|
|
|
# Library-provided routers: /auth/login + /auth/logout (cookie session),
|
|
|
|
|
|
# /auth/register (signup), and /users/me (session check / profile). All under the
|
|
|
|
|
|
# same v2 prefix as the rest of the API, so the frontend calls them base-relative.
|
|
|
|
|
|
app.include_router(
|
|
|
|
|
|
users.fastapi_users.get_auth_router(users.auth_backend),
|
|
|
|
|
|
prefix=f"{BASE}/api/v2/auth", tags=["auth"],
|
|
|
|
|
|
)
|
|
|
|
|
|
app.include_router(
|
|
|
|
|
|
users.fastapi_users.get_register_router(UserRead, UserCreate),
|
|
|
|
|
|
prefix=f"{BASE}/api/v2/auth", tags=["auth"],
|
|
|
|
|
|
)
|
|
|
|
|
|
app.include_router(
|
|
|
|
|
|
users.fastapi_users.get_users_router(UserRead, UserUpdate),
|
|
|
|
|
|
prefix=f"{BASE}/api/v2/users", tags=["users"],
|
|
|
|
|
|
)
|
|
|
|
|
|
# Subscriptions + notifications (authed, user-scoped).
|
|
|
|
|
|
app.include_router(api_accounts.router, prefix=f"{BASE}/api/v2")
|
2026-07-20 02:26:33 +00:00
|
|
|
|
# Discord account linking (OAuth2, authed).
|
|
|
|
|
|
app.include_router(discord_link.router, prefix=f"{BASE}/api/v2")
|
Account system with weather-notification subscriptions (#89)
* Add account system foundation: email/password auth with cookie sessions
Introduce the app's first authoritative, user-owned data in a separate
data/accounts.sqlite (SQLAlchemy), kept apart from the disposable derived-cache
DB. Wire fastapi-users for email/password signup, cookie-based login/logout, and
a session-check endpoint, backed by a database session strategy so logins survive
restarts and are revocable.
- db.py: async (aiosqlite) + sync SQLAlchemy engines over accounts.sqlite, WAL +
foreign keys, create_db_and_tables().
- models.py: User, AccessToken, Subscription, Notification tables.
- users.py: pwdlib hashing, HttpOnly cookie transport (path-scoped, SameSite=Lax,
Secure via env), DatabaseStrategy sessions, current-user dependencies.
- schemas.py: user + subscription + notification Pydantic models.
- app.py: mount auth/register/users routers on v2, create tables at startup.
- Pin fastapi-users[sqlalchemy]/aiosqlite; ignore data/accounts.sqlite*.
* Add account header entry and auth modal (frontend)
account.js self-injects a header entry (following the units.js pattern) that
shows a Sign in button when logged out and an account menu when logged in, plus
an auth modal reusing the existing .mp-overlay/.mp-modal chrome for email/password
sign-in and account creation. A shared apiFetch helper sends the same-origin
cookie for authed calls; exported getUser/openAuth/onAuthChange back later phases.
Imported by every page entry module. On narrow screens the entry collapses to an
icon-only button so it doesn't crowd the title.
Enforce an 8-character minimum password in the user manager.
* Add subscription CRUD API and the alerts management page
Backend api_accounts.py adds user-scoped, cookie-authenticated endpoints to
create/list/update/delete subscriptions (and the notification reads used next):
POST snaps lat/lon to a grid cell, resolves a label, and rejects a duplicate
location+kind with 409; PATCH/DELETE are ownership-checked (404 on mismatch).
Mounted on the v2 prefix.
Frontend subscriptions.js + subscriptions.html serve the /alerts page: a sign-in
gate when logged out, an add flow that reuses the shared map picker and an editor
modal (kind, watched metrics, 95-99 percentile, two-sided), and a card list with
inline threshold/active edits and remove. Reachable from the account menu.
* Add background subscription evaluation engine
notify.py runs a daemon thread that periodically evaluates every active
subscription: it groups them by grid cell, reads history from the parquet cache
only (never spends archive quota) plus the hourly recent/forecast bundle, and
grades candidate days with the existing grading.grade_day. A watched metric that
lands at or beyond the threshold percentile fires a 'high' alert; a two-sided
subscription also fires 'low' for the symmetric cold/calm/dry tail (precip stays
one-directional). Observed subscriptions look at the last few recorded days,
forecast subscriptions at the coming week.
Two guards keep it quiet: a UNIQUE(subscription, event_date, metric, direction,
kind) constraint dedups repeat events, and a per-subscription weekly cap
(last_notified_at) limits each alert to one notification per 7 days. The loop
tolerates a bad cell or an upstream rate limit without aborting the pass. Started
and stopped from the app lifespan; gated by THERMOGRAPH_ENABLE_NOTIFIER.
* Add in-app notification center (header bell)
Extend account.js with a notification bell beside the account menu: an unread
badge, a dropdown listing recent notifications (title, body, relative time), a
per-item mark-read on click, and a Mark all read action, all through the
cookie-authed notifications API. Unread state refreshes on open and polls every
two minutes while signed in; polling stops on sign-out. Styled to match the app,
responsive down to mobile.
* Harden accounts: expired-session cleanup, engine tests, ops docs
- notify.py sweeps expired login sessions (access tokens past their lifetime)
once per evaluation pass.
- Add hermetic unit tests for the evaluation engine's trigger detection (high/low
tails, precip one-directional, normal = no trigger) and notification wording.
- Document accounts.sqlite (authoritative, back it up), the single-worker
requirement for the in-process evaluator, and the new env vars in DEPLOY.md.
2026-07-15 18:46:46 +00:00
|
|
|
|
|
2026-07-11 00:29:47 +00:00
|
|
|
|
|
|
|
|
|
|
# --- static frontend (served under BASE) -----------------------------------
|
|
|
|
|
|
def _page(name):
|
2026-07-11 15:59:14 +00:00
|
|
|
|
"""Route handler for one HTML page. Serves the file with its __ORIGIN__
|
|
|
|
|
|
placeholders (the link-preview/Open Graph tags) filled in as the request's
|
|
|
|
|
|
scheme://host + BASE — preview crawlers (Discord, Slack, …) need absolute
|
|
|
|
|
|
URLs, and the host differs between LAN and prod. The scheme comes from
|
|
|
|
|
|
X-Forwarded-Proto when a reverse proxy (Caddy) fronts the plain-HTTP
|
|
|
|
|
|
uvicorn; the proxy passes the original Host header through untouched."""
|
|
|
|
|
|
path = os.path.join(FRONTEND_DIR, name)
|
|
|
|
|
|
|
|
|
|
|
|
def route(request: Request):
|
|
|
|
|
|
with open(path, encoding="utf-8") as f:
|
|
|
|
|
|
html = f.read()
|
|
|
|
|
|
proto = request.headers.get("x-forwarded-proto") or request.url.scheme
|
|
|
|
|
|
host = request.headers.get("host") or request.url.netloc
|
|
|
|
|
|
html = html.replace("__ORIGIN__", f"{proto}://{host}{BASE}")
|
2026-07-16 18:08:09 +00:00
|
|
|
|
# Search-engine verification <meta> tags (same source as the SEO pages), so
|
|
|
|
|
|
# the homepage Google/Bing verify against carries them too.
|
|
|
|
|
|
verify = content.head_verify_html()
|
|
|
|
|
|
if verify:
|
|
|
|
|
|
html = html.replace("<head>", f"<head>\n {verify}", 1)
|
2026-07-11 15:59:14 +00:00
|
|
|
|
etag = f'W/"{hashlib.sha1(html.encode()).hexdigest()[:20]}"'
|
|
|
|
|
|
if _not_modified(request, etag):
|
|
|
|
|
|
return Response(status_code=304, headers={"ETag": etag})
|
|
|
|
|
|
return Response(html, media_type="text/html", headers={"ETag": etag})
|
|
|
|
|
|
return route
|
2026-07-11 00:29:47 +00:00
|
|
|
|
|
|
|
|
|
|
|
2026-07-11 03:01:16 +00:00
|
|
|
|
# The un-slashed base redirects to BASE/ so the frontend's relative asset URLs
|
2026-07-15 19:58:32 +00:00
|
|
|
|
# resolve correctly. Under a sub-path this keeps the app scoped to BASE (never
|
|
|
|
|
|
# claiming "/", so the domain root stays free for another app via the proxy). When
|
|
|
|
|
|
# BASE is "" the app owns the root: "/" is already the index route below, so there
|
|
|
|
|
|
# is no bare base to redirect (and "" is not a valid route path).
|
2026-07-11 15:59:14 +00:00
|
|
|
|
# Pages also answer HEAD — link-preview crawlers probe with it before fetching.
|
2026-07-15 19:58:32 +00:00
|
|
|
|
if BASE:
|
|
|
|
|
|
app.add_api_route(BASE, lambda: RedirectResponse(url=f"{BASE}/"), methods=["GET", "HEAD"], include_in_schema=False)
|
Rebuild the homepage as a distribution landing; add an SMTP seam (#178)
The homepage was a bare tool: a find-bar and an empty panel reading "Find a
location to begin." A visitor arriving from a search result or a shared link
learned nothing about what the product does before deciding to leave.
Rebuild it around the Weekly view, which is untouched:
- Hero with the question as the h1, and a grade card showing a real graded
example — the most unusual city we're currently tracking, either tail. The
card's frame and text slots are server-rendered with reserved heights, so
app.js re-pointing it at the visitor's own place shifts nothing.
- "Unusual right now" strip, CSS scroll-snap, no JS carousel. A cold-tail city
is force-included whenever one qualifies.
- Stance line, how-it-works, explore cards, and 12 city chips linking into the
~1000-page /climate surface.
- Monthly digest form, in the footer of every page.
Serve / from Jinja instead of a static file with placeholder substitution, so
crawlers and no-JS readers get the whole page as real HTML. home.html.j2
extends base.html.j2 and carries app.js's DOM contract over verbatim; the brand
degrades to a <p> so the hero owns the sole h1. frontend/index.html is deleted
rather than left behind the static mount, where it would keep serving indexable
duplicate content.
"Where is it most unusual right now" has no cheap answer at request time —
percentiles live inside zlib-compressed payload blobs with no column to sort on.
So homepage.py sweeps the warm cache and writes data/homepage.json, read by the
template. The sweep is strictly cache-only (climate.load_cached_recent_forecast
is new, the sibling of load_cached_history), so grading ~1000 cities costs zero
upstream requests. It rides the notifier's timer behind an hourly guard rather
than starting a second daemon, and also runs at the tail of warm_cities so a
fresh deploy has a populated feed.
Instrumentation: metrics gains a product-event counter keyed by (event,
referrer domain, UTC day), behind an allowlist and a per-IP rate limit, fed by
POST /api/v2/event. The referrer is taken from the request's own header, never
from the client. The beacon gets its own inbound category that record_inbound
ignores, so reporting an interaction doesn't also count as traffic. The
dashboard grows an events block with per-referrer attribution.
Email is scaffolded but sends nothing yet. mailer.py talks stdlib smtplib to a
local Postfix null client on 127.0.0.1:25 (deploy/provision-mail.sh), so the
choice between direct-to-MX and relaying through a provider stays a Postfix
config change with no code change. The backend defaults to "console", which
logs and sends nothing, so dev and tests exercise the whole signup path safely.
Signups land in pending_digest unconfirmed; collecting the list shouldn't wait
on delivery.
Also adds /privacy, linked from the footer and kept out of the sitemap.
The strip's classes are named unusual-* rather than record-*: .record-card is
already the SEO records page's, and reusing it leaked layout rules onto
/climate/<slug>/records.
2026-07-18 07:39:47 +00:00
|
|
|
|
# NOTE: "/" is not here — the homepage is server-rendered by content.register()
|
|
|
|
|
|
# below, so its hero, records strip and city links reach crawlers and no-JS
|
|
|
|
|
|
# readers as real HTML.
|
2026-07-11 15:59:14 +00:00
|
|
|
|
app.add_api_route(f"{BASE}/calendar", _page("calendar.html"), methods=["GET", "HEAD"], include_in_schema=False)
|
|
|
|
|
|
app.add_api_route(f"{BASE}/day", _page("day.html"), methods=["GET", "HEAD"], include_in_schema=False)
|
Add a climate-score page from recent-vs-baseline percentile divergence (#196)
Score how far a location's last 6 years have drifted from its full 45-year
record. For each metric and percentile category (p10/p25/p50/p75/p90), the
recent-years value is placed on the baseline distribution and the gap from the
expected percentile is the divergence — unit-free, so metrics compare directly.
Scored per meteorological season plus annual, weighted into per-metric and
overall scores (temps, humidity and feels-like weighted heaviest).
- backend/scoring.py: divergence math, seasonal slicing, precip zero-inflation
split (wet-day frequency + amount), tier mapping onto the existing temp scale.
- climate.py: derive a wet-bulb column (Stull 2011) at the read boundary, before
the humidity column is converted to absolute — via a shared _derive_metrics
wrapper at all four read sites.
- api/v2/score endpoint + build_score payload, cached on the history token with
a scoring-version key.
- frontend score page: overall hero, per-metric cards, by-season chips, and a
button-revealed summary (sentences + metrics×season table + per-percentile
detail). Score nav link across all headers.
- Tests for the scoring math, wet-bulb formula, payload shape and route.
2026-07-19 23:02:33 +00:00
|
|
|
|
app.add_api_route(f"{BASE}/score", _page("score.html"), methods=["GET", "HEAD"], include_in_schema=False)
|
2026-07-11 15:59:14 +00:00
|
|
|
|
app.add_api_route(f"{BASE}/compare", _page("compare.html"), methods=["GET", "HEAD"], include_in_schema=False)
|
|
|
|
|
|
app.add_api_route(f"{BASE}/legend", _page("legend.html"), methods=["GET", "HEAD"], include_in_schema=False)
|
Account system with weather-notification subscriptions (#89)
* Add account system foundation: email/password auth with cookie sessions
Introduce the app's first authoritative, user-owned data in a separate
data/accounts.sqlite (SQLAlchemy), kept apart from the disposable derived-cache
DB. Wire fastapi-users for email/password signup, cookie-based login/logout, and
a session-check endpoint, backed by a database session strategy so logins survive
restarts and are revocable.
- db.py: async (aiosqlite) + sync SQLAlchemy engines over accounts.sqlite, WAL +
foreign keys, create_db_and_tables().
- models.py: User, AccessToken, Subscription, Notification tables.
- users.py: pwdlib hashing, HttpOnly cookie transport (path-scoped, SameSite=Lax,
Secure via env), DatabaseStrategy sessions, current-user dependencies.
- schemas.py: user + subscription + notification Pydantic models.
- app.py: mount auth/register/users routers on v2, create tables at startup.
- Pin fastapi-users[sqlalchemy]/aiosqlite; ignore data/accounts.sqlite*.
* Add account header entry and auth modal (frontend)
account.js self-injects a header entry (following the units.js pattern) that
shows a Sign in button when logged out and an account menu when logged in, plus
an auth modal reusing the existing .mp-overlay/.mp-modal chrome for email/password
sign-in and account creation. A shared apiFetch helper sends the same-origin
cookie for authed calls; exported getUser/openAuth/onAuthChange back later phases.
Imported by every page entry module. On narrow screens the entry collapses to an
icon-only button so it doesn't crowd the title.
Enforce an 8-character minimum password in the user manager.
* Add subscription CRUD API and the alerts management page
Backend api_accounts.py adds user-scoped, cookie-authenticated endpoints to
create/list/update/delete subscriptions (and the notification reads used next):
POST snaps lat/lon to a grid cell, resolves a label, and rejects a duplicate
location+kind with 409; PATCH/DELETE are ownership-checked (404 on mismatch).
Mounted on the v2 prefix.
Frontend subscriptions.js + subscriptions.html serve the /alerts page: a sign-in
gate when logged out, an add flow that reuses the shared map picker and an editor
modal (kind, watched metrics, 95-99 percentile, two-sided), and a card list with
inline threshold/active edits and remove. Reachable from the account menu.
* Add background subscription evaluation engine
notify.py runs a daemon thread that periodically evaluates every active
subscription: it groups them by grid cell, reads history from the parquet cache
only (never spends archive quota) plus the hourly recent/forecast bundle, and
grades candidate days with the existing grading.grade_day. A watched metric that
lands at or beyond the threshold percentile fires a 'high' alert; a two-sided
subscription also fires 'low' for the symmetric cold/calm/dry tail (precip stays
one-directional). Observed subscriptions look at the last few recorded days,
forecast subscriptions at the coming week.
Two guards keep it quiet: a UNIQUE(subscription, event_date, metric, direction,
kind) constraint dedups repeat events, and a per-subscription weekly cap
(last_notified_at) limits each alert to one notification per 7 days. The loop
tolerates a bad cell or an upstream rate limit without aborting the pass. Started
and stopped from the app lifespan; gated by THERMOGRAPH_ENABLE_NOTIFIER.
* Add in-app notification center (header bell)
Extend account.js with a notification bell beside the account menu: an unread
badge, a dropdown listing recent notifications (title, body, relative time), a
per-item mark-read on click, and a Mark all read action, all through the
cookie-authed notifications API. Unread state refreshes on open and polls every
two minutes while signed in; polling stops on sign-out. Styled to match the app,
responsive down to mobile.
* Harden accounts: expired-session cleanup, engine tests, ops docs
- notify.py sweeps expired login sessions (access tokens past their lifetime)
once per evaluation pass.
- Add hermetic unit tests for the evaluation engine's trigger detection (high/low
tails, precip one-directional, normal = no trigger) and notification wording.
- Document accounts.sqlite (authoritative, back it up), the single-worker
requirement for the in-process evaluator, and the new env vars in DEPLOY.md.
2026-07-15 18:46:46 +00:00
|
|
|
|
app.add_api_route(f"{BASE}/alerts", _page("subscriptions.html"), methods=["GET", "HEAD"], include_in_schema=False)
|
2026-07-11 00:29:47 +00:00
|
|
|
|
|
Rebuild the homepage as a distribution landing; add an SMTP seam (#178)
The homepage was a bare tool: a find-bar and an empty panel reading "Find a
location to begin." A visitor arriving from a search result or a shared link
learned nothing about what the product does before deciding to leave.
Rebuild it around the Weekly view, which is untouched:
- Hero with the question as the h1, and a grade card showing a real graded
example — the most unusual city we're currently tracking, either tail. The
card's frame and text slots are server-rendered with reserved heights, so
app.js re-pointing it at the visitor's own place shifts nothing.
- "Unusual right now" strip, CSS scroll-snap, no JS carousel. A cold-tail city
is force-included whenever one qualifies.
- Stance line, how-it-works, explore cards, and 12 city chips linking into the
~1000-page /climate surface.
- Monthly digest form, in the footer of every page.
Serve / from Jinja instead of a static file with placeholder substitution, so
crawlers and no-JS readers get the whole page as real HTML. home.html.j2
extends base.html.j2 and carries app.js's DOM contract over verbatim; the brand
degrades to a <p> so the hero owns the sole h1. frontend/index.html is deleted
rather than left behind the static mount, where it would keep serving indexable
duplicate content.
"Where is it most unusual right now" has no cheap answer at request time —
percentiles live inside zlib-compressed payload blobs with no column to sort on.
So homepage.py sweeps the warm cache and writes data/homepage.json, read by the
template. The sweep is strictly cache-only (climate.load_cached_recent_forecast
is new, the sibling of load_cached_history), so grading ~1000 cities costs zero
upstream requests. It rides the notifier's timer behind an hourly guard rather
than starting a second daemon, and also runs at the tail of warm_cities so a
fresh deploy has a populated feed.
Instrumentation: metrics gains a product-event counter keyed by (event,
referrer domain, UTC day), behind an allowlist and a per-IP rate limit, fed by
POST /api/v2/event. The referrer is taken from the request's own header, never
from the client. The beacon gets its own inbound category that record_inbound
ignores, so reporting an interaction doesn't also count as traffic. The
dashboard grows an events block with per-referrer attribution.
Email is scaffolded but sends nothing yet. mailer.py talks stdlib smtplib to a
local Postfix null client on 127.0.0.1:25 (deploy/provision-mail.sh), so the
choice between direct-to-MX and relaying through a provider stays a Postfix
config change with no code change. The backend defaults to "console", which
logs and sends nothing, so dev and tests exercise the whole signup path safely.
Signups land in pending_digest unconfirmed; collecting the list shouldn't wait
on delivery.
Also adds /privacy, linked from the footer and kept out of the sitemap.
The strip's classes are named unusual-* rather than record-*: .record-card is
already the SEO records page's, and reusing it leaked layout rules onto
/climate/<slug>/records.
2026-07-18 07:39:47 +00:00
|
|
|
|
# Monthly-digest signup (the footer form on every page). Answers the same generic
|
|
|
|
|
|
# message whatever the outcome, so it can't be probed for whether an address is
|
|
|
|
|
|
# already subscribed.
|
|
|
|
|
|
app.add_api_route(f"{BASE}/digest", digest.signup, methods=["POST"], include_in_schema=False)
|
|
|
|
|
|
|
|
|
|
|
|
# Crawlable, server-rendered SEO pages (the homepage, climate hub / per-city /
|
|
|
|
|
|
# month / records / glossary / about / privacy) + robots.txt + sitemap.xml.
|
|
|
|
|
|
# Registered before the static mount so these explicit routes win.
|
SEO: crawlable programmatic climate pages + technical hygiene (#96)
* SEO: generate curated city set for crawlable climate pages
gen_cities.py reuses the GeoNames index places.py already parses to select the top
~500 metros by population, assigns each a stable URL-safe slug (dropping admin1 when
it repeats the city name), and writes committed backend/cities.json. cities.py loads
it lazily with slug lookup, all_slugs(), display_name(), and by_country() grouping
for the upcoming hub + sitemap.
* SEO: rendering core, robots.txt, sitemap.xml, and metadata hygiene
- content.py: Jinja2 environment + HTML responder (ETag/304), dynamic /robots.txt
(disallows /api and /alerts, points at the sitemap) and /sitemap.xml (enumerates
the home/static pages plus every city, month, and records URL from cities.py).
Registered on the app before the StaticFiles mount so the routes win.
- templates/base.html.j2: shared layout with unique title/description, self-
referential canonical, Open Graph, favicon/manifest, header nav (adds a Climate
link) and a footer link graph.
- Give each existing page a unique <meta description> (were 5x identical) and a
self-referential <link rel=canonical>; add WebApplication JSON-LD to the home page.
- Pin jinja2.
* SEO: server-rendered per-city climate page (/climate/{slug})
The keystone crawlable page: for a city it snaps to the grid cell, loads the
archive (fetching once if missing, self-healing), and renders as real HTML — a
'how today compares' block (grade + percentile per metric from grade_day, tinted
by tier), a monthly normals table (climatology at each month's 15th, shown in °F
and °C), all-time records (new grading.all_time_records helper), a breadcrumb,
Dataset+Place+BreadcrumbList JSON-LD, self-referential canonical, and links into
the interactive tool + month/records pages. Content-page CSS added to style.css
(renamed the table class to avoid colliding with the app's .normals flex row).
* SEO: month (/climate/{slug}/{month}) and records (/climate/{slug}/records) pages
Month pages render the exact-month long-tail ('average weather in {city} in
{month}') with that month's average high/low, typical p10-p90 range, month-specific
records, and prev/next month links. Records pages show all-time record highs/lows
per metric with dates (grading.all_time_records). Shared _resolve_city helper; the
literal /records route is registered before the {month} param and month names are
validated (unknown month -> 404).
* SEO: climate hub, weather glossary, and about/methodology pages
- /climate: crawlable directory of all ~500 cities grouped by country — the
internal-link graph that lets search engines discover every city page.
- /glossary + /glossary/{term}: plain-language definitions (climate normal,
percentile, temperature anomaly, feels-like, heat index, wind chill, humidity,
reanalysis) with DefinedTerm JSON-LD and cross-links into the tool.
- /about: methodology page (ERA5 data source, 45-year baseline, +/-7-day window,
percentile grading) for E-E-A-T. All linked from the shared footer.
* SEO: archive warmer, content-page tests, and deploy docs
- warm_cities.py: paced, idempotent offline warmer that pre-fetches each city
cell's archive so /climate pages serve from cache and a crawl can't burst the
archive quota (pages self-heal if hit before warming).
- tests/test_content.py: city-set slug uniqueness/lookup, robots.txt, sitemap
enumerating city/month/records URLs, and that a rendered city page carries the
stats + canonical + Dataset JSON-LD in the HTML; plus month/records/hub/glossary/
about routing and 404s.
- DEPLOY.md: document the content pages, the warm step, and submitting the sitemap.
2026-07-15 23:53:11 +00:00
|
|
|
|
content.register(app)
|
|
|
|
|
|
|
2026-07-11 00:29:47 +00:00
|
|
|
|
# Everything else under BASE (app.js, style.css, nav.js, …) is a static asset.
|
2026-07-15 19:58:32 +00:00
|
|
|
|
# Registered last so the explicit page routes above win. Mounts must start with
|
|
|
|
|
|
# "/", so at the root (BASE == "") mount at "/" — the earlier routes still win
|
|
|
|
|
|
# because Starlette matches in registration order.
|
|
|
|
|
|
app.mount(BASE or "/", StaticFiles(directory=FRONTEND_DIR), name="static")
|