Run Thermograph as a docker-compose stack (app + Postgres 18) and standardize the
data layer on Postgres, while keeping the test suite on SQLite.
- accounts/db.py: DSN-driven engines. On Postgres, a per-worker read-write +
read-only asyncpg pair (the RO engine pins read-only transactions, used by the
pure-GET endpoints) plus a sync psycopg engine for the notifier thread; the
SQLite path is preserved for tests/local (selected when THERMOGRAPH_DATABASE_URL
is unset). models.py: boolean server_default -> sa.false().
- store.py / metrics.py: dialect-flexible — Postgres UNLOGGED tables via psycopg
when configured, else the existing raw-sqlite3 paths byte-for-byte; sync
interfaces and every fail-soft contract preserved.
- Alembic (backend/alembic/) manages the accounts schema; the container entrypoint
runs `alembic upgrade head` before uvicorn (4 workers). migrate_accounts_to_pg.py
copies the accounts data SQLite->PG through the ORM (UUID/bool/JSON coerced),
skips access_token, and resets identity sequences.
- Dockerfile + docker-compose.yml: app image (uvicorn, 4 workers, loopback 8137)
and a Postgres 18 db (2 CPUs) running pg_duckdb (deploy/db/) so the parquet
climate cache is queryable in-DB via read_parquet('/parquet/cache/*.parquet').
- deploy.sh/thermograph.service rewired to manage the compose stack; env example,
Makefile targets (up/down/db-up), and deploy/POSTGRES-MIGRATION.md cutover runbook.
Tests stay on SQLite (dialect fallback) — 323 pass. The full Postgres stack was
verified via docker compose: alembic migrations, register/login, the RO endpoint,
store/metrics round-trips, and the accounts data migration.
Adds Discord slash commands with no bot process and no gateway connection: Discord
POSTs each interaction to a FastAPI route, and the app answers it. First command is
/grade <city>, returning today's grade for a curated city from the warm cache
(reusing homepage._grade_city, so it answers well within the 3-second deadline and
costs no upstream quota).
- backend/discord_interactions.py: Ed25519 verification (PyNaCl) over the RAW
request body — Discord probes the endpoint with bad signatures and disables it if
they aren't rejected with 401. Routes PING to PONG and application-commands to
their handler; unknown/unsupported interactions are acknowledged, not errored.
City lookup is exact-name-then-prefix over the population-sorted city set; unknown
or not-yet-warm cities get an ephemeral note.
- app.py: POST {BASE}/discord/interactions, reading request.body() (not json()) so
the bytes match the signature.
- scripts/register_discord_commands.py: one-off upsert of the command definitions
via Discord REST (app id + bot token).
- PyNaCl added to requirements; Discord public-key / app-id / bot-token documented
in the env example. Endpoint URL: https://thermograph.org/discord/interactions.
Claude-Session: https://claude.ai/code/session_013dRZmX9D3JEntfMKWMTWZ8
* SEO: generate curated city set for crawlable climate pages
gen_cities.py reuses the GeoNames index places.py already parses to select the top
~500 metros by population, assigns each a stable URL-safe slug (dropping admin1 when
it repeats the city name), and writes committed backend/cities.json. cities.py loads
it lazily with slug lookup, all_slugs(), display_name(), and by_country() grouping
for the upcoming hub + sitemap.
* SEO: rendering core, robots.txt, sitemap.xml, and metadata hygiene
- content.py: Jinja2 environment + HTML responder (ETag/304), dynamic /robots.txt
(disallows /api and /alerts, points at the sitemap) and /sitemap.xml (enumerates
the home/static pages plus every city, month, and records URL from cities.py).
Registered on the app before the StaticFiles mount so the routes win.
- templates/base.html.j2: shared layout with unique title/description, self-
referential canonical, Open Graph, favicon/manifest, header nav (adds a Climate
link) and a footer link graph.
- Give each existing page a unique <meta description> (were 5x identical) and a
self-referential <link rel=canonical>; add WebApplication JSON-LD to the home page.
- Pin jinja2.
* SEO: server-rendered per-city climate page (/climate/{slug})
The keystone crawlable page: for a city it snaps to the grid cell, loads the
archive (fetching once if missing, self-healing), and renders as real HTML — a
'how today compares' block (grade + percentile per metric from grade_day, tinted
by tier), a monthly normals table (climatology at each month's 15th, shown in °F
and °C), all-time records (new grading.all_time_records helper), a breadcrumb,
Dataset+Place+BreadcrumbList JSON-LD, self-referential canonical, and links into
the interactive tool + month/records pages. Content-page CSS added to style.css
(renamed the table class to avoid colliding with the app's .normals flex row).
* SEO: month (/climate/{slug}/{month}) and records (/climate/{slug}/records) pages
Month pages render the exact-month long-tail ('average weather in {city} in
{month}') with that month's average high/low, typical p10-p90 range, month-specific
records, and prev/next month links. Records pages show all-time record highs/lows
per metric with dates (grading.all_time_records). Shared _resolve_city helper; the
literal /records route is registered before the {month} param and month names are
validated (unknown month -> 404).
* SEO: climate hub, weather glossary, and about/methodology pages
- /climate: crawlable directory of all ~500 cities grouped by country — the
internal-link graph that lets search engines discover every city page.
- /glossary + /glossary/{term}: plain-language definitions (climate normal,
percentile, temperature anomaly, feels-like, heat index, wind chill, humidity,
reanalysis) with DefinedTerm JSON-LD and cross-links into the tool.
- /about: methodology page (ERA5 data source, 45-year baseline, +/-7-day window,
percentile grading) for E-E-A-T. All linked from the shared footer.
* SEO: archive warmer, content-page tests, and deploy docs
- warm_cities.py: paced, idempotent offline warmer that pre-fetches each city
cell's archive so /climate pages serve from cache and a crawl can't burst the
archive quota (pages self-heal if hit before warming).
- tests/test_content.py: city-set slug uniqueness/lookup, robots.txt, sitemap
enumerating city/month/records URLs, and that a rendered city page carries the
stats + canonical + Dataset JSON-LD in the HTML; plus month/records/hub/glossary/
about routing and 404s.
- DEPLOY.md: document the content pages, the warm step, and submitting the sitemap.
Make the app installable and deliver existing alert notifications as OS
push, alongside the in-app bell.
Backend:
- PushSubscription model (per-device endpoint + keys, owned by a user) and
register/unregister/test endpoints under /api/v2/push, cookie-auth scoped
to the user like the subscription routes.
- push.py: VAPID key management (env -> data/vapid.json -> generated) and a
pywebpush send helper that reports gone endpoints for pruning. No DB coupling.
- notify.py: after creating an in-app Notification, dispatch Web Push to the
user's devices (guarded — a push failure never affects the in-app write;
endpoints reported gone are pruned).
- Serve the .webmanifest with the correct media type.
Frontend:
- manifest.webmanifest + 192/maskable icons; <link rel="manifest"> on all pages.
- sw.js: push + notificationclick handlers (push-only; no fetch caching, so it
doesn't fight the existing IndexedDB cache). Registered globally in nav.js in
secure contexts.
- push-client.js + a "Notifications on this device" toggle and test-send on the
/alerts page, subscribing through the existing cookie-aware apiFetch.
Push and service workers require a secure context, so this is active over HTTPS
(or http://localhost) and cleanly no-ops on a plain-HTTP LAN origin.
* Migrate backend dataframe layer from pandas to polars
Replace pandas with polars across the backend, dropping both pandas and its
pyarrow parquet engine from the dependency set. numpy stays (the grading
percentile math is unchanged).
- climate.py: parquet IO, source→frame mappings, cache read/topup on polars.
New _normalize_read casts the cached `date` column to pl.Date (older files
were written by pandas as datetime64[ns]); frames now unify missing values as
null so the grading boundary drops them consistently across sources.
- grading.py: keep the numpy percentile core; swap the frame→numpy bridge to
.to_numpy()/.drop_nulls(), day-of-year/year to polars dt expressions, and the
per-row loop to iter_rows(named=True).
- views.py: filter/anti-join/concat replace boolean-mask, isin and pd.concat;
scalar dates are stdlib datetime.date; a local _months_before helper replaces
DateOffset(months=) for the calendar-range default.
- app.py, migrate.py: request-date parsing uses datetime.date, removing pandas
from the endpoint and migrate layers entirely.
- The date column is pl.Date end to end, eliminating the pandas normalize() calls
and comparing cleanly against stdlib dates.
Payloads are unchanged: calendar, day, grade and forecast responses are
byte-for-byte identical to the pandas implementation on the same cached record.
Tests ported to polars fixtures, with added coverage for the combined feels-like
fallback, calendar month-offset (month-end/leap), and the concat/dedup
"fresher source wins" rule.
* Port notify.py to polars after merging dev's account system
Merge origin/dev (accounts + notification subscriptions) and carry the pandas→
polars migration into the newly added notify.py, which the merge brought in still
using pandas — with pandas removed from requirements this broke its import.
- notify.py: _candidate_rows filters/sorts the recent bundle with polars
expressions and returns iter_rows dicts; date scalars are datetime.date;
history/recent emptiness via is_empty().
- test_notify.py: synthetic history/rows built with polars + datetime.
* Add account system foundation: email/password auth with cookie sessions
Introduce the app's first authoritative, user-owned data in a separate
data/accounts.sqlite (SQLAlchemy), kept apart from the disposable derived-cache
DB. Wire fastapi-users for email/password signup, cookie-based login/logout, and
a session-check endpoint, backed by a database session strategy so logins survive
restarts and are revocable.
- db.py: async (aiosqlite) + sync SQLAlchemy engines over accounts.sqlite, WAL +
foreign keys, create_db_and_tables().
- models.py: User, AccessToken, Subscription, Notification tables.
- users.py: pwdlib hashing, HttpOnly cookie transport (path-scoped, SameSite=Lax,
Secure via env), DatabaseStrategy sessions, current-user dependencies.
- schemas.py: user + subscription + notification Pydantic models.
- app.py: mount auth/register/users routers on v2, create tables at startup.
- Pin fastapi-users[sqlalchemy]/aiosqlite; ignore data/accounts.sqlite*.
* Add account header entry and auth modal (frontend)
account.js self-injects a header entry (following the units.js pattern) that
shows a Sign in button when logged out and an account menu when logged in, plus
an auth modal reusing the existing .mp-overlay/.mp-modal chrome for email/password
sign-in and account creation. A shared apiFetch helper sends the same-origin
cookie for authed calls; exported getUser/openAuth/onAuthChange back later phases.
Imported by every page entry module. On narrow screens the entry collapses to an
icon-only button so it doesn't crowd the title.
Enforce an 8-character minimum password in the user manager.
* Add subscription CRUD API and the alerts management page
Backend api_accounts.py adds user-scoped, cookie-authenticated endpoints to
create/list/update/delete subscriptions (and the notification reads used next):
POST snaps lat/lon to a grid cell, resolves a label, and rejects a duplicate
location+kind with 409; PATCH/DELETE are ownership-checked (404 on mismatch).
Mounted on the v2 prefix.
Frontend subscriptions.js + subscriptions.html serve the /alerts page: a sign-in
gate when logged out, an add flow that reuses the shared map picker and an editor
modal (kind, watched metrics, 95-99 percentile, two-sided), and a card list with
inline threshold/active edits and remove. Reachable from the account menu.
* Add background subscription evaluation engine
notify.py runs a daemon thread that periodically evaluates every active
subscription: it groups them by grid cell, reads history from the parquet cache
only (never spends archive quota) plus the hourly recent/forecast bundle, and
grades candidate days with the existing grading.grade_day. A watched metric that
lands at or beyond the threshold percentile fires a 'high' alert; a two-sided
subscription also fires 'low' for the symmetric cold/calm/dry tail (precip stays
one-directional). Observed subscriptions look at the last few recorded days,
forecast subscriptions at the coming week.
Two guards keep it quiet: a UNIQUE(subscription, event_date, metric, direction,
kind) constraint dedups repeat events, and a per-subscription weekly cap
(last_notified_at) limits each alert to one notification per 7 days. The loop
tolerates a bad cell or an upstream rate limit without aborting the pass. Started
and stopped from the app lifespan; gated by THERMOGRAPH_ENABLE_NOTIFIER.
* Add in-app notification center (header bell)
Extend account.js with a notification bell beside the account menu: an unread
badge, a dropdown listing recent notifications (title, body, relative time), a
per-item mark-read on click, and a Mark all read action, all through the
cookie-authed notifications API. Unread state refreshes on open and polls every
two minutes while signed in; polling stops on sign-out. Styled to match the app,
responsive down to mobile.
* Harden accounts: expired-session cleanup, engine tests, ops docs
- notify.py sweeps expired login sessions (access tokens past their lifetime)
once per evaluation pass.
- Add hermetic unit tests for the evaluation engine's trigger detection (high/low
tails, precip one-directional, normal = no trigger) and notification wording.
- Document accounts.sqlite (authoritative, back it up), the single-worker
requirement for the in-process evaluator, and the new env vars in DEPLOY.md.
Emi Griffith