2026-07-11 00:29:47 +00:00
|
|
|
fastapi==0.115.6
|
|
|
|
|
uvicorn[standard]==0.34.0
|
|
|
|
|
httpx==0.28.1
|
Migrate backend dataframe layer from pandas to polars (#90)
* Migrate backend dataframe layer from pandas to polars
Replace pandas with polars across the backend, dropping both pandas and its
pyarrow parquet engine from the dependency set. numpy stays (the grading
percentile math is unchanged).
- climate.py: parquet IO, source→frame mappings, cache read/topup on polars.
New _normalize_read casts the cached `date` column to pl.Date (older files
were written by pandas as datetime64[ns]); frames now unify missing values as
null so the grading boundary drops them consistently across sources.
- grading.py: keep the numpy percentile core; swap the frame→numpy bridge to
.to_numpy()/.drop_nulls(), day-of-year/year to polars dt expressions, and the
per-row loop to iter_rows(named=True).
- views.py: filter/anti-join/concat replace boolean-mask, isin and pd.concat;
scalar dates are stdlib datetime.date; a local _months_before helper replaces
DateOffset(months=) for the calendar-range default.
- app.py, migrate.py: request-date parsing uses datetime.date, removing pandas
from the endpoint and migrate layers entirely.
- The date column is pl.Date end to end, eliminating the pandas normalize() calls
and comparing cleanly against stdlib dates.
Payloads are unchanged: calendar, day, grade and forecast responses are
byte-for-byte identical to the pandas implementation on the same cached record.
Tests ported to polars fixtures, with added coverage for the combined feels-like
fallback, calendar month-offset (month-end/leap), and the concat/dedup
"fresher source wins" rule.
* Port notify.py to polars after merging dev's account system
Merge origin/dev (accounts + notification subscriptions) and carry the pandas→
polars migration into the newly added notify.py, which the merge brought in still
using pandas — with pandas removed from requirements this broke its import.
- notify.py: _candidate_rows filters/sorts the recent bundle with polars
expressions and returns iter_rows dicts; date scalars are datetime.date;
history/recent emptiness via is_empty().
- test_notify.py: synthetic history/rows built with polars + datetime.
2026-07-15 19:07:38 +00:00
|
|
|
polars==1.42.1
|
2026-07-11 00:29:47 +00:00
|
|
|
numpy==2.2.1
|
Containerize the app and move the databases to PostgreSQL 18 (#220)
Run Thermograph as a docker-compose stack (app + Postgres 18) and standardize the
data layer on Postgres, while keeping the test suite on SQLite.
- accounts/db.py: DSN-driven engines. On Postgres, a per-worker read-write +
read-only asyncpg pair (the RO engine pins read-only transactions, used by the
pure-GET endpoints) plus a sync psycopg engine for the notifier thread; the
SQLite path is preserved for tests/local (selected when THERMOGRAPH_DATABASE_URL
is unset). models.py: boolean server_default -> sa.false().
- store.py / metrics.py: dialect-flexible — Postgres UNLOGGED tables via psycopg
when configured, else the existing raw-sqlite3 paths byte-for-byte; sync
interfaces and every fail-soft contract preserved.
- Alembic (backend/alembic/) manages the accounts schema; the container entrypoint
runs `alembic upgrade head` before uvicorn (4 workers). migrate_accounts_to_pg.py
copies the accounts data SQLite->PG through the ORM (UUID/bool/JSON coerced),
skips access_token, and resets identity sequences.
- Dockerfile + docker-compose.yml: app image (uvicorn, 4 workers, loopback 8137)
and a Postgres 18 db (2 CPUs) running pg_duckdb (deploy/db/) so the parquet
climate cache is queryable in-DB via read_parquet('/parquet/cache/*.parquet').
- deploy.sh/thermograph.service rewired to manage the compose stack; env example,
Makefile targets (up/down/db-up), and deploy/POSTGRES-MIGRATION.md cutover runbook.
Tests stay on SQLite (dialect fallback) — 323 pass. The full Postgres stack was
verified via docker compose: alembic migrations, register/login, the RO endpoint,
store/metrics round-trips, and the accounts data migration.
2026-07-20 06:28:23 +00:00
|
|
|
# Accounts + notification subscriptions (see accounts/db.py, users.py, notify.py).
|
|
|
|
|
# Postgres in prod/containers: asyncpg (async web/store) + psycopg (sync notifier);
|
|
|
|
|
# aiosqlite is kept for the SQLite fallback the test suite runs on. alembic manages
|
|
|
|
|
# the accounts schema.
|
Account system with weather-notification subscriptions (#89)
* Add account system foundation: email/password auth with cookie sessions
Introduce the app's first authoritative, user-owned data in a separate
data/accounts.sqlite (SQLAlchemy), kept apart from the disposable derived-cache
DB. Wire fastapi-users for email/password signup, cookie-based login/logout, and
a session-check endpoint, backed by a database session strategy so logins survive
restarts and are revocable.
- db.py: async (aiosqlite) + sync SQLAlchemy engines over accounts.sqlite, WAL +
foreign keys, create_db_and_tables().
- models.py: User, AccessToken, Subscription, Notification tables.
- users.py: pwdlib hashing, HttpOnly cookie transport (path-scoped, SameSite=Lax,
Secure via env), DatabaseStrategy sessions, current-user dependencies.
- schemas.py: user + subscription + notification Pydantic models.
- app.py: mount auth/register/users routers on v2, create tables at startup.
- Pin fastapi-users[sqlalchemy]/aiosqlite; ignore data/accounts.sqlite*.
* Add account header entry and auth modal (frontend)
account.js self-injects a header entry (following the units.js pattern) that
shows a Sign in button when logged out and an account menu when logged in, plus
an auth modal reusing the existing .mp-overlay/.mp-modal chrome for email/password
sign-in and account creation. A shared apiFetch helper sends the same-origin
cookie for authed calls; exported getUser/openAuth/onAuthChange back later phases.
Imported by every page entry module. On narrow screens the entry collapses to an
icon-only button so it doesn't crowd the title.
Enforce an 8-character minimum password in the user manager.
* Add subscription CRUD API and the alerts management page
Backend api_accounts.py adds user-scoped, cookie-authenticated endpoints to
create/list/update/delete subscriptions (and the notification reads used next):
POST snaps lat/lon to a grid cell, resolves a label, and rejects a duplicate
location+kind with 409; PATCH/DELETE are ownership-checked (404 on mismatch).
Mounted on the v2 prefix.
Frontend subscriptions.js + subscriptions.html serve the /alerts page: a sign-in
gate when logged out, an add flow that reuses the shared map picker and an editor
modal (kind, watched metrics, 95-99 percentile, two-sided), and a card list with
inline threshold/active edits and remove. Reachable from the account menu.
* Add background subscription evaluation engine
notify.py runs a daemon thread that periodically evaluates every active
subscription: it groups them by grid cell, reads history from the parquet cache
only (never spends archive quota) plus the hourly recent/forecast bundle, and
grades candidate days with the existing grading.grade_day. A watched metric that
lands at or beyond the threshold percentile fires a 'high' alert; a two-sided
subscription also fires 'low' for the symmetric cold/calm/dry tail (precip stays
one-directional). Observed subscriptions look at the last few recorded days,
forecast subscriptions at the coming week.
Two guards keep it quiet: a UNIQUE(subscription, event_date, metric, direction,
kind) constraint dedups repeat events, and a per-subscription weekly cap
(last_notified_at) limits each alert to one notification per 7 days. The loop
tolerates a bad cell or an upstream rate limit without aborting the pass. Started
and stopped from the app lifespan; gated by THERMOGRAPH_ENABLE_NOTIFIER.
* Add in-app notification center (header bell)
Extend account.js with a notification bell beside the account menu: an unread
badge, a dropdown listing recent notifications (title, body, relative time), a
per-item mark-read on click, and a Mark all read action, all through the
cookie-authed notifications API. Unread state refreshes on open and polls every
two minutes while signed in; polling stops on sign-out. Styled to match the app,
responsive down to mobile.
* Harden accounts: expired-session cleanup, engine tests, ops docs
- notify.py sweeps expired login sessions (access tokens past their lifetime)
once per evaluation pass.
- Add hermetic unit tests for the evaluation engine's trigger detection (high/low
tails, precip one-directional, normal = no trigger) and notification wording.
- Document accounts.sqlite (authoritative, back it up), the single-worker
requirement for the in-process evaluator, and the new env vars in DEPLOY.md.
2026-07-15 18:46:46 +00:00
|
|
|
fastapi-users[sqlalchemy]==15.0.5
|
|
|
|
|
aiosqlite==0.22.1
|
Containerize the app and move the databases to PostgreSQL 18 (#220)
Run Thermograph as a docker-compose stack (app + Postgres 18) and standardize the
data layer on Postgres, while keeping the test suite on SQLite.
- accounts/db.py: DSN-driven engines. On Postgres, a per-worker read-write +
read-only asyncpg pair (the RO engine pins read-only transactions, used by the
pure-GET endpoints) plus a sync psycopg engine for the notifier thread; the
SQLite path is preserved for tests/local (selected when THERMOGRAPH_DATABASE_URL
is unset). models.py: boolean server_default -> sa.false().
- store.py / metrics.py: dialect-flexible — Postgres UNLOGGED tables via psycopg
when configured, else the existing raw-sqlite3 paths byte-for-byte; sync
interfaces and every fail-soft contract preserved.
- Alembic (backend/alembic/) manages the accounts schema; the container entrypoint
runs `alembic upgrade head` before uvicorn (4 workers). migrate_accounts_to_pg.py
copies the accounts data SQLite->PG through the ORM (UUID/bool/JSON coerced),
skips access_token, and resets identity sequences.
- Dockerfile + docker-compose.yml: app image (uvicorn, 4 workers, loopback 8137)
and a Postgres 18 db (2 CPUs) running pg_duckdb (deploy/db/) so the parquet
climate cache is queryable in-DB via read_parquet('/parquet/cache/*.parquet').
- deploy.sh/thermograph.service rewired to manage the compose stack; env example,
Makefile targets (up/down/db-up), and deploy/POSTGRES-MIGRATION.md cutover runbook.
Tests stay on SQLite (dialect fallback) — 323 pass. The full Postgres stack was
verified via docker compose: alembic migrations, register/login, the RO endpoint,
store/metrics round-trips, and the accounts data migration.
2026-07-20 06:28:23 +00:00
|
|
|
asyncpg==0.30.0
|
|
|
|
|
psycopg[binary]==3.2.3
|
|
|
|
|
alembic==1.14.0
|
2026-07-15 23:21:06 +00:00
|
|
|
# Web Push (VAPID) delivery of notifications (see backend/push.py). Pulls in
|
|
|
|
|
# py-vapid, cryptography, and http-ece.
|
|
|
|
|
pywebpush==2.0.0
|
2026-07-20 02:19:29 +00:00
|
|
|
# Ed25519 verification of Discord interaction webhooks (see backend/discord_interactions.py).
|
|
|
|
|
PyNaCl==1.5.0
|
2026-07-21 22:48:59 +00:00
|
|
|
# Not used by backend itself (its own content.py/content_loader.py were
|
|
|
|
|
# deleted in repo-split Stage 4) -- installed here because frontend_ssr/
|
|
|
|
|
# shares this requirements.txt in the current shared Dockerfile. See
|
|
|
|
|
# frontend_ssr/content.py (server-rendered SEO content pages).
|
SEO: crawlable programmatic climate pages + technical hygiene (#96)
* SEO: generate curated city set for crawlable climate pages
gen_cities.py reuses the GeoNames index places.py already parses to select the top
~500 metros by population, assigns each a stable URL-safe slug (dropping admin1 when
it repeats the city name), and writes committed backend/cities.json. cities.py loads
it lazily with slug lookup, all_slugs(), display_name(), and by_country() grouping
for the upcoming hub + sitemap.
* SEO: rendering core, robots.txt, sitemap.xml, and metadata hygiene
- content.py: Jinja2 environment + HTML responder (ETag/304), dynamic /robots.txt
(disallows /api and /alerts, points at the sitemap) and /sitemap.xml (enumerates
the home/static pages plus every city, month, and records URL from cities.py).
Registered on the app before the StaticFiles mount so the routes win.
- templates/base.html.j2: shared layout with unique title/description, self-
referential canonical, Open Graph, favicon/manifest, header nav (adds a Climate
link) and a footer link graph.
- Give each existing page a unique <meta description> (were 5x identical) and a
self-referential <link rel=canonical>; add WebApplication JSON-LD to the home page.
- Pin jinja2.
* SEO: server-rendered per-city climate page (/climate/{slug})
The keystone crawlable page: for a city it snaps to the grid cell, loads the
archive (fetching once if missing, self-healing), and renders as real HTML — a
'how today compares' block (grade + percentile per metric from grade_day, tinted
by tier), a monthly normals table (climatology at each month's 15th, shown in °F
and °C), all-time records (new grading.all_time_records helper), a breadcrumb,
Dataset+Place+BreadcrumbList JSON-LD, self-referential canonical, and links into
the interactive tool + month/records pages. Content-page CSS added to style.css
(renamed the table class to avoid colliding with the app's .normals flex row).
* SEO: month (/climate/{slug}/{month}) and records (/climate/{slug}/records) pages
Month pages render the exact-month long-tail ('average weather in {city} in
{month}') with that month's average high/low, typical p10-p90 range, month-specific
records, and prev/next month links. Records pages show all-time record highs/lows
per metric with dates (grading.all_time_records). Shared _resolve_city helper; the
literal /records route is registered before the {month} param and month names are
validated (unknown month -> 404).
* SEO: climate hub, weather glossary, and about/methodology pages
- /climate: crawlable directory of all ~500 cities grouped by country — the
internal-link graph that lets search engines discover every city page.
- /glossary + /glossary/{term}: plain-language definitions (climate normal,
percentile, temperature anomaly, feels-like, heat index, wind chill, humidity,
reanalysis) with DefinedTerm JSON-LD and cross-links into the tool.
- /about: methodology page (ERA5 data source, 45-year baseline, +/-7-day window,
percentile grading) for E-E-A-T. All linked from the shared footer.
* SEO: archive warmer, content-page tests, and deploy docs
- warm_cities.py: paced, idempotent offline warmer that pre-fetches each city
cell's archive so /climate pages serve from cache and a crawl can't burst the
archive quota (pages self-heal if hit before warming).
- tests/test_content.py: city-set slug uniqueness/lookup, robots.txt, sitemap
enumerating city/month/records URLs, and that a rendered city page carries the
stats + canonical + Dataset JSON-LD in the HTML; plus month/records/hub/glossary/
about routing and 404s.
- DEPLOY.md: document the content pages, the warm step, and submitting the sitemap.
2026-07-15 23:53:11 +00:00
|
|
|
jinja2==3.1.6
|
Extract the glossary and static-page SEO copy into a schema-validated content/ tree (#241)
The weather-terms glossary (9 entries) and four static pages' SEO title/
description were hardcoded directly in web/content.py - a 1019-line module
that also owns all SSR rendering logic - mixed in with code that changes on
a completely different cadence and for different reasons.
Add content/glossary.yaml and content/pages.yaml (a new repo-root content/
tree, a sibling of backend/ and frontend/ paths.py resolves the same way -
the seed of a future thermograph-copy repo per the architecture decision
doc's own §4) plus web/content_loader.py: a small loader that validates each
file's shape at load time (required fields present and non-empty, no
duplicate glossary slugs) and fails loudly on a malformed edit rather than
rendering a blank glossary card or an empty <title>. content.py's GLOSSARY
dict and the about/privacy/hub/glossary_index page_title/description
literals now come from the loader.
Scope: only content that is genuinely pure static data with no embedded
template logic. cities_flavor.json (Wikipedia extracts, already its own
generated file) and the homepage's title/description (embedded in
home.html.j2 as Jinja block overrides, a heavily-tested product-critical
template) are deliberately left as they are - a future pass, not required
for this one. UI microcopy bound to frontend logic stays in frontend/,
per the doc's own line between content and frontend.
Verified: content/glossary.yaml generated programmatically from the live
GLOSSARY dict (not hand-transcribed) and round-tripped byte-for-byte
identical against it; content/pages.yaml's four entries checked field-by-
field against the original hardcoded strings. Full backend suite green (362
passed, 4 skipped) with zero existing test changes needed beyond one
assertion made escaping-aware (a pre-existing Jinja double-escape quirk on
the one title containing "&", intentionally preserved not fixed). Built
and booted the real Docker image: content/ present at /app/content, and
curled /glossary, /glossary/percentine, /about, /privacy from inside the
running container - all four render with the exact expected title text.
2026-07-21 01:21:11 +00:00
|
|
|
# Structured SSR copy (glossary, static-page SEO meta) — see
|
2026-07-21 22:48:59 +00:00
|
|
|
# frontend_ssr/content_loader.py and content/.
|
Extract the glossary and static-page SEO copy into a schema-validated content/ tree (#241)
The weather-terms glossary (9 entries) and four static pages' SEO title/
description were hardcoded directly in web/content.py - a 1019-line module
that also owns all SSR rendering logic - mixed in with code that changes on
a completely different cadence and for different reasons.
Add content/glossary.yaml and content/pages.yaml (a new repo-root content/
tree, a sibling of backend/ and frontend/ paths.py resolves the same way -
the seed of a future thermograph-copy repo per the architecture decision
doc's own §4) plus web/content_loader.py: a small loader that validates each
file's shape at load time (required fields present and non-empty, no
duplicate glossary slugs) and fails loudly on a malformed edit rather than
rendering a blank glossary card or an empty <title>. content.py's GLOSSARY
dict and the about/privacy/hub/glossary_index page_title/description
literals now come from the loader.
Scope: only content that is genuinely pure static data with no embedded
template logic. cities_flavor.json (Wikipedia extracts, already its own
generated file) and the homepage's title/description (embedded in
home.html.j2 as Jinja block overrides, a heavily-tested product-critical
template) are deliberately left as they are - a future pass, not required
for this one. UI microcopy bound to frontend logic stays in frontend/,
per the doc's own line between content and frontend.
Verified: content/glossary.yaml generated programmatically from the live
GLOSSARY dict (not hand-transcribed) and round-tripped byte-for-byte
identical against it; content/pages.yaml's four entries checked field-by-
field against the original hardcoded strings. Full backend suite green (362
passed, 4 skipped) with zero existing test changes needed beyond one
assertion made escaping-aware (a pre-existing Jinja double-escape quirk on
the one title containing "&", intentionally preserved not fixed). Built
and booted the real Docker image: content/ present at /app/content, and
curled /glossary, /glossary/percentine, /about, /privacy from inside the
running container - all four render with the exact expected title text.
2026-07-21 01:21:11 +00:00
|
|
|
PyYAML==6.0.3
|
2026-07-21 01:26:39 +00:00
|
|
|
# Recurring worker-tier jobs — city warming, IndexNow pings (see
|
|
|
|
|
# notifications/scheduler.py). In-process; no broker/queue needed at this scale.
|
|
|
|
|
apscheduler==3.11.3
|
