warm_cities.py and indexnow.py are one-shot scripts run manually at deploy
today - nothing keeps the curated city set topped up, or pings IndexNow,
between deploys.
Add notifications/scheduler.py: an in-process APScheduler instance driving
two recurring jobs (city warming, daily; IndexNow --if-changed, every 6h by
default), gated by the same leader election as the notifier - only the
process that already won leadership starts it, so the jobs run in exactly
one place under multi-host Swarm rather than once per replica. Neither job
runs immediately on start: warm_cities already runs at deploy time, so an
immediate duplicate on every worker boot/restart would be wasted work.
Extract indexnow.submit_if_changed() from the CLI's --if-changed branch so
the scheduler and the command line share the exact same skip-when-unchanged
logic instead of two copies drifting apart. The CLI's plain (non-flag) path
is unchanged.
APScheduler over a Postgres-native queue: exactly one process ever runs
this, no fan-out/backpressure/dead-letter need, no new infrastructure.
The weather-terms glossary (9 entries) and four static pages' SEO title/
description were hardcoded directly in web/content.py - a 1019-line module
that also owns all SSR rendering logic - mixed in with code that changes on
a completely different cadence and for different reasons.
Add content/glossary.yaml and content/pages.yaml (a new repo-root content/
tree, a sibling of backend/ and frontend/ paths.py resolves the same way -
the seed of a future thermograph-copy repo per the architecture decision
doc's own §4) plus web/content_loader.py: a small loader that validates each
file's shape at load time (required fields present and non-empty, no
duplicate glossary slugs) and fails loudly on a malformed edit rather than
rendering a blank glossary card or an empty <title>. content.py's GLOSSARY
dict and the about/privacy/hub/glossary_index page_title/description
literals now come from the loader.
Scope: only content that is genuinely pure static data with no embedded
template logic. cities_flavor.json (Wikipedia extracts, already its own
generated file) and the homepage's title/description (embedded in
home.html.j2 as Jinja block overrides, a heavily-tested product-critical
template) are deliberately left as they are - a future pass, not required
for this one. UI microcopy bound to frontend logic stays in frontend/,
per the doc's own line between content and frontend.
Verified: content/glossary.yaml generated programmatically from the live
GLOSSARY dict (not hand-transcribed) and round-tripped byte-for-byte
identical against it; content/pages.yaml's four entries checked field-by-
field against the original hardcoded strings. Full backend suite green (362
passed, 4 skipped) with zero existing test changes needed beyond one
assertion made escaping-aware (a pre-existing Jinja double-escape quirk on
the one title containing "&", intentionally preserved not fixed). Built
and booted the real Docker image: content/ present at /app/content, and
curled /glossary, /glossary/percentine, /about, /privacy from inside the
running container - all four render with the exact expected title text.
* Split web/worker duties with THERMOGRAPH_ROLE
Background work (the subscription notifier) is welded to the same process
that serves requests, so scaling the web tier to N replicas would also scale
notifier instances unless something restricts it further than leader
election alone.
Add THERMOGRAPH_ROLE (web|worker|all, default all - unchanged single-process
behavior). Every replica runs the same image; ROLE only gates whether a
process is allowed to own the notifier at all, layered on top of the
existing leader election: web replicas never start it even if they'd win
leader election, worker replicas start it if they win. The decision is
pulled into _should_run_notifier() so it's unit-testable without booting the
full app (DB init, places index, neighbor warmer).
Add a minimal /healthz liveness route (no DB/upstream I/O, not under BASE)
so a worker replica - which serves no real traffic - still has something
Swarm can health-check.
* Add the Swarm interim stack file, a pinnable TimescaleDB tag, and a Caddy health-gate
Three changes toward the hop-1 interim cutover, all inert until Track B
stands up the platform:
docker-stack.yml: the Swarm stack file for the interim cutover, distinct
from docker-compose.yml (today's plain-compose deploy, unaffected). Pulls a
pre-built image (IMAGE_TAG) instead of building in place; app/worker publish
no host port (127.0.0.1:8137:8137 has no Swarm equivalent - Swarm's routing
mesh publishes on 0.0.0.0, which would expose the plaintext app un-fronted),
reaching Caddy only over an MTU-lowered overlay network (VXLAN-over-WireGuard
needs a smaller MTU or large payloads silently stall); db is placement-
pinned to a labelled node; app/worker skip inline migrations
(RUN_MIGRATIONS=0) so the runbook's one-shot migrate task is the only thing
that ever runs Alembic; secrets are real Swarm secrets mounted at
/run/secrets, read by the entrypoint shim rather than plain env vars.
TIMESCALEDB_TAG: docker-compose.yml's db image now reads this (default
latest-pg18, today's behavior unchanged), wired through Terraform
(timescaledb_tag, default "latest-pg18") so it can actually be pinned to an
exact minor without hand-editing the host - required before any host of the
stack could replicate with another (a floating tag risks mismatched
extension minors, which blocks a physical replica and risks compressed-
chunk corruption on restore).
Caddy active health-gate: both the Terraform-rendered Caddyfile and the live
deploy/Caddyfile now health-check the app on the same cheap /healthz route
its own Docker HEALTHCHECK uses (now /healthz instead of the SSR homepage,
so it's cheap enough for a tight interval and works identically for a
worker replica, which serves no public traffic at all) - Caddy won't
forward into a container that's still booting or unhealthy.
Verified live: built and booted the real image via docker compose - both
containers report healthy via the new /healthz-based HEALTHCHECK, and GET /
still renders the full SSR homepage unchanged. Both Caddyfiles validated
with the real caddy binary. docker-stack.yml validated with docker compose
config (required-var guards fire with clear messages; secrets correctly
mount at /run/secrets/<name>, matching the entrypoint shim's mapping).
docker-compose.yml validated with and without TIMESCALEDB_TAG set, alongside
the existing openmeteo overlay. terraform validate + fmt clean.
homepage.json lives on the appdata volume today, written by whichever
process's notifier last refreshed it. Under Swarm each web replica has its
own disk, so a file only one replica's notifier ever writes leaves every
other replica reading a stale or missing feed indefinitely.
On Postgres, refresh()/load() go through store.py's existing derived-payload
table instead (a fixed sentinel key, since the feed isn't cell-scoped) - a
single upsert is already atomic, and every replica reads the same row.
store.IS_POSTGRES is False without THERMOGRAPH_DATABASE_URL, so dev/tests
keep the plain file path entirely unchanged; no existing test needed to
change.
Verified live against real Postgres: refresh() persists into the shared
derived table and load() reads the exact same feed back.
The subscription notifier elects one leader via a host-local flock
(THERMOGRAPH_SINGLETON_LOCK) so multiple uvicorn workers on one host don't
each run it. Under multi-host Swarm that guard is insufficient: each host
would independently elect its own leader, multiplying Open-Meteo quota use
N-fold again.
Add claim_pg(key) alongside the existing claim(lock_path): a cluster-wide
Postgres advisory lock, visible to every host talking to the same database.
The holding connection is dedicated and kept for the process lifetime
(advisory locks are session-scoped); a dead connection is dropped and
re-election retried on the next call.
claim_leader() dispatches between the two mechanisms from env:
THERMOGRAPH_SINGLETON_PG (+ Postgres) -> claim_pg; else
THERMOGRAPH_SINGLETON_LOCK -> claim; else always leader, unchanged. Wired in
at web/app.py in place of the direct claim() call. Off by default, so
today's single-host behavior is unaffected.
Replace the per-cell parquet cache with TimescaleDB hypertables as the
production backend for the raw daily climate record, and drop pg_duckdb.
Parquet stays the backend whenever THERMOGRAPH_DATABASE_URL is not a
Postgres URL (dev, tests, offline tooling), the same dialect switch the
accounts DB and derived store already use, so CI stays Postgres-free.
- data/climate_store.py: psycopg + polars bridge over climate_history
(a hypertable), climate_recent, and climate_sync (per-cell freshness).
Reads via pl.read_database, writes via COPY + ON CONFLICT upsert;
fail-soft to a cache miss so a DB hiccup degrades to upstream refetch.
- data/climate.py: route every cache/mtime touchpoint through a backend
dispatch. recent_stamp becomes int(recent_synced_at) on Postgres; the
stale-serve path still avoids bumping it, so derived-payload tokens
invalidate on exactly the same events as before.
- alembic 0002: CREATE EXTENSION timescaledb plus the hypertable schema
(compression policy on year-old chunks), guarded to no-op off Postgres.
- migrate_cache_to_pg.py (make migrate-cache): idempotent backfill of the
parquet cache into the hypertables, preserving file mtimes as sync
timestamps so recent_stamp is unchanged across cutover.
- db image -> stock timescale/timescaledb:latest-pg18; drop the custom
pg_duckdb Dockerfile, the read-only /parquet mount, and the duckdb
tuning GUC. Docs updated for the new backend and cutover.
Co-authored-by: Claude <noreply@anthropic.com>
A self-hosted Open-Meteo instance that isn't backfilled yet answers historical
requests with all-null values (or only its recent sync window), which
_finalize_frame reduces to a near-empty frame. That frame is not None, so the
loader would accept it as source=open-meteo and cache it indefinitely as a
complete record — never falling back to NASA and never self-healing after the
backfill lands. Require at least MIN_ARCHIVE_DAYS before accepting an archive
result; below that, fall through to the NASA backup and don't cache the short
frame. Guards the prod cutover window.
Get the 45-year historical record off the rate-limited public Open-Meteo
archive API by running a private Open-Meteo instance that serves the
era5_seamless blend (0.1° ERA5-Land + 0.25° ERA5 for gusts) from the
compressed .om archive in object storage, mounted on the host with rclone.
- climate.py: make ARCHIVE_URL env-driven (THERMOGRAPH_ARCHIVE_URL) and pin
models=era5_seamless on the archive fetches only, so a self-hosted instance
serves the same 0.1° resolution; forecast path unchanged. The public API's
default is already seamless, so dev/beta (URL unset) behave identically.
- docker-compose.openmeteo.yml: open-meteo-api + two rolling sync workers
(era5_land 0.1°, era5 0.25° for gusts), bind-mounting the object-storage
mount; the overlay points the app at the local instance.
- Makefile: om-up / om-down / om-backfill (one-time full-history backfill).
- Terraform: per-host openmeteo flag layers the overlay, renders OM_DATA_DIR,
and provisions the host rclone systemd mount from the bucket credentials.
- deploy/openmeteo: operator runbook + rclone mount unit template.
Run Thermograph as a docker-compose stack (app + Postgres 18) and standardize the
data layer on Postgres, while keeping the test suite on SQLite.
- accounts/db.py: DSN-driven engines. On Postgres, a per-worker read-write +
read-only asyncpg pair (the RO engine pins read-only transactions, used by the
pure-GET endpoints) plus a sync psycopg engine for the notifier thread; the
SQLite path is preserved for tests/local (selected when THERMOGRAPH_DATABASE_URL
is unset). models.py: boolean server_default -> sa.false().
- store.py / metrics.py: dialect-flexible — Postgres UNLOGGED tables via psycopg
when configured, else the existing raw-sqlite3 paths byte-for-byte; sync
interfaces and every fail-soft contract preserved.
- Alembic (backend/alembic/) manages the accounts schema; the container entrypoint
runs `alembic upgrade head` before uvicorn (4 workers). migrate_accounts_to_pg.py
copies the accounts data SQLite->PG through the ORM (UUID/bool/JSON coerced),
skips access_token, and resets identity sequences.
- Dockerfile + docker-compose.yml: app image (uvicorn, 4 workers, loopback 8137)
and a Postgres 18 db (2 CPUs) running pg_duckdb (deploy/db/) so the parquet
climate cache is queryable in-DB via read_parquet('/parquet/cache/*.parquet').
- deploy.sh/thermograph.service rewired to manage the compose stack; env example,
Makefile targets (up/down/db-up), and deploy/POSTGRES-MIGRATION.md cutover runbook.
Tests stay on SQLite (dialect fallback) — 323 pass. The full Postgres stack was
verified via docker compose: alembic migrations, register/login, the RO endpoint,
store/metrics round-trips, and the accounts data migration.
The overall climate-shift score was a weighted roll-up (feels-like and humidity
2x, wind and gusts 0.5x, and so on). Make it an equal average instead: every
metric — and, through each metric's own mean over the 10/25/50/75/90 percentiles,
every percentile — counts the same.
- scoring._overall: plain mean of the present metrics' divergences, mapped to
0-100, replacing the weighted sum. Missing metrics still drop out cleanly.
- Drop the now-unused per-metric weights from the METRICS table and the scored
entries (nothing rendered them; only the roll-up read them).
- views.SCORE_VER s4 -> s5 so cached weighted scores are recomputed.
- score.js: the hero note now reads "Every metric counts equally" instead of
"Temperature, feels-like and humidity are weighted most".
Verified against Seattle: overall mad equals the equal mean of the eight metric
mads (7.9), and the entries no longer carry a weight.
Claude-Session: https://claude.ai/code/session_013dRZmX9D3JEntfMKWMTWZ8
* Centralize filesystem paths in a single module
Add paths.py, which resolves the repo root once and derives the cache,
accounts DB, logs, templates, frontend and bundled-city-data locations
from it. Replace the 13 per-module `dirname(__file__)/..` anchors with
references to it, so a module's location no longer determines where the
app reads its data. Env overrides (accounts DB, VAPID, IndexNow) are
unchanged; every resolved path is byte-identical to before.
Groundwork for moving modules into packages without re-pointing paths.
Claude-Session: https://claude.ai/code/session_01XXxmNFy9cZ6Gh8Y9thZn62
* Split the backend into domain packages
Group the flat backend modules into packages that mirror their concerns:
data/ climate, grading, scoring, grid, places, cities,
city_events, store
web/ app, views, homepage, content, schemas
notifications/ notify, digest, push, mailer, discord,
discord_interactions, discord_link
accounts/ models, users, api_accounts, db
core/ metrics, singleton, audit
Intra-project imports are rewritten to the package-qualified form. The
entry scripts (indexnow, warm_cities, migrate, gen_cities, gen_flavor)
and paths.py stay at the backend/ root, and backend/app.py becomes a
shim re-exporting web.app:app so the launch target stays `app:app` —
run.sh, the systemd units, and CI need no change.
Verified: full suite (318) passes, `uvicorn app:app` boots and serves
the home/SEO/static/API surfaces, and every root script imports clean.
Claude-Session: https://claude.ai/code/session_01XXxmNFy9cZ6Gh8Y9thZn62
Rain intensity gains a Severe tier and Dry becomes strictly no-rain:
- The top half of Very Heavy (95th–99th rain-day percentile) becomes a new
"Severe" tier; Very Heavy keeps the 90–95 band. Eight rain tiers now — Trace /
Light / Brisk / Typical / Heavy / Very Heavy / Severe / Extreme — which refill
the wet-2..wet-9 colour ramp contiguously (no gap), so Heavy/Very Heavy shift
one shade lighter and Severe takes the second-darkest.
- A day is Dry only when it didn't rain at all; any measurable rain, however
slight, is at least Trace. _grade_precip splits on > 0 rather than the 0.01"
threshold (which still governs the separate dry-streak metric).
- The distribution strip drops the range under the Dry column — every dry day is
zero, so a "0–0" span was noise.
_precip_ladder derives its percentile marks from RAIN_BANDS now, so adding or
splitting a tier can't leave a hard-coded list behind (that was the bug the 95th
mark would have hit). The detail-view ladder derives from the band table as before.
Claude-Session: https://claude.ai/code/session_013dRZmX9D3JEntfMKWMTWZ8
Move the flat backend/tests/*.py into domain subfolders so the suite
mirrors the code's concerns:
data/ climate, grading, scoring, grid, places, store
web/ api, content, homepage, views
notifications/ notify, digest, discord (+ dm/interactions/link)
accounts/ api_accounts
core/ metrics, singleton, dashboard
conftest.py stays at the tests/ root, so its sys.path setup and shared
fixtures still apply to every subfolder. The four tests that derive repo
paths from __file__ get their depth bumped one level to match their new
location. Pytest discovers the subfolders recursively; the CI command
(python -m pytest backend/tests) is unchanged.
Claude-Session: https://claude.ai/code/session_01XXxmNFy9cZ6Gh8Y9thZn62
The rain-intensity scale drops its two hyphenated compound labels for single
words and loses a tier, going from eight to seven:
Trace / Light / Brisk / Typical / Heavy / Very Heavy / Extreme
- Light–Mod -> Brisk, Moderate -> Typical (renames only; classes unchanged).
- Mod–Heavy is merged up into Heavy, whose floor drops from the 75th to the 60th
rain-day percentile, so Heavy now spans 60–90.
- The lightest tier (already the merged Very Light) is renamed Trace.
grading.py RAIN_BANDS and the frontend SCALE_RAIN mirror stay in lockstep, and
the detail-view ladder derives from the table so it follows automatically. The
now-unreferenced wet-6 colour token is kept so any day still cached under that
class renders until the derived store recomputes; the chart's percentile fan also
keeps it for a smooth gradient.
Claude-Session: https://claude.ai/code/session_013dRZmX9D3JEntfMKWMTWZ8
The account menu offered 'Link Discord' to every signed-in user even on a
server with no Discord OAuth app configured, where it dead-ends (the start route
303-bounces to /alerts). Add GET /api/v2/discord/config reporting whether linking
is enabled, and have the menu render the entry only when it is. On a server
without Discord set up nothing surfaces; setting the OAuth env vars later makes
the entry appear on its own — no code change needed to turn it on.
The rain-intensity scale had a Trace tier (below the 1st percentile of a place's
rain days) sitting under Very Light — a sliver category that mostly showed 1% and
crowded the distribution strip. Fold it into Very Light, which now bottoms out the
scale at 0, so the lightest measurable rain reads as Very Light.
- grading.py: RAIN_BANDS drops the Trace band; Very Light's floor goes 1 -> 0. The
detail-view ladder derives from the table, so it follows automatically.
- shared.js: SCALE_RAIN drops the Trace row; Very Light's range becomes "<10".
Eight rain tiers now instead of nine; the strip shows one fewer column. The wet-1
colour token is kept (unreferenced by new gradings) so any day still cached with
the old class renders until the derived store recomputes.
Claude-Session: https://claude.ai/code/session_013dRZmX9D3JEntfMKWMTWZ8
The note explaining the hidden digest form contained a literal "{# … #}" in its
prose. Jinja doesn't nest comments, so the inner "#}" closed the comment early and
the trailing "wrapper. #}" rendered as visible text at the top of every footer.
Reworded the note without comment delimiters. Adds a test asserting no base-
template page's footer contains a stray "{#"/"#}" (fails against the old template).
Claude-Session: https://claude.ai/code/session_013dRZmX9D3JEntfMKWMTWZ8
Adds Discord DM as a notification channel beside web push, for users who linked
their Discord account and opted in. It rides the same fan-out point as push
(_dispatch_discord next to _dispatch_push in the notifier pass), reusing the
transport-agnostic title/body/deep-link, and stays best-effort and isolated: the
in-app row is already committed, and push/email remain the fallback for anyone
Discord can't reach (its DM rule requires a shared server / user install).
- discord.py: send_dm() opens the DM channel then posts an embed via the bot token,
with one capped 429 retry. Absolute deep links (a DM can't resolve a relative
path the way the service worker can).
- notify.py: _dispatch_discord() delivers only when the subscriber has a linked
discord_id and discord_dm on; no-ops entirely when no bot token is configured.
- models.py: User.discord_dm (opt-in flag) + migration 002. Linking sets it True
(an active opt-in); a new POST /discord/dm mutes it without unlinking, and unlink
clears it. Exposed on UserRead; account popover shows an on/off toggle.
Claude-Session: https://claude.ai/code/session_013dRZmX9D3JEntfMKWMTWZ8
Lets a signed-in user connect their Discord account (OAuth2 identify), storing the
Discord user id that a later feature (DM alerts) will deliver to. Standard
authorization-code flow, all server-side:
- backend/discord_link.py: /discord/link/start redirects to Discord's consent
screen; /discord/link/callback exchanges the code, reads the Discord user id, and
stores it; /discord/unlink forgets it. The `state` is signed with the app auth
secret (stdlib hmac, no new dependency) and carries the Thermograph user id, so a
callback can't be replayed or bound to another account. Every route requires an
active session, so linking acts on whoever is actually logged in.
- models.py: User.discord_id (unique, nullable). schemas.py exposes it on UserRead
so the frontend can show link state.
- app.py: the router under /api/v2/discord.
- account.js: a "Link Discord" / "Unlink Discord" control in the account popover.
- deploy/migrations/001-user-discord-id.sql: the manual column add for the existing
prod accounts DB. This project has no Alembic — create_all only makes missing
tables, so an added column needs a hand-applied migration (documented in-file).
- env example: THERMOGRAPH_DISCORD_CLIENT_SECRET + the redirect to register.
Claude-Session: https://claude.ai/code/session_013dRZmX9D3JEntfMKWMTWZ8
Adds Discord slash commands with no bot process and no gateway connection: Discord
POSTs each interaction to a FastAPI route, and the app answers it. First command is
/grade <city>, returning today's grade for a curated city from the warm cache
(reusing homepage._grade_city, so it answers well within the 3-second deadline and
costs no upstream quota).
- backend/discord_interactions.py: Ed25519 verification (PyNaCl) over the RAW
request body — Discord probes the endpoint with bad signatures and disables it if
they aren't rejected with 401. Routes PING to PONG and application-commands to
their handler; unknown/unsupported interactions are acknowledged, not errored.
City lookup is exact-name-then-prefix over the population-sorted city set; unknown
or not-yet-warm cities get an ephemeral note.
- app.py: POST {BASE}/discord/interactions, reading request.body() (not json()) so
the bytes match the signature.
- scripts/register_discord_commands.py: one-off upsert of the command definitions
via Discord REST (app id + bot token).
- PyNaCl added to requirements; Discord public-key / app-id / bot-token documented
in the env example. Endpoint URL: https://thermograph.org/discord/interactions.
Claude-Session: https://claude.ai/code/session_013dRZmX9D3JEntfMKWMTWZ8
Adds an optional daily Discord post of the day's most anomalous cities, built from
the same data/homepage.json feed the homepage renders — so the post and the site
always agree. Delivery is a single incoming-webhook POST (no bot, no gateway, no
new process): it rides the notifier daemon like the feed refresh does, once per
day after the refresh, leader-only. With no webhook configured it no-ops.
- backend/discord.py: build_embed() renders the top cities as a rich embed (each
line: city, reading in °F and °C, the normal for context, percentile + grade;
non-today readings are dated). post_daily_feed() is best-effort — it skips a
stale/empty feed and never raises, so a Discord failure can't disturb a pass.
- notify.py: a once-a-day guard (_maybe_post_discord) beside the homepage-refresh
guard; only marks the day done once a post lands, so a transient failure retries.
- deploy/thermograph.env.example: THERMOGRAPH_DISCORD_WEBHOOK (the URL is the
credential — env only).
The webhook value is the top anomaly's tail for the accent colour and reuses the
feed's build time as the embed timestamp. httpx (already a dependency) does the POST.
Claude-Session: https://claude.ai/code/session_013dRZmX9D3JEntfMKWMTWZ8
- Explain on the page what wet-bulb temperature measures (the evaporative-cooling
ceiling on shedding heat), so the metric isn't opaque.
- Report the share of heat-stress "wet-bulb" days (peak wet bulb >= 26 C) vs
normal days, recent window vs the full record — mirroring the precip wet-day
frequency.
- Present the overall total as a direction-agnostic net change (magnitude only),
not "warmer/cooler"; per-metric cards still carry direction.
Bumps the score cache version.
Score how far a location's last 6 years have drifted from its full 45-year
record. For each metric and percentile category (p10/p25/p50/p75/p90), the
recent-years value is placed on the baseline distribution and the gap from the
expected percentile is the divergence — unit-free, so metrics compare directly.
Scored per meteorological season plus annual, weighted into per-metric and
overall scores (temps, humidity and feels-like weighted heaviest).
- backend/scoring.py: divergence math, seasonal slicing, precip zero-inflation
split (wet-day frequency + amount), tier mapping onto the existing temp scale.
- climate.py: derive a wet-bulb column (Stull 2011) at the read boundary, before
the humidity column is converted to absolute — via a shared _derive_metrics
wrapper at all four read sites.
- api/v2/score endpoint + build_score payload, cached on the history token with
a scoring-version key.
- frontend score page: overall hero, per-metric cards, by-season chips, and a
button-revealed summary (sentences + metrics×season table + per-percentile
detail). Score nav link across all headers.
- Tests for the scoring math, wet-bulb formula, payload shape and route.
* Hide footer email digest signup for now
Comment out the compact digest signup form in the site footer; restore
by removing the surrounding Jinja comment wrapper.
Claude-Session: https://claude.ai/code/session_01XXxmNFy9cZ6Gh8Y9thZn62
* Update digest test for hidden footer form
The footer signup form is now hidden; assert its absence site-wide.
The /digest endpoint stays live and its tests are unchanged.
Claude-Session: https://claude.ai/code/session_01XXxmNFy9cZ6Gh8Y9thZn62
Two fixes to how measurements read.
Rain goes back to millimetres, rendered as whole numbers — that is how rainfall
is reported, and a tenth of a millimetre is below what the source resolves.
Inches keep two decimals, since a whole inch of rain is a lot to round to.
The distribution strip's low-high span moves from inside each bar to just under
the average, above it. A column is ~38px on a phone, so the old "Max 62° / Min
17°" pill clipped — it had already been shrunk to 8px to cope, and on a
high-rainfall city with longer values it lost both ends of the label, because the
text is centred and overflow-hidden. Above the bar it has the full column width
and needs no scrim to stay legible over nine tier colours.
The span is now bare numbers ("17-62"), since the average directly above it
carries the unit; repeating " g/m³" on both ends is what made it too wide in the
first place. It uses the same precision as that average, or the two lines
disagree ("52mm" over "12.7-136.91"). Sub-1 inch values drop their leading zero
so precip's ten columns still fit a phone.
With nothing inside the bars, the height floor drops from 30px to 14px: it only
has to keep a non-empty bucket visible now, so the heights encode frequency more
honestly. Category names get 2px of side padding, which stops long neighbours
("Light-Mod" beside "Moderate") reading as one word.
Verified at 390/412/1920 in both themes across all four calendar metrics and the
compare strips, in imperial and metric: nothing clips.
Claude-Session: https://claude.ai/code/session_013dRZmX9D3JEntfMKWMTWZ8
Precipitation on a °C page now reads "4.90 cm" where it read "49.0 mm".
Both units keep two decimals. cm and inches are only 2.54x apart, so cm needs the
same resolution inches have — at one decimal every light-rain day would collapse
to 0.0 or 0.1 cm, and the source itself is hundredths of an inch.
Ingest is untouched: climate.py still converts the upstream mm to inches, which
remains the stored unit and the value carried in data-precip-in.
Claude-Session: https://claude.ai/code/session_013dRZmX9D3JEntfMKWMTWZ8
Defaulting climate pages to the city's temperature convention left every page
half-converted: a Delhi reader got °C next to "9 mph" and "0.00 in". One flag now
drives every measure — picking °C also gives mm and km/h.
Absolute humidity stays g/m³ in both systems; there is no imperial unit for it
anyone would recognise, so converting it would only make the page worse.
units.js becomes the single source for all of it. The formatting logic existed in
three independent copies — shared.js's fmtPrecip/fmtWind/fmtHumid, a second set
inside fmtMetricVal, and a third baked into chart.js's series labels, axis ticks
and tooltip — which is why the units drifted apart in the first place. shared.js
now re-exports from units.js so its importers are unchanged, and chart.js carries
a per-series unit `kind` in place of sniffing for a "°" suffix, which could not
have extended to three unit families.
Server-rendered pages gain the carrier they were missing: precipitation and wind
now render as <span class="precip" data-precip-in> / <span class="wind"
data-wind-mph>, the same contract temperature already had, so climate.js can
repaint them on toggle and the attribute stays imperial as the source of truth.
Precision follows the unit: millimetres get one decimal where inches get two.
0.04 in and 1.0 mm are the same amount, and "1.02 mm" would advertise precision
the source doesn't have. Wind rounds half-up to match Math.round, as temperature
already does.
The weekly table's wind and rain cells are bare numbers to keep the grid narrow,
so their row labels now carry the unit and rebuild per render. Static copy that
names a unit — the legend's "(mph)", "(inches)", and a "(°F)" that had been wrong
for °C readers since the country defaults landed — is marked up with
data-unit-label and painted from the same source.
The chart keeps its imperial domain; only labels convert, so point positions and
the fan geometry are untouched.
Push and email bodies are still imperial-only (notify.py): they are composed
server-side with no client to repaint them and no per-user unit stored, which is
the same limitation temperature already has there. Left for a follow-up.
Claude-Session: https://claude.ai/code/session_013dRZmX9D3JEntfMKWMTWZ8
Search Console shows impressions skewing hard to °C countries — India, South
Africa, New Zealand, the UK, Australia — while every climate page rendered °F
into the HTML. Search snippets quote the server-rendered text, so a reader in
Delhi saw a temperature they had to convert in their head before the page was
worth a click.
Climate pages now render temperatures in the city's country convention: °F for
the US and the handful of other Fahrenheit countries, °C everywhere else. This
happens server-side, so it holds with no JS and shows up in the snippet.
data-temp-f keeps the raw Fahrenheit in every case — it stays the conversion
source of truth, and climate.js repaints from it on toggle exactly as before.
The visitor's stored choice still wins over everything; the page default only
sits between that and the locale guess, so a first-time visitor stops seeing a
°F→°C repaint on load for the majority of our traffic.
The unit rides on a ContextVar rather than a parameter because _temp() is
reached from both directions — templates call it as a Jinja global, and the
context builders call it directly to pre-render the records tables into strings.
Threading an argument would mean touching ~20 call sites across three levels of
nesting, where missing one yields a silently wrong unit instead of an error. The
scope wraps the context builder as well as the render, since the builder is
evaluated as an argument and runs first, and it resets in a finally: the page
handlers are sync `def`, so Starlette runs them on a recycled threadpool thread
and a value left set would leak into the next request that thread serves.
Conversion rounds half-up to match JS's Math.round rather than Python's
round-half-to-even, so the server prints the number climate.js would and there
is no flicker on load for a visitor whose unit already matched.
Pages with no single city — the homepage strip, the hub, the interactive views —
set no scope, emit no data-unit-default, and keep today's locale behaviour.
Precipitation, wind and humidity stay imperial; that inconsistency is worth a
follow-up but is outside this change.
Two existing assertions were passing for the wrong reason: base.html.j2 mentions
"°F/°C" in an HTML comment, so a bare `"°F" in html` check succeeded even on a
page whose every temperature was Celsius. They now read the rendered spans.
Claude-Session: https://claude.ai/code/session_013dRZmX9D3JEntfMKWMTWZ8
Search Console's first 48 hours showed city-month pages drawing impressions at
positions 44-77 and converting almost none of them: the title template spent its
width on region and country ("Average weather in London, England, United Kingdom
in July"), so Google cut it before the month a searcher was looking for.
Titles now lead with the bare city name and put the subject next, which keeps
both inside the first ~40 characters:
London in July: normal weather & records
London climate: daily normals, records & how unusual it is now
London weather records: hottest & coldest days since 1980
Add cities.title_name() for the short form. 968 of the 1000 cities have a unique
name and render bare; the 32 that don't are qualified with their country code
("London, GB"). Five names — Columbus, Arlington, Aurora, Glendale, Gorakhpur —
recur *within* one country, where the country code collides too and would hand
two different pages the same title, so those fall back to admin1 ("Columbus,
Ohio"). A test asserts every rendered title is unique across the set.
Meta descriptions now lead with the page's actual numbers rather than restating
the template ("London averages highs of 73°F in July and lows of 38°F in
January."), capped at the ~155 characters a SERP shows.
The records title derives its year from the loaded history rather than a fixed
date, so it can't contradict the page. H1s, URLs, breadcrumbs and JSON-LD are
unchanged; og:title and og:description follow <title>/<meta> automatically
through base.html.j2.
Claude-Session: https://claude.ai/code/session_013dRZmX9D3JEntfMKWMTWZ8
The homepage strip floored percentiles into 1-99 while the Day page rendered
"100th pct" for the same reading. Rather than teach the Day page a second copy
of the rule, put it in grading.pct_ordinal() and have every surface defer to it:
the Day page, the calendar tooltip, the chart labels, the city pages and the
strip.
Two bugs fell out of unifying them:
- The city pages rendered `{{ c.percentile }}th pct` against a float, so all
~1000 of them showed "97.1th pct", "66.4th pct", "16.5th pct" — and would say
"1th"/"2th"/"3th" for the low tail. Live in prod.
- Python's round() is half-to-even and JavaScript's Math.round is half-up, so
16.5 gave "16th" server-side and "17th" client-side. The Python helper uses
floor(x + 0.5) to match the frontend exactly; a cross-language check over every
.5 boundary now agrees on all of them.
The frontend mirrors the rule as pctOrd() in shared.js, replacing the bare ord()
at all five percentile call sites (day.js, calendar.js, chart.js x2, app.js).
ord() stays as the general ordinal helper it always was.
A card read "Shanghai · 30°C · Near Record hot · Low temp · 99th pct" while the
Day page for the same cell said 26°C, 66th pct, Above Normal. Both were right
about the climatology — the normals matched exactly — but the strip was grading
a reading from several days earlier and presenting it as current.
Two causes, both mine:
- load_cached_recent_forecast deliberately ignores how stale the file is, and
_grade_city took the newest row <= today with no age check at all.
- Nothing keeps that cache current for the tracked cities. warm_cities.py skips
any city whose archive is already cached, so it never re-fetches their forecast
bundle, and the notifier only touches subscribed cells. So "zero upstream
calls" was satisfied by reading a cache nothing refreshes.
Fixes:
- Drop readings older than MAX_OBSERVATION_AGE_DAYS (1) instead of ranking them.
Cities span a day of timezones, so a current observation can still be dated
yesterday in UTC — hence 1 rather than 0.
- Top up the stalest cells' recent/forecast each pass, oldest first and capped
(THERMOGRAPH_HOMEPAGE_REFRESH, default 40), the same shape as the notifier's
archive-fetch budget. This is a real change: the sweep is no longer zero-cost
upstream, because it cannot be and still say "right now". Documented in
DEPLOY.md, and refresh(fetch=0) keeps a strictly cache-only rebuild.
- Rank over the top THERMOGRAPH_HOMEPAGE_CITIES (250) rather than all ~1000. The
strip shows ~12 cards; a smaller genuinely-fresh set beats a large stale one.
- Show the observation date on a card when it isn't today's, so a yesterday
reading can't silently pass for current.
Every card in "Unusual right now" read the same: a city, "100th", "Near Record".
Nothing said what was unusual, by how much, or even whether it was hot or cold.
Cards now lead with the reading and say what it is measured against:
Tehran, Iran
108°F
Near Record hot
High temp · 99th pct · normal 97°F
- "Near Record" is the band label at BOTH ends of the scale, so a cold extreme
and a hot one rendered identically and colour was the only thing telling them
apart. Qualify that tier with its direction; the other tiers already read
directionally ("Very High", "Low") and are left alone.
- Carry the ±7-day median through to the card. A percentile means little without
the value it describes and the normal it departs from.
- Floor the percentile label into 1-99. An empirical percentile can round to 100,
and "100th percentile" reads as a measurement error.
- Shorten the city label to "City, Country". display_name keeps admin1 when it
differs from the city, which gave "Dar es Salaam, Dar es Salaam Region,
Tanzania" — truncated to nonsense in a one-line card.
- Temperatures render as .temp spans and the homepage now loads climate.js, so
the strip follows the °F/°C toggle like every other server-rendered page.
Colour: the strip cards and the explore cards take a wash and a border from
their token instead of being uniform grey, and Calendar carries the nine grade
tiers in order — it is a miniature of the calendar itself.
Text colours mix 50% band colour with the foreground. That is the most colour
they can carry and still clear 4.5:1 against the tinted card in both schemes;
the worst case is the deliberately-dark --rec-cold/--rec-hot tiers, which at the
previous 78% measured 2.33:1 in dark mode. Measured, not eyeballed: worst text
contrast is now 4.79 (light) and 4.81 (dark).
The mark and wordmark were already a home link on the Jinja pages (homepage,
about, climate, glossary, privacy), but the five static tool pages — calendar,
day, compare, legend, alerts — built their header separately and left the brand
as a bare <h1>, so clicking it did nothing.
Wrap the lockup in `<a href="./">` there, matching the relative form the nav
already uses: it resolves to the app root whether the app is served at the
domain root or under a base path.
The link wraps the whole lockup, so the mark is clickable too, not just the
word. Its colour/decoration moves from `.site-name a` up to the shared
`.brand h1 a, .brand .site-name a` rule so it covers both the static pages'
<h1> form and the homepage's <p>, with an accent hover so the affordance is
visible.
Adds a parametrized test over every page: the two header implementations are
easy to change independently, so this asserts each one carries a home link that
wraps the mark.
The homepage header lost its wordmark treatment: the mark baseline-aligned
instead of centring (riding ~8px high) and the wordmark fell back to 20px Inter
regular instead of the 26px monospace used everywhere else.
The lockup is styled by `.brand h1` — an element selector. The homepage renders
the brand as `<p class="site-name">` so its hero headline can own the page's
only h1, so it matched none of it: no flex centring, no gap, no monospace, no
weight. Measured against /about at 393px:
homepage <p> Inter 20px w400 display:block mark -8.5px
/about <h1> ui-monospace 26px w700 display:flex mark -1.2px
Match `.brand .site-name` alongside `.brand h1` (and the same for the inner
link and the compact-header size). Both now measure -1.2px, the intended
optical nudge.
Adds a test for the invariant. Nothing else caught this: the page rendered,
every test passed, and only the rendering was wrong — so the test asserts the
lockup rule covers the class the homepage actually uses, and fails if the
selector is narrowed again.
The button was dead on plain HTTP and failed silently everywhere.
Browsers only expose geolocation on secure origins, but `navigator.geolocation`
still EXISTS on http:// — getCurrentPosition just fails with a permission error
("Only secure origins are allowed"). The guard was `if (!navigator.geolocation)
return;`, which passes on http://, so the call went ahead, errored, and landed
in an empty error callback. Nothing happened at all.
`make lan-run` serves plain HTTP on 0.0.0.0:8137 so phones can reach it, so the
button could never work in the normal dev-and-phone workflow, and gave no hint
why. It would have worked on thermograph.org, which is HTTPS.
- Gate on `window.isSecureContext`, not just the API's presence. Where the
browser will refuse, hide the locate button and promote "Pick on the map" to
primary rather than offering a control that cannot work.
- Report failures: a declined prompt, a timeout, and an unavailable fix each get
their own message in an aria-live slot. A silent failure is indistinguishable
from a broken button, which is exactly how this went unnoticed.
- Show a "Locating…" state while the fix is pending; it can take seconds.
To exercise geolocation against the LAN dev server, serve it over TLS:
`make lan-run TLS=1` (self-signed cert into certs/).
Verified by driving the button in a real browser across all three cases:
insecure origin (button hidden, map promoted), permission granted (hero
re-points at the located place), permission denied (message shown, button
restored).
The homepage was a bare tool: a find-bar and an empty panel reading "Find a
location to begin." A visitor arriving from a search result or a shared link
learned nothing about what the product does before deciding to leave.
Rebuild it around the Weekly view, which is untouched:
- Hero with the question as the h1, and a grade card showing a real graded
example — the most unusual city we're currently tracking, either tail. The
card's frame and text slots are server-rendered with reserved heights, so
app.js re-pointing it at the visitor's own place shifts nothing.
- "Unusual right now" strip, CSS scroll-snap, no JS carousel. A cold-tail city
is force-included whenever one qualifies.
- Stance line, how-it-works, explore cards, and 12 city chips linking into the
~1000-page /climate surface.
- Monthly digest form, in the footer of every page.
Serve / from Jinja instead of a static file with placeholder substitution, so
crawlers and no-JS readers get the whole page as real HTML. home.html.j2
extends base.html.j2 and carries app.js's DOM contract over verbatim; the brand
degrades to a <p> so the hero owns the sole h1. frontend/index.html is deleted
rather than left behind the static mount, where it would keep serving indexable
duplicate content.
"Where is it most unusual right now" has no cheap answer at request time —
percentiles live inside zlib-compressed payload blobs with no column to sort on.
So homepage.py sweeps the warm cache and writes data/homepage.json, read by the
template. The sweep is strictly cache-only (climate.load_cached_recent_forecast
is new, the sibling of load_cached_history), so grading ~1000 cities costs zero
upstream requests. It rides the notifier's timer behind an hourly guard rather
than starting a second daemon, and also runs at the tail of warm_cities so a
fresh deploy has a populated feed.
Instrumentation: metrics gains a product-event counter keyed by (event,
referrer domain, UTC day), behind an allowlist and a per-IP rate limit, fed by
POST /api/v2/event. The referrer is taken from the request's own header, never
from the client. The beacon gets its own inbound category that record_inbound
ignores, so reporting an interaction doesn't also count as traffic. The
dashboard grows an events block with per-referrer attribution.
Email is scaffolded but sends nothing yet. mailer.py talks stdlib smtplib to a
local Postfix null client on 127.0.0.1:25 (deploy/provision-mail.sh), so the
choice between direct-to-MX and relaying through a provider stays a Postfix
config change with no code change. The backend defaults to "console", which
logs and sends nothing, so dev and tests exercise the whole signup path safely.
Signups land in pending_digest unconfirmed; collecting the list shouldn't wait
on delivery.
Also adds /privacy, linked from the footer and kept out of the sitemap.
The strip's classes are named unusual-* rather than record-*: .record-card is
already the SEO records page's, and reusing it leaked layout rules onto
/climate/<slug>/records.
conftest redirected AUDIT_DIR and ERROR_DIR to a throwaway tmp dir but left
ACCESS_DIR pointing at the repo's logs/access/. The request-logging middleware
runs under TestClient too, so every suite run appended its deliberate error-path
requests (401/404/409/422/502/503) to the live access log that the ops
dashboard reads, inflating its traffic and error tallies. Point ACCESS_DIR at
the tmp dir like the other two.
Co-authored-by: root <root@vmi3417050.contaboserver.net>
The ops dashboard marked subscription-notifier DOWN ~2/3 of the time after the
notifier was gated to a single leader worker. It judged the daemon by whether
its thread was present on whichever worker answered /api/v2/metrics, but only
the leader runs it — so polls landing on the other two workers saw no thread and
reported DOWN for a perfectly healthy notifier. Per-worker thread checks can't
describe a single-leader daemon, and worse, they could no longer distinguish a
real crash from a normal non-leader.
Report it by liveness instead: the notifier stamps a heartbeat (timestamp +
expected interval) into the shared metrics store each loop iteration, including
once at startup so the dashboard shows it up immediately after a restart. Since
the beat lives in the shared SQLite store, any worker's metrics response carries
it, so the dashboard sees the notifier's health whichever worker it polls.
- metrics.py: both stores record/expose heartbeats (SQLite reuses the meta
table, so every worker's beats share one file); snapshot() converts each beat
to a server-computed age so readers needn't reconcile clock skew.
- notify.py: beat at loop start and every wake.
- dashboard.py: judge heartbeat daemons by freshness (up to 2 missed intervals
before DOWN, with beat age shown); neighbor-warmer keeps its per-worker thread
check, which is correct since every worker runs it.
- Tests: heartbeat store/snapshot shape + cross-worker sharing; dashboard
fresh/stale/missing rendering and the unchanged warmer check. 194 passed.
Co-authored-by: root <root@vmi3417050.contaboserver.net>
Google Search Console flags city pages with 'Missing field item (in
itemListElement)': the country crumb has no page of its own, so its
ListItem was emitted without the required item URL. Google only allows
the final ListItem to omit item, so drop unlinked intermediate crumbs
from the structured data (the visible breadcrumb is unchanged) and
renumber positions. Shared helper now builds the BreadcrumbList for
both the city and records pages, with a regression test parsing the
emitted JSON-LD.
Claude-Session: https://claude.ai/code/session_01KdTZCjpLeD26ZbXe6ApjpR
Prod runs 3 uvicorn workers, but the in-process subscription notifier was
started in every worker's lifespan (it was written assuming a single-worker
deploy). Its sweep fetches recent-forecast/archive data from Open-Meteo on a
15-min timer independent of any request, so running it 3× tripled the
background upstream load against a shared quota — the source of the overnight
429/503 rate-limit errors in logs/errors, and 3× redundant subscription scans.
Elect one worker to own the notifier via a non-blocking exclusive flock on a
lockfile (THERMOGRAPH_SINGLETON_LOCK): the first worker wins and holds the fd
for its lifetime; others stand down; the OS releases the lock if the leader
dies, so a restart re-elects cleanly. Unset (single worker / dev / tests)
always wins, so behavior there is unchanged. Mirrors the shared-metrics store's
env-selected cross-worker coordination.
The neighbor warmer stays per-worker on purpose: each worker drains its own
request-fed queue, so gating it would leave non-leader queues undrained.
- backend/singleton.py: flock-based leader election (claim()).
- backend/app.py: gate notify.start() on singleton.claim().
- deploy/thermograph.service: default THERMOGRAPH_SINGLETON_LOCK to
data/notifier.lock (needs the unit reinstalled on prod to take effect).
- deploy/thermograph.env.example: document it alongside WORKERS.
- Tests: leader election — single-worker default, first-wins, idempotent
re-claim, second-holder stands down, re-election after release. 183 passed.
Co-authored-by: root <root@vmi3417050.contaboserver.net>
* Serve the app on thermograph.org at root; redirect emigriffith.dev/thermograph
Move Thermograph to its own domain. thermograph.org serves the app at its root,
and emigriffith.dev/thermograph* permanently redirects there (prefix stripped,
so deep links map straight across).
- app.py: BASE now supports an empty root prefix. THERMOGRAPH_BASE=/ (or empty)
yields BASE="" so routes sit at "/", "/calendar", "/api/v2/…" with no double
slashes; the bare-base redirect is skipped and the static mount falls back to
"/". Non-empty values keep the existing "/thermograph" sub-path behavior
unchanged (backward compatible).
- deploy/Caddyfile: add a thermograph.org site block reverse-proxying to the
uvicorn on 127.0.0.1:8137; turn emigriffith.dev's /thermograph handler into a
permanent redirect to thermograph.org.
- deploy/thermograph.env.example: default THERMOGRAPH_BASE=/.
- DEPLOY.md: document the two-domain layout and that deploy.sh does not ship the
Caddyfile or env — those are applied on the VPS by hand.
The frontend already uses base-relative URLs, so it follows the root base with no
changes. Verified both modes locally (root serves /, /calendar, /api/v2/*; the
/thermograph sub-path still redirects bare→slash and 404s at root).
* Add dashboard (#144)
* Auto-submit IndexNow on deploy when the URL set changes (#130)
Hook the production deploy (deploy/deploy.sh) to ping IndexNow after the health
check passes, so new/changed pages reach Bing/DuckDuckGo/Yandex without a manual
run. Best-effort — wrapped so it can never fail a deploy — and it sources
/etc/thermograph.env so its key matches the one the running service serves.
To avoid re-blasting ~14k URLs on every code push, indexnow.py gains a
--if-changed mode gated by a signature of the URL set (persisted to
data/indexnow_state.txt): code-only deploys skip; a deploy that adds or removes a
city submits. `make indexnow` still forces a full submit.
* Add ops metrics endpoint + SSH/Termux dashboard (#131)
Adds observability for the running app, viewable over SSH (e.g. Termux):
- backend/metrics.py: thread-safe, best-effort in-process counters for inbound
requests (per category) and outbound calls (per external source), plus a
phase->source map and snapshot(). Hooked into climate._request (outbound, all
six weather/geocode sources) and the existing revalidate_static middleware
(inbound). Never raises into the request path.
- GET /api/v2/metrics: token-gated JSON of the counters + warm-queue depth +
thread names/uptime/pid. Returns 404 unless a valid THERMOGRAPH_METRICS_TOKEN
is presented or the request is direct loopback with no proxy X-Forwarded-*
header, so ops data is never exposed through Caddy on the public domain.
- scripts/dashboard.py (+ scripts/dash, make dashboard): stdlib-only terminal
dashboard. Reads cache (parquet + SQLite), accounts (users + active
subscriptions), and system resources (/proc + systemd) directly, and pulls
traffic from the metrics endpoint over loopback. Live refresh, --once, --json;
mobile-terminal width. Paths resolve relative to the script so the same tool
works on the LAN dev and prod VPS checkouts.
- Monitoring sections in DEPLOY.md / DEPLOY-DEV.md; unit tests for the counters.
* Dashboard: run under the app venv, not the box's default python3 (#132)
The box's default python3 can be a build without _sqlite3 (pyenv 3.10 here), so
./scripts/dash crashed on import. Prefer the repo's .venv interpreter (the Python
the server itself runs on, which always has sqlite3); fall back to python3/python.
Also make the sqlite3 import in dashboard.py optional so a fallback interpreter
degrades to empty cache/accounts panels instead of crashing, and point the
'make dashboard' target at the wrapper.
* Dashboard: redraw live view in place so it stops scrolling to the bottom (#133)
The live loop did a full clear-and-reprint each refresh; when the output is taller
than the terminal (common on a phone) that scrolls the view to the bottom every tick.
Redraw from the top instead: hide the cursor, clear once, then each frame move to home
and rewrite line-by-line (clear-to-EOL + clear-below), capping the body to the window
height so it never prints more lines than fit and never scrolls. Restore the cursor on
exit; piped (non-TTY) output prints plain frames.
* Dashboard: make the live view a scrollable in-place pager (#134)
The live mode is now an alt-screen pager (like top/less): it shows only the latest
snapshot, you scroll it with swipe/arrows/jk/PgUp-PgDn/g/G, and each refresh replaces
the page in place while keeping your scroll position — nothing gets appended below and
the view is never yanked to the top or bottom. On exit it restores the cursor, leaves
the alternate screen, and resets the terminal mode.
--once still prints a single static snapshot (also used automatically when stdout/stdin
aren't a terminal, e.g. piping); --json unchanged.
* Metrics: don't count the dashboard's own /api/v2/metrics polling (#135)
The ops dashboard polls the metrics endpoint every few seconds; counting those hits
just shows the monitor watching itself and inflates the inbound totals. Skip the
'metrics' category in record_inbound (so it's excluded from inbound_total too), and
hide it in the dashboard's inbound list defensively.
* Dashboard: add a STORAGE section (archive + DB sizes) (#136)
Show on-disk storage: the parquet archives (with history/recent split), the derived
SQLite DB, and the accounts SQLite DB individually, plus a total. Archive bytes are
summed during the existing TTL-cached cache scan; DB sizes include the -wal/-shm
sidecars (WAL holds recent writes, so the base file alone understates the footprint).
Drop the now-redundant 'payload db' line from CACHE.
* Dashboard: show STORAGE sizes in fixed GB (0.01 GB floor) (#137)
The auto-scaling formatter only switched to GB once a value passed 1 GB, so the
archives read as MB and the DBs as KB. Format the STORAGE section in fixed GB with two
decimals and a 0.01 GB floor (so the small databases stay visible instead of rounding
to 0.00). Memory readouts keep the auto-scaling formatter.
* Dashboard: show STORAGE under 50 MB in MB, GB above (#138)
Values below 50 MB now render as MB with one decimal (no KB), so the databases read
naturally (derived 7.7MB, accounts 0.4MB) instead of 0.01GB; 50 MB and up stay in GB
with two decimals. Renames the helper _gb -> _storage_size to match.
* Dashboard: move STORAGE MB->GB cutoff to 500 MB (#139)
Values below 500 MB show in MB (one decimal); 500 MB and up in GB.
* Dashboard: accounts totals only; log client IPs per request (#140)
Dashboard ACCOUNTS section now shows only totals (users, active/total subscriptions,
notifications, push subs) — the per-user and per-subscription lists are gone, and the
dashboard no longer queries that PII at all.
Backend: add a per-request access log (logs/access/access-<date>.jsonl) recording the
client IP (real IP via the left-most X-Forwarded-For hop when proxied behind Caddy,
else the peer address), method, path, status, and category. Static assets and the
dashboard's own metrics polling are skipped. Best-effort, never raises into the request
path — retained for later traffic/IP analysis.
* Dashboard: wrap-aware pager so the header stops scrolling off (#141)
The pager counted one screen row per logical line, but on a narrow phone many lines
wrap to two rows — so a page printed more physical rows than the terminal had, scrolling
the top (title + CACHE header) off-screen with no way to scroll back to it. Measure each
line's wrapped height and fit a page to the real physical-row budget; page/End step by
what's actually visible; read the true tty size (not shutil, which honors stale
COLUMNS/LINES). Status line is clipped to one row.
* Add log storage size to the ops dashboard STORAGE section (#142)
The STORAGE section reported parquet archives and both SQLite DBs but
omitted the logs/ footprint, so the "total" understated real disk use.
Add a recursive _tree_size helper and a "logs" row measuring the whole
logs/ tree (audit + errors + access JSONL plus stray *.log files), broken
out by stream, and fold it into the STORAGE total. Flows through --json
automatically via read_storage.
* Dashboard: show last-10-minute traffic per category (#143)
Add a rolling short-window view alongside the since-start totals. metrics.py
keeps per-minute buckets (epoch-minute -> {key: count}) for inbound categories
and outbound sources, summed over the last 10 minutes on snapshot() and exposed
as inbound_recent / outbound_recent / window_minutes. Cheap and bounded — at most
WINDOW+1 tiny dicts, pruned as minutes roll off; self-polling still excluded.
The dashboard's TRAFFIC section now renders that count as a dim column next to
each row's total (header reads "total · last 10m"); idle sources show "-".
* Add dashboard #2 (#145)
* Auto-submit IndexNow on deploy when the URL set changes (#130)
Hook the production deploy (deploy/deploy.sh) to ping IndexNow after the health
check passes, so new/changed pages reach Bing/DuckDuckGo/Yandex without a manual
run. Best-effort — wrapped so it can never fail a deploy — and it sources
/etc/thermograph.env so its key matches the one the running service serves.
To avoid re-blasting ~14k URLs on every code push, indexnow.py gains a
--if-changed mode gated by a signature of the URL set (persisted to
data/indexnow_state.txt): code-only deploys skip; a deploy that adds or removes a
city submits. `make indexnow` still forces a full submit.
* Add ops metrics endpoint + SSH/Termux dashboard (#131)
Adds observability for the running app, viewable over SSH (e.g. Termux):
- backend/metrics.py: thread-safe, best-effort in-process counters for inbound
requests (per category) and outbound calls (per external source), plus a
phase->source map and snapshot(). Hooked into climate._request (outbound, all
six weather/geocode sources) and the existing revalidate_static middleware
(inbound). Never raises into the request path.
- GET /api/v2/metrics: token-gated JSON of the counters + warm-queue depth +
thread names/uptime/pid. Returns 404 unless a valid THERMOGRAPH_METRICS_TOKEN
is presented or the request is direct loopback with no proxy X-Forwarded-*
header, so ops data is never exposed through Caddy on the public domain.
- scripts/dashboard.py (+ scripts/dash, make dashboard): stdlib-only terminal
dashboard. Reads cache (parquet + SQLite), accounts (users + active
subscriptions), and system resources (/proc + systemd) directly, and pulls
traffic from the metrics endpoint over loopback. Live refresh, --once, --json;
mobile-terminal width. Paths resolve relative to the script so the same tool
works on the LAN dev and prod VPS checkouts.
- Monitoring sections in DEPLOY.md / DEPLOY-DEV.md; unit tests for the counters.
* Dashboard: run under the app venv, not the box's default python3 (#132)
The box's default python3 can be a build without _sqlite3 (pyenv 3.10 here), so
./scripts/dash crashed on import. Prefer the repo's .venv interpreter (the Python
the server itself runs on, which always has sqlite3); fall back to python3/python.
Also make the sqlite3 import in dashboard.py optional so a fallback interpreter
degrades to empty cache/accounts panels instead of crashing, and point the
'make dashboard' target at the wrapper.
* Dashboard: redraw live view in place so it stops scrolling to the bottom (#133)
The live loop did a full clear-and-reprint each refresh; when the output is taller
than the terminal (common on a phone) that scrolls the view to the bottom every tick.
Redraw from the top instead: hide the cursor, clear once, then each frame move to home
and rewrite line-by-line (clear-to-EOL + clear-below), capping the body to the window
height so it never prints more lines than fit and never scrolls. Restore the cursor on
exit; piped (non-TTY) output prints plain frames.
* Dashboard: make the live view a scrollable in-place pager (#134)
The live mode is now an alt-screen pager (like top/less): it shows only the latest
snapshot, you scroll it with swipe/arrows/jk/PgUp-PgDn/g/G, and each refresh replaces
the page in place while keeping your scroll position — nothing gets appended below and
the view is never yanked to the top or bottom. On exit it restores the cursor, leaves
the alternate screen, and resets the terminal mode.
--once still prints a single static snapshot (also used automatically when stdout/stdin
aren't a terminal, e.g. piping); --json unchanged.
* Metrics: don't count the dashboard's own /api/v2/metrics polling (#135)
The ops dashboard polls the metrics endpoint every few seconds; counting those hits
just shows the monitor watching itself and inflates the inbound totals. Skip the
'metrics' category in record_inbound (so it's excluded from inbound_total too), and
hide it in the dashboard's inbound list defensively.
* Dashboard: add a STORAGE section (archive + DB sizes) (#136)
Show on-disk storage: the parquet archives (with history/recent split), the derived
SQLite DB, and the accounts SQLite DB individually, plus a total. Archive bytes are
summed during the existing TTL-cached cache scan; DB sizes include the -wal/-shm
sidecars (WAL holds recent writes, so the base file alone understates the footprint).
Drop the now-redundant 'payload db' line from CACHE.
* Dashboard: show STORAGE sizes in fixed GB (0.01 GB floor) (#137)
The auto-scaling formatter only switched to GB once a value passed 1 GB, so the
archives read as MB and the DBs as KB. Format the STORAGE section in fixed GB with two
decimals and a 0.01 GB floor (so the small databases stay visible instead of rounding
to 0.00). Memory readouts keep the auto-scaling formatter.
* Dashboard: show STORAGE under 50 MB in MB, GB above (#138)
Values below 50 MB now render as MB with one decimal (no KB), so the databases read
naturally (derived 7.7MB, accounts 0.4MB) instead of 0.01GB; 50 MB and up stay in GB
with two decimals. Renames the helper _gb -> _storage_size to match.
* Dashboard: move STORAGE MB->GB cutoff to 500 MB (#139)
Values below 500 MB show in MB (one decimal); 500 MB and up in GB.
* Dashboard: accounts totals only; log client IPs per request (#140)
Dashboard ACCOUNTS section now shows only totals (users, active/total subscriptions,
notifications, push subs) — the per-user and per-subscription lists are gone, and the
dashboard no longer queries that PII at all.
Backend: add a per-request access log (logs/access/access-<date>.jsonl) recording the
client IP (real IP via the left-most X-Forwarded-For hop when proxied behind Caddy,
else the peer address), method, path, status, and category. Static assets and the
dashboard's own metrics polling are skipped. Best-effort, never raises into the request
path — retained for later traffic/IP analysis.
* Dashboard: wrap-aware pager so the header stops scrolling off (#141)
The pager counted one screen row per logical line, but on a narrow phone many lines
wrap to two rows — so a page printed more physical rows than the terminal had, scrolling
the top (title + CACHE header) off-screen with no way to scroll back to it. Measure each
line's wrapped height and fit a page to the real physical-row budget; page/End step by
what's actually visible; read the true tty size (not shutil, which honors stale
COLUMNS/LINES). Status line is clipped to one row.
* Add log storage size to the ops dashboard STORAGE section (#142)
The STORAGE section reported parquet archives and both SQLite DBs but
omitted the logs/ footprint, so the "total" understated real disk use.
Add a recursive _tree_size helper and a "logs" row measuring the whole
logs/ tree (audit + errors + access JSONL plus stray *.log files), broken
out by stream, and fold it into the STORAGE total. Flows through --json
automatically via read_storage.
* Dashboard: show last-10-minute traffic per category (#143)
Add a rolling short-window view alongside the since-start totals. metrics.py
keeps per-minute buckets (epoch-minute -> {key: count}) for inbound categories
and outbound sources, summed over the last 10 minutes on snapshot() and exposed
as inbound_recent / outbound_recent / window_minutes. Cheap and bounded — at most
WINDOW+1 tiny dicts, pruned as minutes roll off; self-polling still excluded.
The dashboard's TRAFFIC section now renders that count as a dim column next to
each row's total (header reads "total · last 10m"); idle sources show "-".
* Run prod on 3 uvicorn workers with a shared metrics store
A single slow upstream weather fetch (cache-miss during an open-meteo/NASA
degradation) blocked the one uvicorn worker and took the whole app down for
~2 min — real users got aborted connections. Give prod concurrency headroom so
one slow request can't freeze the rest.
- deploy/thermograph.service: worker count is env-driven (`--workers ${WORKERS}`,
default 1); prod sets WORKERS=3 in /etc/thermograph.env. Dev's separate
--user unit is untouched (stays single-worker).
- metrics.py: with multiple workers, per-process counters would split the tally
and the ops dashboard (polls one random worker) would see only a fraction.
Add a shared SQLite store selected by THERMOGRAPH_METRICS_DB so every worker
tallies into one place; unset keeps the zero-dependency in-memory store for
dev/tests/single-worker. Public API and snapshot shape unchanged.
- The unit defaults THERMOGRAPH_METRICS_DB and clears it on each (re)start via
ExecStartPre, so bumping WORKERS can never silently fragment the dashboard and
"since start" tallies keep their old meaning.
- Tests: cover the SQLite store — cross-worker aggregation, window roll-off,
and env-based selection. Full suite: 178 passed.
Note: applying to prod also needs the unit reinstalled + WORKERS=3 in
/etc/thermograph.env — deploy.sh only pulls+restarts, it doesn't reinstall the
systemd unit.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016mRqdHWRdUjEvEdfwig3wg
---------
Co-authored-by: root <root@vmi3417050.contaboserver.net>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
A failed web push (e.g. VAPID key mismatch -> 401/403) was swallowed: /push/test still
returned 202 and the UI showed 'Sent', while the only trace was a journald WARNING.
Now: push.send logs failures to the errors JSONL (phase=push) so they show up like other
errors; /push/test returns a 'failed' count; and the 'Send test' button reports 'No
device' / 'Sent' / 'Failed' from the real result. Document the VAPID env vars (missing
from thermograph.env.example) and how to pin/diagnose them.
On prod (served at root) the dashboard first probes /thermograph/api/v2/metrics before
falling back to root; that 404 was classified as inbound 'other' and shown as an error —
the monitor polluting the traffic it displays. Two fixes: classify any */api/v2/metrics
path as 'metrics' (excluded) whatever prefix it arrives under; and cache the working
metrics URL in the dashboard so it stops re-probing the failing base every refresh.
Add a rolling short-window view alongside the since-start totals. metrics.py
keeps per-minute buckets (epoch-minute -> {key: count}) for inbound categories
and outbound sources, summed over the last 10 minutes on snapshot() and exposed
as inbound_recent / outbound_recent / window_minutes. Cheap and bounded — at most
WINDOW+1 tiny dicts, pruned as minutes roll off; self-polling still excluded.
The dashboard's TRAFFIC section now renders that count as a dim column next to
each row's total (header reads "total · last 10m"); idle sources show "-".
The ops dashboard polls the metrics endpoint every few seconds; counting those hits
just shows the monitor watching itself and inflates the inbound totals. Skip the
'metrics' category in record_inbound (so it's excluded from inbound_total too), and
hide it in the dashboard's inbound list defensively.
Adds observability for the running app, viewable over SSH (e.g. Termux):
- backend/metrics.py: thread-safe, best-effort in-process counters for inbound
requests (per category) and outbound calls (per external source), plus a
phase->source map and snapshot(). Hooked into climate._request (outbound, all
six weather/geocode sources) and the existing revalidate_static middleware
(inbound). Never raises into the request path.
- GET /api/v2/metrics: token-gated JSON of the counters + warm-queue depth +
thread names/uptime/pid. Returns 404 unless a valid THERMOGRAPH_METRICS_TOKEN
is presented or the request is direct loopback with no proxy X-Forwarded-*
header, so ops data is never exposed through Caddy on the public domain.
- scripts/dashboard.py (+ scripts/dash, make dashboard): stdlib-only terminal
dashboard. Reads cache (parquet + SQLite), accounts (users + active
subscriptions), and system resources (/proc + systemd) directly, and pulls
traffic from the metrics endpoint over loopback. Live refresh, --once, --json;
mobile-terminal width. Paths resolve relative to the script so the same tool
works on the LAN dev and prod VPS checkouts.
- Monitoring sections in DEPLOY.md / DEPLOY-DEV.md; unit tests for the counters.
Hook the production deploy (deploy/deploy.sh) to ping IndexNow after the health
check passes, so new/changed pages reach Bing/DuckDuckGo/Yandex without a manual
run. Best-effort — wrapped so it can never fail a deploy — and it sources
/etc/thermograph.env so its key matches the one the running service serves.
To avoid re-blasting ~14k URLs on every code push, indexnow.py gains a
--if-changed mode gated by a signature of the URL set (persisted to
data/indexnow_state.txt): code-only deploys skip; a deploy that adds or removes a
city submits. `make indexnow` still forces a full submit.
Faster search-engine indexing for the ~14k climate/city/record URLs:
- IndexNow (backend/indexnow.py): instantly notify Bing/DuckDuckGo/Yandex of new
or changed URLs. Per-host key resolved env → gitignored file → generated (mirrors
push.py), served at /{key}.txt, and echoed in submissions. `submit_all()` + a
`make indexnow` CLI push every indexable URL, batched under the 10k cap. Google
doesn't use IndexNow, so it stays on the sitemap.
- Sitemap: replace the per-request today() <lastmod> (which churns every fetch and
trains crawlers to ignore lastmod) with a stable content-build date; factor the
URL list into public_paths() shared with IndexNow so the two never drift.
- Search-console verification: render google-site-verification + msvalidate.01
<meta> tags from env into every page's <head> — the SEO pages via base.html.j2
and the static pages (incl. the homepage Google verifies) via _page().
- Docs: env example documents the three new vars; .gitignore ignores the key.
Tests: key-file route, stable single-date lastmod, IndexNow payload/batching, and
verification meta on both SEO and static pages.
The server-rendered SEO pages shared base.html.j2's nav but carried no JS, so
they lacked the °F/°C toggle and account/bell, and their temperatures were baked
as dual-unit strings that couldn't respond to a toggle.
- Load units.js + account.js + climate.js on base.html.j2, so every SEO page gets
the same header as the interactive pages (nav + °F/°C toggle + account + bell).
- Emit each temperature as <span class="temp" data-temp-f> (default °F text, real
for crawlers/no-JS); climate.js rewrites them to the active unit on load and on
toggle. Tint classes stay pinned to absolute °F. Drops the inline "(NN°C)".
- Make account.js base-aware: derive the app base from import.meta.url so its
api/v2 fetches and the alerts links resolve from deep /climate/{slug} URLs
instead of 404ing. Equivalent on the depth-1 interactive pages.
- Default the unit from the browser locale for first-time visitors (no stored
pref): en-US → °F, everyone else → °C. A stored choice always wins.
Complete the forecast fallback chain: Open-Meteo → MET Norway → stale cache. When
both live sources are unavailable, serve the last cached recent/forecast bundle
even if it is past its 1-hour TTL — an hours-old bundle (which still carries the
recent observed days MET Norway lacks) beats a hard 503, mirroring the archive
path's stale-serve in _load_history.
The stale bundle is returned WITHOUT rewriting it, so its mtime stays old and the
next request retries the live sources first rather than treating the stale copy as
fresh. Only when there is no cache at all does the typed WeatherUnavailable surface.
