The ops dashboard marked subscription-notifier DOWN ~2/3 of the time after the
notifier was gated to a single leader worker. It judged the daemon by whether
its thread was present on whichever worker answered /api/v2/metrics, but only
the leader runs it — so polls landing on the other two workers saw no thread and
reported DOWN for a perfectly healthy notifier. Per-worker thread checks can't
describe a single-leader daemon, and worse, they could no longer distinguish a
real crash from a normal non-leader.
Report it by liveness instead: the notifier stamps a heartbeat (timestamp +
expected interval) into the shared metrics store each loop iteration, including
once at startup so the dashboard shows it up immediately after a restart. Since
the beat lives in the shared SQLite store, any worker's metrics response carries
it, so the dashboard sees the notifier's health whichever worker it polls.
- metrics.py: both stores record/expose heartbeats (SQLite reuses the meta
table, so every worker's beats share one file); snapshot() converts each beat
to a server-computed age so readers needn't reconcile clock skew.
- notify.py: beat at loop start and every wake.
- dashboard.py: judge heartbeat daemons by freshness (up to 2 missed intervals
before DOWN, with beat age shown); neighbor-warmer keeps its per-worker thread
check, which is correct since every worker runs it.
- Tests: heartbeat store/snapshot shape + cross-worker sharing; dashboard
fresh/stale/missing rendering and the unchanged warmer check. 194 passed.
Co-authored-by: root <root@vmi3417050.contaboserver.net>
Google Search Console flags city pages with 'Missing field item (in
itemListElement)': the country crumb has no page of its own, so its
ListItem was emitted without the required item URL. Google only allows
the final ListItem to omit item, so drop unlinked intermediate crumbs
from the structured data (the visible breadcrumb is unchanged) and
renumber positions. Shared helper now builds the BreadcrumbList for
both the city and records pages, with a regression test parsing the
emitted JSON-LD.
Claude-Session: https://claude.ai/code/session_01KdTZCjpLeD26ZbXe6ApjpR
* Serve the app on thermograph.org at root; redirect emigriffith.dev/thermograph
Move Thermograph to its own domain. thermograph.org serves the app at its root,
and emigriffith.dev/thermograph* permanently redirects there (prefix stripped,
so deep links map straight across).
- app.py: BASE now supports an empty root prefix. THERMOGRAPH_BASE=/ (or empty)
yields BASE="" so routes sit at "/", "/calendar", "/api/v2/…" with no double
slashes; the bare-base redirect is skipped and the static mount falls back to
"/". Non-empty values keep the existing "/thermograph" sub-path behavior
unchanged (backward compatible).
- deploy/Caddyfile: add a thermograph.org site block reverse-proxying to the
uvicorn on 127.0.0.1:8137; turn emigriffith.dev's /thermograph handler into a
permanent redirect to thermograph.org.
- deploy/thermograph.env.example: default THERMOGRAPH_BASE=/.
- DEPLOY.md: document the two-domain layout and that deploy.sh does not ship the
Caddyfile or env — those are applied on the VPS by hand.
The frontend already uses base-relative URLs, so it follows the root base with no
changes. Verified both modes locally (root serves /, /calendar, /api/v2/*; the
/thermograph sub-path still redirects bare→slash and 404s at root).
* Add dashboard (#144)
* Auto-submit IndexNow on deploy when the URL set changes (#130)
Hook the production deploy (deploy/deploy.sh) to ping IndexNow after the health
check passes, so new/changed pages reach Bing/DuckDuckGo/Yandex without a manual
run. Best-effort — wrapped so it can never fail a deploy — and it sources
/etc/thermograph.env so its key matches the one the running service serves.
To avoid re-blasting ~14k URLs on every code push, indexnow.py gains a
--if-changed mode gated by a signature of the URL set (persisted to
data/indexnow_state.txt): code-only deploys skip; a deploy that adds or removes a
city submits. `make indexnow` still forces a full submit.
* Add ops metrics endpoint + SSH/Termux dashboard (#131)
Adds observability for the running app, viewable over SSH (e.g. Termux):
- backend/metrics.py: thread-safe, best-effort in-process counters for inbound
requests (per category) and outbound calls (per external source), plus a
phase->source map and snapshot(). Hooked into climate._request (outbound, all
six weather/geocode sources) and the existing revalidate_static middleware
(inbound). Never raises into the request path.
- GET /api/v2/metrics: token-gated JSON of the counters + warm-queue depth +
thread names/uptime/pid. Returns 404 unless a valid THERMOGRAPH_METRICS_TOKEN
is presented or the request is direct loopback with no proxy X-Forwarded-*
header, so ops data is never exposed through Caddy on the public domain.
- scripts/dashboard.py (+ scripts/dash, make dashboard): stdlib-only terminal
dashboard. Reads cache (parquet + SQLite), accounts (users + active
subscriptions), and system resources (/proc + systemd) directly, and pulls
traffic from the metrics endpoint over loopback. Live refresh, --once, --json;
mobile-terminal width. Paths resolve relative to the script so the same tool
works on the LAN dev and prod VPS checkouts.
- Monitoring sections in DEPLOY.md / DEPLOY-DEV.md; unit tests for the counters.
* Dashboard: run under the app venv, not the box's default python3 (#132)
The box's default python3 can be a build without _sqlite3 (pyenv 3.10 here), so
./scripts/dash crashed on import. Prefer the repo's .venv interpreter (the Python
the server itself runs on, which always has sqlite3); fall back to python3/python.
Also make the sqlite3 import in dashboard.py optional so a fallback interpreter
degrades to empty cache/accounts panels instead of crashing, and point the
'make dashboard' target at the wrapper.
* Dashboard: redraw live view in place so it stops scrolling to the bottom (#133)
The live loop did a full clear-and-reprint each refresh; when the output is taller
than the terminal (common on a phone) that scrolls the view to the bottom every tick.
Redraw from the top instead: hide the cursor, clear once, then each frame move to home
and rewrite line-by-line (clear-to-EOL + clear-below), capping the body to the window
height so it never prints more lines than fit and never scrolls. Restore the cursor on
exit; piped (non-TTY) output prints plain frames.
* Dashboard: make the live view a scrollable in-place pager (#134)
The live mode is now an alt-screen pager (like top/less): it shows only the latest
snapshot, you scroll it with swipe/arrows/jk/PgUp-PgDn/g/G, and each refresh replaces
the page in place while keeping your scroll position — nothing gets appended below and
the view is never yanked to the top or bottom. On exit it restores the cursor, leaves
the alternate screen, and resets the terminal mode.
--once still prints a single static snapshot (also used automatically when stdout/stdin
aren't a terminal, e.g. piping); --json unchanged.
* Metrics: don't count the dashboard's own /api/v2/metrics polling (#135)
The ops dashboard polls the metrics endpoint every few seconds; counting those hits
just shows the monitor watching itself and inflates the inbound totals. Skip the
'metrics' category in record_inbound (so it's excluded from inbound_total too), and
hide it in the dashboard's inbound list defensively.
* Dashboard: add a STORAGE section (archive + DB sizes) (#136)
Show on-disk storage: the parquet archives (with history/recent split), the derived
SQLite DB, and the accounts SQLite DB individually, plus a total. Archive bytes are
summed during the existing TTL-cached cache scan; DB sizes include the -wal/-shm
sidecars (WAL holds recent writes, so the base file alone understates the footprint).
Drop the now-redundant 'payload db' line from CACHE.
* Dashboard: show STORAGE sizes in fixed GB (0.01 GB floor) (#137)
The auto-scaling formatter only switched to GB once a value passed 1 GB, so the
archives read as MB and the DBs as KB. Format the STORAGE section in fixed GB with two
decimals and a 0.01 GB floor (so the small databases stay visible instead of rounding
to 0.00). Memory readouts keep the auto-scaling formatter.
* Dashboard: show STORAGE under 50 MB in MB, GB above (#138)
Values below 50 MB now render as MB with one decimal (no KB), so the databases read
naturally (derived 7.7MB, accounts 0.4MB) instead of 0.01GB; 50 MB and up stay in GB
with two decimals. Renames the helper _gb -> _storage_size to match.
* Dashboard: move STORAGE MB->GB cutoff to 500 MB (#139)
Values below 500 MB show in MB (one decimal); 500 MB and up in GB.
* Dashboard: accounts totals only; log client IPs per request (#140)
Dashboard ACCOUNTS section now shows only totals (users, active/total subscriptions,
notifications, push subs) — the per-user and per-subscription lists are gone, and the
dashboard no longer queries that PII at all.
Backend: add a per-request access log (logs/access/access-<date>.jsonl) recording the
client IP (real IP via the left-most X-Forwarded-For hop when proxied behind Caddy,
else the peer address), method, path, status, and category. Static assets and the
dashboard's own metrics polling are skipped. Best-effort, never raises into the request
path — retained for later traffic/IP analysis.
* Dashboard: wrap-aware pager so the header stops scrolling off (#141)
The pager counted one screen row per logical line, but on a narrow phone many lines
wrap to two rows — so a page printed more physical rows than the terminal had, scrolling
the top (title + CACHE header) off-screen with no way to scroll back to it. Measure each
line's wrapped height and fit a page to the real physical-row budget; page/End step by
what's actually visible; read the true tty size (not shutil, which honors stale
COLUMNS/LINES). Status line is clipped to one row.
* Add log storage size to the ops dashboard STORAGE section (#142)
The STORAGE section reported parquet archives and both SQLite DBs but
omitted the logs/ footprint, so the "total" understated real disk use.
Add a recursive _tree_size helper and a "logs" row measuring the whole
logs/ tree (audit + errors + access JSONL plus stray *.log files), broken
out by stream, and fold it into the STORAGE total. Flows through --json
automatically via read_storage.
* Dashboard: show last-10-minute traffic per category (#143)
Add a rolling short-window view alongside the since-start totals. metrics.py
keeps per-minute buckets (epoch-minute -> {key: count}) for inbound categories
and outbound sources, summed over the last 10 minutes on snapshot() and exposed
as inbound_recent / outbound_recent / window_minutes. Cheap and bounded — at most
WINDOW+1 tiny dicts, pruned as minutes roll off; self-polling still excluded.
The dashboard's TRAFFIC section now renders that count as a dim column next to
each row's total (header reads "total · last 10m"); idle sources show "-".
* Add dashboard #2 (#145)
* Auto-submit IndexNow on deploy when the URL set changes (#130)
Hook the production deploy (deploy/deploy.sh) to ping IndexNow after the health
check passes, so new/changed pages reach Bing/DuckDuckGo/Yandex without a manual
run. Best-effort — wrapped so it can never fail a deploy — and it sources
/etc/thermograph.env so its key matches the one the running service serves.
To avoid re-blasting ~14k URLs on every code push, indexnow.py gains a
--if-changed mode gated by a signature of the URL set (persisted to
data/indexnow_state.txt): code-only deploys skip; a deploy that adds or removes a
city submits. `make indexnow` still forces a full submit.
* Add ops metrics endpoint + SSH/Termux dashboard (#131)
Adds observability for the running app, viewable over SSH (e.g. Termux):
- backend/metrics.py: thread-safe, best-effort in-process counters for inbound
requests (per category) and outbound calls (per external source), plus a
phase->source map and snapshot(). Hooked into climate._request (outbound, all
six weather/geocode sources) and the existing revalidate_static middleware
(inbound). Never raises into the request path.
- GET /api/v2/metrics: token-gated JSON of the counters + warm-queue depth +
thread names/uptime/pid. Returns 404 unless a valid THERMOGRAPH_METRICS_TOKEN
is presented or the request is direct loopback with no proxy X-Forwarded-*
header, so ops data is never exposed through Caddy on the public domain.
- scripts/dashboard.py (+ scripts/dash, make dashboard): stdlib-only terminal
dashboard. Reads cache (parquet + SQLite), accounts (users + active
subscriptions), and system resources (/proc + systemd) directly, and pulls
traffic from the metrics endpoint over loopback. Live refresh, --once, --json;
mobile-terminal width. Paths resolve relative to the script so the same tool
works on the LAN dev and prod VPS checkouts.
- Monitoring sections in DEPLOY.md / DEPLOY-DEV.md; unit tests for the counters.
* Dashboard: run under the app venv, not the box's default python3 (#132)
The box's default python3 can be a build without _sqlite3 (pyenv 3.10 here), so
./scripts/dash crashed on import. Prefer the repo's .venv interpreter (the Python
the server itself runs on, which always has sqlite3); fall back to python3/python.
Also make the sqlite3 import in dashboard.py optional so a fallback interpreter
degrades to empty cache/accounts panels instead of crashing, and point the
'make dashboard' target at the wrapper.
* Dashboard: redraw live view in place so it stops scrolling to the bottom (#133)
The live loop did a full clear-and-reprint each refresh; when the output is taller
than the terminal (common on a phone) that scrolls the view to the bottom every tick.
Redraw from the top instead: hide the cursor, clear once, then each frame move to home
and rewrite line-by-line (clear-to-EOL + clear-below), capping the body to the window
height so it never prints more lines than fit and never scrolls. Restore the cursor on
exit; piped (non-TTY) output prints plain frames.
* Dashboard: make the live view a scrollable in-place pager (#134)
The live mode is now an alt-screen pager (like top/less): it shows only the latest
snapshot, you scroll it with swipe/arrows/jk/PgUp-PgDn/g/G, and each refresh replaces
the page in place while keeping your scroll position — nothing gets appended below and
the view is never yanked to the top or bottom. On exit it restores the cursor, leaves
the alternate screen, and resets the terminal mode.
--once still prints a single static snapshot (also used automatically when stdout/stdin
aren't a terminal, e.g. piping); --json unchanged.
* Metrics: don't count the dashboard's own /api/v2/metrics polling (#135)
The ops dashboard polls the metrics endpoint every few seconds; counting those hits
just shows the monitor watching itself and inflates the inbound totals. Skip the
'metrics' category in record_inbound (so it's excluded from inbound_total too), and
hide it in the dashboard's inbound list defensively.
* Dashboard: add a STORAGE section (archive + DB sizes) (#136)
Show on-disk storage: the parquet archives (with history/recent split), the derived
SQLite DB, and the accounts SQLite DB individually, plus a total. Archive bytes are
summed during the existing TTL-cached cache scan; DB sizes include the -wal/-shm
sidecars (WAL holds recent writes, so the base file alone understates the footprint).
Drop the now-redundant 'payload db' line from CACHE.
* Dashboard: show STORAGE sizes in fixed GB (0.01 GB floor) (#137)
The auto-scaling formatter only switched to GB once a value passed 1 GB, so the
archives read as MB and the DBs as KB. Format the STORAGE section in fixed GB with two
decimals and a 0.01 GB floor (so the small databases stay visible instead of rounding
to 0.00). Memory readouts keep the auto-scaling formatter.
* Dashboard: show STORAGE under 50 MB in MB, GB above (#138)
Values below 50 MB now render as MB with one decimal (no KB), so the databases read
naturally (derived 7.7MB, accounts 0.4MB) instead of 0.01GB; 50 MB and up stay in GB
with two decimals. Renames the helper _gb -> _storage_size to match.
* Dashboard: move STORAGE MB->GB cutoff to 500 MB (#139)
Values below 500 MB show in MB (one decimal); 500 MB and up in GB.
* Dashboard: accounts totals only; log client IPs per request (#140)
Dashboard ACCOUNTS section now shows only totals (users, active/total subscriptions,
notifications, push subs) — the per-user and per-subscription lists are gone, and the
dashboard no longer queries that PII at all.
Backend: add a per-request access log (logs/access/access-<date>.jsonl) recording the
client IP (real IP via the left-most X-Forwarded-For hop when proxied behind Caddy,
else the peer address), method, path, status, and category. Static assets and the
dashboard's own metrics polling are skipped. Best-effort, never raises into the request
path — retained for later traffic/IP analysis.
* Dashboard: wrap-aware pager so the header stops scrolling off (#141)
The pager counted one screen row per logical line, but on a narrow phone many lines
wrap to two rows — so a page printed more physical rows than the terminal had, scrolling
the top (title + CACHE header) off-screen with no way to scroll back to it. Measure each
line's wrapped height and fit a page to the real physical-row budget; page/End step by
what's actually visible; read the true tty size (not shutil, which honors stale
COLUMNS/LINES). Status line is clipped to one row.
* Add log storage size to the ops dashboard STORAGE section (#142)
The STORAGE section reported parquet archives and both SQLite DBs but
omitted the logs/ footprint, so the "total" understated real disk use.
Add a recursive _tree_size helper and a "logs" row measuring the whole
logs/ tree (audit + errors + access JSONL plus stray *.log files), broken
out by stream, and fold it into the STORAGE total. Flows through --json
automatically via read_storage.
* Dashboard: show last-10-minute traffic per category (#143)
Add a rolling short-window view alongside the since-start totals. metrics.py
keeps per-minute buckets (epoch-minute -> {key: count}) for inbound categories
and outbound sources, summed over the last 10 minutes on snapshot() and exposed
as inbound_recent / outbound_recent / window_minutes. Cheap and bounded — at most
WINDOW+1 tiny dicts, pruned as minutes roll off; self-polling still excluded.
The dashboard's TRAFFIC section now renders that count as a dim column next to
each row's total (header reads "total · last 10m"); idle sources show "-".
* Run prod on 3 uvicorn workers with a shared metrics store
A single slow upstream weather fetch (cache-miss during an open-meteo/NASA
degradation) blocked the one uvicorn worker and took the whole app down for
~2 min — real users got aborted connections. Give prod concurrency headroom so
one slow request can't freeze the rest.
- deploy/thermograph.service: worker count is env-driven (`--workers ${WORKERS}`,
default 1); prod sets WORKERS=3 in /etc/thermograph.env. Dev's separate
--user unit is untouched (stays single-worker).
- metrics.py: with multiple workers, per-process counters would split the tally
and the ops dashboard (polls one random worker) would see only a fraction.
Add a shared SQLite store selected by THERMOGRAPH_METRICS_DB so every worker
tallies into one place; unset keeps the zero-dependency in-memory store for
dev/tests/single-worker. Public API and snapshot shape unchanged.
- The unit defaults THERMOGRAPH_METRICS_DB and clears it on each (re)start via
ExecStartPre, so bumping WORKERS can never silently fragment the dashboard and
"since start" tallies keep their old meaning.
- Tests: cover the SQLite store — cross-worker aggregation, window roll-off,
and env-based selection. Full suite: 178 passed.
Note: applying to prod also needs the unit reinstalled + WORKERS=3 in
/etc/thermograph.env — deploy.sh only pulls+restarts, it doesn't reinstall the
systemd unit.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016mRqdHWRdUjEvEdfwig3wg
---------
Co-authored-by: root <root@vmi3417050.contaboserver.net>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Prod runs 3 uvicorn workers, but the in-process subscription notifier was
started in every worker's lifespan (it was written assuming a single-worker
deploy). Its sweep fetches recent-forecast/archive data from Open-Meteo on a
15-min timer independent of any request, so running it 3× tripled the
background upstream load against a shared quota — the source of the overnight
429/503 rate-limit errors in logs/errors, and 3× redundant subscription scans.
Elect one worker to own the notifier via a non-blocking exclusive flock on a
lockfile (THERMOGRAPH_SINGLETON_LOCK): the first worker wins and holds the fd
for its lifetime; others stand down; the OS releases the lock if the leader
dies, so a restart re-elects cleanly. Unset (single worker / dev / tests)
always wins, so behavior there is unchanged. Mirrors the shared-metrics store's
env-selected cross-worker coordination.
The neighbor warmer stays per-worker on purpose: each worker drains its own
request-fed queue, so gating it would leave non-leader queues undrained.
- backend/singleton.py: flock-based leader election (claim()).
- backend/app.py: gate notify.start() on singleton.claim().
- deploy/thermograph.service: default THERMOGRAPH_SINGLETON_LOCK to
data/notifier.lock (needs the unit reinstalled on prod to take effect).
- deploy/thermograph.env.example: document it alongside WORKERS.
- Tests: leader election — single-worker default, first-wins, idempotent
re-claim, second-holder stands down, re-election after release. 183 passed.
Co-authored-by: root <root@vmi3417050.contaboserver.net>
* Serve the app on thermograph.org at root; redirect emigriffith.dev/thermograph
Move Thermograph to its own domain. thermograph.org serves the app at its root,
and emigriffith.dev/thermograph* permanently redirects there (prefix stripped,
so deep links map straight across).
- app.py: BASE now supports an empty root prefix. THERMOGRAPH_BASE=/ (or empty)
yields BASE="" so routes sit at "/", "/calendar", "/api/v2/…" with no double
slashes; the bare-base redirect is skipped and the static mount falls back to
"/". Non-empty values keep the existing "/thermograph" sub-path behavior
unchanged (backward compatible).
- deploy/Caddyfile: add a thermograph.org site block reverse-proxying to the
uvicorn on 127.0.0.1:8137; turn emigriffith.dev's /thermograph handler into a
permanent redirect to thermograph.org.
- deploy/thermograph.env.example: default THERMOGRAPH_BASE=/.
- DEPLOY.md: document the two-domain layout and that deploy.sh does not ship the
Caddyfile or env — those are applied on the VPS by hand.
The frontend already uses base-relative URLs, so it follows the root base with no
changes. Verified both modes locally (root serves /, /calendar, /api/v2/*; the
/thermograph sub-path still redirects bare→slash and 404s at root).
* Add dashboard (#144)
* Auto-submit IndexNow on deploy when the URL set changes (#130)
Hook the production deploy (deploy/deploy.sh) to ping IndexNow after the health
check passes, so new/changed pages reach Bing/DuckDuckGo/Yandex without a manual
run. Best-effort — wrapped so it can never fail a deploy — and it sources
/etc/thermograph.env so its key matches the one the running service serves.
To avoid re-blasting ~14k URLs on every code push, indexnow.py gains a
--if-changed mode gated by a signature of the URL set (persisted to
data/indexnow_state.txt): code-only deploys skip; a deploy that adds or removes a
city submits. `make indexnow` still forces a full submit.
* Add ops metrics endpoint + SSH/Termux dashboard (#131)
Adds observability for the running app, viewable over SSH (e.g. Termux):
- backend/metrics.py: thread-safe, best-effort in-process counters for inbound
requests (per category) and outbound calls (per external source), plus a
phase->source map and snapshot(). Hooked into climate._request (outbound, all
six weather/geocode sources) and the existing revalidate_static middleware
(inbound). Never raises into the request path.
- GET /api/v2/metrics: token-gated JSON of the counters + warm-queue depth +
thread names/uptime/pid. Returns 404 unless a valid THERMOGRAPH_METRICS_TOKEN
is presented or the request is direct loopback with no proxy X-Forwarded-*
header, so ops data is never exposed through Caddy on the public domain.
- scripts/dashboard.py (+ scripts/dash, make dashboard): stdlib-only terminal
dashboard. Reads cache (parquet + SQLite), accounts (users + active
subscriptions), and system resources (/proc + systemd) directly, and pulls
traffic from the metrics endpoint over loopback. Live refresh, --once, --json;
mobile-terminal width. Paths resolve relative to the script so the same tool
works on the LAN dev and prod VPS checkouts.
- Monitoring sections in DEPLOY.md / DEPLOY-DEV.md; unit tests for the counters.
* Dashboard: run under the app venv, not the box's default python3 (#132)
The box's default python3 can be a build without _sqlite3 (pyenv 3.10 here), so
./scripts/dash crashed on import. Prefer the repo's .venv interpreter (the Python
the server itself runs on, which always has sqlite3); fall back to python3/python.
Also make the sqlite3 import in dashboard.py optional so a fallback interpreter
degrades to empty cache/accounts panels instead of crashing, and point the
'make dashboard' target at the wrapper.
* Dashboard: redraw live view in place so it stops scrolling to the bottom (#133)
The live loop did a full clear-and-reprint each refresh; when the output is taller
than the terminal (common on a phone) that scrolls the view to the bottom every tick.
Redraw from the top instead: hide the cursor, clear once, then each frame move to home
and rewrite line-by-line (clear-to-EOL + clear-below), capping the body to the window
height so it never prints more lines than fit and never scrolls. Restore the cursor on
exit; piped (non-TTY) output prints plain frames.
* Dashboard: make the live view a scrollable in-place pager (#134)
The live mode is now an alt-screen pager (like top/less): it shows only the latest
snapshot, you scroll it with swipe/arrows/jk/PgUp-PgDn/g/G, and each refresh replaces
the page in place while keeping your scroll position — nothing gets appended below and
the view is never yanked to the top or bottom. On exit it restores the cursor, leaves
the alternate screen, and resets the terminal mode.
--once still prints a single static snapshot (also used automatically when stdout/stdin
aren't a terminal, e.g. piping); --json unchanged.
* Metrics: don't count the dashboard's own /api/v2/metrics polling (#135)
The ops dashboard polls the metrics endpoint every few seconds; counting those hits
just shows the monitor watching itself and inflates the inbound totals. Skip the
'metrics' category in record_inbound (so it's excluded from inbound_total too), and
hide it in the dashboard's inbound list defensively.
* Dashboard: add a STORAGE section (archive + DB sizes) (#136)
Show on-disk storage: the parquet archives (with history/recent split), the derived
SQLite DB, and the accounts SQLite DB individually, plus a total. Archive bytes are
summed during the existing TTL-cached cache scan; DB sizes include the -wal/-shm
sidecars (WAL holds recent writes, so the base file alone understates the footprint).
Drop the now-redundant 'payload db' line from CACHE.
* Dashboard: show STORAGE sizes in fixed GB (0.01 GB floor) (#137)
The auto-scaling formatter only switched to GB once a value passed 1 GB, so the
archives read as MB and the DBs as KB. Format the STORAGE section in fixed GB with two
decimals and a 0.01 GB floor (so the small databases stay visible instead of rounding
to 0.00). Memory readouts keep the auto-scaling formatter.
* Dashboard: show STORAGE under 50 MB in MB, GB above (#138)
Values below 50 MB now render as MB with one decimal (no KB), so the databases read
naturally (derived 7.7MB, accounts 0.4MB) instead of 0.01GB; 50 MB and up stay in GB
with two decimals. Renames the helper _gb -> _storage_size to match.
* Dashboard: move STORAGE MB->GB cutoff to 500 MB (#139)
Values below 500 MB show in MB (one decimal); 500 MB and up in GB.
* Dashboard: accounts totals only; log client IPs per request (#140)
Dashboard ACCOUNTS section now shows only totals (users, active/total subscriptions,
notifications, push subs) — the per-user and per-subscription lists are gone, and the
dashboard no longer queries that PII at all.
Backend: add a per-request access log (logs/access/access-<date>.jsonl) recording the
client IP (real IP via the left-most X-Forwarded-For hop when proxied behind Caddy,
else the peer address), method, path, status, and category. Static assets and the
dashboard's own metrics polling are skipped. Best-effort, never raises into the request
path — retained for later traffic/IP analysis.
* Dashboard: wrap-aware pager so the header stops scrolling off (#141)
The pager counted one screen row per logical line, but on a narrow phone many lines
wrap to two rows — so a page printed more physical rows than the terminal had, scrolling
the top (title + CACHE header) off-screen with no way to scroll back to it. Measure each
line's wrapped height and fit a page to the real physical-row budget; page/End step by
what's actually visible; read the true tty size (not shutil, which honors stale
COLUMNS/LINES). Status line is clipped to one row.
* Add log storage size to the ops dashboard STORAGE section (#142)
The STORAGE section reported parquet archives and both SQLite DBs but
omitted the logs/ footprint, so the "total" understated real disk use.
Add a recursive _tree_size helper and a "logs" row measuring the whole
logs/ tree (audit + errors + access JSONL plus stray *.log files), broken
out by stream, and fold it into the STORAGE total. Flows through --json
automatically via read_storage.
* Dashboard: show last-10-minute traffic per category (#143)
Add a rolling short-window view alongside the since-start totals. metrics.py
keeps per-minute buckets (epoch-minute -> {key: count}) for inbound categories
and outbound sources, summed over the last 10 minutes on snapshot() and exposed
as inbound_recent / outbound_recent / window_minutes. Cheap and bounded — at most
WINDOW+1 tiny dicts, pruned as minutes roll off; self-polling still excluded.
The dashboard's TRAFFIC section now renders that count as a dim column next to
each row's total (header reads "total · last 10m"); idle sources show "-".
* Add dashboard #2 (#145)
* Auto-submit IndexNow on deploy when the URL set changes (#130)
Hook the production deploy (deploy/deploy.sh) to ping IndexNow after the health
check passes, so new/changed pages reach Bing/DuckDuckGo/Yandex without a manual
run. Best-effort — wrapped so it can never fail a deploy — and it sources
/etc/thermograph.env so its key matches the one the running service serves.
To avoid re-blasting ~14k URLs on every code push, indexnow.py gains a
--if-changed mode gated by a signature of the URL set (persisted to
data/indexnow_state.txt): code-only deploys skip; a deploy that adds or removes a
city submits. `make indexnow` still forces a full submit.
* Add ops metrics endpoint + SSH/Termux dashboard (#131)
Adds observability for the running app, viewable over SSH (e.g. Termux):
- backend/metrics.py: thread-safe, best-effort in-process counters for inbound
requests (per category) and outbound calls (per external source), plus a
phase->source map and snapshot(). Hooked into climate._request (outbound, all
six weather/geocode sources) and the existing revalidate_static middleware
(inbound). Never raises into the request path.
- GET /api/v2/metrics: token-gated JSON of the counters + warm-queue depth +
thread names/uptime/pid. Returns 404 unless a valid THERMOGRAPH_METRICS_TOKEN
is presented or the request is direct loopback with no proxy X-Forwarded-*
header, so ops data is never exposed through Caddy on the public domain.
- scripts/dashboard.py (+ scripts/dash, make dashboard): stdlib-only terminal
dashboard. Reads cache (parquet + SQLite), accounts (users + active
subscriptions), and system resources (/proc + systemd) directly, and pulls
traffic from the metrics endpoint over loopback. Live refresh, --once, --json;
mobile-terminal width. Paths resolve relative to the script so the same tool
works on the LAN dev and prod VPS checkouts.
- Monitoring sections in DEPLOY.md / DEPLOY-DEV.md; unit tests for the counters.
* Dashboard: run under the app venv, not the box's default python3 (#132)
The box's default python3 can be a build without _sqlite3 (pyenv 3.10 here), so
./scripts/dash crashed on import. Prefer the repo's .venv interpreter (the Python
the server itself runs on, which always has sqlite3); fall back to python3/python.
Also make the sqlite3 import in dashboard.py optional so a fallback interpreter
degrades to empty cache/accounts panels instead of crashing, and point the
'make dashboard' target at the wrapper.
* Dashboard: redraw live view in place so it stops scrolling to the bottom (#133)
The live loop did a full clear-and-reprint each refresh; when the output is taller
than the terminal (common on a phone) that scrolls the view to the bottom every tick.
Redraw from the top instead: hide the cursor, clear once, then each frame move to home
and rewrite line-by-line (clear-to-EOL + clear-below), capping the body to the window
height so it never prints more lines than fit and never scrolls. Restore the cursor on
exit; piped (non-TTY) output prints plain frames.
* Dashboard: make the live view a scrollable in-place pager (#134)
The live mode is now an alt-screen pager (like top/less): it shows only the latest
snapshot, you scroll it with swipe/arrows/jk/PgUp-PgDn/g/G, and each refresh replaces
the page in place while keeping your scroll position — nothing gets appended below and
the view is never yanked to the top or bottom. On exit it restores the cursor, leaves
the alternate screen, and resets the terminal mode.
--once still prints a single static snapshot (also used automatically when stdout/stdin
aren't a terminal, e.g. piping); --json unchanged.
* Metrics: don't count the dashboard's own /api/v2/metrics polling (#135)
The ops dashboard polls the metrics endpoint every few seconds; counting those hits
just shows the monitor watching itself and inflates the inbound totals. Skip the
'metrics' category in record_inbound (so it's excluded from inbound_total too), and
hide it in the dashboard's inbound list defensively.
* Dashboard: add a STORAGE section (archive + DB sizes) (#136)
Show on-disk storage: the parquet archives (with history/recent split), the derived
SQLite DB, and the accounts SQLite DB individually, plus a total. Archive bytes are
summed during the existing TTL-cached cache scan; DB sizes include the -wal/-shm
sidecars (WAL holds recent writes, so the base file alone understates the footprint).
Drop the now-redundant 'payload db' line from CACHE.
* Dashboard: show STORAGE sizes in fixed GB (0.01 GB floor) (#137)
The auto-scaling formatter only switched to GB once a value passed 1 GB, so the
archives read as MB and the DBs as KB. Format the STORAGE section in fixed GB with two
decimals and a 0.01 GB floor (so the small databases stay visible instead of rounding
to 0.00). Memory readouts keep the auto-scaling formatter.
* Dashboard: show STORAGE under 50 MB in MB, GB above (#138)
Values below 50 MB now render as MB with one decimal (no KB), so the databases read
naturally (derived 7.7MB, accounts 0.4MB) instead of 0.01GB; 50 MB and up stay in GB
with two decimals. Renames the helper _gb -> _storage_size to match.
* Dashboard: move STORAGE MB->GB cutoff to 500 MB (#139)
Values below 500 MB show in MB (one decimal); 500 MB and up in GB.
* Dashboard: accounts totals only; log client IPs per request (#140)
Dashboard ACCOUNTS section now shows only totals (users, active/total subscriptions,
notifications, push subs) — the per-user and per-subscription lists are gone, and the
dashboard no longer queries that PII at all.
Backend: add a per-request access log (logs/access/access-<date>.jsonl) recording the
client IP (real IP via the left-most X-Forwarded-For hop when proxied behind Caddy,
else the peer address), method, path, status, and category. Static assets and the
dashboard's own metrics polling are skipped. Best-effort, never raises into the request
path — retained for later traffic/IP analysis.
* Dashboard: wrap-aware pager so the header stops scrolling off (#141)
The pager counted one screen row per logical line, but on a narrow phone many lines
wrap to two rows — so a page printed more physical rows than the terminal had, scrolling
the top (title + CACHE header) off-screen with no way to scroll back to it. Measure each
line's wrapped height and fit a page to the real physical-row budget; page/End step by
what's actually visible; read the true tty size (not shutil, which honors stale
COLUMNS/LINES). Status line is clipped to one row.
* Add log storage size to the ops dashboard STORAGE section (#142)
The STORAGE section reported parquet archives and both SQLite DBs but
omitted the logs/ footprint, so the "total" understated real disk use.
Add a recursive _tree_size helper and a "logs" row measuring the whole
logs/ tree (audit + errors + access JSONL plus stray *.log files), broken
out by stream, and fold it into the STORAGE total. Flows through --json
automatically via read_storage.
* Dashboard: show last-10-minute traffic per category (#143)
Add a rolling short-window view alongside the since-start totals. metrics.py
keeps per-minute buckets (epoch-minute -> {key: count}) for inbound categories
and outbound sources, summed over the last 10 minutes on snapshot() and exposed
as inbound_recent / outbound_recent / window_minutes. Cheap and bounded — at most
WINDOW+1 tiny dicts, pruned as minutes roll off; self-polling still excluded.
The dashboard's TRAFFIC section now renders that count as a dim column next to
each row's total (header reads "total · last 10m"); idle sources show "-".
* Run prod on 3 uvicorn workers with a shared metrics store
A single slow upstream weather fetch (cache-miss during an open-meteo/NASA
degradation) blocked the one uvicorn worker and took the whole app down for
~2 min — real users got aborted connections. Give prod concurrency headroom so
one slow request can't freeze the rest.
- deploy/thermograph.service: worker count is env-driven (`--workers ${WORKERS}`,
default 1); prod sets WORKERS=3 in /etc/thermograph.env. Dev's separate
--user unit is untouched (stays single-worker).
- metrics.py: with multiple workers, per-process counters would split the tally
and the ops dashboard (polls one random worker) would see only a fraction.
Add a shared SQLite store selected by THERMOGRAPH_METRICS_DB so every worker
tallies into one place; unset keeps the zero-dependency in-memory store for
dev/tests/single-worker. Public API and snapshot shape unchanged.
- The unit defaults THERMOGRAPH_METRICS_DB and clears it on each (re)start via
ExecStartPre, so bumping WORKERS can never silently fragment the dashboard and
"since start" tallies keep their old meaning.
- Tests: cover the SQLite store — cross-worker aggregation, window roll-off,
and env-based selection. Full suite: 178 passed.
Note: applying to prod also needs the unit reinstalled + WORKERS=3 in
/etc/thermograph.env — deploy.sh only pulls+restarts, it doesn't reinstall the
systemd unit.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016mRqdHWRdUjEvEdfwig3wg
---------
Co-authored-by: root <root@vmi3417050.contaboserver.net>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
A failed web push (e.g. VAPID key mismatch -> 401/403) was swallowed: /push/test still
returned 202 and the UI showed 'Sent', while the only trace was a journald WARNING.
Now: push.send logs failures to the errors JSONL (phase=push) so they show up like other
errors; /push/test returns a 'failed' count; and the 'Send test' button reports 'No
device' / 'Sent' / 'Failed' from the real result. Document the VAPID env vars (missing
from thermograph.env.example) and how to pin/diagnose them.
Put Alerts in the main navigation (after Climate) across the interactive pages
and the SEO/climate pages, so it's a discoverable entry point rather than only the
header bell. The bell stays; this is an additional way in.
The signed-out /alerts gate now leads with a clear "What are weather alerts?"
explainer — what they do, how the percentile threshold works, and an example —
above the sign-in / create-account button, so visitors understand the feature
before signing in.
The extra nav item pushed the inline header past ~800px into a horizontal
scrollbar; trim the tab padding in the 641–1000px band so the row fits again
(phones already collapse into the hamburger).
On prod (served at root) the dashboard first probes /thermograph/api/v2/metrics before
falling back to root; that 404 was classified as inbound 'other' and shown as an error —
the monitor polluting the traffic it displays. Two fixes: classify any */api/v2/metrics
path as 'metrics' (excluded) whatever prefix it arrives under; and cache the working
metrics URL in the dashboard so it stops re-probing the failing base every refresh.
Add a rolling short-window view alongside the since-start totals. metrics.py
keeps per-minute buckets (epoch-minute -> {key: count}) for inbound categories
and outbound sources, summed over the last 10 minutes on snapshot() and exposed
as inbound_recent / outbound_recent / window_minutes. Cheap and bounded — at most
WINDOW+1 tiny dicts, pruned as minutes roll off; self-polling still excluded.
The dashboard's TRAFFIC section now renders that count as a dim column next to
each row's total (header reads "total · last 10m"); idle sources show "-".
Dashboard ACCOUNTS section now shows only totals (users, active/total subscriptions,
notifications, push subs) — the per-user and per-subscription lists are gone, and the
dashboard no longer queries that PII at all.
Backend: add a per-request access log (logs/access/access-<date>.jsonl) recording the
client IP (real IP via the left-most X-Forwarded-For hop when proxied behind Caddy,
else the peer address), method, path, status, and category. Static assets and the
dashboard's own metrics polling are skipped. Best-effort, never raises into the request
path — retained for later traffic/IP analysis.
The ops dashboard polls the metrics endpoint every few seconds; counting those hits
just shows the monitor watching itself and inflates the inbound totals. Skip the
'metrics' category in record_inbound (so it's excluded from inbound_total too), and
hide it in the dashboard's inbound list defensively.
Adds observability for the running app, viewable over SSH (e.g. Termux):
- backend/metrics.py: thread-safe, best-effort in-process counters for inbound
requests (per category) and outbound calls (per external source), plus a
phase->source map and snapshot(). Hooked into climate._request (outbound, all
six weather/geocode sources) and the existing revalidate_static middleware
(inbound). Never raises into the request path.
- GET /api/v2/metrics: token-gated JSON of the counters + warm-queue depth +
thread names/uptime/pid. Returns 404 unless a valid THERMOGRAPH_METRICS_TOKEN
is presented or the request is direct loopback with no proxy X-Forwarded-*
header, so ops data is never exposed through Caddy on the public domain.
- scripts/dashboard.py (+ scripts/dash, make dashboard): stdlib-only terminal
dashboard. Reads cache (parquet + SQLite), accounts (users + active
subscriptions), and system resources (/proc + systemd) directly, and pulls
traffic from the metrics endpoint over loopback. Live refresh, --once, --json;
mobile-terminal width. Paths resolve relative to the script so the same tool
works on the LAN dev and prod VPS checkouts.
- Monitoring sections in DEPLOY.md / DEPLOY-DEV.md; unit tests for the counters.
Hook the production deploy (deploy/deploy.sh) to ping IndexNow after the health
check passes, so new/changed pages reach Bing/DuckDuckGo/Yandex without a manual
run. Best-effort — wrapped so it can never fail a deploy — and it sources
/etc/thermograph.env so its key matches the one the running service serves.
To avoid re-blasting ~14k URLs on every code push, indexnow.py gains a
--if-changed mode gated by a signature of the URL set (persisted to
data/indexnow_state.txt): code-only deploys skip; a deploy that adds or removes a
city submits. `make indexnow` still forces a full submit.
Faster search-engine indexing for the ~14k climate/city/record URLs:
- IndexNow (backend/indexnow.py): instantly notify Bing/DuckDuckGo/Yandex of new
or changed URLs. Per-host key resolved env → gitignored file → generated (mirrors
push.py), served at /{key}.txt, and echoed in submissions. `submit_all()` + a
`make indexnow` CLI push every indexable URL, batched under the 10k cap. Google
doesn't use IndexNow, so it stays on the sitemap.
- Sitemap: replace the per-request today() <lastmod> (which churns every fetch and
trains crawlers to ignore lastmod) with a stable content-build date; factor the
URL list into public_paths() shared with IndexNow so the two never drift.
- Search-console verification: render google-site-verification + msvalidate.01
<meta> tags from env into every page's <head> — the SEO pages via base.html.j2
and the static pages (incl. the homepage Google verifies) via _page().
- Docs: env example documents the three new vars; .gitignore ignores the key.
Tests: key-file route, stable single-date lastmod, IndexNow payload/batching, and
verification meta on both SEO and static pages.
The server-rendered SEO pages shared base.html.j2's nav but carried no JS, so
they lacked the °F/°C toggle and account/bell, and their temperatures were baked
as dual-unit strings that couldn't respond to a toggle.
- Load units.js + account.js + climate.js on base.html.j2, so every SEO page gets
the same header as the interactive pages (nav + °F/°C toggle + account + bell).
- Emit each temperature as <span class="temp" data-temp-f> (default °F text, real
for crawlers/no-JS); climate.js rewrites them to the active unit on load and on
toggle. Tint classes stay pinned to absolute °F. Drops the inline "(NN°C)".
- Make account.js base-aware: derive the app base from import.meta.url so its
api/v2 fetches and the alerts links resolve from deep /climate/{slug} URLs
instead of 404ing. Equivalent on the depth-1 interactive pages.
- Default the unit from the browser locale for first-time visitors (no stored
pref): en-US → °F, everyone else → °C. A stored choice always wins.
- Compact the mobile menu: narrower panel (208px), tighter rows, and the nav
list now stretches to the panel width and left-aligns (it was centering because
the base .view-nav align-self:center overrode the panel's stretch).
- Stylish separation between entries: an inset hairline between each nav item,
suppressed around the active pill so nothing crosses it.
- Rename the "Day" view to "Day Detail" across the app nav, and add it to the
climate/city page nav (base.html.j2), which was missing it.
On phones the header spanned two rows: the title block plus a control bar of
the °F/°C toggle, bell, account and hamburger. Wrap the nav in a .nav-panel
inside the <details> menu and inject the °F/°C toggle and account/bell into that
panel instead of into the header row. On desktop the panel is display:contents,
so the inline top bar renders exactly as before (order restored via flex order);
on phones the panel becomes a right-anchored, viewport-clamped dropdown and the
header collapses to a single row (logo + name + hamburger, tagline hidden).
The viewport clamp also fixes the SEO/city pages, whose menu dropdown previously
ran off-screen.
- Collapse the view navigation into a hamburger menu on phones. The nav is
wrapped in a <details> (pure CSS, no JS — so it works on the crawlable SEO
pages too): transparent (display:contents) on desktop so the tabs render
inline, a dropdown behind a hamburger under 640px. The °F/°C toggle, bell and
account stay in the header bar; units.js/account.js now anchor their injected
controls on the wrapper.
- Replace the header logo: the ▚ glyph was a rotated Unicode char that fonts
rendered inconsistently (warped). Use a crisp inline SVG of the two accent
squares (currentColor) across all headers.
- Month pages: turn the plain "Visiting <city> in <month>?" line into a bordered
callout with a CTA button, matching the city page's travel box.
Verified in Firefox and Chromium at desktop and 390px, light and dark: desktop
nav inline, mobile hamburger opens the dropdown, logo crisp, travel note styled.
Complete the forecast fallback chain: Open-Meteo → MET Norway → stale cache. When
both live sources are unavailable, serve the last cached recent/forecast bundle
even if it is past its 1-hour TTL — an hours-old bundle (which still carries the
recent observed days MET Norway lacks) beats a hard 503, mirroring the archive
path's stale-serve in _load_history.
The stale bundle is returned WITHOUT rewriting it, so its mtime stays old and the
next request retries the live sources first rather than treating the stale copy as
fresh. Only when there is no cache at all does the typed WeatherUnavailable surface.
The forecast path had no backup — unlike history, which falls back to NASA POWER.
When Open-Meteo's forecast API was rate-limited or down, the recent/forecast
bundle (and every endpoint that grades future days) failed with a 503.
Add MET Norway (yr.no) Locationforecast as a keyless, global forecast backup,
mirroring the NASA POWER role for history:
- _metno_to_frame: aggregates MET Norway's sub-daily timeseries into the daily
schema, converting units (°C→°F, mm→in, m/s→mph) and deriving feels-like from
the NWS heat index / wind chill (MET has no gusts or apparent temperature).
Precip prefers the 1-hour block and falls back to the 6-hour block so the
hourly→6-hourly resolution switch never double-counts.
- _fetch_forecast_metno: the backup fetch, with the ToS-required identifying
User-Agent and coordinates rounded to 4 decimals.
- _load_recent_forecast: on any Open-Meteo forecast failure, try MET Norway
before surfacing the error; a shared rate limit still raises the typed,
daily-aware WeatherUnavailable.
MET Norway is forecast-only (no recent past days), so it's a degraded-but-working
fallback: the forecast / day-ahead views keep serving during an Open-Meteo outage.
Tests cover the daily aggregation + unit conversion (incl. the no-double-count
precip rule) and the fallback wiring.
The Wikipedia "Read more →" link in the notable-weather section could break
between "Read" and "more" at narrow widths. Join it with non-breaking spaces so
it stays on one line.
Claude-Session: https://claude.ai/code/session_01XXxmNFy9cZ6Gh8Y9thZn62
The per-month records section listed only the warmest and coldest temperature.
Extend it to a card per metric (High, Low, Feels-like, Humidity, Wind, Gust,
Precip), each showing that calendar month's record high and low with the date.
Precip is special-cased: a per-day record low is just zero, and the dry-streak
helper would bridge year boundaries on month-filtered rows, so the wettest and
driest sides use the month's largest and smallest total accumulation, dated to
the year. Partial current months (< 26 days) are excluded so an unfinished month
can't win "driest" on a fraction of its rainfall.
Reuses the records page's card grid, so it's already vertical on mobile.
Claude-Session: https://claude.ai/code/session_01XXxmNFy9cZ6Gh8Y9thZn62
The month and season records tables showed only the record high (of the daytime
high) and the record low (of the overnight low) — one extreme each. Show both
ends of both metrics: for the daytime high, its record hottest (▲) and the
coldest a day ever stayed (▼, its record-low high); for the overnight low, its
mildest night (▲, its record-high low) and its record low (▼). All four already
came out of all_time_records; they were just being dropped.
Restructure each table to Period · Daytime high · Overnight low, with each metric
cell carrying two tinted, dated chips (▲ warmest / ▼ coldest). Fits mobile in
three columns; the all-time cards already showed both extremes, so this brings
the per-period tables in line.
The monthly temperature-range visualization now spans the 10th-percentile daily low
to the 90th-percentile daily high (from climatology's p10/p90) — the band most days
fall within — instead of just the average low to average high, giving a truer sense
of each month's spread. The normals table keeps the average high/low figures.
Add a search box to /climate that filters to a flat, alphabetical list of matching
cities (by city or country name), shown with their country; clearing it restores the
collapsible directory. The index is built from the already-crawlable links, so it's
a pure progressive enhancement (no SEO impact). Also sort cities alphabetically
within each country in the directory (were population-ordered).
Two mobile polish issues on the climate pages:
- The season records table's "(Dec–Feb)" span overflowed its narrow first
column (nowrap) and slid under the tinted value cell. Stack the span under the
season name on phones, top-align the record cells, and let the first column
wrap — so the period label sits cleanly in its own column.
- The monthly temperature-range strip stretched edge-to-edge with the values
floating far to the right. Cap the strip width so bars + values pack into a
tight group, shrink the value labels to fit (compact "41°–66°", no reserved
92px column). The overrides now sit after the base range rules so they win.
The all-time-records redesign (#106) removed the min-width floor that let the
wide records tables scroll, so the month and season records tables — still five
columns (value + separate date, twice) — compressed below their content on
phones and overlapped/clipped.
Fold each record's date under its value so those tables become three columns
(period · record high · record low), which fit any width without horizontal
scroll. Also tighten all climate tables under 560px (smaller font + padding) so
a hot city's 3-digit °F values with °C in parens never collide in the city
page's normals table.
* SEO: monthly & seasonal records on the city records page
The /climate/<city>/records page showed only all-time records per metric.
Expand it into a full records page: record high/low for each of the 12
months (each linking its month page) and for each meteorological season,
alongside the existing all-time table.
Seasons are hemisphere-aware — Dec–Feb reads as winter for northern cities
and summer for southern ones (picked from the city's latitude). Records
reuse grading.all_time_records over a month-filtered archive, so no new
data fetching or warming is needed. Adds a Dataset + breadcrumb JSON-LD
block and richer title/description for the expanded coverage.
* SEO: colour-code climate & records pages (heat-map + range strip)
The city, records and month pages were plain muted tables — generic climate-site
styling. Bring the interactive grader's diverging cold→hot palette onto them so
they read as heat maps in the site's own visual language:
- Map absolute °F to the 9-tier palette (temp_class, a Jinja global) and tint the
temperature cells across the monthly-normals, monthly/seasonal/all-time records
tables. The value stays in the ink token; the tint is a wash behind it. Non-temp
rows (humidity/wind/precip) and date columns stay untinted.
- Add a monthly temperature-range strip on the city page: one gradient bar per
month spanning the average low→high on a shared −10..115°F axis, so a city's
whole-year rhythm (and hot-vs-cold character) reads at a glance.
- Tint the month page's hero high/low and its record values inline.
Palette, light/dark, and 390–1920px layouts verified by rendering hot (Phoenix)
and cold (Anchorage) cities.
Replace the 5-column records table (which forced horizontal scrolling on mobile —
only high OR low visible) with a responsive card grid: one card per metric, each
with a red-accented 'Highest' row and a blue-accented 'Lowest' row (tier colors).
Grids on desktop, stacks to a single column on mobile with high/low stacked
vertically — no horizontal scroll. Precip reads Wettest / longest dry spell.
Record-low rainfall is meaningless (always ~0), so the precip row's low side now
shows the longest run of consecutive days without measurable rain and the date that
dry spell began (grading.longest_dry_streak), keeping the wettest day on the high
side. A footnote explains the substitution.
Drop the open-by-default on the first three countries (every <details> starts
collapsed) and order the country sections alphabetically instead of by largest
city's population.
- Climate hub (/climate) now uses native <details> per country — collapsible
expand/shrink sections with a city count and a rotating marker (top 3 countries
open by default). Links stay in the DOM, so crawling/SEO is unaffected.
- Fix nested F/C parentheses in the city lede (e.g. 'July (71°F (22°C) average
high)') by rephrasing so the already-parenthesized temperature isn't wrapped in
another paren.
- Fix the 5-column records table overlapping on mobile: give it a min-width so it
scrolls inside .table-wrap on narrow screens instead of colliding.
Add EXCLUDE_CC + KEEP_SLUGS to gen_cities.py: drop cities in countries that
criminalize LGBTQ+ people (plus China, Pakistan, Indonesia by choice), except a
hand-kept list of notable / tourist / high-English hubs — Lagos & Abuja; Nairobi &
Mombasa; Cairo, Giza, Alexandria; Dubai & Abu Dhabi; Kuala Lumpur/Malacca/Kota
Kinabalu/Kuching; Dar es Salaam/Zanzibar/Arusha; Casablanca/Fes/Rabat; Tehran;
Jeddah; Baghdad; Kabul; 7 Pakistani cities (Karachi, Lahore, Islamabad, Faisalabad,
Rawalpindi, Multan, Hyderabad); Jakarta + Surabaya + Bekasi; and China's 5 most
English-friendly cities (Shanghai, Beijing, Shenzhen, Guangzhou, Chengdu). Removed
slots backfill from the next-ranked non-excluded cities, so cities.json stays 1000.
Flavor regenerated (prunes removed, fetches backfilled).
Common-sense coverage pass on city_events.py: the list was 10/31 US and heavily
Anglosphere with no Latin America, no Middle East, and only London/Paris for all of
continental Europe. Add 9 verified events filling those gaps — Los Angeles (2025
wildfires), Moscow (2010 heat/wildfires), Madrid (2021 Storm Filomena), Athens (2018
Attica fires), Seoul (2022 floods), Dubai (2024 floods), Jeddah (2009 floods), Sao
Paulo (2014-17 drought), Rio de Janeiro (2010 floods). Every Wikipedia link
verified to resolve. 40 events total; nothing removed (no factual errors found).
Auto-sourcing a notable weather event from Wikipedia search proved unreliable
(wrong matches, low coverage), so city_events.py is a small, fact-checked list of
one iconic event per city (Katrina/New Orleans, Harvey/Houston, the 1952 Great
Smog/London, the 2021 heat dome/Seattle, ...) — 31 cities, each with a Wikipedia
link. The city page shows a 'Notable weather in {city}' callout when one exists and
otherwise just the flavor blurb. Tests check the curated slugs are all valid and the
callout renders (and is absent for uncurated cities).
Add a third tier to gen_cities.py: the top-population cities from the remaining
English-official countries plus countries where >35% speak English (Eurobarometer/
EF), that weren't already chosen. cities.json grows to 1000 (500 global + 250
core-English + 250 extended), pulling in the biggest cities of India, Nigeria,
Pakistan, the Philippines, plus Amsterdam/Stockholm/Nairobi/Tel Aviv, etc.
gen_flavor.py is now incremental (fetches only cities missing a blurb, prunes stale
ones; --full to rebuild); cities_flavor.json refreshed to cover 942/1000.
Add unique editorial content so the programmatic pages don't read as templated:
- gen_flavor.py seeds backend/cities_flavor.json with a short descriptive blurb per
city from Wikipedia's free REST summary API (no key), validating each match by
comparing article coordinates to the city's lat/lon so the wrong 'Springfield'
never attaches. Retries + modest concurrency; ~700/750 cities get a blurb, the
rest render without one. Text is CC BY-SA, attributed with a 'via Wikipedia' link.
- City pages show the blurb under the intro and a travel callout that deep-links to
the compare page with the city pre-loaded (/compare#loc=lat,lon) — the visitor
just adds their own city. Month pages get the same seasonal 'visiting in {month}?'
compare link. Both add unique per-page text and internal links.
- cities.py gains flavor(slug); tests cover the blurb + attribution + compare CTA.
The population-ranked global top-500 skewed to Asian megacities and missed
high-English-search-demand cities. gen_cities.py now tops up with the top ~250
cities from English-speaking countries (US/GB/CA/AU/NZ/IE/ZA) not already in the
global set, so US coverage goes 13->146, GB 2->42, CA 3->29, etc. (Seattle, Boston,
Manchester, Melbourne, Auckland, Dublin, ...). cities.json regenerated to 750.
Both deploy scripts now launch warm_cities.py automatically after the health check,
detached (dev: a systemd --user transient unit; prod: setsid/nohup), so the city
pages serve from cache without a manual step; idempotent, so only the first deploy
does the full warm. DEPLOY.md updated.
* SEO: generate curated city set for crawlable climate pages
gen_cities.py reuses the GeoNames index places.py already parses to select the top
~500 metros by population, assigns each a stable URL-safe slug (dropping admin1 when
it repeats the city name), and writes committed backend/cities.json. cities.py loads
it lazily with slug lookup, all_slugs(), display_name(), and by_country() grouping
for the upcoming hub + sitemap.
* SEO: rendering core, robots.txt, sitemap.xml, and metadata hygiene
- content.py: Jinja2 environment + HTML responder (ETag/304), dynamic /robots.txt
(disallows /api and /alerts, points at the sitemap) and /sitemap.xml (enumerates
the home/static pages plus every city, month, and records URL from cities.py).
Registered on the app before the StaticFiles mount so the routes win.
- templates/base.html.j2: shared layout with unique title/description, self-
referential canonical, Open Graph, favicon/manifest, header nav (adds a Climate
link) and a footer link graph.
- Give each existing page a unique <meta description> (were 5x identical) and a
self-referential <link rel=canonical>; add WebApplication JSON-LD to the home page.
- Pin jinja2.
* SEO: server-rendered per-city climate page (/climate/{slug})
The keystone crawlable page: for a city it snaps to the grid cell, loads the
archive (fetching once if missing, self-healing), and renders as real HTML — a
'how today compares' block (grade + percentile per metric from grade_day, tinted
by tier), a monthly normals table (climatology at each month's 15th, shown in °F
and °C), all-time records (new grading.all_time_records helper), a breadcrumb,
Dataset+Place+BreadcrumbList JSON-LD, self-referential canonical, and links into
the interactive tool + month/records pages. Content-page CSS added to style.css
(renamed the table class to avoid colliding with the app's .normals flex row).
* SEO: month (/climate/{slug}/{month}) and records (/climate/{slug}/records) pages
Month pages render the exact-month long-tail ('average weather in {city} in
{month}') with that month's average high/low, typical p10-p90 range, month-specific
records, and prev/next month links. Records pages show all-time record highs/lows
per metric with dates (grading.all_time_records). Shared _resolve_city helper; the
literal /records route is registered before the {month} param and month names are
validated (unknown month -> 404).
* SEO: climate hub, weather glossary, and about/methodology pages
- /climate: crawlable directory of all ~500 cities grouped by country — the
internal-link graph that lets search engines discover every city page.
- /glossary + /glossary/{term}: plain-language definitions (climate normal,
percentile, temperature anomaly, feels-like, heat index, wind chill, humidity,
reanalysis) with DefinedTerm JSON-LD and cross-links into the tool.
- /about: methodology page (ERA5 data source, 45-year baseline, +/-7-day window,
percentile grading) for E-E-A-T. All linked from the shared footer.
* SEO: archive warmer, content-page tests, and deploy docs
- warm_cities.py: paced, idempotent offline warmer that pre-fetches each city
cell's archive so /climate pages serve from cache and a crawl can't burst the
archive quota (pages self-heal if hit before warming).
- tests/test_content.py: city-set slug uniqueness/lookup, robots.txt, sitemap
enumerating city/month/records URLs, and that a rendered city page carries the
stats + canonical + Dataset JSON-LD in the HTML; plus month/records/hub/glossary/
about routing and 404s.
- DEPLOY.md: document the content pages, the warm step, and submitting the sitemap.
Make the app installable and deliver existing alert notifications as OS
push, alongside the in-app bell.
Backend:
- PushSubscription model (per-device endpoint + keys, owned by a user) and
register/unregister/test endpoints under /api/v2/push, cookie-auth scoped
to the user like the subscription routes.
- push.py: VAPID key management (env -> data/vapid.json -> generated) and a
pywebpush send helper that reports gone endpoints for pruning. No DB coupling.
- notify.py: after creating an in-app Notification, dispatch Web Push to the
user's devices (guarded — a push failure never affects the in-app write;
endpoints reported gone are pruned).
- Serve the .webmanifest with the correct media type.
Frontend:
- manifest.webmanifest + 192/maskable icons; <link rel="manifest"> on all pages.
- sw.js: push + notificationclick handlers (push-only; no fetch caching, so it
doesn't fight the existing IndexedDB cache). Registered globally in nav.js in
secure contexts.
- push-client.js + a "Notifications on this device" toggle and test-send on the
/alerts page, subscribing through the existing cookie-aware apiFetch.
Push and service workers require a secure context, so this is active over HTTPS
(or http://localhost) and cleanly no-ops on a plain-HTTP LAN origin.
A subscription to a city that had never been viewed had no cached ~45-year
archive, so the evaluator skipped it and it never fired. Now when a subscribed
cell has no cached archive, the pass fetches it once via get_history (which caches
it); every later pass reads it from cache and never re-fetches. A per-pass budget
(THERMOGRAPH_NOTIFY_ARCHIVE_FETCHES, default 4) caps how many missing archives one
pass will fetch so a burst of new subscriptions can't exhaust the archive quota,
and a rate-limited fetch just retries on the next pass. The recent/forecast bundle
keeps refreshing on its own hourly cadence.
Add integration tests covering fetch-when-missing and never-refetch-when-cached.
The notification dropdown is anchored to the bell button, which on mobile sits
left of the account button rather than at the screen edge, so the wide panel
overflowed off the left of the viewport (title/text cut off). On narrow screens,
drop .notif's positioning context so the dropdown anchors to the .acct cluster at
the header's right edge, keeping it fully on-screen.
Also add route-level tests for the accounts feature (auth flow, subscription CRUD,
duplicate/validation, ownership 404s, notifications), plus a
THERMOGRAPH_ACCOUNTS_DB override so the suite writes to a throwaway DB and stays
hermetic.
Move Thermograph to its own domain. thermograph.org now serves the app at its
root, and emigriffith.dev/thermograph* permanently redirects there (prefix
stripped, so deep links map straight across).
- app.py: BASE now supports an empty root prefix. THERMOGRAPH_BASE=/ (or empty)
yields BASE="" so routes sit at "/", "/calendar", "/api/v2/…" with no double
slashes; the bare-base redirect is skipped and the static mount falls back to
"/". Non-empty values keep the existing "/thermograph" sub-path behavior
unchanged (backward compatible).
- deploy/Caddyfile: add a thermograph.org site block reverse-proxying to the
uvicorn on 127.0.0.1:8137; turn emigriffith.dev's /thermograph handler into a
permanent redirect to thermograph.org (handle_path strips the prefix; the bare
/thermograph goes to the root).
- deploy/thermograph.env.example: default THERMOGRAPH_BASE=/ (app owns the domain).
- DEPLOY.md: document the two-domain layout and that deploy.sh does not ship the
Caddyfile or env — those are applied on the VPS by hand.
The frontend already uses base-relative URLs, so it follows the root base with no
changes. Verified both modes locally (root serves /, /calendar, /api/v2/*; the
/thermograph sub-path still redirects bare→slash and 404s at root) and the full
test suite (123) passes.
* Migrate backend dataframe layer from pandas to polars
Replace pandas with polars across the backend, dropping both pandas and its
pyarrow parquet engine from the dependency set. numpy stays (the grading
percentile math is unchanged).
- climate.py: parquet IO, source→frame mappings, cache read/topup on polars.
New _normalize_read casts the cached `date` column to pl.Date (older files
were written by pandas as datetime64[ns]); frames now unify missing values as
null so the grading boundary drops them consistently across sources.
- grading.py: keep the numpy percentile core; swap the frame→numpy bridge to
.to_numpy()/.drop_nulls(), day-of-year/year to polars dt expressions, and the
per-row loop to iter_rows(named=True).
- views.py: filter/anti-join/concat replace boolean-mask, isin and pd.concat;
scalar dates are stdlib datetime.date; a local _months_before helper replaces
DateOffset(months=) for the calendar-range default.
- app.py, migrate.py: request-date parsing uses datetime.date, removing pandas
from the endpoint and migrate layers entirely.
- The date column is pl.Date end to end, eliminating the pandas normalize() calls
and comparing cleanly against stdlib dates.
Payloads are unchanged: calendar, day, grade and forecast responses are
byte-for-byte identical to the pandas implementation on the same cached record.
Tests ported to polars fixtures, with added coverage for the combined feels-like
fallback, calendar month-offset (month-end/leap), and the concat/dedup
"fresher source wins" rule.
* Port notify.py to polars after merging dev's account system
Merge origin/dev (accounts + notification subscriptions) and carry the pandas→
polars migration into the newly added notify.py, which the merge brought in still
using pandas — with pandas removed from requirements this broke its import.
- notify.py: _candidate_rows filters/sorts the recent bundle with polars
expressions and returns iter_rows dicts; date scalars are datetime.date;
history/recent emptiness via is_empty().
- test_notify.py: synthetic history/rows built with polars + datetime.
* Add account system foundation: email/password auth with cookie sessions
Introduce the app's first authoritative, user-owned data in a separate
data/accounts.sqlite (SQLAlchemy), kept apart from the disposable derived-cache
DB. Wire fastapi-users for email/password signup, cookie-based login/logout, and
a session-check endpoint, backed by a database session strategy so logins survive
restarts and are revocable.
- db.py: async (aiosqlite) + sync SQLAlchemy engines over accounts.sqlite, WAL +
foreign keys, create_db_and_tables().
- models.py: User, AccessToken, Subscription, Notification tables.
- users.py: pwdlib hashing, HttpOnly cookie transport (path-scoped, SameSite=Lax,
Secure via env), DatabaseStrategy sessions, current-user dependencies.
- schemas.py: user + subscription + notification Pydantic models.
- app.py: mount auth/register/users routers on v2, create tables at startup.
- Pin fastapi-users[sqlalchemy]/aiosqlite; ignore data/accounts.sqlite*.
* Add account header entry and auth modal (frontend)
account.js self-injects a header entry (following the units.js pattern) that
shows a Sign in button when logged out and an account menu when logged in, plus
an auth modal reusing the existing .mp-overlay/.mp-modal chrome for email/password
sign-in and account creation. A shared apiFetch helper sends the same-origin
cookie for authed calls; exported getUser/openAuth/onAuthChange back later phases.
Imported by every page entry module. On narrow screens the entry collapses to an
icon-only button so it doesn't crowd the title.
Enforce an 8-character minimum password in the user manager.
* Add subscription CRUD API and the alerts management page
Backend api_accounts.py adds user-scoped, cookie-authenticated endpoints to
create/list/update/delete subscriptions (and the notification reads used next):
POST snaps lat/lon to a grid cell, resolves a label, and rejects a duplicate
location+kind with 409; PATCH/DELETE are ownership-checked (404 on mismatch).
Mounted on the v2 prefix.
Frontend subscriptions.js + subscriptions.html serve the /alerts page: a sign-in
gate when logged out, an add flow that reuses the shared map picker and an editor
modal (kind, watched metrics, 95-99 percentile, two-sided), and a card list with
inline threshold/active edits and remove. Reachable from the account menu.
* Add background subscription evaluation engine
notify.py runs a daemon thread that periodically evaluates every active
subscription: it groups them by grid cell, reads history from the parquet cache
only (never spends archive quota) plus the hourly recent/forecast bundle, and
grades candidate days with the existing grading.grade_day. A watched metric that
lands at or beyond the threshold percentile fires a 'high' alert; a two-sided
subscription also fires 'low' for the symmetric cold/calm/dry tail (precip stays
one-directional). Observed subscriptions look at the last few recorded days,
forecast subscriptions at the coming week.
Two guards keep it quiet: a UNIQUE(subscription, event_date, metric, direction,
kind) constraint dedups repeat events, and a per-subscription weekly cap
(last_notified_at) limits each alert to one notification per 7 days. The loop
tolerates a bad cell or an upstream rate limit without aborting the pass. Started
and stopped from the app lifespan; gated by THERMOGRAPH_ENABLE_NOTIFIER.
* Add in-app notification center (header bell)
Extend account.js with a notification bell beside the account menu: an unread
badge, a dropdown listing recent notifications (title, body, relative time), a
per-item mark-read on click, and a Mark all read action, all through the
cookie-authed notifications API. Unread state refreshes on open and polls every
two minutes while signed in; polling stops on sign-out. Styled to match the app,
responsive down to mobile.
* Harden accounts: expired-session cleanup, engine tests, ops docs
- notify.py sweeps expired login sessions (access tokens past their lifetime)
once per evaluation pass.
- Add hermetic unit tests for the evaluation engine's trigger detection (high/low
tails, precip one-directional, normal = no trigger) and notification wording.
- Document accounts.sqlite (authoritative, back it up), the single-worker
requirement for the in-process evaluator, and the new env vars in DEPLOY.md.
- Fix the mobile width blow-out: the distribution grid item (.cmp-dist-row) had no
min-width:0, so each row expanded to the 9–10-column connected bar chart's
min-content and widened the whole page (zoom-out, clipped filter sheet). Add
grid-template-columns: minmax(0,1fr) + min-width:0 so the strip scrolls inside its
own .ct-strip, and tighten .ct-col to 34px on phones.
- Add a Load/Refresh button inside the compare filter sheet's date-range block, wired
to the same refresh()/isDirty(); editing the range on mobile no longer needs the
sheet closed. The existing button stays in the controls (loads added places).
- Default date range on both pages is now January six years back → the present. On
the calendar this is an explicit chunked range (the months=24 path is server-capped
at ~2yr).
- Names now include country: reverse_geocode appends it to the place label; revgeo
cache key gets a v2 prefix and PAYLOAD_VER bumps to p2 so labels/payloads
repopulate.
- Location names read as a hierarchy: compact chips show just the lead segment (full
name in the title/aria), and each ranked card shows the lead segment bold over a
muted "city · region · country" line.
Frontend + a small backend name/cache change; no schema migration.
cache.js contained a JavaScript clone of backend/grid.py's snapping math
(its own comment said so) to compute the 8 surrounding cells and fire 8
staggered prefetch requests — grid geometry had two homes, one per
language, plus a client-side guess at Nominatim pacing.
The server now owns it: grid.neighbors(cell) steps one cell width from
the center and re-snaps (adjacent rows have different longitude steps;
poles and the antimeridian handled by snap), and /api/v2/cell grew a
neighbors=1 flag that enqueues those cells for a single background
worker. The warm-only guarantee matches prefetch=1 — a cell with no
cached archive is skipped, so no weather-API quota is ever spent — and
reverse_geocode's own lock paces the at-most-one Nominatim call per
never-labeled cell. Re-enqueues are TTL-deduped; the worker starts from
the lifespan hook, so tests and offline importers never spawn it.
The client now sends its one conditional bundle request with
neighbors=1 (a warm spot costs an empty 304) instead of skipping the
bundle and firing 8 extra requests; the grid-math clone and the
now-unused hasFreshCache are deleted.
Tests (114): grid.neighbors mid-latitude/pole/antimeridian, the
neighbors=1 enqueue + TTL dedupe, _warm_cell materializing the
history-derived store rows, and the never-fetch-upstream guarantee.
Verified with the headless-Chromium smoke across all five pages.
climate.py repeated the same four mechanical patterns:
- the Open-Meteo daily params dict (3x) -> _om_daily_params(cell, **window)
- the doy attach line (6x) -> _with_doy
- makedirs + drop-doy + zstd to_parquet (3x) -> _write_cache
- the identical _to_frame/_nasa_to_frame tail (feels-like, valid-day
filter, doy) -> _finalize_frame
grading.py encoded the tier tables twice — TEMP_BANDS/RAIN_BANDS plus the
hand-aligned _TEMP_LADDER/_RAIN_LADDER ('kept aligned' by comment). The
ladders are now derived from the bands (_ladder_from; verified
byte-identical to the old tables before landing), so tier boundaries have
exactly one definition. grade_range's inline dry-streak walk is replaced
with the existing dry_streaks(); its per-(doy,var) sample cache now also
memoizes the window mask per doy instead of recomputing it once per
metric (9x per day-of-year).
New tests pin the refactor: _to_frame schema/day-filter/missing-series
tolerance, the combined feels-like side selection, NASA unit conversions
and fill-sentinel handling, _om_daily_params windows, and _write_cache
stripping the derived doy column.
api_suggest carried ~80 lines of ranking policy — the exactness-boosted
scoring, local/upstream blending with dedupe, and the token-respelling
correction loop — all consuming the match: prefix/fuzzy vocabulary that
places.py produces. Producer and consumer now live together:
places.suggest(q, upstream, limit) implements the whole policy with the
upstream geocoder lookup injected as a callable, so places stays free of
the fetch layer and the policy is testable without FastAPI.
The endpoint is the HTTP shim: call places.suggest, map the
nothing-at-all-to-serve failure to a 502. Behavior unchanged.
Policy tests: upstream skipped when the local answer convinces, blend +
dedupe, the exactness boost (a hamlet spelled like the typo can't beat
Seattle), single-typo queries answered by the fuzzy matcher without a
correction, the correction path verified via the upstream probe,
upstream failure degrading to local results, and the terminal
error-propagation case.
Rate-limit handling crossed the climate→app boundary as prose: climate
raised RuntimeError(cooldown text), and app._weather_fetch_error re-parsed
it by keyword ('429'/'rate-limited'/'daily'/'tomorrow') while also calling
climate's private _is_rate_limit/_rate_limit_reason — and the user-facing
daily-quota copy existed verbatim in both modules.
climate now raises WeatherUnavailable (a RuntimeError subclass carrying
the user-facing text and a daily flag): from _load_history when both
sources fail with no stale cache, and from the forecast fetch on a 429.
limit_message() is the single home of the rate-limit copy; is_rate_limit
is public for the one remaining raw-429 fallback. app maps the typed
error to 503 by isinstance — no message parsing, no private imports.
The burst-limit copy is unified on 'The weather service is rate-limited…'
(the archive path previously said 'weather archive' internally but the
API always rewrote it; user-visible text is unchanged).
Tests: daily-quota 503 carries the 'tomorrow' copy, an unclassified raw
429 still maps to a clean 503, and a genuine fault stays a 502 with the
raw error.
