beta (Forgejo's pinned node) is also today's live thermograph.org host,
with Caddy already bound to ports 80/443. A second reverse proxy
binding those same ports would collide with it. Drop Traefik entirely:
forgejo's web port now publishes to 127.0.0.1:3080 only, and a new
Caddy site block (deploy/forgejo/caddy-git.conf) reverse-proxies
git.thermograph.org to it, reusing Caddy's existing automatic-HTTPS
instead of a second ACME flow.
That same Caddy block resolves the registry-exposure question (hazard
#15 in the hop1 runbook) as its first listed option: /v2/* (the
registry API) is blocked to everything outside the WireGuard mesh
CIDR, while the git/web UI stays public. README documents the
/etc/hosts override mesh clients need so registry traffic actually
routes over the tunnel rather than the public IP.
Bring the Swarm+Forgejo layer in line with the canonical topology in
docs/runbooks/implementation-handoff.md: three nodes (prod, beta, and
the desktop LAN dev machine) instead of two, with the Forgejo Actions
runner registered on the desktop as a plain systemd service rather than
a Swarm-scheduled Docker-in-Docker sidecar.
- setup-wireguard.sh: full N-peer mesh instead of point-to-point
- docker-stack.yml: drop the runner/runner-dind services, volumes, and
runner-token secret; only forgejo_db_password remains
- register-lan-runner.sh: register under both docker and
thermograph-lan labels, since one runner now covers both job types
- init-swarm.sh, join-swarm.sh, swarm/README.md, forgejo/README.md,
INFRA.md: updated node lists, order of operations, and access-state
table for three nodes
- deploy.yml: renamed to "Deploy to beta VPS" and reconciled the
main-vs-release branch question against terraform.tfvars.example
(this workflow already targets beta; a release-triggered prod
deploy doesn't exist yet and isn't invented here)
warm_cities.py and indexnow.py are one-shot scripts run manually at deploy
today - nothing keeps the curated city set topped up, or pings IndexNow,
between deploys.
Add notifications/scheduler.py: an in-process APScheduler instance driving
two recurring jobs (city warming, daily; IndexNow --if-changed, every 6h by
default), gated by the same leader election as the notifier - only the
process that already won leadership starts it, so the jobs run in exactly
one place under multi-host Swarm rather than once per replica. Neither job
runs immediately on start: warm_cities already runs at deploy time, so an
immediate duplicate on every worker boot/restart would be wasted work.
Extract indexnow.submit_if_changed() from the CLI's --if-changed branch so
the scheduler and the command line share the exact same skip-when-unchanged
logic instead of two copies drifting apart. The CLI's plain (non-flag) path
is unchanged.
APScheduler over a Postgres-native queue: exactly one process ever runs
this, no fan-out/backpressure/dead-letter need, no new infrastructure.
Terraform generates the secrets that have no external meaning
(POSTGRES_PASSWORD, AUTH_SECRET, METRICS_TOKEN, INDEXNOW_KEY) via the random
provider instead of requiring the operator to hand-generate and paste each
into terraform.tfvars. Each is pinned with a static keepers value (secrets.tf)
so apply never regenerates a value already in use - the exact incident class
this guards against: every session invalidated, the app<->DB password
mismatched. Rotation is now a deliberate keepers edit, never a side effect.
postgres_password/auth_secret move from required inputs to optional (default
"") - explicit var wins when supplied (seeding an EXISTING live secret during
a migration onto Terraform, hop-1 cutover runbook Stage 0), else Terraform
generates and owns it. metrics_token/indexnow_key are new: neither existed in
Terraform before, both previously left for the app's own fallback generation.
VAPID deliberately stays a required, non-generated input - an EC keypair
where regeneration breaks every existing push subscription outright, unlike
an opaque token.
Sizing tiers: a locals.sizes t-shirt map (nano/small/medium/large ->
{workers, app_cpus, db_cpus, db_memory}), toward the target Proxmox
sizing-tier model (architecture doc SS6) ahead of actually provisioning VMs -
Proxmox itself stays deferred; today a tier just sizes container caps on the
existing SSH-managed hosts. A host can reference one by name (hosts.<name>.
size) or keep hand-picking the four fields, so existing tfvars are
unaffected; prod's example now uses size = "large" (identical numbers),
beta keeps explicit numbers, and a commented uat example demonstrates the
shortcut for a future ephemeral host.
Strengthened terraform/README.md's local-state caveat: more Terraform-
generated secrets landing in tfstate raises the stakes of the existing
never-commit-cleartext-state guidance, not just the sizing.
Verified: terraform validate + fmt clean. A real `terraform plan` against
fake hosts (prod/beta/uat, mixing size="large"/explicit-numbers/size="nano")
resolved every sizing correctly (prod 8/8/4/16g, beta 4/4/2/8g, uat
1/1/1/1g) and planned exactly one instance of each random_password/random_id
resource. Applied just those four resources (real generation, -target to
avoid touching the fake SSH-only host resources) and re-planned: "No
changes" - confirming the keepers pinning holds. Adding an explicit
postgres_password override afterward left the random_password resource
itself completely untouched (0 replace/destroy), confirming the override
path never disturbs the generated resource.
The Swarm/Forgejo standup (a parallel infra track, see INFRA.md) landed real
.forgejo/workflows/{build,pr-build,deploy,deploy-dev}.yml before this PR
merged, making ci.yml a redundant duplicate of their build.yml (same build
gate, same job). Drop it.
Fix the remaining two files to match the real, already-deployed conventions
those files established rather than the guesses this PR shipped with:
runs-on: [self-hosted, thermograph] -> docker (the actual Docker-in-Docker
Swarm-hosted runner label), and appleboy/ssh-action referenced by full GitHub
URL (confirmed not mirrored on this Forgejo instance's default action
registry, per deploy.yml's own header comment) rather than the short form.
actions/checkout needed no change - already confirmed to resolve unchanged
from Forgejo's default mirror.
build-push.yml (image registry push) and ops-cron.yml (backup + IndexNow)
stay: neither duplicates anything in the real mirror, which faithfully
replicates the OLD git-checkout-and-build-in-place deploy model rather than
the registry-based one, and has no scheduled ops jobs at all.
Records the concrete, live state of Track 1 rather than just the setup
procedure: both boxes' roles/IPs, the ssh invocation, where the private key
lives (~/.ssh/thermograph_agent_ed25519, same convention DEPLOY.md already
uses for the CI deploy key), and how to rotate it. Also notes that neither
box has Docker yet and Terraform hasn't been applied to either, which is why
Track 2 is currently paused.
Checks off the two verification-checklist items that are actually confirmed
now (SSH access, password auth dead) rather than leaving them unchecked.
Three workflows for the self-hosted Forgejo instance (Track A chunk 7 of the
infra-design implementation handoff), mirroring/extending the existing
.github/workflows without touching them - GitHub stays the primary repo and
its own build.yml/ci-cd.yml keep gating auto-merge.
ci.yml mirrors build.yml's build gate (deps, backend tests, frontend JS
syntax check, boot + page/API health check) on a Forgejo runner.
build-push.yml builds the app image and pushes it to Forgejo's built-in
registry, tagged by git SHA (every push) and semver (version tags) - the
registry half of the hop-1 cutover's build-once/deploy-everywhere model.
Runs alongside the existing deploy.yml/deploy.sh (git-checkout-and-build-in-
place) without touching it, per the cutover runbook's Stage C.
ops-cron.yml runs a daily pg_dump backup and an IndexNow --if-changed ping,
both via SSH into the prod host (the same SSH secrets deploy.yml already
uses) using the app's already-running compose stack - no new network
exposure, no separate dependency install. These are ops/infra concerns
(scheduled via Forgejo Actions cron), distinct from the app-domain jobs the
worker's own APScheduler runs.
All three need a runner registered with the `thermograph` label and the
registry's public/mesh exposure resolved (Track B).
Verified: raw YAML syntax valid; actionlint clean except the pre-existing,
already-present-on-.github/workflows "unknown custom runner label" warning
(not something these files introduce - the existing thermograph-lan label
triggers the identical warning). One real finding fixed pre-commit: an
unused shellcheck-flagged loop variable in the boot-check retry loop.
* Split web/worker duties with THERMOGRAPH_ROLE
Background work (the subscription notifier) is welded to the same process
that serves requests, so scaling the web tier to N replicas would also scale
notifier instances unless something restricts it further than leader
election alone.
Add THERMOGRAPH_ROLE (web|worker|all, default all - unchanged single-process
behavior). Every replica runs the same image; ROLE only gates whether a
process is allowed to own the notifier at all, layered on top of the
existing leader election: web replicas never start it even if they'd win
leader election, worker replicas start it if they win. The decision is
pulled into _should_run_notifier() so it's unit-testable without booting the
full app (DB init, places index, neighbor warmer).
Add a minimal /healthz liveness route (no DB/upstream I/O, not under BASE)
so a worker replica - which serves no real traffic - still has something
Swarm can health-check.
* Add the Swarm interim stack file, a pinnable TimescaleDB tag, and a Caddy health-gate
Three changes toward the hop-1 interim cutover, all inert until Track B
stands up the platform:
docker-stack.yml: the Swarm stack file for the interim cutover, distinct
from docker-compose.yml (today's plain-compose deploy, unaffected). Pulls a
pre-built image (IMAGE_TAG) instead of building in place; app/worker publish
no host port (127.0.0.1:8137:8137 has no Swarm equivalent - Swarm's routing
mesh publishes on 0.0.0.0, which would expose the plaintext app un-fronted),
reaching Caddy only over an MTU-lowered overlay network (VXLAN-over-WireGuard
needs a smaller MTU or large payloads silently stall); db is placement-
pinned to a labelled node; app/worker skip inline migrations
(RUN_MIGRATIONS=0) so the runbook's one-shot migrate task is the only thing
that ever runs Alembic; secrets are real Swarm secrets mounted at
/run/secrets, read by the entrypoint shim rather than plain env vars.
TIMESCALEDB_TAG: docker-compose.yml's db image now reads this (default
latest-pg18, today's behavior unchanged), wired through Terraform
(timescaledb_tag, default "latest-pg18") so it can actually be pinned to an
exact minor without hand-editing the host - required before any host of the
stack could replicate with another (a floating tag risks mismatched
extension minors, which blocks a physical replica and risks compressed-
chunk corruption on restore).
Caddy active health-gate: both the Terraform-rendered Caddyfile and the live
deploy/Caddyfile now health-check the app on the same cheap /healthz route
its own Docker HEALTHCHECK uses (now /healthz instead of the SSR homepage,
so it's cheap enough for a tight interval and works identically for a
worker replica, which serves no public traffic at all) - Caddy won't
forward into a container that's still booting or unhealthy.
Verified live: built and booted the real image via docker compose - both
containers report healthy via the new /healthz-based HEALTHCHECK, and GET /
still renders the full SSR homepage unchanged. Both Caddyfiles validated
with the real caddy binary. docker-stack.yml validated with docker compose
config (required-var guards fire with clear messages; secrets correctly
mount at /run/secrets/<name>, matching the entrypoint shim's mapping).
docker-compose.yml validated with and without TIMESCALEDB_TAG set, alongside
the existing openmeteo overlay. terraform validate + fmt clean.
Three additive infrastructure layers on top of the two VPS boxes Terraform
already provisions (prod: new 48 GB/12-core box, thermograph.org; beta: old
VPS, 75.119.132.91). None of this touches backend/, Dockerfile,
docker-compose*.yml, terraform/, or deploy/db/ — that stays owned by the
app-containerization work in flight elsewhere; this is strictly the layer on
top. See INFRA.md for the full runbook and order of operations.
- deploy/provision-agent-access.sh: a dedicated, auditable full-sudo login
(not raw root) for agent-driven ops — passwordless sudo under a distinct
username, sshd hardened to key-only auth, auditd logging every
root-effective command. One line to revoke.
- deploy/swarm/: a 2-node Swarm (prod=manager, beta=worker) joined over a
WireGuard tunnel rather than trusting the public internet with the
overlay data plane, which Docker's own guidance says should never face it
directly. Swarm ports locked to the tunnel interface once joined. This
cluster's only workload is Forgejo — it does not orchestrate the
Terraform-managed app deploys, so nothing here can strand the app's
single-writer database.
- deploy/forgejo/ + .forgejo/workflows/: Forgejo + Traefik + a
Docker-in-Docker-sandboxed runner as a Swarm stack pinned to beta, plus
Forgejo Actions workflows mirroring .github/workflows/*.yml. The custom
auto-merge workflow step is dropped — it existed only to work around
GitHub's paywalled branch protection on private free-tier repos, which
Forgejo has no such tier for; native "auto merge when checks succeed"
replaces it, and as a real git push (unlike GitHub's non-triggering
token-merge) it fires the LAN deploy naturally with no double-trigger
logic needed. appleboy/ssh-action is referenced by full URL (not mirrored
on Forgejo's default action registry); actions/checkout and
actions/setup-python resolve unchanged.
Migration is mirror-first: the GitHub repo import and workflow files land
here, but cutting deploy secrets over and retiring GitHub happens only after
verification (INFRA.md 3d) — GitHub stays live as a fallback throughout.
One flagged, unresolved mismatch: deploy.yml still triggers on `main`, but
terraform.tfvars.example names prod's deploy branch `release`. Left as a
faithful mirror rather than guessed at — reconcile with whoever's driving
Terraform/deploy.
The subscription notifier elects one leader via a host-local flock
(THERMOGRAPH_SINGLETON_LOCK) so multiple uvicorn workers on one host don't
each run it. Under multi-host Swarm that guard is insufficient: each host
would independently elect its own leader, multiplying Open-Meteo quota use
N-fold again.
Add claim_pg(key) alongside the existing claim(lock_path): a cluster-wide
Postgres advisory lock, visible to every host talking to the same database.
The holding connection is dedicated and kept for the process lifetime
(advisory locks are session-scoped); a dead connection is dropped and
re-election retried on the next call.
claim_leader() dispatches between the two mechanisms from env:
THERMOGRAPH_SINGLETON_PG (+ Postgres) -> claim_pg; else
THERMOGRAPH_SINGLETON_LOCK -> claim; else always leader, unchanged. Wired in
at web/app.py in place of the direct claim() call. Off by default, so
today's single-host behavior is unaffected.
Replace the per-cell parquet cache with TimescaleDB hypertables as the
production backend for the raw daily climate record, and drop pg_duckdb.
Parquet stays the backend whenever THERMOGRAPH_DATABASE_URL is not a
Postgres URL (dev, tests, offline tooling), the same dialect switch the
accounts DB and derived store already use, so CI stays Postgres-free.
- data/climate_store.py: psycopg + polars bridge over climate_history
(a hypertable), climate_recent, and climate_sync (per-cell freshness).
Reads via pl.read_database, writes via COPY + ON CONFLICT upsert;
fail-soft to a cache miss so a DB hiccup degrades to upstream refetch.
- data/climate.py: route every cache/mtime touchpoint through a backend
dispatch. recent_stamp becomes int(recent_synced_at) on Postgres; the
stale-serve path still avoids bumping it, so derived-payload tokens
invalidate on exactly the same events as before.
- alembic 0002: CREATE EXTENSION timescaledb plus the hypertable schema
(compression policy on year-old chunks), guarded to no-op off Postgres.
- migrate_cache_to_pg.py (make migrate-cache): idempotent backfill of the
parquet cache into the hypertables, preserving file mtimes as sync
timestamps so recent_stamp is unchanged across cutover.
- db image -> stock timescale/timescaledb:latest-pg18; drop the custom
pg_duckdb Dockerfile, the read-only /parquet mount, and the duckdb
tuning GUC. Docs updated for the new backend and cutover.
Co-authored-by: Claude <noreply@anthropic.com>
Two prod-readiness hardening changes:
DB tuning scales with the container budget. Replace the fixed 8 GB
20-tuning.sql with 20-tuning.sh, which derives shared_buffers (25%),
effective_cache_size (75%), work_mem, maintenance_work_mem and
duckdb.max_memory (50%) from the DB_MEMORY the compose db service now passes in.
The ratios reproduce the historical 8 GB tuning exactly and scale linearly, so
the 48 GB prod box (db_memory 16g) gets shared_buffers 4 GB / duckdb 8 GB with
no separate edit. Beta/local (8g default) are unchanged. Docs that told
operators to raise the tuning by hand are updated.
Boot ordering for the self-hosted archive. On an openmeteo host, install a
docker.service drop-in (Wants/After rclone-om.service) so Docker starts after
the object-storage mount is ready on every boot — the restart-policy containers
never bind an empty mount point. rclone-om is Type=notify, so After waits for
the mount to actually be ready. Cleaned up when openmeteo is toggled off.
Get the 45-year historical record off the rate-limited public Open-Meteo
archive API by running a private Open-Meteo instance that serves the
era5_seamless blend (0.1° ERA5-Land + 0.25° ERA5 for gusts) from the
compressed .om archive in object storage, mounted on the host with rclone.
- climate.py: make ARCHIVE_URL env-driven (THERMOGRAPH_ARCHIVE_URL) and pin
models=era5_seamless on the archive fetches only, so a self-hosted instance
serves the same 0.1° resolution; forecast path unchanged. The public API's
default is already seamless, so dev/beta (URL unset) behave identically.
- docker-compose.openmeteo.yml: open-meteo-api + two rolling sync workers
(era5_land 0.1°, era5 0.25° for gusts), bind-mounting the object-storage
mount; the overlay points the app at the local instance.
- Makefile: om-up / om-down / om-backfill (one-time full-history backfill).
- Terraform: per-host openmeteo flag layers the overlay, renders OM_DATA_DIR,
and provisions the host rclone systemd mount from the bucket credentials.
- deploy/openmeteo: operator runbook + rclone mount unit template.
Terraform config under terraform/ manages the two existing VPS hosts and hands the
app to docker-compose, with local state:
- prod: the new 48GB/12-core VPS (release branch, thermograph.org), sized larger.
- beta: the old VPS 75.119.132.91 (main branch, testing tier), no public domain.
- The LAN dev box stays on deploy/deploy-dev.sh (dev branch) — out of Terraform.
A reusable module (modules/thermograph-host) SSHes each host to install docker/
compose/ufw (+ Caddy when a domain is set), sync the checkout to the host's branch,
render /etc/thermograph.env from Terraform variables (secrets pushed via provisioner
content, never on local disk), `docker compose up -d`, and health-check. Named
volumes are preserved on re-apply, so the Postgres data is never recreated.
Container resources are now env-driven in docker-compose.yml (APP_CPUS/DB_CPUS/
DB_MEMORY/WORKERS) with unchanged defaults, so Terraform can size each host.
deploy/db/init/20-tuning.sql sets the memory budget via ALTER SYSTEM (applied on a
fresh volume, effective after the post-init restart): shared_buffers 2GB,
effective_cache_size 6GB, work_mem 64MB, maintenance_work_mem 512MB, max_wal_size
4GB, plus duckdb.max_memory 4GB for parquet processing.
Compose: the db container gets an 8g memory ceiling + 1g shm_size (parallel-query
shared memory) in prod; the dev overlay resets mem_limit so dev memory is uncapped
too (the ~8 GB budget still comes from the tuning). Verified the settings take
effect and pg_duckdb still loads.
Run Thermograph as a docker-compose stack (app + Postgres 18) and standardize the
data layer on Postgres, while keeping the test suite on SQLite.
- accounts/db.py: DSN-driven engines. On Postgres, a per-worker read-write +
read-only asyncpg pair (the RO engine pins read-only transactions, used by the
pure-GET endpoints) plus a sync psycopg engine for the notifier thread; the
SQLite path is preserved for tests/local (selected when THERMOGRAPH_DATABASE_URL
is unset). models.py: boolean server_default -> sa.false().
- store.py / metrics.py: dialect-flexible — Postgres UNLOGGED tables via psycopg
when configured, else the existing raw-sqlite3 paths byte-for-byte; sync
interfaces and every fail-soft contract preserved.
- Alembic (backend/alembic/) manages the accounts schema; the container entrypoint
runs `alembic upgrade head` before uvicorn (4 workers). migrate_accounts_to_pg.py
copies the accounts data SQLite->PG through the ORM (UUID/bool/JSON coerced),
skips access_token, and resets identity sequences.
- Dockerfile + docker-compose.yml: app image (uvicorn, 4 workers, loopback 8137)
and a Postgres 18 db (2 CPUs) running pg_duckdb (deploy/db/) so the parquet
climate cache is queryable in-DB via read_parquet('/parquet/cache/*.parquet').
- deploy.sh/thermograph.service rewired to manage the compose stack; env example,
Makefile targets (up/down/db-up), and deploy/POSTGRES-MIGRATION.md cutover runbook.
Tests stay on SQLite (dialect fallback) — 323 pass. The full Postgres stack was
verified via docker compose: alembic migrations, register/login, the RO endpoint,
store/metrics round-trips, and the accounts data migration.
The deploy scripts pulled code and restarted but never applied the
hand-written SQL migrations in deploy/migrations/, so column additions to
the long-lived accounts DB had to be run by hand.
Add deploy/migrate-db.py, an idempotent runner that resolves the accounts
DB the same way the app does (THERMOGRAPH_ACCOUNTS_DB or
<repo>/data/accounts.sqlite), applies any files not yet recorded in a
schema_migrations table, and:
- backs the DB up before touching it;
- baselines a fresh DB (no file, or no user table) rather than ALTERing
a table create_all() will build with the current schema on next start;
- tolerates an already-present column/index (hand-applied or model-built)
by recording it instead of failing.
Wire it into deploy.sh (prod) and deploy-dev.sh (dev) just before the
service restarts, so new code never queries a column an older DB lacks.
No env or systemd changes: the migration runs as the deploy user, who owns
the data dir.
Claude-Session: https://claude.ai/code/session_01XXxmNFy9cZ6Gh8Y9thZn62
Lets a signed-in user connect their Discord account (OAuth2 identify), storing the
Discord user id that a later feature (DM alerts) will deliver to. Standard
authorization-code flow, all server-side:
- backend/discord_link.py: /discord/link/start redirects to Discord's consent
screen; /discord/link/callback exchanges the code, reads the Discord user id, and
stores it; /discord/unlink forgets it. The `state` is signed with the app auth
secret (stdlib hmac, no new dependency) and carries the Thermograph user id, so a
callback can't be replayed or bound to another account. Every route requires an
active session, so linking acts on whoever is actually logged in.
- models.py: User.discord_id (unique, nullable). schemas.py exposes it on UserRead
so the frontend can show link state.
- app.py: the router under /api/v2/discord.
- account.js: a "Link Discord" / "Unlink Discord" control in the account popover.
- deploy/migrations/001-user-discord-id.sql: the manual column add for the existing
prod accounts DB. This project has no Alembic — create_all only makes missing
tables, so an added column needs a hand-applied migration (documented in-file).
- env example: THERMOGRAPH_DISCORD_CLIENT_SECRET + the redirect to register.
Claude-Session: https://claude.ai/code/session_013dRZmX9D3JEntfMKWMTWZ8
Adds Discord slash commands with no bot process and no gateway connection: Discord
POSTs each interaction to a FastAPI route, and the app answers it. First command is
/grade <city>, returning today's grade for a curated city from the warm cache
(reusing homepage._grade_city, so it answers well within the 3-second deadline and
costs no upstream quota).
- backend/discord_interactions.py: Ed25519 verification (PyNaCl) over the RAW
request body — Discord probes the endpoint with bad signatures and disables it if
they aren't rejected with 401. Routes PING to PONG and application-commands to
their handler; unknown/unsupported interactions are acknowledged, not errored.
City lookup is exact-name-then-prefix over the population-sorted city set; unknown
or not-yet-warm cities get an ephemeral note.
- app.py: POST {BASE}/discord/interactions, reading request.body() (not json()) so
the bytes match the signature.
- scripts/register_discord_commands.py: one-off upsert of the command definitions
via Discord REST (app id + bot token).
- PyNaCl added to requirements; Discord public-key / app-id / bot-token documented
in the env example. Endpoint URL: https://thermograph.org/discord/interactions.
Claude-Session: https://claude.ai/code/session_013dRZmX9D3JEntfMKWMTWZ8
Adds an optional daily Discord post of the day's most anomalous cities, built from
the same data/homepage.json feed the homepage renders — so the post and the site
always agree. Delivery is a single incoming-webhook POST (no bot, no gateway, no
new process): it rides the notifier daemon like the feed refresh does, once per
day after the refresh, leader-only. With no webhook configured it no-ops.
- backend/discord.py: build_embed() renders the top cities as a rich embed (each
line: city, reading in °F and °C, the normal for context, percentile + grade;
non-today readings are dated). post_daily_feed() is best-effort — it skips a
stale/empty feed and never raises, so a Discord failure can't disturb a pass.
- notify.py: a once-a-day guard (_maybe_post_discord) beside the homepage-refresh
guard; only marks the day done once a post lands, so a transient failure retries.
- deploy/thermograph.env.example: THERMOGRAPH_DISCORD_WEBHOOK (the URL is the
credential — env only).
The webhook value is the top anomaly's tail for the accent colour and reuses the
feed's build time as the embed timestamp. httpx (already a dependency) does the POST.
Claude-Session: https://claude.ai/code/session_013dRZmX9D3JEntfMKWMTWZ8
The homepage was a bare tool: a find-bar and an empty panel reading "Find a
location to begin." A visitor arriving from a search result or a shared link
learned nothing about what the product does before deciding to leave.
Rebuild it around the Weekly view, which is untouched:
- Hero with the question as the h1, and a grade card showing a real graded
example — the most unusual city we're currently tracking, either tail. The
card's frame and text slots are server-rendered with reserved heights, so
app.js re-pointing it at the visitor's own place shifts nothing.
- "Unusual right now" strip, CSS scroll-snap, no JS carousel. A cold-tail city
is force-included whenever one qualifies.
- Stance line, how-it-works, explore cards, and 12 city chips linking into the
~1000-page /climate surface.
- Monthly digest form, in the footer of every page.
Serve / from Jinja instead of a static file with placeholder substitution, so
crawlers and no-JS readers get the whole page as real HTML. home.html.j2
extends base.html.j2 and carries app.js's DOM contract over verbatim; the brand
degrades to a <p> so the hero owns the sole h1. frontend/index.html is deleted
rather than left behind the static mount, where it would keep serving indexable
duplicate content.
"Where is it most unusual right now" has no cheap answer at request time —
percentiles live inside zlib-compressed payload blobs with no column to sort on.
So homepage.py sweeps the warm cache and writes data/homepage.json, read by the
template. The sweep is strictly cache-only (climate.load_cached_recent_forecast
is new, the sibling of load_cached_history), so grading ~1000 cities costs zero
upstream requests. It rides the notifier's timer behind an hourly guard rather
than starting a second daemon, and also runs at the tail of warm_cities so a
fresh deploy has a populated feed.
Instrumentation: metrics gains a product-event counter keyed by (event,
referrer domain, UTC day), behind an allowlist and a per-IP rate limit, fed by
POST /api/v2/event. The referrer is taken from the request's own header, never
from the client. The beacon gets its own inbound category that record_inbound
ignores, so reporting an interaction doesn't also count as traffic. The
dashboard grows an events block with per-referrer attribution.
Email is scaffolded but sends nothing yet. mailer.py talks stdlib smtplib to a
local Postfix null client on 127.0.0.1:25 (deploy/provision-mail.sh), so the
choice between direct-to-MX and relaying through a provider stays a Postfix
config change with no code change. The backend defaults to "console", which
logs and sends nothing, so dev and tests exercise the whole signup path safely.
Signups land in pending_digest unconfirmed; collecting the list shouldn't wait
on delivery.
Also adds /privacy, linked from the footer and kept out of the sitemap.
The strip's classes are named unusual-* rather than record-*: .record-card is
already the SEO records page's, and reusing it leaked layout rules onto
/climate/<slug>/records.
* Serve the app on thermograph.org at root; redirect emigriffith.dev/thermograph
Move Thermograph to its own domain. thermograph.org serves the app at its root,
and emigriffith.dev/thermograph* permanently redirects there (prefix stripped,
so deep links map straight across).
- app.py: BASE now supports an empty root prefix. THERMOGRAPH_BASE=/ (or empty)
yields BASE="" so routes sit at "/", "/calendar", "/api/v2/…" with no double
slashes; the bare-base redirect is skipped and the static mount falls back to
"/". Non-empty values keep the existing "/thermograph" sub-path behavior
unchanged (backward compatible).
- deploy/Caddyfile: add a thermograph.org site block reverse-proxying to the
uvicorn on 127.0.0.1:8137; turn emigriffith.dev's /thermograph handler into a
permanent redirect to thermograph.org.
- deploy/thermograph.env.example: default THERMOGRAPH_BASE=/.
- DEPLOY.md: document the two-domain layout and that deploy.sh does not ship the
Caddyfile or env — those are applied on the VPS by hand.
The frontend already uses base-relative URLs, so it follows the root base with no
changes. Verified both modes locally (root serves /, /calendar, /api/v2/*; the
/thermograph sub-path still redirects bare→slash and 404s at root).
* Add dashboard (#144)
* Auto-submit IndexNow on deploy when the URL set changes (#130)
Hook the production deploy (deploy/deploy.sh) to ping IndexNow after the health
check passes, so new/changed pages reach Bing/DuckDuckGo/Yandex without a manual
run. Best-effort — wrapped so it can never fail a deploy — and it sources
/etc/thermograph.env so its key matches the one the running service serves.
To avoid re-blasting ~14k URLs on every code push, indexnow.py gains a
--if-changed mode gated by a signature of the URL set (persisted to
data/indexnow_state.txt): code-only deploys skip; a deploy that adds or removes a
city submits. `make indexnow` still forces a full submit.
* Add ops metrics endpoint + SSH/Termux dashboard (#131)
Adds observability for the running app, viewable over SSH (e.g. Termux):
- backend/metrics.py: thread-safe, best-effort in-process counters for inbound
requests (per category) and outbound calls (per external source), plus a
phase->source map and snapshot(). Hooked into climate._request (outbound, all
six weather/geocode sources) and the existing revalidate_static middleware
(inbound). Never raises into the request path.
- GET /api/v2/metrics: token-gated JSON of the counters + warm-queue depth +
thread names/uptime/pid. Returns 404 unless a valid THERMOGRAPH_METRICS_TOKEN
is presented or the request is direct loopback with no proxy X-Forwarded-*
header, so ops data is never exposed through Caddy on the public domain.
- scripts/dashboard.py (+ scripts/dash, make dashboard): stdlib-only terminal
dashboard. Reads cache (parquet + SQLite), accounts (users + active
subscriptions), and system resources (/proc + systemd) directly, and pulls
traffic from the metrics endpoint over loopback. Live refresh, --once, --json;
mobile-terminal width. Paths resolve relative to the script so the same tool
works on the LAN dev and prod VPS checkouts.
- Monitoring sections in DEPLOY.md / DEPLOY-DEV.md; unit tests for the counters.
* Dashboard: run under the app venv, not the box's default python3 (#132)
The box's default python3 can be a build without _sqlite3 (pyenv 3.10 here), so
./scripts/dash crashed on import. Prefer the repo's .venv interpreter (the Python
the server itself runs on, which always has sqlite3); fall back to python3/python.
Also make the sqlite3 import in dashboard.py optional so a fallback interpreter
degrades to empty cache/accounts panels instead of crashing, and point the
'make dashboard' target at the wrapper.
* Dashboard: redraw live view in place so it stops scrolling to the bottom (#133)
The live loop did a full clear-and-reprint each refresh; when the output is taller
than the terminal (common on a phone) that scrolls the view to the bottom every tick.
Redraw from the top instead: hide the cursor, clear once, then each frame move to home
and rewrite line-by-line (clear-to-EOL + clear-below), capping the body to the window
height so it never prints more lines than fit and never scrolls. Restore the cursor on
exit; piped (non-TTY) output prints plain frames.
* Dashboard: make the live view a scrollable in-place pager (#134)
The live mode is now an alt-screen pager (like top/less): it shows only the latest
snapshot, you scroll it with swipe/arrows/jk/PgUp-PgDn/g/G, and each refresh replaces
the page in place while keeping your scroll position — nothing gets appended below and
the view is never yanked to the top or bottom. On exit it restores the cursor, leaves
the alternate screen, and resets the terminal mode.
--once still prints a single static snapshot (also used automatically when stdout/stdin
aren't a terminal, e.g. piping); --json unchanged.
* Metrics: don't count the dashboard's own /api/v2/metrics polling (#135)
The ops dashboard polls the metrics endpoint every few seconds; counting those hits
just shows the monitor watching itself and inflates the inbound totals. Skip the
'metrics' category in record_inbound (so it's excluded from inbound_total too), and
hide it in the dashboard's inbound list defensively.
* Dashboard: add a STORAGE section (archive + DB sizes) (#136)
Show on-disk storage: the parquet archives (with history/recent split), the derived
SQLite DB, and the accounts SQLite DB individually, plus a total. Archive bytes are
summed during the existing TTL-cached cache scan; DB sizes include the -wal/-shm
sidecars (WAL holds recent writes, so the base file alone understates the footprint).
Drop the now-redundant 'payload db' line from CACHE.
* Dashboard: show STORAGE sizes in fixed GB (0.01 GB floor) (#137)
The auto-scaling formatter only switched to GB once a value passed 1 GB, so the
archives read as MB and the DBs as KB. Format the STORAGE section in fixed GB with two
decimals and a 0.01 GB floor (so the small databases stay visible instead of rounding
to 0.00). Memory readouts keep the auto-scaling formatter.
* Dashboard: show STORAGE under 50 MB in MB, GB above (#138)
Values below 50 MB now render as MB with one decimal (no KB), so the databases read
naturally (derived 7.7MB, accounts 0.4MB) instead of 0.01GB; 50 MB and up stay in GB
with two decimals. Renames the helper _gb -> _storage_size to match.
* Dashboard: move STORAGE MB->GB cutoff to 500 MB (#139)
Values below 500 MB show in MB (one decimal); 500 MB and up in GB.
* Dashboard: accounts totals only; log client IPs per request (#140)
Dashboard ACCOUNTS section now shows only totals (users, active/total subscriptions,
notifications, push subs) — the per-user and per-subscription lists are gone, and the
dashboard no longer queries that PII at all.
Backend: add a per-request access log (logs/access/access-<date>.jsonl) recording the
client IP (real IP via the left-most X-Forwarded-For hop when proxied behind Caddy,
else the peer address), method, path, status, and category. Static assets and the
dashboard's own metrics polling are skipped. Best-effort, never raises into the request
path — retained for later traffic/IP analysis.
* Dashboard: wrap-aware pager so the header stops scrolling off (#141)
The pager counted one screen row per logical line, but on a narrow phone many lines
wrap to two rows — so a page printed more physical rows than the terminal had, scrolling
the top (title + CACHE header) off-screen with no way to scroll back to it. Measure each
line's wrapped height and fit a page to the real physical-row budget; page/End step by
what's actually visible; read the true tty size (not shutil, which honors stale
COLUMNS/LINES). Status line is clipped to one row.
* Add log storage size to the ops dashboard STORAGE section (#142)
The STORAGE section reported parquet archives and both SQLite DBs but
omitted the logs/ footprint, so the "total" understated real disk use.
Add a recursive _tree_size helper and a "logs" row measuring the whole
logs/ tree (audit + errors + access JSONL plus stray *.log files), broken
out by stream, and fold it into the STORAGE total. Flows through --json
automatically via read_storage.
* Dashboard: show last-10-minute traffic per category (#143)
Add a rolling short-window view alongside the since-start totals. metrics.py
keeps per-minute buckets (epoch-minute -> {key: count}) for inbound categories
and outbound sources, summed over the last 10 minutes on snapshot() and exposed
as inbound_recent / outbound_recent / window_minutes. Cheap and bounded — at most
WINDOW+1 tiny dicts, pruned as minutes roll off; self-polling still excluded.
The dashboard's TRAFFIC section now renders that count as a dim column next to
each row's total (header reads "total · last 10m"); idle sources show "-".
* Add dashboard #2 (#145)
* Auto-submit IndexNow on deploy when the URL set changes (#130)
Hook the production deploy (deploy/deploy.sh) to ping IndexNow after the health
check passes, so new/changed pages reach Bing/DuckDuckGo/Yandex without a manual
run. Best-effort — wrapped so it can never fail a deploy — and it sources
/etc/thermograph.env so its key matches the one the running service serves.
To avoid re-blasting ~14k URLs on every code push, indexnow.py gains a
--if-changed mode gated by a signature of the URL set (persisted to
data/indexnow_state.txt): code-only deploys skip; a deploy that adds or removes a
city submits. `make indexnow` still forces a full submit.
* Add ops metrics endpoint + SSH/Termux dashboard (#131)
Adds observability for the running app, viewable over SSH (e.g. Termux):
- backend/metrics.py: thread-safe, best-effort in-process counters for inbound
requests (per category) and outbound calls (per external source), plus a
phase->source map and snapshot(). Hooked into climate._request (outbound, all
six weather/geocode sources) and the existing revalidate_static middleware
(inbound). Never raises into the request path.
- GET /api/v2/metrics: token-gated JSON of the counters + warm-queue depth +
thread names/uptime/pid. Returns 404 unless a valid THERMOGRAPH_METRICS_TOKEN
is presented or the request is direct loopback with no proxy X-Forwarded-*
header, so ops data is never exposed through Caddy on the public domain.
- scripts/dashboard.py (+ scripts/dash, make dashboard): stdlib-only terminal
dashboard. Reads cache (parquet + SQLite), accounts (users + active
subscriptions), and system resources (/proc + systemd) directly, and pulls
traffic from the metrics endpoint over loopback. Live refresh, --once, --json;
mobile-terminal width. Paths resolve relative to the script so the same tool
works on the LAN dev and prod VPS checkouts.
- Monitoring sections in DEPLOY.md / DEPLOY-DEV.md; unit tests for the counters.
* Dashboard: run under the app venv, not the box's default python3 (#132)
The box's default python3 can be a build without _sqlite3 (pyenv 3.10 here), so
./scripts/dash crashed on import. Prefer the repo's .venv interpreter (the Python
the server itself runs on, which always has sqlite3); fall back to python3/python.
Also make the sqlite3 import in dashboard.py optional so a fallback interpreter
degrades to empty cache/accounts panels instead of crashing, and point the
'make dashboard' target at the wrapper.
* Dashboard: redraw live view in place so it stops scrolling to the bottom (#133)
The live loop did a full clear-and-reprint each refresh; when the output is taller
than the terminal (common on a phone) that scrolls the view to the bottom every tick.
Redraw from the top instead: hide the cursor, clear once, then each frame move to home
and rewrite line-by-line (clear-to-EOL + clear-below), capping the body to the window
height so it never prints more lines than fit and never scrolls. Restore the cursor on
exit; piped (non-TTY) output prints plain frames.
* Dashboard: make the live view a scrollable in-place pager (#134)
The live mode is now an alt-screen pager (like top/less): it shows only the latest
snapshot, you scroll it with swipe/arrows/jk/PgUp-PgDn/g/G, and each refresh replaces
the page in place while keeping your scroll position — nothing gets appended below and
the view is never yanked to the top or bottom. On exit it restores the cursor, leaves
the alternate screen, and resets the terminal mode.
--once still prints a single static snapshot (also used automatically when stdout/stdin
aren't a terminal, e.g. piping); --json unchanged.
* Metrics: don't count the dashboard's own /api/v2/metrics polling (#135)
The ops dashboard polls the metrics endpoint every few seconds; counting those hits
just shows the monitor watching itself and inflates the inbound totals. Skip the
'metrics' category in record_inbound (so it's excluded from inbound_total too), and
hide it in the dashboard's inbound list defensively.
* Dashboard: add a STORAGE section (archive + DB sizes) (#136)
Show on-disk storage: the parquet archives (with history/recent split), the derived
SQLite DB, and the accounts SQLite DB individually, plus a total. Archive bytes are
summed during the existing TTL-cached cache scan; DB sizes include the -wal/-shm
sidecars (WAL holds recent writes, so the base file alone understates the footprint).
Drop the now-redundant 'payload db' line from CACHE.
* Dashboard: show STORAGE sizes in fixed GB (0.01 GB floor) (#137)
The auto-scaling formatter only switched to GB once a value passed 1 GB, so the
archives read as MB and the DBs as KB. Format the STORAGE section in fixed GB with two
decimals and a 0.01 GB floor (so the small databases stay visible instead of rounding
to 0.00). Memory readouts keep the auto-scaling formatter.
* Dashboard: show STORAGE under 50 MB in MB, GB above (#138)
Values below 50 MB now render as MB with one decimal (no KB), so the databases read
naturally (derived 7.7MB, accounts 0.4MB) instead of 0.01GB; 50 MB and up stay in GB
with two decimals. Renames the helper _gb -> _storage_size to match.
* Dashboard: move STORAGE MB->GB cutoff to 500 MB (#139)
Values below 500 MB show in MB (one decimal); 500 MB and up in GB.
* Dashboard: accounts totals only; log client IPs per request (#140)
Dashboard ACCOUNTS section now shows only totals (users, active/total subscriptions,
notifications, push subs) — the per-user and per-subscription lists are gone, and the
dashboard no longer queries that PII at all.
Backend: add a per-request access log (logs/access/access-<date>.jsonl) recording the
client IP (real IP via the left-most X-Forwarded-For hop when proxied behind Caddy,
else the peer address), method, path, status, and category. Static assets and the
dashboard's own metrics polling are skipped. Best-effort, never raises into the request
path — retained for later traffic/IP analysis.
* Dashboard: wrap-aware pager so the header stops scrolling off (#141)
The pager counted one screen row per logical line, but on a narrow phone many lines
wrap to two rows — so a page printed more physical rows than the terminal had, scrolling
the top (title + CACHE header) off-screen with no way to scroll back to it. Measure each
line's wrapped height and fit a page to the real physical-row budget; page/End step by
what's actually visible; read the true tty size (not shutil, which honors stale
COLUMNS/LINES). Status line is clipped to one row.
* Add log storage size to the ops dashboard STORAGE section (#142)
The STORAGE section reported parquet archives and both SQLite DBs but
omitted the logs/ footprint, so the "total" understated real disk use.
Add a recursive _tree_size helper and a "logs" row measuring the whole
logs/ tree (audit + errors + access JSONL plus stray *.log files), broken
out by stream, and fold it into the STORAGE total. Flows through --json
automatically via read_storage.
* Dashboard: show last-10-minute traffic per category (#143)
Add a rolling short-window view alongside the since-start totals. metrics.py
keeps per-minute buckets (epoch-minute -> {key: count}) for inbound categories
and outbound sources, summed over the last 10 minutes on snapshot() and exposed
as inbound_recent / outbound_recent / window_minutes. Cheap and bounded — at most
WINDOW+1 tiny dicts, pruned as minutes roll off; self-polling still excluded.
The dashboard's TRAFFIC section now renders that count as a dim column next to
each row's total (header reads "total · last 10m"); idle sources show "-".
* Run prod on 3 uvicorn workers with a shared metrics store
A single slow upstream weather fetch (cache-miss during an open-meteo/NASA
degradation) blocked the one uvicorn worker and took the whole app down for
~2 min — real users got aborted connections. Give prod concurrency headroom so
one slow request can't freeze the rest.
- deploy/thermograph.service: worker count is env-driven (`--workers ${WORKERS}`,
default 1); prod sets WORKERS=3 in /etc/thermograph.env. Dev's separate
--user unit is untouched (stays single-worker).
- metrics.py: with multiple workers, per-process counters would split the tally
and the ops dashboard (polls one random worker) would see only a fraction.
Add a shared SQLite store selected by THERMOGRAPH_METRICS_DB so every worker
tallies into one place; unset keeps the zero-dependency in-memory store for
dev/tests/single-worker. Public API and snapshot shape unchanged.
- The unit defaults THERMOGRAPH_METRICS_DB and clears it on each (re)start via
ExecStartPre, so bumping WORKERS can never silently fragment the dashboard and
"since start" tallies keep their old meaning.
- Tests: cover the SQLite store — cross-worker aggregation, window roll-off,
and env-based selection. Full suite: 178 passed.
Note: applying to prod also needs the unit reinstalled + WORKERS=3 in
/etc/thermograph.env — deploy.sh only pulls+restarts, it doesn't reinstall the
systemd unit.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016mRqdHWRdUjEvEdfwig3wg
---------
Co-authored-by: root <root@vmi3417050.contaboserver.net>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Prod runs 3 uvicorn workers, but the in-process subscription notifier was
started in every worker's lifespan (it was written assuming a single-worker
deploy). Its sweep fetches recent-forecast/archive data from Open-Meteo on a
15-min timer independent of any request, so running it 3× tripled the
background upstream load against a shared quota — the source of the overnight
429/503 rate-limit errors in logs/errors, and 3× redundant subscription scans.
Elect one worker to own the notifier via a non-blocking exclusive flock on a
lockfile (THERMOGRAPH_SINGLETON_LOCK): the first worker wins and holds the fd
for its lifetime; others stand down; the OS releases the lock if the leader
dies, so a restart re-elects cleanly. Unset (single worker / dev / tests)
always wins, so behavior there is unchanged. Mirrors the shared-metrics store's
env-selected cross-worker coordination.
The neighbor warmer stays per-worker on purpose: each worker drains its own
request-fed queue, so gating it would leave non-leader queues undrained.
- backend/singleton.py: flock-based leader election (claim()).
- backend/app.py: gate notify.start() on singleton.claim().
- deploy/thermograph.service: default THERMOGRAPH_SINGLETON_LOCK to
data/notifier.lock (needs the unit reinstalled on prod to take effect).
- deploy/thermograph.env.example: document it alongside WORKERS.
- Tests: leader election — single-worker default, first-wins, idempotent
re-claim, second-holder stands down, re-election after release. 183 passed.
Co-authored-by: root <root@vmi3417050.contaboserver.net>
* Serve the app on thermograph.org at root; redirect emigriffith.dev/thermograph
Move Thermograph to its own domain. thermograph.org serves the app at its root,
and emigriffith.dev/thermograph* permanently redirects there (prefix stripped,
so deep links map straight across).
- app.py: BASE now supports an empty root prefix. THERMOGRAPH_BASE=/ (or empty)
yields BASE="" so routes sit at "/", "/calendar", "/api/v2/…" with no double
slashes; the bare-base redirect is skipped and the static mount falls back to
"/". Non-empty values keep the existing "/thermograph" sub-path behavior
unchanged (backward compatible).
- deploy/Caddyfile: add a thermograph.org site block reverse-proxying to the
uvicorn on 127.0.0.1:8137; turn emigriffith.dev's /thermograph handler into a
permanent redirect to thermograph.org.
- deploy/thermograph.env.example: default THERMOGRAPH_BASE=/.
- DEPLOY.md: document the two-domain layout and that deploy.sh does not ship the
Caddyfile or env — those are applied on the VPS by hand.
The frontend already uses base-relative URLs, so it follows the root base with no
changes. Verified both modes locally (root serves /, /calendar, /api/v2/*; the
/thermograph sub-path still redirects bare→slash and 404s at root).
* Add dashboard (#144)
* Auto-submit IndexNow on deploy when the URL set changes (#130)
Hook the production deploy (deploy/deploy.sh) to ping IndexNow after the health
check passes, so new/changed pages reach Bing/DuckDuckGo/Yandex without a manual
run. Best-effort — wrapped so it can never fail a deploy — and it sources
/etc/thermograph.env so its key matches the one the running service serves.
To avoid re-blasting ~14k URLs on every code push, indexnow.py gains a
--if-changed mode gated by a signature of the URL set (persisted to
data/indexnow_state.txt): code-only deploys skip; a deploy that adds or removes a
city submits. `make indexnow` still forces a full submit.
* Add ops metrics endpoint + SSH/Termux dashboard (#131)
Adds observability for the running app, viewable over SSH (e.g. Termux):
- backend/metrics.py: thread-safe, best-effort in-process counters for inbound
requests (per category) and outbound calls (per external source), plus a
phase->source map and snapshot(). Hooked into climate._request (outbound, all
six weather/geocode sources) and the existing revalidate_static middleware
(inbound). Never raises into the request path.
- GET /api/v2/metrics: token-gated JSON of the counters + warm-queue depth +
thread names/uptime/pid. Returns 404 unless a valid THERMOGRAPH_METRICS_TOKEN
is presented or the request is direct loopback with no proxy X-Forwarded-*
header, so ops data is never exposed through Caddy on the public domain.
- scripts/dashboard.py (+ scripts/dash, make dashboard): stdlib-only terminal
dashboard. Reads cache (parquet + SQLite), accounts (users + active
subscriptions), and system resources (/proc + systemd) directly, and pulls
traffic from the metrics endpoint over loopback. Live refresh, --once, --json;
mobile-terminal width. Paths resolve relative to the script so the same tool
works on the LAN dev and prod VPS checkouts.
- Monitoring sections in DEPLOY.md / DEPLOY-DEV.md; unit tests for the counters.
* Dashboard: run under the app venv, not the box's default python3 (#132)
The box's default python3 can be a build without _sqlite3 (pyenv 3.10 here), so
./scripts/dash crashed on import. Prefer the repo's .venv interpreter (the Python
the server itself runs on, which always has sqlite3); fall back to python3/python.
Also make the sqlite3 import in dashboard.py optional so a fallback interpreter
degrades to empty cache/accounts panels instead of crashing, and point the
'make dashboard' target at the wrapper.
* Dashboard: redraw live view in place so it stops scrolling to the bottom (#133)
The live loop did a full clear-and-reprint each refresh; when the output is taller
than the terminal (common on a phone) that scrolls the view to the bottom every tick.
Redraw from the top instead: hide the cursor, clear once, then each frame move to home
and rewrite line-by-line (clear-to-EOL + clear-below), capping the body to the window
height so it never prints more lines than fit and never scrolls. Restore the cursor on
exit; piped (non-TTY) output prints plain frames.
* Dashboard: make the live view a scrollable in-place pager (#134)
The live mode is now an alt-screen pager (like top/less): it shows only the latest
snapshot, you scroll it with swipe/arrows/jk/PgUp-PgDn/g/G, and each refresh replaces
the page in place while keeping your scroll position — nothing gets appended below and
the view is never yanked to the top or bottom. On exit it restores the cursor, leaves
the alternate screen, and resets the terminal mode.
--once still prints a single static snapshot (also used automatically when stdout/stdin
aren't a terminal, e.g. piping); --json unchanged.
* Metrics: don't count the dashboard's own /api/v2/metrics polling (#135)
The ops dashboard polls the metrics endpoint every few seconds; counting those hits
just shows the monitor watching itself and inflates the inbound totals. Skip the
'metrics' category in record_inbound (so it's excluded from inbound_total too), and
hide it in the dashboard's inbound list defensively.
* Dashboard: add a STORAGE section (archive + DB sizes) (#136)
Show on-disk storage: the parquet archives (with history/recent split), the derived
SQLite DB, and the accounts SQLite DB individually, plus a total. Archive bytes are
summed during the existing TTL-cached cache scan; DB sizes include the -wal/-shm
sidecars (WAL holds recent writes, so the base file alone understates the footprint).
Drop the now-redundant 'payload db' line from CACHE.
* Dashboard: show STORAGE sizes in fixed GB (0.01 GB floor) (#137)
The auto-scaling formatter only switched to GB once a value passed 1 GB, so the
archives read as MB and the DBs as KB. Format the STORAGE section in fixed GB with two
decimals and a 0.01 GB floor (so the small databases stay visible instead of rounding
to 0.00). Memory readouts keep the auto-scaling formatter.
* Dashboard: show STORAGE under 50 MB in MB, GB above (#138)
Values below 50 MB now render as MB with one decimal (no KB), so the databases read
naturally (derived 7.7MB, accounts 0.4MB) instead of 0.01GB; 50 MB and up stay in GB
with two decimals. Renames the helper _gb -> _storage_size to match.
* Dashboard: move STORAGE MB->GB cutoff to 500 MB (#139)
Values below 500 MB show in MB (one decimal); 500 MB and up in GB.
* Dashboard: accounts totals only; log client IPs per request (#140)
Dashboard ACCOUNTS section now shows only totals (users, active/total subscriptions,
notifications, push subs) — the per-user and per-subscription lists are gone, and the
dashboard no longer queries that PII at all.
Backend: add a per-request access log (logs/access/access-<date>.jsonl) recording the
client IP (real IP via the left-most X-Forwarded-For hop when proxied behind Caddy,
else the peer address), method, path, status, and category. Static assets and the
dashboard's own metrics polling are skipped. Best-effort, never raises into the request
path — retained for later traffic/IP analysis.
* Dashboard: wrap-aware pager so the header stops scrolling off (#141)
The pager counted one screen row per logical line, but on a narrow phone many lines
wrap to two rows — so a page printed more physical rows than the terminal had, scrolling
the top (title + CACHE header) off-screen with no way to scroll back to it. Measure each
line's wrapped height and fit a page to the real physical-row budget; page/End step by
what's actually visible; read the true tty size (not shutil, which honors stale
COLUMNS/LINES). Status line is clipped to one row.
* Add log storage size to the ops dashboard STORAGE section (#142)
The STORAGE section reported parquet archives and both SQLite DBs but
omitted the logs/ footprint, so the "total" understated real disk use.
Add a recursive _tree_size helper and a "logs" row measuring the whole
logs/ tree (audit + errors + access JSONL plus stray *.log files), broken
out by stream, and fold it into the STORAGE total. Flows through --json
automatically via read_storage.
* Dashboard: show last-10-minute traffic per category (#143)
Add a rolling short-window view alongside the since-start totals. metrics.py
keeps per-minute buckets (epoch-minute -> {key: count}) for inbound categories
and outbound sources, summed over the last 10 minutes on snapshot() and exposed
as inbound_recent / outbound_recent / window_minutes. Cheap and bounded — at most
WINDOW+1 tiny dicts, pruned as minutes roll off; self-polling still excluded.
The dashboard's TRAFFIC section now renders that count as a dim column next to
each row's total (header reads "total · last 10m"); idle sources show "-".
* Add dashboard #2 (#145)
* Auto-submit IndexNow on deploy when the URL set changes (#130)
Hook the production deploy (deploy/deploy.sh) to ping IndexNow after the health
check passes, so new/changed pages reach Bing/DuckDuckGo/Yandex without a manual
run. Best-effort — wrapped so it can never fail a deploy — and it sources
/etc/thermograph.env so its key matches the one the running service serves.
To avoid re-blasting ~14k URLs on every code push, indexnow.py gains a
--if-changed mode gated by a signature of the URL set (persisted to
data/indexnow_state.txt): code-only deploys skip; a deploy that adds or removes a
city submits. `make indexnow` still forces a full submit.
* Add ops metrics endpoint + SSH/Termux dashboard (#131)
Adds observability for the running app, viewable over SSH (e.g. Termux):
- backend/metrics.py: thread-safe, best-effort in-process counters for inbound
requests (per category) and outbound calls (per external source), plus a
phase->source map and snapshot(). Hooked into climate._request (outbound, all
six weather/geocode sources) and the existing revalidate_static middleware
(inbound). Never raises into the request path.
- GET /api/v2/metrics: token-gated JSON of the counters + warm-queue depth +
thread names/uptime/pid. Returns 404 unless a valid THERMOGRAPH_METRICS_TOKEN
is presented or the request is direct loopback with no proxy X-Forwarded-*
header, so ops data is never exposed through Caddy on the public domain.
- scripts/dashboard.py (+ scripts/dash, make dashboard): stdlib-only terminal
dashboard. Reads cache (parquet + SQLite), accounts (users + active
subscriptions), and system resources (/proc + systemd) directly, and pulls
traffic from the metrics endpoint over loopback. Live refresh, --once, --json;
mobile-terminal width. Paths resolve relative to the script so the same tool
works on the LAN dev and prod VPS checkouts.
- Monitoring sections in DEPLOY.md / DEPLOY-DEV.md; unit tests for the counters.
* Dashboard: run under the app venv, not the box's default python3 (#132)
The box's default python3 can be a build without _sqlite3 (pyenv 3.10 here), so
./scripts/dash crashed on import. Prefer the repo's .venv interpreter (the Python
the server itself runs on, which always has sqlite3); fall back to python3/python.
Also make the sqlite3 import in dashboard.py optional so a fallback interpreter
degrades to empty cache/accounts panels instead of crashing, and point the
'make dashboard' target at the wrapper.
* Dashboard: redraw live view in place so it stops scrolling to the bottom (#133)
The live loop did a full clear-and-reprint each refresh; when the output is taller
than the terminal (common on a phone) that scrolls the view to the bottom every tick.
Redraw from the top instead: hide the cursor, clear once, then each frame move to home
and rewrite line-by-line (clear-to-EOL + clear-below), capping the body to the window
height so it never prints more lines than fit and never scrolls. Restore the cursor on
exit; piped (non-TTY) output prints plain frames.
* Dashboard: make the live view a scrollable in-place pager (#134)
The live mode is now an alt-screen pager (like top/less): it shows only the latest
snapshot, you scroll it with swipe/arrows/jk/PgUp-PgDn/g/G, and each refresh replaces
the page in place while keeping your scroll position — nothing gets appended below and
the view is never yanked to the top or bottom. On exit it restores the cursor, leaves
the alternate screen, and resets the terminal mode.
--once still prints a single static snapshot (also used automatically when stdout/stdin
aren't a terminal, e.g. piping); --json unchanged.
* Metrics: don't count the dashboard's own /api/v2/metrics polling (#135)
The ops dashboard polls the metrics endpoint every few seconds; counting those hits
just shows the monitor watching itself and inflates the inbound totals. Skip the
'metrics' category in record_inbound (so it's excluded from inbound_total too), and
hide it in the dashboard's inbound list defensively.
* Dashboard: add a STORAGE section (archive + DB sizes) (#136)
Show on-disk storage: the parquet archives (with history/recent split), the derived
SQLite DB, and the accounts SQLite DB individually, plus a total. Archive bytes are
summed during the existing TTL-cached cache scan; DB sizes include the -wal/-shm
sidecars (WAL holds recent writes, so the base file alone understates the footprint).
Drop the now-redundant 'payload db' line from CACHE.
* Dashboard: show STORAGE sizes in fixed GB (0.01 GB floor) (#137)
The auto-scaling formatter only switched to GB once a value passed 1 GB, so the
archives read as MB and the DBs as KB. Format the STORAGE section in fixed GB with two
decimals and a 0.01 GB floor (so the small databases stay visible instead of rounding
to 0.00). Memory readouts keep the auto-scaling formatter.
* Dashboard: show STORAGE under 50 MB in MB, GB above (#138)
Values below 50 MB now render as MB with one decimal (no KB), so the databases read
naturally (derived 7.7MB, accounts 0.4MB) instead of 0.01GB; 50 MB and up stay in GB
with two decimals. Renames the helper _gb -> _storage_size to match.
* Dashboard: move STORAGE MB->GB cutoff to 500 MB (#139)
Values below 500 MB show in MB (one decimal); 500 MB and up in GB.
* Dashboard: accounts totals only; log client IPs per request (#140)
Dashboard ACCOUNTS section now shows only totals (users, active/total subscriptions,
notifications, push subs) — the per-user and per-subscription lists are gone, and the
dashboard no longer queries that PII at all.
Backend: add a per-request access log (logs/access/access-<date>.jsonl) recording the
client IP (real IP via the left-most X-Forwarded-For hop when proxied behind Caddy,
else the peer address), method, path, status, and category. Static assets and the
dashboard's own metrics polling are skipped. Best-effort, never raises into the request
path — retained for later traffic/IP analysis.
* Dashboard: wrap-aware pager so the header stops scrolling off (#141)
The pager counted one screen row per logical line, but on a narrow phone many lines
wrap to two rows — so a page printed more physical rows than the terminal had, scrolling
the top (title + CACHE header) off-screen with no way to scroll back to it. Measure each
line's wrapped height and fit a page to the real physical-row budget; page/End step by
what's actually visible; read the true tty size (not shutil, which honors stale
COLUMNS/LINES). Status line is clipped to one row.
* Add log storage size to the ops dashboard STORAGE section (#142)
The STORAGE section reported parquet archives and both SQLite DBs but
omitted the logs/ footprint, so the "total" understated real disk use.
Add a recursive _tree_size helper and a "logs" row measuring the whole
logs/ tree (audit + errors + access JSONL plus stray *.log files), broken
out by stream, and fold it into the STORAGE total. Flows through --json
automatically via read_storage.
* Dashboard: show last-10-minute traffic per category (#143)
Add a rolling short-window view alongside the since-start totals. metrics.py
keeps per-minute buckets (epoch-minute -> {key: count}) for inbound categories
and outbound sources, summed over the last 10 minutes on snapshot() and exposed
as inbound_recent / outbound_recent / window_minutes. Cheap and bounded — at most
WINDOW+1 tiny dicts, pruned as minutes roll off; self-polling still excluded.
The dashboard's TRAFFIC section now renders that count as a dim column next to
each row's total (header reads "total · last 10m"); idle sources show "-".
* Run prod on 3 uvicorn workers with a shared metrics store
A single slow upstream weather fetch (cache-miss during an open-meteo/NASA
degradation) blocked the one uvicorn worker and took the whole app down for
~2 min — real users got aborted connections. Give prod concurrency headroom so
one slow request can't freeze the rest.
- deploy/thermograph.service: worker count is env-driven (`--workers ${WORKERS}`,
default 1); prod sets WORKERS=3 in /etc/thermograph.env. Dev's separate
--user unit is untouched (stays single-worker).
- metrics.py: with multiple workers, per-process counters would split the tally
and the ops dashboard (polls one random worker) would see only a fraction.
Add a shared SQLite store selected by THERMOGRAPH_METRICS_DB so every worker
tallies into one place; unset keeps the zero-dependency in-memory store for
dev/tests/single-worker. Public API and snapshot shape unchanged.
- The unit defaults THERMOGRAPH_METRICS_DB and clears it on each (re)start via
ExecStartPre, so bumping WORKERS can never silently fragment the dashboard and
"since start" tallies keep their old meaning.
- Tests: cover the SQLite store — cross-worker aggregation, window roll-off,
and env-based selection. Full suite: 178 passed.
Note: applying to prod also needs the unit reinstalled + WORKERS=3 in
/etc/thermograph.env — deploy.sh only pulls+restarts, it doesn't reinstall the
systemd unit.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016mRqdHWRdUjEvEdfwig3wg
---------
Co-authored-by: root <root@vmi3417050.contaboserver.net>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
A failed web push (e.g. VAPID key mismatch -> 401/403) was swallowed: /push/test still
returned 202 and the UI showed 'Sent', while the only trace was a journald WARNING.
Now: push.send logs failures to the errors JSONL (phase=push) so they show up like other
errors; /push/test returns a 'failed' count; and the 'Send test' button reports 'No
device' / 'Sent' / 'Failed' from the real result. Document the VAPID env vars (missing
from thermograph.env.example) and how to pin/diagnose them.
Hook the production deploy (deploy/deploy.sh) to ping IndexNow after the health
check passes, so new/changed pages reach Bing/DuckDuckGo/Yandex without a manual
run. Best-effort — wrapped so it can never fail a deploy — and it sources
/etc/thermograph.env so its key matches the one the running service serves.
To avoid re-blasting ~14k URLs on every code push, indexnow.py gains a
--if-changed mode gated by a signature of the URL set (persisted to
data/indexnow_state.txt): code-only deploys skip; a deploy that adds or removes a
city submits. `make indexnow` still forces a full submit.
Faster search-engine indexing for the ~14k climate/city/record URLs:
- IndexNow (backend/indexnow.py): instantly notify Bing/DuckDuckGo/Yandex of new
or changed URLs. Per-host key resolved env → gitignored file → generated (mirrors
push.py), served at /{key}.txt, and echoed in submissions. `submit_all()` + a
`make indexnow` CLI push every indexable URL, batched under the 10k cap. Google
doesn't use IndexNow, so it stays on the sitemap.
- Sitemap: replace the per-request today() <lastmod> (which churns every fetch and
trains crawlers to ignore lastmod) with a stable content-build date; factor the
URL list into public_paths() shared with IndexNow so the two never drift.
- Search-console verification: render google-site-verification + msvalidate.01
<meta> tags from env into every page's <head> — the SEO pages via base.html.j2
and the static pages (incl. the homepage Google verifies) via _page().
- Docs: env example documents the three new vars; .gitignore ignores the key.
Tests: key-file route, stable single-date lastmod, IndexNow payload/batching, and
verification meta on both SEO and static pages.
The population-ranked global top-500 skewed to Asian megacities and missed
high-English-search-demand cities. gen_cities.py now tops up with the top ~250
cities from English-speaking countries (US/GB/CA/AU/NZ/IE/ZA) not already in the
global set, so US coverage goes 13->146, GB 2->42, CA 3->29, etc. (Seattle, Boston,
Manchester, Melbourne, Auckland, Dublin, ...). cities.json regenerated to 750.
Both deploy scripts now launch warm_cities.py automatically after the health check,
detached (dev: a systemd --user transient unit; prod: setsid/nohup), so the city
pages serve from cache without a manual step; idempotent, so only the first deploy
does the full warm. DEPLOY.md updated.
Move Thermograph to its own domain. thermograph.org now serves the app at its
root, and emigriffith.dev/thermograph* permanently redirects there (prefix
stripped, so deep links map straight across).
- app.py: BASE now supports an empty root prefix. THERMOGRAPH_BASE=/ (or empty)
yields BASE="" so routes sit at "/", "/calendar", "/api/v2/…" with no double
slashes; the bare-base redirect is skipped and the static mount falls back to
"/". Non-empty values keep the existing "/thermograph" sub-path behavior
unchanged (backward compatible).
- deploy/Caddyfile: add a thermograph.org site block reverse-proxying to the
uvicorn on 127.0.0.1:8137; turn emigriffith.dev's /thermograph handler into a
permanent redirect to thermograph.org (handle_path strips the prefix; the bare
/thermograph goes to the root).
- deploy/thermograph.env.example: default THERMOGRAPH_BASE=/ (app owns the domain).
- DEPLOY.md: document the two-domain layout and that deploy.sh does not ship the
Caddyfile or env — those are applied on the VPS by hand.
The frontend already uses base-relative URLs, so it follows the root base with no
changes. Verified both modes locally (root serves /, /calendar, /api/v2/*; the
/thermograph sub-path still redirects bare→slash and 404s at root) and the full
test suite (123) passes.
Requests to http://<vps-ip>/... were bounced by the default HTTP->HTTPS rule to
https on the bare IP, which has no certificate and fails the TLS handshake, so
pre-domain bookmarks dead-ended with a blank page. Add an http://<ip> site block
that 301-redirects to https://emigriffith.dev, preserving the path.
The handle directive takes a single matcher token; two space-separated paths
(/thermograph /thermograph/*) fail to parse with 'Wrong argument count'. Match
both via a named @thermograph path matcher instead. Verified with caddy validate.
- Drop the app-level "/" redirect so the FastAPI app no longer claims the
domain root; it stays scoped to THERMOGRAPH_BASE (/thermograph). Root now
404s at the app, leaving it for another service behind the proxy.
- Caddyfile: serve a static portfolio at / and reverse-proxy /thermograph* to
uvicorn, on emigriffith.dev with automatic Let's Encrypt TLS.
- Default THERMOGRAPH_BASE to /thermograph in the env example to match. (#11)
* Add dev CI/CD pipeline deploying to a LAN server (#6)
* Weekly page: no-rain visuals adopt the dry-streak look
On the Weekly page, no-rain (0") days now warm with the dry-streak ramp
(tan→red, deepening to a 14-day cap) instead of a flat blue/tan, matching
the Dry chart and calendar's dry-period language:
- Precip trend chart: no-rain day dots colored by drynessColor(dsr).
- "Daily, graded" timeline strip: Rain-row dry cells tinted by dsr via a
new optional color override on rdCell.
Rain days keep their intensity-tier colors; unknown streaks fall back to
the prior flat color, so nothing regresses.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019VP23wKmjS2ozk1g5a9g1Z
* Calendar totals: compact per-category status lines
Replace the stacked share bar + labeled percentage chips above the
calendar grid with a compact deviation strip: one thin status-colored
line per category (height scaled to the tallest non-median category),
with each share printed above it in that category's own status color.
The median tier — the neutral "Normal" center of the diverging
temperature scale — draws no line, just a faint baseline tick, so it
reads as the reference the other categories deviate from. Precip and
dry-streak have no natural median, so every category there keeps a line.
Percentage text is lifted toward the theme text color (color-mix) so
even the darkest/lightest tiers stay legible in both light and dark.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Calendar totals strip: wider bars, 0.1%-precision small shares (#5)
Widen the per-category deviation bars (3px→10px) so each reads
clearly, and print sub-1% shares as e.g. "0.4%" instead of "<1%".
Claude-Session: https://claude.ai/code/session_019VP23wKmjS2ozk1g5a9g1Z
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* Weekly timeline: label dry streak instead of a dot
On the precip row of the recent/forecast timeline, dry days now show the
running dry-streak count ("3d" = 3 days since measurable rain), tinted by
the same dryness ramp as the chart, rather than a bare "·".
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Add dev CI/CD pipeline deploying to a LAN server
PRs into dev run a build + boot/health check, auto-merge on green, and
deploy the merged branch to a self-hosted runner on the LAN box, which
runs the app as a sudo-free systemd --user service on 0.0.0.0:8137.
- ci-cd.yml: build -> auto-merge -> deploy (self-hosted)
- deploy-dev.yml: deploy on direct pushes / manual dispatch
- deploy-dev.sh + thermograph-dev.service + provision-dev-lan.sh
- CLAUDE.md: dev is the PR base branch; commit/PR message conventions
- DEPLOY-DEV.md: pipeline docs
* Match runner service name in docs to the installed user unit
* Pin CI Python to 3.12 for prebuilt pyarrow/pandas wheels
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* Build LAN dev venv on pinned Python 3.12 via uv (#7)
This machine's default python3 is 3.14, which has no prebuilt pyarrow/pandas
wheels, so plain venv + pip fell back to a failing source build. Use uv to pin
the interpreter (fetching a managed CPython 3.12 if needed), independent of the
runner's PATH and pyenv state.
* Add comfort-temperature compare view; simplify Feels calendar filter (#9)
Compare page (frontend/compare.{html,js} + /thermograph/compare route):
line up several places over a date range and rank which best matches a
comfort temperature. Per location it pulls the same daily record the
Calendar uses and, per day, takes a chosen temperature (daytime high /
daily mean / overnight low / feels-like) against the comfort target. A
day "hits comfort" when it lands within an adjustable band; otherwise it
counts as colder or warmer and the average miss is tracked. Results are
ranked by comfort-day share (tie-broken by the smaller typical miss),
with a diverging below/comfort/above bar and per-bucket stats. Comfort,
band and judged temperature re-rank instantly client-side; only the
location set or date range trigger a data load (shared calendar cache).
Feels calendar filter: drop the comfort-temperature slider and the
client-side felt-high/felt-low re-pick. The Feels metric now colors by
the server's combined feels-like value like the other metrics, so its
tab matches the rest of the metric selector.
* Scope app to /thermograph, free the domain root for a portfolio (#10)
- Drop the app-level "/" redirect so the FastAPI app no longer claims the
domain root; it stays scoped to THERMOGRAPH_BASE (/thermograph). Root now
404s at the app, leaving it for another service behind the proxy.
- Caddyfile: serve a static portfolio at / and reverse-proxy /thermograph* to
uvicorn, on emigriffith.dev with automatic Let's Encrypt TLS.
- Default THERMOGRAPH_BASE to /thermograph in the env example to match.
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
