From 1838527854c5115d0755eeaee96faf4276cfc076 Mon Sep 17 00:00:00 2001 From: Emi Griffith Date: Fri, 17 Jul 2026 05:54:55 -0700 Subject: [PATCH] Gate the subscription notifier to one worker (#161) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Prod runs 3 uvicorn workers, but the in-process subscription notifier was started in every worker's lifespan (it was written assuming a single-worker deploy). Its sweep fetches recent-forecast/archive data from Open-Meteo on a 15-min timer independent of any request, so running it 3× tripled the background upstream load against a shared quota — the source of the overnight 429/503 rate-limit errors in logs/errors, and 3× redundant subscription scans. Elect one worker to own the notifier via a non-blocking exclusive flock on a lockfile (THERMOGRAPH_SINGLETON_LOCK): the first worker wins and holds the fd for its lifetime; others stand down; the OS releases the lock if the leader dies, so a restart re-elects cleanly. Unset (single worker / dev / tests) always wins, so behavior there is unchanged. Mirrors the shared-metrics store's env-selected cross-worker coordination. The neighbor warmer stays per-worker on purpose: each worker drains its own request-fed queue, so gating it would leave non-leader queues undrained. - backend/singleton.py: flock-based leader election (claim()). - backend/app.py: gate notify.start() on singleton.claim(). - deploy/thermograph.service: default THERMOGRAPH_SINGLETON_LOCK to data/notifier.lock (needs the unit reinstalled on prod to take effect). - deploy/thermograph.env.example: document it alongside WORKERS. - Tests: leader election — single-worker default, first-wins, idempotent re-claim, second-holder stands down, re-election after release. 183 passed. Co-authored-by: root --- deploy/thermograph.env.example | 4 +++- deploy/thermograph.service | 4 ++++ 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/deploy/thermograph.env.example b/deploy/thermograph.env.example index 2738f9f..832cb9e 100644 --- a/deploy/thermograph.env.example +++ b/deploy/thermograph.env.example @@ -8,7 +8,9 @@ PORT=8137 # (e.g. a cache-miss weather lookup) from blocking every other request — the cause of # past brief outages. Prod runs 3; leave unset (defaults to 1) on a small box. Workers # share one metrics store (THERMOGRAPH_METRICS_DB, defaulted in the systemd unit) so the -# ops dashboard still sees the whole picture. ~200 MB RAM per worker. +# ops dashboard still sees the whole picture, and elect one leader for the subscription +# notifier via a lockfile (THERMOGRAPH_SINGLETON_LOCK, also defaulted in the unit) so its +# timer-driven upstream sweep runs once, not once per worker. ~200 MB RAM per worker. WORKERS=3 # Base path the app is served under. diff --git a/deploy/thermograph.service b/deploy/thermograph.service index e08e8d5..58c1bd4 100644 --- a/deploy/thermograph.service +++ b/deploy/thermograph.service @@ -16,6 +16,10 @@ WorkingDirectory=/opt/thermograph/backend # dashboard, polling one random worker, would see only a fraction of the traffic. Environment=WORKERS=1 Environment=THERMOGRAPH_METRICS_DB=/opt/thermograph/data/metrics.db +# Elect one worker to run the subscription notifier (its timer-driven upstream sweep +# must run once across the deploy, not once per worker — else it multiplies the +# Open-Meteo quota use). Workers race for this lockfile; the winner runs the notifier. +Environment=THERMOGRAPH_SINGLETON_LOCK=/opt/thermograph/data/notifier.lock # THERMOGRAPH_BASE, PORT, WORKERS, etc. See deploy/thermograph.env.example. EnvironmentFile=/etc/thermograph.env # Clear the shared metrics DB on each (re)start so the dashboard's "since start" tallies