Gate the subscription notifier to one worker (#161)
Prod runs 3 uvicorn workers, but the in-process subscription notifier was
started in every worker's lifespan (it was written assuming a single-worker
deploy). Its sweep fetches recent-forecast/archive data from Open-Meteo on a
15-min timer independent of any request, so running it 3× tripled the
background upstream load against a shared quota — the source of the overnight
429/503 rate-limit errors in logs/errors, and 3× redundant subscription scans.
Elect one worker to own the notifier via a non-blocking exclusive flock on a
lockfile (THERMOGRAPH_SINGLETON_LOCK): the first worker wins and holds the fd
for its lifetime; others stand down; the OS releases the lock if the leader
dies, so a restart re-elects cleanly. Unset (single worker / dev / tests)
always wins, so behavior there is unchanged. Mirrors the shared-metrics store's
env-selected cross-worker coordination.
The neighbor warmer stays per-worker on purpose: each worker drains its own
request-fed queue, so gating it would leave non-leader queues undrained.
- backend/singleton.py: flock-based leader election (claim()).
- backend/app.py: gate notify.start() on singleton.claim().
- deploy/thermograph.service: default THERMOGRAPH_SINGLETON_LOCK to
data/notifier.lock (needs the unit reinstalled on prod to take effect).
- deploy/thermograph.env.example: document it alongside WORKERS.
- Tests: leader election — single-worker default, first-wins, idempotent
re-claim, second-holder stands down, re-election after release. 183 passed.
Co-authored-by: root <root@vmi3417050.contaboserver.net>
2026-07-17 12:54:55 +00:00
|
|
|
"""Unit tests for the single-instance leader election (singleton.py)."""
|
|
|
|
|
import fcntl
|
|
|
|
|
import importlib
|
|
|
|
|
import os
|
|
|
|
|
|
Split the backend into domain packages (#217)
* Centralize filesystem paths in a single module
Add paths.py, which resolves the repo root once and derives the cache,
accounts DB, logs, templates, frontend and bundled-city-data locations
from it. Replace the 13 per-module `dirname(__file__)/..` anchors with
references to it, so a module's location no longer determines where the
app reads its data. Env overrides (accounts DB, VAPID, IndexNow) are
unchanged; every resolved path is byte-identical to before.
Groundwork for moving modules into packages without re-pointing paths.
Claude-Session: https://claude.ai/code/session_01XXxmNFy9cZ6Gh8Y9thZn62
* Split the backend into domain packages
Group the flat backend modules into packages that mirror their concerns:
data/ climate, grading, scoring, grid, places, cities,
city_events, store
web/ app, views, homepage, content, schemas
notifications/ notify, digest, push, mailer, discord,
discord_interactions, discord_link
accounts/ models, users, api_accounts, db
core/ metrics, singleton, audit
Intra-project imports are rewritten to the package-qualified form. The
entry scripts (indexnow, warm_cities, migrate, gen_cities, gen_flavor)
and paths.py stay at the backend/ root, and backend/app.py becomes a
shim re-exporting web.app:app so the launch target stays `app:app` —
run.sh, the systemd units, and CI need no change.
Verified: full suite (318) passes, `uvicorn app:app` boots and serves
the home/SEO/static/API surfaces, and every root script imports clean.
Claude-Session: https://claude.ai/code/session_01XXxmNFy9cZ6Gh8Y9thZn62
2026-07-20 05:31:03 +00:00
|
|
|
from core import singleton
|
Gate the subscription notifier to one worker (#161)
Prod runs 3 uvicorn workers, but the in-process subscription notifier was
started in every worker's lifespan (it was written assuming a single-worker
deploy). Its sweep fetches recent-forecast/archive data from Open-Meteo on a
15-min timer independent of any request, so running it 3× tripled the
background upstream load against a shared quota — the source of the overnight
429/503 rate-limit errors in logs/errors, and 3× redundant subscription scans.
Elect one worker to own the notifier via a non-blocking exclusive flock on a
lockfile (THERMOGRAPH_SINGLETON_LOCK): the first worker wins and holds the fd
for its lifetime; others stand down; the OS releases the lock if the leader
dies, so a restart re-elects cleanly. Unset (single worker / dev / tests)
always wins, so behavior there is unchanged. Mirrors the shared-metrics store's
env-selected cross-worker coordination.
The neighbor warmer stays per-worker on purpose: each worker drains its own
request-fed queue, so gating it would leave non-leader queues undrained.
- backend/singleton.py: flock-based leader election (claim()).
- backend/app.py: gate notify.start() on singleton.claim().
- deploy/thermograph.service: default THERMOGRAPH_SINGLETON_LOCK to
data/notifier.lock (needs the unit reinstalled on prod to take effect).
- deploy/thermograph.env.example: document it alongside WORKERS.
- Tests: leader election — single-worker default, first-wins, idempotent
re-claim, second-holder stands down, re-election after release. 183 passed.
Co-authored-by: root <root@vmi3417050.contaboserver.net>
2026-07-17 12:54:55 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
def _fresh():
|
|
|
|
|
# _held is module-level state; reload for an isolated starting point per test.
|
|
|
|
|
return importlib.reload(singleton)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def test_no_lock_path_is_always_leader():
|
|
|
|
|
s = _fresh()
|
|
|
|
|
assert s.claim(None) is True
|
|
|
|
|
assert s.claim("") is True
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def test_first_claimer_wins_and_holds_the_lock(tmp_path):
|
|
|
|
|
s = _fresh()
|
|
|
|
|
lock = str(tmp_path / "notifier.lock")
|
|
|
|
|
assert s.claim(lock) is True
|
|
|
|
|
# Held for the process lifetime: the fd stays open and flock'd.
|
|
|
|
|
assert lock in s._held
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def test_reclaiming_same_path_is_idempotent(tmp_path):
|
|
|
|
|
s = _fresh()
|
|
|
|
|
lock = str(tmp_path / "notifier.lock")
|
|
|
|
|
assert s.claim(lock) is True
|
|
|
|
|
fd = s._held[lock]
|
|
|
|
|
# Re-claiming must not re-open or drop the lock — same fd, still True.
|
|
|
|
|
assert s.claim(lock) is True
|
|
|
|
|
assert s._held[lock] == fd
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def test_second_holder_stands_down(tmp_path):
|
|
|
|
|
"""When another holder already owns the lock, claim() returns False and leaks no
|
|
|
|
|
fd. Simulate the other worker by taking the exclusive flock on a separate open
|
|
|
|
|
file description (a fresh os.open, as a distinct process would)."""
|
|
|
|
|
s = _fresh()
|
|
|
|
|
lock = str(tmp_path / "notifier.lock")
|
|
|
|
|
other = os.open(lock, os.O_RDWR | os.O_CREAT, 0o644)
|
|
|
|
|
fcntl.flock(other, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
|
|
|
|
try:
|
|
|
|
|
assert s.claim(lock) is False
|
|
|
|
|
assert lock not in s._held # did not record a role it doesn't own
|
|
|
|
|
finally:
|
|
|
|
|
fcntl.flock(other, fcntl.LOCK_UN)
|
|
|
|
|
os.close(other)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def test_reelection_after_holder_releases(tmp_path):
|
|
|
|
|
"""A restart drops the OS lock; the next claim() then wins."""
|
|
|
|
|
s = _fresh()
|
|
|
|
|
lock = str(tmp_path / "notifier.lock")
|
|
|
|
|
other = os.open(lock, os.O_RDWR | os.O_CREAT, 0o644)
|
|
|
|
|
fcntl.flock(other, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
|
|
|
|
assert s.claim(lock) is False
|
|
|
|
|
# Prior leader exits -> OS releases the lock.
|
|
|
|
|
fcntl.flock(other, fcntl.LOCK_UN)
|
|
|
|
|
os.close(other)
|
|
|
|
|
assert s.claim(lock) is True
|