1838527854
Prod runs 3 uvicorn workers, but the in-process subscription notifier was started in every worker's lifespan (it was written assuming a single-worker deploy). Its sweep fetches recent-forecast/archive data from Open-Meteo on a 15-min timer independent of any request, so running it 3× tripled the background upstream load against a shared quota — the source of the overnight 429/503 rate-limit errors in logs/errors, and 3× redundant subscription scans. Elect one worker to own the notifier via a non-blocking exclusive flock on a lockfile (THERMOGRAPH_SINGLETON_LOCK): the first worker wins and holds the fd for its lifetime; others stand down; the OS releases the lock if the leader dies, so a restart re-elects cleanly. Unset (single worker / dev / tests) always wins, so behavior there is unchanged. Mirrors the shared-metrics store's env-selected cross-worker coordination. The neighbor warmer stays per-worker on purpose: each worker drains its own request-fed queue, so gating it would leave non-leader queues undrained. - backend/singleton.py: flock-based leader election (claim()). - backend/app.py: gate notify.start() on singleton.claim(). - deploy/thermograph.service: default THERMOGRAPH_SINGLETON_LOCK to data/notifier.lock (needs the unit reinstalled on prod to take effect). - deploy/thermograph.env.example: document it alongside WORKERS. - Tests: leader election — single-worker default, first-wins, idempotent re-claim, second-holder stands down, re-election after release. 183 passed. Co-authored-by: root <root@vmi3417050.contaboserver.net>
41 lines
1.9 KiB
Desktop File
41 lines
1.9 KiB
Desktop File
[Unit]
|
|
Description=Thermograph (FastAPI/uvicorn)
|
|
After=network-online.target
|
|
Wants=network-online.target
|
|
|
|
[Service]
|
|
Type=exec
|
|
User=deploy
|
|
Group=deploy
|
|
# The repo checkout. uvicorn is run from backend/ so `app:app` resolves.
|
|
WorkingDirectory=/opt/thermograph/backend
|
|
# Defaults; /etc/thermograph.env (below) overrides. WORKERS is how many uvicorn
|
|
# worker processes to run — prod sets WORKERS=3 there; a single-worker box can leave
|
|
# it. Multiple workers each keep their own in-process state, so metrics go to a shared
|
|
# SQLite DB (THERMOGRAPH_METRICS_DB) that every worker tallies into — otherwise the ops
|
|
# dashboard, polling one random worker, would see only a fraction of the traffic.
|
|
Environment=WORKERS=1
|
|
Environment=THERMOGRAPH_METRICS_DB=/opt/thermograph/data/metrics.db
|
|
# Elect one worker to run the subscription notifier (its timer-driven upstream sweep
|
|
# must run once across the deploy, not once per worker — else it multiplies the
|
|
# Open-Meteo quota use). Workers race for this lockfile; the winner runs the notifier.
|
|
Environment=THERMOGRAPH_SINGLETON_LOCK=/opt/thermograph/data/notifier.lock
|
|
# THERMOGRAPH_BASE, PORT, WORKERS, etc. See deploy/thermograph.env.example.
|
|
EnvironmentFile=/etc/thermograph.env
|
|
# Clear the shared metrics DB on each (re)start so the dashboard's "since start" tallies
|
|
# reflect the running workers, matching how the old in-process counters behaved.
|
|
ExecStartPre=/usr/bin/rm -f /opt/thermograph/data/metrics.db /opt/thermograph/data/metrics.db-wal /opt/thermograph/data/metrics.db-shm
|
|
# Bind loopback only — Caddy terminates TLS and reverse-proxies to us.
|
|
ExecStart=/opt/thermograph/.venv/bin/uvicorn app:app --host 127.0.0.1 --port ${PORT} --workers ${WORKERS}
|
|
Restart=on-failure
|
|
RestartSec=2
|
|
# Hardening
|
|
NoNewPrivileges=true
|
|
PrivateTmp=true
|
|
ProtectSystem=full
|
|
ProtectHome=read-only
|
|
# The parquet cache must stay writable across deploys.
|
|
ReadWritePaths=/opt/thermograph/data /opt/thermograph/logs
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target
|