3 commits

Author	SHA1	Message	Date
centralis-agent	ba11423fd9	alerting: base web/worker liveness on the app heartbeat, not container stdout ... Three of the prod liveness rules were measuring streams that no longer say anything about the thing they claim to watch. ProdWorkerContainerSilent was unsatisfiable by construction: prod's alloy config drops the thermograph_worker container on purpose, so the stream it queried never reaches Loki and the rule returned zero regardless of worker health. Firing continuously since 2026-07-24T21:05Z. ProdWebContainerSilent measured a floor that stopped existing when the app moved request logging off stdout onto app-json; a web container now emits ~15 uvicorn banner lines for its whole lifetime. It cleared only for the few minutes after each deploy/autoscale event, producing 20+ false pages in a day. NotifierHeartbeatMissing{Prod,Beta} matched a bare tag=heartbeat, which was unambiguous until _start_heartbeat shipped at 2026-07-25T17:24Z and every uvicorn process began beating under the same tag. The rule that had just caught a real 17.5h notifier outage would have gone silent on the next one. Both liveness rules now read the app's own heartbeat filtered by daemon, and the notifier rules are pinned to daemon=subscription-notifier. db and frontend keep their container-log rules, which remain valid. Queries verified against live Loki: web 16, worker 4, notifier 2 per 20m window; beta notifier never below 1 across 37 consecutive buckets.	2026-07-25 10:46:35 -07:00
Emi Griffith	e4693dce58	observability: add the estate's first alerting; supervise Postfix ... The estate had zero alert rules. The only contact point was Grafana's factory default pointing at the literal string <example@email.com>, and beta's untracked compose override routed Grafana's SMTP at prod's Postfix — so alerts, had any existed, would have been delivered by the box most likely to be on fire. Prod served 86 5xx in 24h including five /healthz failures and nobody was told. Alerting (deploys to beta, which is where Grafana runs): - 12 Loki-based rules. There is no Prometheus in this estate and Loki is the only datasource, so every rule is log-derived. - Thresholds come from a 24h backtest that happens to contain a real ~20min prod outage at 04:20Z. Absolute counts, not ratios: prod runs ~7 req/min, so one bad request is 1.4% and a ratio alert would scream all night. The 5xx burst rule fires on the outage's 45 and 22 buckets and on nothing else in the day; the largest benign bucket all day was 5. - Routed to a new private #ops-alerts channel, not to any existing channel — #weather-events, #announcements and #prod are product surfaces that notify real subscribers. - AlertingWatchdog is a dead-man's switch; its value is its absence. - Delivery is proven, not assumed: the rules were provisioned into a throwaway Grafana against beta's live Loki and a real alert arrived in Discord. This matters because GET /api/v1/provisioning/contact-points returns [REDACTED] for the URL — a contact point holding an uninterpolated env var looks perfectly healthy and pages nobody. The only proof is a message arriving. CI gains a structural check, because the existing one only proves YAML parses: an alert rule whose condition names a missing refId is valid YAML, provisions cleanly, and never fires. It also hard-fails on a literal Discord webhook in the repo. Verified against all three breakages deliberately introduced. Postfix supervision: - The 13h outage was a boot-ordering race, not a Docker renumbering: postfix started at 08:09:49, wg0 came up at :51, postfix fataled at :52 on a missing docker_gwbridge address, and dockerd did not finish starting until 08:10:16. Stock postfix@.service is ordered only After=network-online.target and ships no Restart=, so one lost race became a permanent outage. - An ExecStartPre gate now blocks up to 60s until every inet_interfaces address actually exists, which absorbs the transient case inside a single start attempt. That makes bounded retry correct: 5 attempts in 600s, then failed — a genuinely broken config reaches a visible failed state in ~100s instead of re-fataling every 15s forever. - A 5-minute watchdog timer retries indefinitely and runs reset-failed, so "failed" still self-heals. Worst case is ~5 minutes, not 13 hours. - Wants=, not Requires=: a dockerd failure must not take down the loopback and mesh listeners that do not depend on Docker at all. Health checks must read config with `postconf -c`, never postmulti/postqueue/ postfix — those three RESOLVE inet_interfaces and so fatal precisely when an address is missing, which made the first version of this check report status=ok bound=0/0. A health check that fails open is worse than none. DEPLOY.md carries the monitoring contract, including that systemctl is-active postfix is a known-false signal: postfix.service is a wrapper whose ExecStart=/bin/true, so it reports active forever while the real postfix@- instance is failed with zero listeners. Reproduced live. Known gap, documented not fixed: Alloy ships Docker stdout, Caddy files and app JSONL, not journald — so no Postfix line reaches Loki and the mail rules cannot fire until loki.source.journal is added. Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY	2026-07-24 13:19:19 -07:00
Emi Griffith	f2fd8f6835	Subtree-merge thermograph-observability (origin/main) into observability/ ... git-subtree-dir: observability git-subtree-mainline: ae1d9bb534 git-subtree-split: 19e74af9ca	2026-07-22 22:01:11 -07:00