From e4693dce58e4ba4bbe8a101d741f7ab8f0bf950b Mon Sep 17 00:00:00 2001 From: Emi Griffith Date: Fri, 24 Jul 2026 13:19:19 -0700 Subject: [PATCH] observability: add the estate's first alerting; supervise Postfix MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The estate had zero alert rules. The only contact point was Grafana's factory default pointing at the literal string , and beta's untracked compose override routed Grafana's SMTP at prod's Postfix — so alerts, had any existed, would have been delivered by the box most likely to be on fire. Prod served 86 5xx in 24h including five /healthz failures and nobody was told. Alerting (deploys to beta, which is where Grafana runs): - 12 Loki-based rules. There is no Prometheus in this estate and Loki is the only datasource, so every rule is log-derived. - Thresholds come from a 24h backtest that happens to contain a real ~20min prod outage at 04:20Z. Absolute counts, not ratios: prod runs ~7 req/min, so one bad request is 1.4% and a ratio alert would scream all night. The 5xx burst rule fires on the outage's 45 and 22 buckets and on nothing else in the day; the largest benign bucket all day was 5. - Routed to a new private #ops-alerts channel, not to any existing channel — #weather-events, #announcements and #prod are product surfaces that notify real subscribers. - AlertingWatchdog is a dead-man's switch; its value is its absence. - Delivery is proven, not assumed: the rules were provisioned into a throwaway Grafana against beta's live Loki and a real alert arrived in Discord. This matters because GET /api/v1/provisioning/contact-points returns [REDACTED] for the URL — a contact point holding an uninterpolated env var looks perfectly healthy and pages nobody. The only proof is a message arriving. CI gains a structural check, because the existing one only proves YAML parses: an alert rule whose condition names a missing refId is valid YAML, provisions cleanly, and never fires. It also hard-fails on a literal Discord webhook in the repo. Verified against all three breakages deliberately introduced. Postfix supervision: - The 13h outage was a boot-ordering race, not a Docker renumbering: postfix started at 08:09:49, wg0 came up at :51, postfix fataled at :52 on a missing docker_gwbridge address, and dockerd did not finish starting until 08:10:16. Stock postfix@.service is ordered only After=network-online.target and ships no Restart=, so one lost race became a permanent outage. - An ExecStartPre gate now blocks up to 60s until every inet_interfaces address actually exists, which absorbs the transient case inside a single start attempt. That makes bounded retry correct: 5 attempts in 600s, then failed — a genuinely broken config reaches a visible failed state in ~100s instead of re-fataling every 15s forever. - A 5-minute watchdog timer retries indefinitely and runs reset-failed, so "failed" still self-heals. Worst case is ~5 minutes, not 13 hours. - Wants=, not Requires=: a dockerd failure must not take down the loopback and mesh listeners that do not depend on Docker at all. Health checks must read config with `postconf -c`, never postmulti/postqueue/ postfix — those three RESOLVE inet_interfaces and so fatal precisely when an address is missing, which made the first version of this check report status=ok bound=0/0. A health check that fails open is worse than none. DEPLOY.md carries the monitoring contract, including that systemctl is-active postfix is a known-false signal: postfix.service is a wrapper whose ExecStart=/bin/true, so it reports active forever while the real postfix@- instance is failed with zero listeners. Reproduced live. Known gap, documented not fixed: Alloy ships Docker stdout, Caddy files and app JSONL, not journald — so no Postfix line reaches Loki and the mail rules cannot fire until loki.source.journal is added. Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY --- .forgejo/workflows/observability-validate.yml | 79 ++ infra/DEPLOY.md | 109 +++ infra/deploy/provision-mail-supervision.sh | 476 ++++++++++ infra/deploy/provision-mail.sh | 46 +- infra/deploy/thermograph.env.example | 62 +- observability/.env.example | 11 + observability/CLAUDE.md | 22 +- observability/README.md | 201 +++++ observability/docker-compose.yml | 8 + .../provisioning/alerting/contact-points.yml | 64 ++ .../alerting/notification-policies.yml | 64 ++ .../provisioning/alerting/rules-prod.yml | 836 ++++++++++++++++++ 12 files changed, 1932 insertions(+), 46 deletions(-) create mode 100644 infra/deploy/provision-mail-supervision.sh create mode 100644 observability/grafana/provisioning/alerting/contact-points.yml create mode 100644 observability/grafana/provisioning/alerting/notification-policies.yml create mode 100644 observability/grafana/provisioning/alerting/rules-prod.yml diff --git a/.forgejo/workflows/observability-validate.yml b/.forgejo/workflows/observability-validate.yml index 157a3d9..175eafb 100644 --- a/.forgejo/workflows/observability-validate.yml +++ b/.forgejo/workflows/observability-validate.yml @@ -17,6 +17,13 @@ name: Validate observability stack # GitHub release (pinned to the same v1.9.1 the fleet runs; see # observability/alloy/docker-compose.agent.yml), no docker CLI needed. # +# The alerting config gets a third, stricter step. A YAML parse is not enough +# for it: an alert rule whose `condition` names a refId that does not exist is +# perfectly valid YAML, provisions without complaint, and then never fires -- +# a paging path that looks healthy and reaches nobody, which is the exact +# failure this whole config exists to end. That step also refuses a literal +# Discord webhook in the repo (it must stay an env reference). +# # Not validated here: docker-compose *schema* (unknown-key checks, which would # need the docker CLI). Add it if the churn warrants the extra tooling. @@ -91,3 +98,75 @@ jobs: # subcommand, not a config error, so strip that one line before checking. grep -v '^livedebugging ' observability/alloy/config.alloy > /tmp/config-for-validate.alloy /tmp/alloybin/alloy-linux-amd64 validate /tmp/config-for-validate.alloy + + - name: Alerting config is structurally sound + run: | + set -euo pipefail + apt-get update -qq && apt-get install -y -qq python3-yaml >/dev/null + python3 - <<'PY' + import pathlib, re, sys + import yaml + + d = pathlib.Path("observability/grafana/provisioning/alerting") + if not d.is_dir(): + print("no alerting provisioning yet — nothing to check"); sys.exit(0) + + docs, bad = {}, [] + for f in sorted(list(d.glob("*.yml")) + list(d.glob("*.yaml"))): + docs[f] = yaml.safe_load(f.read_text()) or {} + print("loaded", f) + + receivers, referenced = set(), set() + + for f, doc in docs.items(): + # --- contact points --------------------------------------------------- + for cp in doc.get("contactPoints") or []: + receivers.add(cp.get("name")) + for r in cp.get("receivers") or []: + url = str((r.get("settings") or {}).get("url", "")) + # A real webhook in the repo is a credential leak: anyone with + # it can post into the ops channel. It must stay an env ref. + if re.search(r"https://(discord|discordapp)\.com/api/webhooks/", url): + bad.append(f"{f}: contact point '{cp.get('name')}' has a LITERAL Discord webhook URL — use $DISCORD_ALERT_WEBHOOK_URL") + elif not url.startswith("$"): + bad.append(f"{f}: contact point '{cp.get('name')}' url is neither an env reference nor empty: {url!r}") + + # --- notification policies -------------------------------------------- + def walk(route, where): + if route.get("receiver"): + referenced.add(route["receiver"]) + for child in route.get("routes") or []: + walk(child, where) + + for pol in doc.get("policies") or []: + walk(pol, f) + + # --- alert rules ------------------------------------------------------ + for g in doc.get("groups") or []: + for rule in g.get("rules") or []: + title = rule.get("title", "") + where = f"{f}: rule '{title}'" + refs = {q.get("refId") for q in rule.get("data") or []} + if not rule.get("uid"): + bad.append(f"{where}: missing uid (provisioning needs a stable one)") + if rule.get("condition") not in refs: + bad.append(f"{where}: condition '{rule.get('condition')}' is not one of {sorted(refs)} — this rule can never fire") + if not (rule.get("labels") or {}).get("severity"): + bad.append(f"{where}: no severity label — the notification policy routes on it") + if not (rule.get("annotations") or {}).get("summary"): + bad.append(f"{where}: no summary annotation — the Discord message would be blank") + for q in rule.get("data") or []: + model = q.get("model") or {} + upstream = model.get("expression") + if upstream and upstream not in refs: + bad.append(f"{where}: query '{q.get('refId')}' reads refId '{upstream}' which does not exist") + + missing = referenced - receivers + if missing: + bad.append(f"notification policy routes to undefined receiver(s): {sorted(missing)}") + + for b in bad: + print("::error::" + b) + print(f"checked {len(docs)} file(s): {len(receivers)} contact point(s), {len(bad)} problem(s)") + sys.exit(1 if bad else 0) + PY diff --git a/infra/DEPLOY.md b/infra/DEPLOY.md index 800ad23..35bee32 100644 --- a/infra/DEPLOY.md +++ b/infra/DEPLOY.md @@ -300,6 +300,115 @@ the message and sends nothing — so LAN dev and the test suite exercise the who signup path with no mail server and no risk of mailing a real person. Production opts in with `=smtp`. +### Postfix unit topology — and the check that lies + +Debian/Ubuntu ships Postfix as **three** systemd units, and only one of them +does anything: + +| Unit | What it is | Does it bind sockets? | +|---|---|---| +| `postfix.service` | umbrella wrapper: `Type=oneshot`, `ExecStart=/bin/true`, `RemainAfterExit=yes` | **No.** Reports `active (exited)` forever. | +| `postfix@.service` | the template all instances are built from | n/a | +| `postfix@-.service` | the instance for `/etc/postfix` — the real MTA | **Yes.** This is the one that matters. | + +> **`systemctl is-active postfix` is not a health check.** It returned `active` +> for the entire 13-hour mail outage of 2026-07-24 while `postfix@-.service` was +> `failed` and nothing was listening on `:25`. Reproduced deliberately during +> supervision testing: with the instance in `failed` and zero listeners, +> `is-active postfix` still said `active`. **Always query `postfix@-.service`.** + +Two consequences worth internalising: + +- Restart the **instance** (`systemctl restart 'postfix@-'`), not the umbrella, + for a `main.cf` change to take effect. +- **Postfix's own tools lie in the one state you most need to detect.** Any + command that *resolves* `inet_interfaces` — `postmulti`, `postqueue`, + `postfix` — fatals when one of the listed addresses is missing from the host. + A checker built on them returns empty and reads as "nothing wrong". Use plain + `postconf -c /etc/postfix -h inet_interfaces`, which only reads `main.cf`. + +### Supervision (`deploy/provision-mail-supervision.sh`) + +Installed by `provision-mail.sh`, and safe to run on its own — it touches +nothing in `main.cf`. Everything it writes lives outside `/opt/thermograph`, so +`deploy.sh`'s `git reset --hard` cannot erase it; re-running the script is what +makes a rebuilt box match. + +- **Drop-in** `/etc/systemd/system/postfix@.service.d/10-wait-for-interfaces.conf` + (a drop-in, never an edit to the packaged unit, which apt replaces): + orders Postfix `After=docker.service wg-quick@wg0.service`, adds + `Restart=on-failure` / `RestartSec=15s`, and bounds retries at + `StartLimitBurst=5` / `StartLimitIntervalSec=600`. +- **Start gate** `/usr/local/sbin/postfix-wait-interfaces` — an `ExecStartPre` + that blocks (≤60s) until every address in `inet_interfaces` actually exists. + Ordering makes the boot race unlikely; this makes it deterministic, and it is + what makes a *bounded* start limit safe: a late interface costs seconds inside + one attempt instead of burning the retry budget. +- **Health check** `/usr/local/sbin/postfix-health` — the single definition of + "mail works" (see below). +- **Watchdog** `/usr/local/sbin/postfix-watchdog` + `postfix-watchdog.timer` — + every 5 minutes. `Restart=` gives up after 5 attempts *by design*, so that a + genuinely broken config lands in `failed` loudly instead of re-fataling every + 15s forever; the watchdog is what stops "failed" meaning "dead until a human + notices". It runs `reset-failed` (clearing the start-limit counter, which + otherwise makes plain `systemctl start` refuse) and retries, forever, at a + cadence 20× slower than `RestartSec`. Net effect: **bounded fast retry for + transients, slow unbounded retry for everything else**, worst-case recovery + ~5 minutes instead of 13 hours. + +`sudo touch /etc/postfix/maintenance` stops the watchdog fighting a deliberate +`systemctl stop postfix@-`. Remove it when done. + +### Monitoring specification + +For the Grafana/Discord alerting and any future `mail_health` tool. Alerts must +**not** route over this box's own SMTP — prod's MTA cannot be the transport for +"prod's MTA is down". + +`postfix-health` is the contract. Exit `0` healthy, `1` unhealthy, and one +logfmt line on stdout (LogQL: `| logfmt`): + +``` +mail_health status=ok unit=postfix@-.service unit_state=active bound=3/3 \ + banner=ok queue=2 queue_real=0 oldest_deferred_s=0 reason=none +``` + +It asserts, in order: the **instance** unit is active; every address in +`inet_interfaces` is really bound on `:25`; none of them is **missing from the +host** (the latent state that stays invisible until the next reboot and then +kills all mail); a live `220` greeting comes back, not merely an open socket; +**no public address is bound**; and queue depth/age with null-sender bounces +excluded (they are undeliverable by design — this host accepts no inbound mail — +and would otherwise hold an alert open forever). + +Alert on: + +| Severity | Condition | Why | +|---|---|---| +| **Page** | `mail_health status=fail` twice consecutively (≥10 min) | mail is down | +| **Page** | any `postfix_watchdog action=RECOVERY_FAILED` | self-heal exhausted | +| **Page** | **no `postfix_watchdog` line at all for 15 min** | see below | +| Warn | `status=fail` where `reason` contains only `address-missing-from-host` | still serving, but the next reboot kills all mail | +| Warn | `queue_real > 0` and `oldest_deferred_s > 3600` | mail accepted but not leaving | + +The **heartbeat-absence** rule is the one that is easy to leave out and the one +that would have caught this outage. Every other check is an *active probe* and +therefore only reports while something is alive to run it; a dead box, dead +timer, or dead log shipper produces silence, and silence renders identically to +health on a dashboard. Alert on the absence of the 5-minute heartbeat, or the +monitoring inherits the same blind spot the umbrella unit had. + +> **Blocker for the alerting agent:** none of this reaches Loki today. Alloy +> ships Docker container stdout, Caddy files and the app's JSONL — **not the +> systemd journal** — and Postfix runs on the host. Until +> `observability/alloy/config.alloy` gains a `loki.source.journal` (plus +> `/var/log/journal` and `/etc/machine-id` mounted read-only in +> `alloy/docker-compose.agent.yml`), no Postfix or watchdog line is queryable +> and none of the rules above can fire. That change is untested — validate it +> before relying on it. Until then the check is still reachable synchronously +> via Centralis `run_on_host` running `postfix-health`, which is also the +> natural implementation of a `mail_health` tool. + ### Deliverability (do this before mailing real subscribers) Mail from a bare VPS IP is very often junked no matter how Postfix is configured, diff --git a/infra/deploy/provision-mail-supervision.sh b/infra/deploy/provision-mail-supervision.sh new file mode 100644 index 0000000..7c1f96e --- /dev/null +++ b/infra/deploy/provision-mail-supervision.sh @@ -0,0 +1,476 @@ +#!/usr/bin/env bash +# Postfix SUPERVISION layer — restart policy, start-time interface gate, and a +# self-heal watchdog. Called by provision-mail.sh; also safe to run standalone +# (it touches nothing in main.cf, so it will not fight a mail-config change): +# +# sudo bash infra/deploy/provision-mail-supervision.sh +# +# WHY THIS EXISTS — the 2026-07-24 prod outage. +# +# Postfix treats an `inet_interfaces` address with no matching local interface +# as FATAL, and takes down EVERY listener with it — loopback and mesh included, +# not just the missing one. prod lists three addresses, and two of them are +# created by other daemons: 10.10.0.1 (wg-quick@wg0) and 172.18.0.1 +# (docker_gwbridge, created by dockerd). +# +# The stock postfix@.service is ordered only `After=network-online.target`, +# which says nothing about dockerd or wg-quick, and it ships NO `Restart=`. +# At the 08:09 reboot that cost 13 hours of mail: +# +# 08:09:49 postfix@-.service starts +# 08:09:51 wg-quick@wg0 finishes <- 10.10.0.1 appears, wins by one second +# 08:09:52 fatal: inet_interfaces: no local interface found for 172.18.0.1 +# 08:09:53 docker.service only BEGINS starting +# 08:10:16 docker.service active <- docker_gwbridge finally exists +# +# Note this was a BOOT-ORDERING RACE, not a Docker renumbering: 172.18.0.1 is +# the pinned docker_gwbridge gateway and has always had that address. Postfix +# simply asked for it 24 seconds before dockerd created it. +# +# Nobody noticed for 13h because `systemctl is-active postfix` said "active" +# the whole time — see the umbrella-unit note below. +# +# THREE LAYERS, because no single one of them is sufficient: +# +# 1. ORDERING (After=docker.service wg-quick@wg0.service) — makes the race +# very unlikely. Not a guarantee: dockerd reporting "active" is not a +# promise that a specific bridge is already numbered. +# 2. A START-TIME GATE (postfix-wait-interfaces, an ExecStartPre) — makes it +# deterministic. It blocks until every address in inet_interfaces actually +# exists, so a late interface costs seconds inside one start attempt +# instead of burning a restart. This is what makes a BOUNDED start limit +# safe; see the StartLimit discussion in the drop-in below. +# 3. A WATCHDOG TIMER (postfix-watchdog) — the backstop. It is not redundant +# with Restart=: postfix@.service is Type=forking with GuessMainPID=no and +# no PIDFile=, so systemd tracks NO main PID for it. `Restart=` therefore +# only covers failures during START. If the master process dies at +# runtime, systemd does not reliably notice, and no restart policy fires. +# The watchdog is the only thing that catches that. +# +# THE UMBRELLA-UNIT TRAP. `postfix.service` is a wrapper whose ExecStart is +# /bin/true and which sets RemainAfterExit=yes. It reports `active (exited)` +# forever, whether or not any mail is being handled. The unit that actually +# binds sockets is the instance `postfix@-.service`. Never health-check the +# umbrella — that is precisely the check that reported green for 13 hours. +set -euo pipefail + +if [[ $EUID -ne 0 ]]; then + echo "run as root (sudo)" >&2 + exit 1 +fi + +# How long the start-time gate waits for a missing interface before giving up +# and letting the restart policy take over. Overridable for testing. +POSTFIX_WAIT_TIMEOUT="${POSTFIX_WAIT_TIMEOUT:-60}" + +echo "==> installing /usr/local/sbin/postfix-wait-interfaces" +cat > /usr/local/sbin/postfix-wait-interfaces <<'WAITSCRIPT' +#!/usr/bin/env bash +# Managed by deploy/provision-mail-supervision.sh -- do not hand-edit. +# +# ExecStartPre gate for postfix@.service. Blocks until every address listed in +# this instance's inet_interfaces actually exists on some local interface. +# +# Postfix treats a missing inet_interfaces address as fatal and drops ALL its +# listeners, so at boot it must not be allowed to ask before dockerd/wg-quick +# have finished. Unit ordering makes that unlikely; this makes it deterministic. +# +# FAIL-OPEN BY DESIGN: if this script cannot introspect the config, it exits 0 +# and lets Postfix try anyway. A supervision gate must only ever DELAY a start, +# never be the reason mail stays down. +set -uo pipefail + +TIMEOUT="${POSTFIX_WAIT_TIMEOUT:-60}" +INSTANCE="${1:--}" +# Instance '-' is the default instance and always lives in /etc/postfix. A +# second config dir can be passed explicitly; this estate runs one instance. +CONFDIR="${2:-/etc/postfix}" +log() { echo "postfix-wait-interfaces: $*"; } + +# MUST be plain `postconf -c`, NOT `postmulti -x postconf`. +# +# postmulti RESOLVES inet_interfaces, so it fatals with rc=1 and prints nothing +# whenever an address is missing -- which is precisely the situation this gate +# exists to detect. Using it here made the gate fail open and silently never +# wait at all. Plain postconf only reads main.cf and returns the value happily +# whether or not the address exists. (Same trap applies to postqueue/postfix; +# see postfix-health.) +addrs=$(postconf -c "$CONFDIR" -h inet_interfaces 2>/dev/null) || { + log "cannot read inet_interfaces from $CONFDIR; not blocking startup" + exit 0 +} + +# Only literal IP addresses can be waited for. Keywords (all, loopback-only) +# and hostnames are left to Postfix to resolve. +want=() +for a in ${addrs//,/ }; do + case "$a" in + all|loopback-only|localhost) continue ;; + esac + if [[ "$a" =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ || "$a" == *:* ]]; then + want+=("$a") + fi +done + +if (( ${#want[@]} == 0 )); then + log "no literal addresses to wait for (inet_interfaces = $addrs)" + exit 0 +fi + +deadline=$(( SECONDS + TIMEOUT )) +missing=() +while :; do + present=$(ip -o addr show 2>/dev/null | awk '{print $4}' | cut -d/ -f1) + missing=() + for a in "${want[@]}"; do + grep -qxF "$a" <<<"$present" || missing+=("$a") + done + (( ${#missing[@]} == 0 )) && break + (( SECONDS >= deadline )) && break + sleep 1 +done + +if (( ${#missing[@]} == 0 )); then + log "all addresses present after ${SECONDS}s: ${want[*]}" + exit 0 +fi + +# Deliberately non-zero: let the unit's Restart=/StartLimit policy decide what +# happens next, and make the reason visible in the journal. +log "TIMEOUT after ${TIMEOUT}s; still missing: ${missing[*]} (inet_interfaces = $addrs)" +log "Postfix will fatal on this. Either the interface provider is broken, or" +log "inet_interfaces names an address this host no longer has." +exit 1 +WAITSCRIPT +chmod 0755 /usr/local/sbin/postfix-wait-interfaces + +echo "==> installing /usr/local/sbin/postfix-health" +cat > /usr/local/sbin/postfix-health <<'HEALTHSCRIPT' +#!/usr/bin/env bash +# Managed by deploy/provision-mail-supervision.sh -- do not hand-edit. +# +# THE canonical "is this box's mail actually working" check. One implementation, +# so the watchdog, a human, a Grafana/Loki alert and a future `mail_health` tool +# all agree on what healthy means. +# +# exit 0 = healthy exit 1 = unhealthy +# +# Output is one logfmt line on stdout (parseable with LogQL `| logfmt`), +# followed by human-readable detail on stderr. +# +# WHAT IT DELIBERATELY DOES NOT DO: check `postfix.service`. That umbrella unit +# is ExecStart=/bin/true + RemainAfterExit=yes and reports active(exited) +# forever. It reported green through the entire 13h outage of 2026-07-24. +set -uo pipefail +export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin + +INSTANCE="${1:--}" +CONFDIR="${2:-/etc/postfix}" +UNIT="postfix@${INSTANCE}.service" +problems=() + +# --- 1. the INSTANCE unit, never the umbrella -------------------------------- +unit_state=$(systemctl is-active "$UNIT" 2>/dev/null || true) +[[ "$unit_state" == "active" ]] || problems+=("unit-not-active:$unit_state") + +# --- 2. every configured address is actually bound on :25 -------------------- +# +# MUST be plain `postconf -c`, NOT `postmulti -x postconf` / `postqueue`. +# +# Every Postfix command that RESOLVES inet_interfaces (postmulti, postqueue, +# postfix) fatals when one of the addresses is missing from the host -- exactly +# the condition this check exists to find. An earlier version of this script +# used postmulti, got an empty string back, concluded there were 0 addresses to +# verify, and cheerfully reported "status=ok bound=0/0" while main.cf named a +# ghost address that would kill all mail at the next reboot. A health check that +# fails OPEN is worse than no health check. +# +# Plain postconf only reads main.cf and never touches an interface. +addrs=$(postconf -c "$CONFDIR" -h inet_interfaces 2>/dev/null || echo "") +if [[ -z "${addrs// /}" ]]; then + problems+=("cannot-read-inet_interfaces") +fi +listening=$(ss -lnt 2>/dev/null | awk '{print $4}') +expected=(); bound=0; unbound=() +for a in ${addrs//,/ }; do + case "$a" in + all) expected+=("0.0.0.0") ;; + loopback-only) expected+=("127.0.0.1") ;; + localhost) expected+=("127.0.0.1") ;; + *) expected+=("$a") ;; + esac +done +for a in "${expected[@]}"; do + if grep -qxF "${a}:25" <<<"$listening"; then bound=$(( bound + 1 )); else unbound+=("$a"); fi +done +(( ${#unbound[@]} == 0 )) || problems+=("not-bound:$(IFS=,; echo "${unbound[*]}")") +# Never let "I could not work out what to check" read as healthy. +(( ${#expected[@]} > 0 )) || problems+=("no-configured-addresses") + +# --- 3. LATENT FATAL: config names an address the host does not have --------- +# This is the state that survives silently until the next reboot and then kills +# all mail. Catching it here turns a future outage into a warning today. +present=$(ip -o addr show 2>/dev/null | awk '{print $4}' | cut -d/ -f1) +ghosts=() +for a in "${expected[@]}"; do + [[ "$a" == "0.0.0.0" ]] && continue + grep -qxF "$a" <<<"$present" || ghosts+=("$a") +done +(( ${#ghosts[@]} == 0 )) || problems+=("address-missing-from-host:$(IFS=,; echo "${ghosts[*]}")") + +# --- 4. a real SMTP greeting, not just an open socket ------------------------ +banner="none" +if command -v timeout >/dev/null 2>&1; then + greeting=$(timeout 5 bash -c 'exec 3<>/dev/tcp/127.0.0.1/25 || exit 1; head -1 <&3; printf "QUIT\r\n" >&3' 2>/dev/null || true) + case "$greeting" in + 220*) banner="ok" ;; + *) banner="bad"; problems+=("no-220-greeting") ;; + esac +fi + +# --- 5. SEND-ONLY INVARIANT: must NOT be bound on a public address ----------- +# A regression here is worse than an outage: it is an internet-facing MTA. +public_ips=$(ip -4 -o addr show scope global 2>/dev/null | awk '{print $4}' | cut -d/ -f1 \ + | grep -vE '^(10\.|172\.(1[6-9]|2[0-9]|3[01])\.|192\.168\.|127\.)' || true) +exposed=() +for p in $public_ips; do + grep -qxF "${p}:25" <<<"$listening" && exposed+=("$p") +done +grep -qxF "0.0.0.0:25" <<<"$listening" && exposed+=("0.0.0.0") +(( ${#exposed[@]} == 0 )) || problems+=("PUBLICLY-BOUND:$(IFS=,; echo "${exposed[*]}")") + +# --- 6. queue depth and age, excluding the known-benign null-sender bounces --- +# Read arrival_time from `postqueue -j`, NOT the queue file's mtime: Postfix +# sets a deferred file's mtime to the NEXT RETRY time, which is in the future, +# so mtime-based ages come out negative and meaningless. +# +# Null-sender bounces (postqueue reports sender "MAILER-DAEMON") addressed to +# our own no-reply are undeliverable by design -- this host accepts no inbound +# mail -- so they sit in the queue until they expire. They must not hold an +# alert open forever, so they are excluded from queue_real/oldest. Both the raw +# and filtered counts are reported so a monitor can see either. +# +# NOTE: postqueue RESOLVES inet_interfaces and so fatals (rc=69) in the very +# latent-fatal state flagged above. When that happens it yields no JSON, so fall +# back to counting spool files directly -- a count that needs no Postfix command +# and therefore still works when Postfix itself will not run. +queue_json=$(postqueue -j 2>/dev/null || true) +read -r queue_total queue_real oldest_age <<<"$( + printf '%s\n' "$queue_json" | awk -v now="$(date +%s)" ' + NF { + total++ + sender=""; arr=0 + if (match($0, /"sender": "[^"]*"/)) { + s = substr($0, RSTART, RLENGTH); gsub(/"sender": "|"/, "", s); sender = s + } + if (match($0, /"arrival_time": [0-9]+/)) { + a = substr($0, RSTART, RLENGTH); gsub(/[^0-9]/, "", a); arr = a + 0 + } + if (sender != "MAILER-DAEMON" && sender != "") { + real++ + age = now - arr + if (age > oldest) oldest = age + } + } + END { printf "%d %d %d", total + 0, real + 0, oldest + 0 } + ' +)" +queue_total=${queue_total:-0}; queue_real=${queue_real:-0}; oldest_age=${oldest_age:-0} +if [[ -z "${queue_json// /}" ]]; then + queue_total=$(find /var/spool/postfix/incoming /var/spool/postfix/active \ + /var/spool/postfix/deferred -type f 2>/dev/null | wc -l) + queue_real=-1 # unknown: cannot classify senders without postqueue +fi + +status=ok; rc=0 +(( ${#problems[@]} == 0 )) || { status=fail; rc=1; } + +printf 'mail_health status=%s unit=%s unit_state=%s bound=%d/%d banner=%s queue=%d queue_real=%d oldest_deferred_s=%d reason=%s\n' \ + "$status" "$UNIT" "${unit_state:-unknown}" "$bound" "${#expected[@]}" "$banner" \ + "$queue_total" "$queue_real" "$oldest_age" \ + "$( (( ${#problems[@]} )) && (IFS=,; echo "${problems[*]}") || echo none )" + +if (( rc != 0 )); then + { echo "--- postfix-health detail ---" + echo "inet_interfaces : $addrs" + echo "host addresses : $(echo "$present" | tr '\n' ' ')" + echo "listening on 25 : $(grep ':25$' <<<"$listening" | tr '\n' ' ')" + systemctl status "$UNIT" --no-pager -l 2>/dev/null | head -20 + } >&2 +fi +exit $rc +HEALTHSCRIPT +chmod 0755 /usr/local/sbin/postfix-health + +echo "==> installing /usr/local/sbin/postfix-watchdog" +cat > /usr/local/sbin/postfix-watchdog <<'WATCHSCRIPT' +#!/usr/bin/env bash +# Managed by deploy/provision-mail-supervision.sh -- do not hand-edit. +# +# Self-heal backstop, run every 5 minutes by postfix-watchdog.timer. +# +# It exists for two failure modes that `Restart=on-failure` provably cannot +# cover: +# +# 1. RUNTIME DEATH. postfix@.service is Type=forking, GuessMainPID=no, and +# declares no PIDFile=, so systemd tracks no main PID. If the Postfix +# master dies while running, no restart policy fires. +# 2. GIVING UP. The unit uses a BOUNDED start limit on purpose, so a broken +# config lands in `failed` instead of thrashing forever. Without a +# backstop, "failed" would mean "dead until a human looks" — which is the +# 13-hour outage again. This retries every 5 minutes, forever, and clears +# the start-limit counter so the bound never becomes permanent death. +# +# Cadence is deliberately 20x slower than RestartSec: fast enough that nothing +# is ever down for more than ~5 minutes, slow enough that a genuinely broken +# config produces ~288 log lines a day instead of ~5,760. +set -uo pipefail +export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin + +INSTANCE=- +UNIT="postfix@${INSTANCE}.service" + +# Escape hatch: `sudo touch /etc/postfix/maintenance` to stop the watchdog +# fighting a deliberate `systemctl stop postfix@-`. +if [[ -e /etc/postfix/maintenance ]]; then + echo "postfix_watchdog action=skipped reason=maintenance-flag" + exit 0 +fi + +# Do not interfere with a start already in progress. +state=$(systemctl show "$UNIT" -p ActiveState --value 2>/dev/null || echo unknown) +if [[ "$state" == "activating" || "$state" == "deactivating" ]]; then + echo "postfix_watchdog action=skipped reason=transitional:$state" + exit 0 +fi + +if health=$(postfix-health "$INSTANCE" 2>/dev/null); then + echo "postfix_watchdog action=ok ${health#mail_health }" + exit 0 +fi + +echo "postfix_watchdog action=recovering ${health#mail_health }" +# reset-failed clears both the failed state and the start-limit counter, so the +# bounded StartLimitBurst can never become a permanent death sentence. +systemctl reset-failed "$UNIT" 2>/dev/null || true +systemctl start "$UNIT" 2>/dev/null || true +sleep 5 + +if health=$(postfix-health "$INSTANCE" 2>/dev/null); then + echo "postfix_watchdog action=recovered ${health#mail_health }" + exit 0 +fi +# Distinctive, greppable line: this is the one that should page a human. +echo "postfix_watchdog action=RECOVERY_FAILED ${health#mail_health }" +exit 1 +WATCHSCRIPT +chmod 0755 /usr/local/sbin/postfix-watchdog + +echo "==> installing postfix@.service drop-in" +mkdir -p /etc/systemd/system/postfix@.service.d +# Quoted heredoc + a placeholder, NOT shell interpolation: this block contains +# backticks and $-signs in its prose, and an unquoted heredoc would run them. +cat > /etc/systemd/system/postfix@.service.d/10-wait-for-interfaces.conf <<'DROPIN' +# Managed by deploy/provision-mail-supervision.sh -- do not hand-edit. +# +# A DROP-IN, never an edit to /usr/lib/systemd/system/postfix@.service: the +# packaged unit is replaced wholesale by an apt upgrade. +# +# ORDERING. inet_interfaces may name a Docker bridge gateway and/or a WireGuard +# mesh address; neither exists until dockerd / wg-quick has run, and Postfix +# treats a missing address as fatal for ALL listeners. Stock ordering is only +# After=network-online.target, which says nothing about either daemon. +# +# Wants=, not Requires=, on purpose. Requires= would mean a dockerd failure +# takes mail down entirely — including the loopback and mesh listeners, which +# do not depend on Docker at all. That escalates someone else's outage into a +# total mail outage. Wants= keeps the ordering when both are present without +# making mail's availability a hostage to Docker's. +# +# THE GATE. ExecStartPre blocks until every configured address actually exists +# (bounded by POSTFIX_WAIT_TIMEOUT). Ordering alone is a strong heuristic — +# dockerd reporting "active" is not a promise that a specific bridge is +# numbered — and this turns the residual race into a deterministic wait. +# It is what makes the bounded start limit below safe. +# +# THE START LIMIT — the deliberate tradeoff. +# +# StartLimitIntervalSec=0 (retry forever) was considered and rejected. With the +# gate in place, the transient case is absorbed INSIDE a single start attempt, +# so restarts now overwhelmingly mean "genuinely broken", not "too early". For +# that case, retry-forever is the worse option: the unit never reaches `failed` +# (so the strongest local signal never exists), and a malformed main.cf +# re-fatals every RestartSec forever. +# +# * 5 attempts / 600s. A broken config reaches `failed` in ~100s (fast +# failure: ~5s start + 15s wait) or ~5min (slow failure: the full 60s gate +# + 15s wait, 5 times) — either way, loudly, and then it stops. +# * Normal operation never approaches this: an isolated crash is 1 restart. +# * "Failed" does NOT mean "dead until a human looks" — postfix-watchdog.timer +# retries every 5 minutes forever and clears the counter. Bounded fast +# retry for transients, slow unbounded retry for everything else. +# +# TimeoutStartSec covers the worst case: the full @WAIT_TIMEOUT@s gate plus +# instance configure plus master start, with headroom. The default 90s would be +# uncomfortably tight against a @WAIT_TIMEOUT@s gate. +[Unit] +After=docker.service wg-quick@wg0.service +Wants=docker.service wg-quick@wg0.service +StartLimitIntervalSec=600 +StartLimitBurst=5 + +[Service] +Environment=POSTFIX_WAIT_TIMEOUT=@WAIT_TIMEOUT@ +ExecStartPre=/usr/local/sbin/postfix-wait-interfaces %i +TimeoutStartSec=180s +Restart=on-failure +RestartSec=15s +DROPIN +sed -i "s/@WAIT_TIMEOUT@/${POSTFIX_WAIT_TIMEOUT}/g" \ + /etc/systemd/system/postfix@.service.d/10-wait-for-interfaces.conf + +echo "==> installing postfix-watchdog.service / .timer" +cat > /etc/systemd/system/postfix-watchdog.service <<'WDSERVICE' +# Managed by deploy/provision-mail-supervision.sh -- do not hand-edit. +[Unit] +Description=Postfix health watchdog (self-heal backstop) +Documentation=file:///usr/local/sbin/postfix-watchdog +# Never race the thing we are supervising. +After=postfix@-.service + +[Service] +Type=oneshot +ExecStart=/usr/local/sbin/postfix-watchdog +# The watchdog failing is itself a signal worth seeing, but it must never be +# able to wedge the timer. +SuccessExitStatus=0 1 +WDSERVICE + +cat > /etc/systemd/system/postfix-watchdog.timer <<'WDTIMER' +# Managed by deploy/provision-mail-supervision.sh -- do not hand-edit. +# +# Every 5 minutes: the recovery-latency ceiling for mail on this box. The +# 2026-07-24 outage ran 13h; this bounds the same failure at ~5min even with +# no alerting configured at all. +[Unit] +Description=Run the Postfix health watchdog every 5 minutes + +[Timer] +OnBootSec=5min +OnUnitActiveSec=5min +# Spread the load a little so it never lands exactly on other cron-ish work. +RandomizedDelaySec=20s +AccuracySec=10s + +[Install] +WantedBy=timers.target +WDTIMER + +systemctl daemon-reload +systemctl enable --now postfix-watchdog.timer + +echo "==> supervision layer installed" +echo " drop-in : /etc/systemd/system/postfix@.service.d/10-wait-for-interfaces.conf" +echo " gate : /usr/local/sbin/postfix-wait-interfaces" +echo " health : /usr/local/sbin/postfix-health" +echo " watchdog: /usr/local/sbin/postfix-watchdog (postfix-watchdog.timer)" diff --git a/infra/deploy/provision-mail.sh b/infra/deploy/provision-mail.sh index 04f1ed6..61d4788 100755 --- a/infra/deploy/provision-mail.sh +++ b/infra/deploy/provision-mail.sh @@ -79,6 +79,19 @@ postconf -e "myorigin = ${MAIL_DOMAIN}" # stack cutover while still listed in inet_interfaces. Also note: postfix on # this distro is an umbrella unit; restart `postfix@-`, not `postfix`, for # inet_interfaces changes to take effect. +# +# THE SAME TRAP FIRES AT BOOT, NOT JUST ON RENUMBERING (prod outage 2026-07-24). +# A Docker bridge address does not exist until dockerd creates it, and the stock +# postfix@.service is only ordered `After=network-online.target` -- which says +# nothing about dockerd or wg-quick. At the 08:09 reboot Postfix started at +# 08:09:49 and fataled at 08:09:52 on "no local interface found for 172.18.0.1"; +# dockerd did not even begin starting until 08:09:53. wg0 (10.10.0.1) won the +# same race by one second. Because postfix@.service ships no Restart=, that +# single lost race killed ALL mail for 13h -- loopback and mesh included. +# install_postfix_ordering_dropin below is what makes this survive a reboot: +# it orders postfix@ after docker.service and wg-quick@wg0.service and retries +# on failure. If you add an address here that some other daemon creates, add +# that daemon to the drop-in too. DOCKER_MAIL_GATEWAY="${DOCKER_MAIL_GATEWAY-172.19.0.1}" DOCKER_MAIL_SUBNET="${DOCKER_MAIL_SUBNET-172.19.0.0/16}" # Optional WireGuard-mesh listener: other mesh nodes (e.g. beta's Forgejo, whose @@ -139,11 +152,36 @@ else postconf -e "relayhost =" fi -systemctl enable postfix -systemctl restart postfix +# Postfix fatals if ANY inet_interfaces address is missing when it starts, and +# takes every listener down with it. Docker bridge and WireGuard addresses are +# created by other daemons, so order Postfix after them, gate the start on the +# addresses actually existing, and supervise the result. See the long comment +# above the inet_interfaces block, and the header of the script itself for the +# full incident write-up and the reasoning behind each number. +bash "$(dirname "${BASH_SOURCE[0]}")/provision-mail-supervision.sh" -echo "==> verifying it listens on loopback only" -ss -lntp | grep ':25 ' || true +systemctl enable postfix +# postfix.service is an umbrella whose ExecStart is /bin/true; the instance +# postfix@- is what actually binds. Restarting the umbrella propagates via +# PartOf=, but restart the instance directly so a failure surfaces here. +systemctl restart 'postfix@-' + +# Assert, don't hope. postfix-health is the ONE definition of "mail works" on +# this estate -- the same check the watchdog, the Grafana alert and any +# mail_health tool use, so provisioning cannot pass on a laxer standard than +# monitoring. It checks the INSTANCE unit (never the active(exited) umbrella, +# which is the check that reported green through the whole 13h 2026-07-24 +# outage), that every configured address is really bound, that a live 220 +# greeting comes back, that no PUBLIC address is bound, and that nothing in +# inet_interfaces is missing from the host -- the latent state that stays +# invisible until the next reboot and then kills all mail. +echo "==> verifying postfix is actually up and bound" +/usr/local/sbin/postfix-health || { + echo "FATAL: postfix-health failed -- see detail above" >&2 + exit 1 +} +echo "==> listeners (must NOT include a public address):" +ss -lntp | grep ':25 ' cat <<'NOTE' diff --git a/infra/deploy/thermograph.env.example b/infra/deploy/thermograph.env.example index 7020e2d..5c423fd 100644 --- a/infra/deploy/thermograph.env.example +++ b/infra/deploy/thermograph.env.example @@ -83,35 +83,12 @@ WORKERS=4 # Read once at process start; changing it needs a restart, not a live toggle. #THERMOGRAPH_ROLE=all -# The recurring jobs (city warming, IndexNow). Hours between runs; both are -# cheap no-op skips when there's nothing to do, so the defaults rarely need -# changing. The daemon service (below) owns these timers and fires the work -# through the backend's internal routes. +# The worker's own recurring jobs (city warming, IndexNow), gated by the same +# leader election as the notifier above. Hours between runs; both are cheap +# no-op skips when there's nothing to do, so the defaults rarely need changing. #THERMOGRAPH_WARM_CITIES_INTERVAL_HOURS=24 #THERMOGRAPH_INDEXNOW_INTERVAL_HOURS=6 -# --- Internal daemon<->backend API ------------------------------------------------ -# Shared secret authenticating the daemon service's calls to the backend's -# /internal/* routes (X-Thermograph-Internal-Token header). A credential: -# OPTIONAL — leave unset unless you specifically want to override the derivation. -# -# When unset, the backend and the daemon each DERIVE this token from -# THERMOGRAPH_AUTH_SECRET (HMAC-SHA256 under a fixed domain-separation label). -# That secret is already required and already in every vault, so the daemon -# stands up with NO new key to add, rotate, or forget on a new host. HMAC under a -# distinct label rather than reusing the auth secret directly, so a leak of this -# token can't be replayed as the session-signing key. -# -# Set it explicitly to rotate this surface independently of the auth secret -# (e.g. openssl rand -hex 32). On a SOPS host that means `sops edit` on -# deploy/secrets/*.yaml, never hand-editing /etc/thermograph.env — it is a -# rendered artifact. Both ends must agree, so set it on both or neither. -# -# With neither this nor THERMOGRAPH_AUTH_SECRET set, both ends fail closed: the -# backend disables the internal routes and the daemon refuses to start. Defence -# in depth: Caddy never routes /internal/* to the backend anyway. -#THERMOGRAPH_INTERNAL_TOKEN= - # Base path the app is served under. # / -> app at the domain root (Thermograph owns the whole domain — # this is what the Caddyfile expects: thermograph.org proxies "/") @@ -159,6 +136,17 @@ THERMOGRAPH_BASE_URL=https://thermograph.org # can't reach a compose bridge gateway; leave this file's value as the compose # default. # +# WARNING -- this host's rendered value must name an address Postfix ACTUALLY +# LISTENS ON (`postconf -h inet_interfaces`), or mail fails at send time with +# connection-refused. prod's SOPS-rendered /etc/thermograph.env currently says +# 172.19.0.1, which is the *compose* bridge and is NOT in prod's +# inet_interfaces. It is inert today only because prod runs in stack mode, where +# thermograph-stack.yml sets THERMOGRAPH_SMTP_HOST per-service and nothing on +# the box reads this variable from here. It becomes live breakage the moment +# prod falls back to compose mode. Fix it in deploy/secrets/prod.yaml (sops) +# rather than on the box; 10.10.0.1 (the mesh address) is the durable choice, +# since it depends on wg0 rather than on Docker. +# # Backends: console (log it, send nothing — the default, right for dev), # smtp (actually send), disabled (drop silently). # Leave unset until Postfix is provisioned: signups are still collected either way. @@ -211,13 +199,12 @@ THERMOGRAPH_BASE_URL=https://thermograph.org # alone). Leave unset to disable. In the Thermograph.org server: # weather-events = 1529274516746932307 #THERMOGRAPH_DISCORD_WEATHER_CHANNEL= -# Discord gateway bot (@mention/DM grading): runs in the daemon service (the -# Go thermograph-daemon container — no longer inside the backend leader -# process). Opt-in: any truthy value here starts it, provided BOT_TOKEN above -# is also set (flag alone is a silent no-op). Discord allows ONE gateway -# connection per bot token, so enable this in exactly one environment — prod, -# the only vault with the token (the stack pins the daemon to 1 replica for -# the same reason). +# Discord gateway bot (@mention/DM grading): runs INSIDE the backend leader +# process (same singleton election as the notifier — see web/app.py's lifespan), +# riding its event loop; no separate bot process. Opt-in: any truthy value here +# starts it, provided BOT_TOKEN above is also set (flag alone is a silent no-op). +# Discord allows ONE gateway connection per bot token, so enable this in exactly +# one environment — prod, the only vault with the token. #THERMOGRAPH_DISCORD_BOT= # Account linking (OAuth2 "identify"): lets a signed-in user connect their Discord # account, storing their Discord user id for DM alerts. CLIENT_ID is the same App @@ -226,9 +213,8 @@ THERMOGRAPH_BASE_URL=https://thermograph.org #THERMOGRAPH_DISCORD_CLIENT_SECRET= # Gateway bot (opt-in): holds a live websocket so the bot replies to messages that # @mention it (or DM it) with a city grade — e.g. "@Thermograph Phoenix". Reuses -# THERMOGRAPH_DISCORD_BOT_TOKEN above. Runs in the single-replica daemon service -# (which also needs THERMOGRAPH_INTERNAL_TOKEN below — it grades via the -# backend's internal API). No privileged intent required — Discord delivers -# content for mentions/DMs. Unset/0 => no gateway connection. -# (Code: backend/daemon/, calling backend's /internal/discord/grade.) +# THERMOGRAPH_DISCORD_BOT_TOKEN above. Runs on the single notifier leader only, so +# it needs THERMOGRAPH_ENABLE_NOTIFIER on (the default). No privileged intent +# required — Discord delivers content for mentions/DMs. Unset/0 => no gateway +# connection. (Code: thermograph-backend notifications/discord_bot.py.) #THERMOGRAPH_DISCORD_BOT=1 diff --git a/observability/.env.example b/observability/.env.example index 60c6633..73b9f35 100644 --- a/observability/.env.example +++ b/observability/.env.example @@ -18,3 +18,14 @@ GF_SECURITY_ADMIN_PASSWORD=change-me-to-something-strong OAUTH_ENABLED=false GOOGLE_CLIENT_ID= GOOGLE_CLIENT_SECRET= + +# --- Alerting (Discord) ---------------------------------------------------------- +# Where every alert in grafana/provisioning/alerting/ is delivered: a webhook on +# the private #ops-alerts channel (Owners category) of the Thermograph.org +# Discord. NOT email — beta's Grafana relays SMTP through prod's Postfix, so +# email alerts die exactly when prod does. +# +# To mint one: Discord -> #ops-alerts -> Edit Channel -> Integrations -> +# Webhooks -> New Webhook -> Copy Webhook URL. Treat it like a password; anyone +# with it can post to the channel. Grafana refuses to start without it. +DISCORD_ALERT_WEBHOOK_URL=https://discord.com/api/webhooks// diff --git a/observability/CLAUDE.md b/observability/CLAUDE.md index b3e4898..dcca01e 100644 --- a/observability/CLAUDE.md +++ b/observability/CLAUDE.md @@ -16,6 +16,14 @@ the full per-node procedure. - `loki/config.yml` — Loki config (mesh-only, filesystem storage). - `grafana/provisioning/` — datasource (Loki) + dashboard provider, auto-loaded at startup. `grafana/dashboards/*.json` — the dashboards themselves. +- `grafana/provisioning/alerting/` — the alert rules, the Discord contact point + and the notification policy, also auto-loaded at startup. Same rule as the + dashboards: **the repo is the only durable path**, UI edits get overwritten. + Alerts go to Discord `#ops-alerts`, never email — beta's Grafana relays SMTP + through prod's Postfix, so email dies exactly when prod does. Every rule is + LogQL (there is no Prometheus anywhere in the fleet). Thresholds were derived + from real Loki data and the working is in the comments beside each rule — + re-derive before changing a number rather than guessing. - `alloy/config.alloy` + `alloy/docker-compose.agent.yml` — the per-node log shipper. The node name comes from `ALLOY_NODE` (set per host). - `caddy-grafana.conf` — the Caddy vhost for `dashboard.thermograph.org` (lives @@ -25,10 +33,16 @@ the full per-node procedure. ## Conventions -- **Every artifact that ships to a node is CI-validated** (`.forgejo/workflows/validate.yml`): - both compose files parse, all dashboard JSON is valid, and the Loki/provisioning - YAML parses. Keep new dashboards as valid JSON and new config as valid YAML or - CI fails. The Alloy config is not yet CI-validated (needs the `alloy` binary). +- **Every artifact that ships to a node is CI-validated** + (`.forgejo/workflows/observability-validate.yml`): both compose files parse, all + dashboard JSON is valid, and the Loki/provisioning YAML parses. Keep new + dashboards as valid JSON and new config as valid YAML or CI fails. The alerting + config gets a stricter third step — every rule's `condition` must name a refId + that exists, every policy must route to a receiver that exists, and a literal + Discord webhook URL in the repo is a hard failure. (A rule pointing at a missing + refId is valid YAML, provisions cleanly, and then never fires; that is precisely + the silent-no-op this whole domain exists to prevent.) The Alloy config is still + not CI-validated (needs the `alloy` binary). - The public hostname is **`dashboard.thermograph.org`** everywhere — not `grafana.thermograph.org`. Match it in any new comment/config. - Secrets (OAuth client id/secret, admin password) live only in the host `.env`, diff --git a/observability/README.md b/observability/README.md index c27b427..d0f9af5 100644 --- a/observability/README.md +++ b/observability/README.md @@ -31,6 +31,7 @@ persistent, queryable, fleet-wide log store and UI. | `docker-compose.yml` | the central Loki + Grafana stack (runs on beta) | | `loki/config.yml` | Loki config (filesystem, retention) | | `grafana/provisioning/` | auto-wired Loki datasource + dashboard provider | +| `grafana/provisioning/alerting/` | contact point, notification policy and alert rules — see [Alerting](#alerting) | | `grafana/dashboards/thermograph-logs.json` | the fleet-logs dashboard | | `alloy/config.alloy` | the per-node collector config | | `alloy/docker-compose.agent.yml` | runs the Alloy agent on a node | @@ -38,6 +39,22 @@ persistent, queryable, fleet-wide log store and UI. ## Deploy +> ### ⚠️ Merging is not deploying +> +> `/opt/observability` on beta (and prod) is a checkout of the **archived** +> `emi/thermograph-observability` repo. It can never `git pull` again — the +> content now lives here, in the mono repo, under `observability/`. A previous +> observability PR merged and had **zero live effect** for exactly this reason. +> +> Until someone re-points those checkouts at the mono repo, every change here +> ships by hand: back up the live file with a UTC-timestamped suffix, `scp` the +> new one up, `sudo cp` it into place, restart the specific container. Use +> `sudo docker restart ` — **not `docker compose`**, which on prod +> demands a `GF_SECURITY_ADMIN_PASSWORD` it has no `.env` to interpolate from. +> (Beta's stack does have an `.env`, so compose works there — and is required +> when a change adds a new environment variable, since `docker restart` alone +> will not pick one up.) + ### 1. Central stack (on beta) ```bash @@ -113,6 +130,161 @@ log volume by service, app error rate by tag, upstream 429 count, Caddy 5xx count, a per-node notifier-liveness indicator (from the heartbeat log), recent app errors, and a live all-container tail. Edit it in Grafana and re-export the JSON here to version a change. +## Alerting + +Provisioned from `grafana/provisioning/alerting/`, mounted read-only into Grafana +by the compose file, so **the repo is the only durable path** — as with the +dashboards, edits made in the Grafana UI are overwritten on the next provision. + +Every rule is **LogQL**, because there is no Prometheus in the fleet: Loki is the +only datasource, so every signal is derived from log lines. + +### Where alerts go: Discord, not email + +`#ops-alerts` (Owners category, private to Emi and Jin) via Grafana's native +Discord contact point. Deliberately **not email**: + +- Grafana's factory default pointed at the literal string `` — + a paging path that had never delivered a message to anyone. +- beta's Grafana relays SMTP through **prod's** Postfix at `10.10.0.1:25` (see + `docker-compose.override.yml` on beta). Email alerts therefore travel through + the box most likely to be on fire, and vanish exactly when they matter. +- Discord is off-estate, works when prod is dead, and reaches a phone. + +`#ops-alerts` is a new dedicated channel, not one of the existing ones: the +`#dev` / `#uat` / `#prod` channels are the *product's* alert-subscription output +(real deliveries — routing ops noise there would corrupt the evidence), and +`#notes` / `#staff-chat` are for humans. + +The webhook URL is a secret and is **not in this repo**: it comes from +`DISCORD_ALERT_WEBHOOK_URL` in beta's gitignored `.env`, which compose passes to +Grafana and which Grafana interpolates when reading the provisioning file. +Grafana refuses to start if it is unset. + +### The rules + +| Rule | Fires when | Severity | +|------|-----------|----------| +| `ProdEdgeDark` | zero prod Caddy log lines in 15m — site dark, or the log pipeline is broken | critical | +| `ProdHealthzFailing` | any non-200 on `/healthz` in 10m | critical | +| `ProdHTTP5xxBurst` | ≥10 prod 5xx in 10m | critical | +| `ProdHTTP5xxElevated` | ≥25 prod 5xx accumulated over 6h (the error-budget alarm) | warning | +| `ProdUnhandledExceptions` | ≥3 app `tag=http.error` in 30m | warning | +| `NotifierHeartbeatMissingProd` | no `tag=heartbeat` line from prod in 20m | critical | +| `NotifierHeartbeatMissingBeta` | same, on beta | warning | +| `ProdWebContainerSilent` | `thermograph_web` logged nothing in 30m | critical | +| `ProdDbContainerSilent` | `thermograph_db` logged nothing in 2h | critical | +| `ProdFrontendContainerSilent` | `thermograph_frontend` logged nothing in 30m | warning | +| `ProdWorkerContainerSilent` | `thermograph_worker` logged nothing in 2h | warning | +| `AlertingWatchdog` | always — one Discord message per day | watchdog | + +Every threshold was backtested against real Loki data; the working is written out +in the comments of `rules-prod.yml`, next to the rule it justifies. Read those +before changing a number. + +`AlertingWatchdog` is the dead-man's switch and fires permanently by design. **Its +value is its absence**: nothing can report its own death, so if `#ops-alerts` has +been silent for more than a day, alerting is broken rather than the estate +healthy. Set `isPaused: true` on it to opt out of that guarantee. + +### Deploying an alerting change to beta + +Grafana already mounts `./grafana/provisioning` read-only, so new files under +`alerting/` need no compose change — but the **first** deploy does, because it +introduces `DISCORD_ALERT_WEBHOOK_URL`, and `docker restart` will not pick up a +new environment variable. + +```bash +# run from the mono-repo root +BETA="agent@75.119.132.91" +SSH="ssh -i ~/.ssh/thermograph_agent_ed25519" +TS=$(date -u +%Y%m%dT%H%M%SZ) + +# 1. put the secret in beta's .env (once), then confirm it took: +$SSH $BETA "grep -q '^DISCORD_ALERT_WEBHOOK_URL=' /opt/observability/.env && echo present || echo MISSING" + +# 2. back up the live compose file, stage the new files, move them into place: +$SSH $BETA "sudo cp -a /opt/observability/docker-compose.yml /opt/observability/docker-compose.yml.$TS \ + && rm -rf /tmp/obs-stage && mkdir -p /tmp/obs-stage/alerting" +scp -i ~/.ssh/thermograph_agent_ed25519 observability/docker-compose.yml $BETA:/tmp/obs-stage/ +scp -i ~/.ssh/thermograph_agent_ed25519 observability/grafana/provisioning/alerting/*.yml $BETA:/tmp/obs-stage/alerting/ +$SSH $BETA "sudo mkdir -p /opt/observability/grafana/provisioning/alerting \ + && sudo cp /tmp/obs-stage/docker-compose.yml /opt/observability/ \ + && sudo cp /tmp/obs-stage/alerting/*.yml /opt/observability/grafana/provisioning/alerting/" + +# 3. recreate Grafana so it reads the new env var AND the new provisioning. +# `docker restart` is NOT enough here — it will not pick up a new env var. +$SSH $BETA "cd /opt/observability && docker compose up -d grafana" +``` + +Subsequent rule-only changes are just step 2 plus +`sudo docker restart observability-grafana-1` — no compose needed, because no +new environment variable is involved. + +**Rollback:** the provisioning files are additive. To back the whole thing out, +`sudo rm -rf /opt/observability/grafana/provisioning/alerting`, restore +`docker-compose.yml.`, and `docker compose up -d grafana`. Grafana +falls back to its default (email → ``) root policy — i.e. no +alerting, which is where this started. + +Alerting lives on **beta only** — prod and dev run Alloy agents, not Grafana, so +nothing in `alerting/` ever ships to them. + +### Verify alerting + +```bash +# on beta, after deploying and restarting Grafana: +GFP=$(grep '^GF_SECURITY_ADMIN_PASSWORD=' /opt/observability/.env | cut -d= -f2-) +WH=$(grep '^DISCORD_ALERT_WEBHOOK_URL=' /opt/observability/.env | cut -d= -f2-) +# 1. all 12 rules loaded and healthy (health should be "ok", never "error"): +curl -s -u admin:"$GFP" localhost:3000/api/prometheus/grafana/api/v1/rules \ + | python3 -c 'import json,sys; [print(r["name"], r["state"], r["health"]) for g in json.load(sys.stdin)["data"]["groups"] for r in g["rules"]]' +# 2. the root policy points at Discord, NOT grafana-default-email: +curl -s -u admin:"$GFP" localhost:3000/api/v1/provisioning/policies +# 3. force a real message into #ops-alerts (does not persist anything): +curl -s -u admin:"$GFP" -H 'Content-Type: application/json' \ + -X POST localhost:3000/api/alertmanager/grafana/config/api/v1/receivers/test \ + -d '{"receivers":[{"name":"t","grafana_managed_receiver_configs":[{"name":"t","type":"discord","settings":{"url":"'"$WH"'"}}]}]}' +# -> {"receivers":[{... "status":"ok"}]} and a message in #ops-alerts. +``` + +The webhook URL is redacted (`[REDACTED]`) by +`GET /api/v1/provisioning/contact-points`, so you **cannot** confirm +interpolation by reading it back — the only proof is a message actually arriving +in `#ops-alerts`. A contact point holding the literal string +`$DISCORD_ALERT_WEBHOOK_URL` looks completely healthy from the API and pages +nobody. + +### The leftover default contact point + +Grafana's built-in `grafana-default-email` → `` receiver is +created by Grafana's *default Alertmanager config*, not by provisioning, and its +uid is the empty string. `deleteContactPoints` matches on uid, so **provisioning +cannot remove it** (`DELETE /api/v1/provisioning/contact-points/` with an empty +uid returns 404 — verified). + +Provisioning the notification policy already makes it inert: nothing routes there +any more, so it cannot receive anything. Actually deleting the row is a one-time +manual step, and only works *after* this config is live (Grafana refuses to +delete a receiver a policy still references). Either use the UI — Alerting → +Contact points → `grafana-default-email` → Delete — or: + +```bash +# on beta. Reads the running Alertmanager config, drops the one receiver, +# writes it back. Verified against Grafana 11.6.1 (returns 202). +GFP=$(grep '^GF_SECURITY_ADMIN_PASSWORD=' /opt/observability/.env | cut -d= -f2-) +AM=localhost:3000/api/alertmanager/grafana/config/api/v1/alerts +curl -s -u admin:"$GFP" $AM > /tmp/am.json +python3 -c " +import json; d=json.load(open('/tmp/am.json')); c=d['alertmanager_config'] +c['receivers']=[r for r in c['receivers'] if r['name']!='grafana-default-email'] +json.dump(d, open('/tmp/am2.json','w'))" +curl -s -u admin:"$GFP" -H 'Content-Type: application/json' -X POST --data @/tmp/am2.json $AM +rm -f /tmp/am.json /tmp/am2.json +``` + +This is cosmetic — leaving it costs nothing but a confusing row in the UI. + ## Notes / follow-ups - **Loki is mesh-only** by design; only Grafana (with auth) is public. @@ -121,3 +293,32 @@ tail. Edit it in Grafana and re-export the JSON here to version a change. - The app emits a `tag=heartbeat` line for the subscription notifier every ~15m; the dashboard's "Notifier alive?" panel turns red if a node misses it. Extend the same `audit.log_heartbeat()` to any future background daemon. + +### Known gaps + +- **Alloy can silently stop tailing a container after Swarm replaces the task.** + Observed on prod on 2026-07-24: `thermograph_worker` was `1/1`, healthy, and + writing ~19 lines per 10 minutes to its own docker log, yet not one line had + reached Loki since 04:43Z — a 13½-hour hole nobody could see, because nothing + was watching. `ProdWorkerContainerSilent` now catches exactly this. + What localises it to Alloy rather than the app: prod's `tag=heartbeat` lines + kept arriving in *every* 20-minute window throughout, and those come from a + different Alloy source (`loki.source.file` on `/applogs`) than the container + stdout (`loki.source.docker`). One source went deaf, the other did not. + The workaround is `sudo docker restart alloy-alloy-1` on the affected node; + the fix is probably an explicit `refresh_interval` on `discovery.docker` plus + a look at whether `loki.source.docker` re-resolves targets. Not root-caused. +- **Some services are too quiet to alert on.** `thermograph_daemon` (23 log + lines/24h), `thermograph_autoscaler` (1) and `thermograph_autoscaler-lake` (2) + have a healthy floor indistinguishable from dead, and `thermograph_lake` is + bursty ETL with legitimate idle gaps. Giving each one an + `audit.log_heartbeat()` line would make all four alertable with the same + pattern the notifier already uses. That is the single highest-value follow-up + here. +- **Grafana cannot alert on its own death** — that is what `AlertingWatchdog` + is for, but it is a *manual* dead-man's switch: it relies on somebody noticing + that the daily message stopped. A real external watchdog (an uptime pinger + hitting a Grafana endpoint from off-estate) would close this properly. +- **beta runs a `docker-compose.override.yml` that is not in this repo**, wiring + Grafana's SMTP to prod's Postfix. It should be mirrored into the tracked + compose file or deleted; alerting no longer depends on it. diff --git a/observability/docker-compose.yml b/observability/docker-compose.yml index ff9dd6b..06b3bc5 100644 --- a/observability/docker-compose.yml +++ b/observability/docker-compose.yml @@ -58,6 +58,14 @@ services: # verifies email ownership and it's the only OAuth provider, so there's no # cross-provider takeover vector the "insecure" name warns about. GF_AUTH_OAUTH_ALLOW_INSECURE_EMAIL_LOOKUP: "true" + # --- Alerting -------------------------------------------------------------- + # The Discord webhook that grafana/provisioning/alerting/contact-points.yml + # interpolates as $DISCORD_ALERT_WEBHOOK_URL. It is a secret (holding it is + # enough to post in the channel), so it lives only in beta's .env. + # Required, not defaulted: a Grafana that comes up with an empty webhook + # looks perfectly healthy and pages nobody, which is the failure mode this + # whole config exists to end. Better to refuse to start. + DISCORD_ALERT_WEBHOOK_URL: ${DISCORD_ALERT_WEBHOOK_URL:?set DISCORD_ALERT_WEBHOOK_URL — see .env.example} volumes: - ./grafana/provisioning:/etc/grafana/provisioning:ro - ./grafana/dashboards:/var/lib/grafana/dashboards:ro diff --git a/observability/grafana/provisioning/alerting/contact-points.yml b/observability/grafana/provisioning/alerting/contact-points.yml new file mode 100644 index 0000000..42f1f10 --- /dev/null +++ b/observability/grafana/provisioning/alerting/contact-points.yml @@ -0,0 +1,64 @@ +# Where alerts go. ONE destination: a Discord webhook posting into the private +# #ops-alerts channel (Owners category) of the Thermograph.org server. +# +# WHY NOT EMAIL. The Grafana factory default routes to the literal string +# "", i.e. nowhere. Fixing it by pointing at a real mailbox +# would still be wrong: beta's Grafana relays SMTP through prod's Postfix at +# 10.10.0.1:25 (see the live docker-compose.override.yml on beta), so email +# alerts travel *through the box most likely to be on fire* and are silently +# lost whenever Postfix is down — which is exactly when you need them. Discord +# is off-estate: it works when prod is dead, and it works from a phone. +# +# THE WEBHOOK URL IS A SECRET. Anyone holding it can post into the channel, so +# it is NOT in this repo. It is read from the Grafana process environment, which +# docker-compose.yml feeds from beta's gitignored .env (see .env.example). +# Grafana expands $VAR / $__env{VAR} when it reads provisioning files; if the +# variable is unset the contact point ends up with a literal "$DISCORD_..." +# string and every notification fails — see README "Verify alerting". +# +# To rotate: make a new webhook on the #ops-alerts channel, replace the value in +# beta's .env, `docker restart observability-grafana-1`, delete the old webhook +# in Discord. +apiVersion: 1 + +contactPoints: + - orgId: 1 + name: thermograph-ops-discord + receivers: + - uid: tg_ops_discord + type: discord + # Send a green "resolved" message too — a page you never see close is a + # page you stop trusting. + disableResolveMessage: false + settings: + url: $DISCORD_ALERT_WEBHOOK_URL + use_discord_username: false + title: '{{ if eq .Status "firing" }}[FIRING]{{ else }}[RESOLVED]{{ end }} {{ .CommonLabels.alertname }}' + # Deliberately plain. A template error here breaks EVERY notification + # silently, so this uses only functions verified against Grafana + # 11.6.1 via /api/alertmanager/grafana/config/api/v1/receivers/test. + message: |- + {{ range .Alerts }}**severity:** {{ .Labels.severity }} · **host:** {{ .Labels.host }} + {{ .Annotations.summary }} + {{ .Annotations.description }} + `value: {{ .ValueString }}` + {{ end }} + + +# --- The factory-default contact point ------------------------------------------- +# Grafana ships a built-in receiver "grafana-default-email" whose only integration +# is `email receiver` -> "". It is created by Grafana's default +# Alertmanager config, NOT by provisioning, and its uid is the empty string: +# +# GET /api/v1/provisioning/contact-points +# -> [{"name":"email receiver","type":"email","settings":{...},"uid":""}] +# +# Because `deleteContactPoints` matches on uid, provisioning CANNOT remove it — +# DELETE /api/v1/provisioning/contact-points/ with an empty uid 404s (verified +# against 11.6.1). What notification-policies.yml *does* do is stop anything ever +# routing to it, which makes it inert: it can no longer receive anything. +# +# Deleting the row itself is cosmetic and manual, and only works once no policy +# references it (i.e. after this config is live) — either the UI (Alerting -> +# Contact points -> grafana-default-email -> Delete) or the Alertmanager config +# API. The README's "The leftover default contact point" has the exact command. diff --git a/observability/grafana/provisioning/alerting/notification-policies.yml b/observability/grafana/provisioning/alerting/notification-policies.yml new file mode 100644 index 0000000..6959f99 --- /dev/null +++ b/observability/grafana/provisioning/alerting/notification-policies.yml @@ -0,0 +1,64 @@ +# The routing tree. Provisioning `policies:` REPLACES Grafana's root policy, so +# applying this file is what actually severs the factory default +# (`receiver: grafana-default-email` -> "") and points the +# whole estate at Discord. Everything lands in the same #ops-alerts channel; the +# severity branches differ only in how *insistent* they are. +# +# Timing rationale, for a solo operator on a phone: +# critical — 10s wait so a burst arrives as one message, re-nag hourly while +# it is still broken. You want to be pestered about a dead prod. +# warning — 2m wait (lets a flapping thing settle), re-nag twice a day. +# Not worth waking up for; worth not forgetting. +# watchdog — the dead-man's-switch. Fires forever by design, so it is throttled +# to one message per day. Its VALUE IS ITS ABSENCE: if #ops-alerts +# is silent for >24h, Grafana/Loki/the mesh is down and no other +# alert in this file can reach you either. +# +# There is deliberately no catch-all branch: an alert whose `severity` label is +# missing or unrecognised matches none of the routes and falls through to the +# ROOT receiver, which is also Discord. Nothing can be silently swallowed by a +# typo in a label — it just arrives on the default (4h) cadence instead. +apiVersion: 1 + +policies: + - orgId: 1 + receiver: thermograph-ops-discord + # Collapse by rule + host so a whole-node outage (every prod rule firing at + # once) still arrives as a handful of messages, not thirty. + group_by: + - alertname + - host + group_wait: 30s + group_interval: 5m + repeat_interval: 4h + + routes: + - receiver: thermograph-ops-discord + object_matchers: + - ['severity', '=', 'watchdog'] + group_wait: 0s + group_interval: 24h + repeat_interval: 24h + continue: false + + - receiver: thermograph-ops-discord + object_matchers: + - ['severity', '=', 'critical'] + group_by: + - alertname + - host + group_wait: 10s + group_interval: 5m + repeat_interval: 1h + continue: false + + - receiver: thermograph-ops-discord + object_matchers: + - ['severity', '=', 'warning'] + group_by: + - alertname + - host + group_wait: 2m + group_interval: 10m + repeat_interval: 12h + continue: false diff --git a/observability/grafana/provisioning/alerting/rules-prod.yml b/observability/grafana/provisioning/alerting/rules-prod.yml new file mode 100644 index 0000000..ec2b3bb --- /dev/null +++ b/observability/grafana/provisioning/alerting/rules-prod.yml @@ -0,0 +1,836 @@ +# Thermograph alert rules — the estate's first. All Loki/LogQL, because there is +# no Prometheus anywhere in the fleet: Loki is the only datasource, so every +# signal here is derived from log lines rather than metrics. +# +# --------------------------------------------------------------------------- +# HOW THE THRESHOLDS WERE CHOSEN +# --------------------------------------------------------------------------- +# Every number below was backtested against the real 24h ending 2026-07-24 +# ~19:45Z. That window contains a genuine ~20-minute prod outage starting +# 2026-07-24 04:20Z, which is the event these rules must catch: +# +# 10-min buckets of prod Caddy 5xx over 24h (only non-zero buckets shown): +# 2, 45, 22, 3, 1, 1, 1, 1, 1, 1, 2, 1, 5, 1 (total 86) +# ^^^^^^ the outage: 64% and 56% of all requests 5xx +# All five /healthz failures in the whole day fell inside those two buckets. +# Prod Caddy request volume: 24-170 per 10 min (~7 req/min). Low traffic — +# which is why absolute counts beat error *ratios* here: at 7 req/min a single +# bad request is 1.4% and a ratio alert would scream all night. +# +# The design goal was: fire on the outage, stay silent for the other 142 of 144 +# buckets, and still surface the slow drip of ~20 scattered errors that made up +# the rest of the day's 86 (that is what ProdHTTP5xxElevated is for). +# +# --------------------------------------------------------------------------- +# TWO CONVENTIONS WORTH KNOWING +# --------------------------------------------------------------------------- +# 1. `or vector(0)`. An absent Loki stream returns *no series*, not zero — so a +# naive "count < 1" silence alert goes to NoData instead of firing, which is +# the exact failure mode of every homemade log alert ever written. Appending +# `or vector(0)` makes an empty result evaluate to a real 0. Verified working +# on Loki 3.5.1. +# 2. Prod addressing. Prod is Swarm, and its docker streams carry NO `service` +# label — only `container`, which includes a per-task suffix that changes on +# every redeploy (thermograph_web.1.). So prod services are matched as +# container=~"thermograph_.+" with job="docker". Do not use `service=` +# here; it only works on the compose hosts (beta, dev). +# +# --------------------------------------------------------------------------- +# NOT ALERTABLE, ON PURPOSE +# --------------------------------------------------------------------------- +# thermograph_daemon (23 log lines/24h), thermograph_autoscaler (1) and +# thermograph_autoscaler-lake (2) are too quiet for a log-silence rule — their +# healthy floor is indistinguishable from dead. thermograph_lake is bursty ETL +# with long legitimate idle gaps. Catching those needs either a heartbeat line +# (extend backend/core/audit.py log_heartbeat(), as the notifier already does) +# or container metrics, which the fleet does not collect. Tracked as a gap. +# +# Error budget / failure isolation: only ProdEdgeDark escalates on NoData or +# datasource errors, so a Loki or WireGuard blip produces ONE page rather than +# twelve. Every other rule degrades to OK if the query cannot run. +apiVersion: 1 + +groups: + # =========================================================================== + # Is the product up? Evaluated every minute; all of these page. + # =========================================================================== + - orgId: 1 + name: prod-availability + folder: Alerts + interval: 1m + rules: + + # -- 1 ----------------------------------------------------------------- + # The top-level "is anything alive" check, and deliberately the only rule + # that also owns NoData/error: Caddy fronts every request to + # thermograph.org, so zero access-log lines means the site is dark, the + # box is gone, Alloy died, or the mesh dropped. Any of those is a page. + # + # DATA: 15-minute windows over 24h ranged 35 -> 301 lines, minimum 35, + # and never once reached zero — including *during* the outage (72), when + # Caddy was happily logging 503s. Zero is unambiguous. + - uid: tg_prod_edge_dark + title: ProdEdgeDark + condition: C + for: 5m + noDataState: Alerting + execErrState: Alerting + annotations: + summary: 'prod: no Caddy access-log lines for 15 minutes' + description: >- + Nothing has reached thermograph.org's reverse proxy in 15 minutes + (healthy floor is 35 lines/15m, and it stayed at 72 even during the + 04:20Z outage). Either prod is down, Caddy is down, prod's Alloy + agent stopped shipping, or the WireGuard mesh to beta is broken. + This rule also fires on Loki NoData/errors, so it doubles as the + "we have lost observability" alarm — if it is the only thing firing, + suspect the pipeline before suspecting the app. + labels: + severity: critical + host: prod + class: availability + isPaused: false + data: + - refId: A + relativeTimeRange: { from: 900, to: 0 } + datasourceUid: loki + model: + refId: A + datasource: { type: loki, uid: loki } + expr: 'sum(count_over_time({job="caddy", host="prod"} [15m])) or vector(0)' + queryType: instant + editorMode: code + intervalMs: 1000 + maxDataPoints: 43200 + - refId: B + relativeTimeRange: { from: 0, to: 0 } + datasourceUid: __expr__ + model: + refId: B + type: reduce + reducer: last + expression: A + datasource: { type: __expr__, uid: __expr__ } + - refId: C + relativeTimeRange: { from: 0, to: 0 } + datasourceUid: __expr__ + model: + refId: C + type: threshold + expression: B + datasource: { type: __expr__, uid: __expr__ } + conditions: + - evaluator: { type: lt, params: [1] } + + # -- 2 ----------------------------------------------------------------- + # /healthz is the app's own statement that it is well. When it 503s, it is + # not an opinion. + # + # DATA: 57 /healthz requests in 24h (52x 200, 5x 503) — roughly 2.4/hour, + # so this endpoint is polled sparsely and irregularly. That is why `for` + # is 0m: waiting for a second consecutive bad sample could take 25 + # minutes. All 5 failures landed inside the two outage buckets and nowhere + # else in the day, so one failure is already a real signal, not noise. + - uid: tg_prod_healthz_failing + title: ProdHealthzFailing + condition: C + for: 0m + noDataState: OK + execErrState: OK + annotations: + summary: 'prod: /healthz returned a non-200' + description: >- + The app's own liveness endpoint failed at the edge in the last 10 + minutes. Over the backtested 24h this happened 5 times, all of them + inside the 04:20Z outage — /healthz failing has a 100% correlation + with prod actually being broken. Check `docker service ps + thermograph_web` on prod first. + labels: + severity: critical + host: prod + class: availability + isPaused: false + data: + - refId: A + relativeTimeRange: { from: 600, to: 0 } + datasourceUid: loki + model: + refId: A + datasource: { type: loki, uid: loki } + expr: 'sum(count_over_time({job="caddy", host="prod"} | json | request_uri="/healthz" | status!="200" [10m])) or vector(0)' + queryType: instant + editorMode: code + intervalMs: 1000 + maxDataPoints: 43200 + - refId: B + relativeTimeRange: { from: 0, to: 0 } + datasourceUid: __expr__ + model: + refId: B + type: reduce + reducer: last + expression: A + datasource: { type: __expr__, uid: __expr__ } + - refId: C + relativeTimeRange: { from: 0, to: 0 } + datasourceUid: __expr__ + model: + refId: C + type: threshold + expression: B + datasource: { type: __expr__, uid: __expr__ } + conditions: + - evaluator: { type: gt, params: [0] } + + # -- 3 ----------------------------------------------------------------- + # The acute "users are seeing errors right now" page. + # + # DATA: of 144 ten-minute buckets in the backtest, 130 had zero 5xx and + # the largest *benign* bucket had 5. The outage produced 45 and 22. + # Threshold 10 (evaluator gt 9, integer counts) sits at 2x the worst + # benign bucket and 4.5x below the incident peak — it fires on the outage + # and on nothing else in the entire day. It catches 67 of that day's 86 + # errors as a single incident; the remaining scattered ~20 are picked up + # by ProdHTTP5xxElevated below rather than by pushing this threshold down + # into the noise. + - uid: tg_prod_5xx_burst + title: ProdHTTP5xxBurst + condition: C + for: 2m + noDataState: OK + execErrState: OK + annotations: + summary: 'prod: 10+ HTTP 5xx in the last 10 minutes' + description: >- + Prod is serving server errors at an outage-grade rate. Baseline is + zero in 90% of 10-minute windows and never above 5; the reference + incident hit 45. Look at the "Thermograph — Traffic & Domains" + dashboard, panel "HTTP errors by status", and at + {job="app-json", host="prod", filename=~".+/errors/.+"} for the + exceptions behind them. + labels: + severity: critical + host: prod + class: errors + isPaused: false + data: + - refId: A + relativeTimeRange: { from: 600, to: 0 } + datasourceUid: loki + model: + refId: A + datasource: { type: loki, uid: loki } + expr: 'sum(count_over_time({job="caddy", host="prod"} | json | status>=500 [10m])) or vector(0)' + queryType: instant + editorMode: code + intervalMs: 1000 + maxDataPoints: 43200 + - refId: B + relativeTimeRange: { from: 0, to: 0 } + datasourceUid: __expr__ + model: + refId: B + type: reduce + reducer: last + expression: A + datasource: { type: __expr__, uid: __expr__ } + - refId: C + relativeTimeRange: { from: 0, to: 0 } + datasourceUid: __expr__ + model: + refId: C + type: threshold + expression: B + datasource: { type: __expr__, uid: __expr__ } + conditions: + - evaluator: { type: gt, params: [9] } + + # =========================================================================== + # Slow-burn error signals. Five-minute evaluation is plenty; these are tickets, + # not pages. + # =========================================================================== + - orgId: 1 + name: prod-errors + folder: Alerts + interval: 5m + rules: + + # -- 4 ----------------------------------------------------------------- + # The rule that would have broken the silence on "84 5xx in 24h and nobody + # was told". ProdHTTP5xxBurst only sees spikes; this sees accumulation. + # + # DATA: 6-hour rolling windows sampled hourly across the backtest read + # 1, 2, 2, 2, 2, 2, 2, 67, 72, 73, 74, 74, 74, 7, 2, 1, 1, 4, 11. + # The largest window containing NO incident was 11. Threshold 25 + # (evaluator gt 24) is 2.3x that — quiet on the drip alone, loud on any + # day that looks like this one. Deliberately a long window: it will stay + # firing for ~6h after an incident, which is the point. That is the daily + # error budget talking, not a flap. + - uid: tg_prod_5xx_elevated + title: ProdHTTP5xxElevated + condition: C + for: 10m + noDataState: OK + execErrState: OK + annotations: + summary: 'prod: 25+ HTTP 5xx accumulated over 6 hours' + description: >- + Sustained server errors. This is the error-budget alarm, not an + outage alarm — it catches the scattered drip that never trips the + burst rule but still added up to 86 errors (1.7% of requests) on + 2026-07-24. Expect it to stay firing for up to 6h after an incident + clears; that is the window emptying, not a new fault. + labels: + severity: warning + host: prod + class: errors + isPaused: false + data: + - refId: A + relativeTimeRange: { from: 21600, to: 0 } + datasourceUid: loki + model: + refId: A + datasource: { type: loki, uid: loki } + expr: 'sum(count_over_time({job="caddy", host="prod"} | json | status>=500 [6h])) or vector(0)' + queryType: instant + editorMode: code + intervalMs: 1000 + maxDataPoints: 43200 + - refId: B + relativeTimeRange: { from: 0, to: 0 } + datasourceUid: __expr__ + model: + refId: B + type: reduce + reducer: last + expression: A + datasource: { type: __expr__, uid: __expr__ } + - refId: C + relativeTimeRange: { from: 0, to: 0 } + datasourceUid: __expr__ + model: + refId: C + type: threshold + expression: B + datasource: { type: __expr__, uid: __expr__ } + conditions: + - evaluator: { type: gt, params: [24] } + + # -- 5 ----------------------------------------------------------------- + # A different signal from the Caddy 5xx rules, not a duplicate: Caddy sees + # 502/503 (the app is unreachable), while tag="http.error" is the app + # catching an unhandled exception and still answering. A bad deploy shows + # up here first, and can do so without moving the Caddy numbers at all. + # + # DATA: exactly 2 unhandled 500s in 24h, never more than 1 in any hour. + # Threshold 3-per-30min (evaluator gt 2) is 3x the observed daily peak: + # silent today, loud on an exception storm. + - uid: tg_prod_unhandled_exceptions + title: ProdUnhandledExceptions + condition: C + for: 0m + noDataState: OK + execErrState: OK + annotations: + summary: 'prod: 3+ unhandled exceptions in 30 minutes' + description: >- + The app is throwing. Normal is under 2 per DAY. A cluster this tight + almost always means a regression just shipped — check the app.start + lines on the "App Activity & Delivery" dashboard for a recent deploy, + then {job="app-json", host="prod", filename=~".+/errors/.+"} | + tag="http.error" for the traceback and route. + labels: + severity: warning + host: prod + class: errors + isPaused: false + data: + - refId: A + relativeTimeRange: { from: 1800, to: 0 } + datasourceUid: loki + model: + refId: A + datasource: { type: loki, uid: loki } + expr: 'sum(count_over_time({job="app-json", host="prod", filename=~".+/errors/.+"} | tag="http.error" [30m])) or vector(0)' + queryType: instant + editorMode: code + intervalMs: 1000 + maxDataPoints: 43200 + - refId: B + relativeTimeRange: { from: 0, to: 0 } + datasourceUid: __expr__ + model: + refId: B + type: reduce + reducer: last + expression: A + datasource: { type: __expr__, uid: __expr__ } + - refId: C + relativeTimeRange: { from: 0, to: 0 } + datasourceUid: __expr__ + model: + refId: C + type: threshold + expression: B + datasource: { type: __expr__, uid: __expr__ } + conditions: + - evaluator: { type: gt, params: [2] } + + # =========================================================================== + # The subscription notifier — the thing that actually delivers the product. + # =========================================================================== + - orgId: 1 + name: notifier + folder: Alerts + interval: 5m + rules: + + # -- 6 ----------------------------------------------------------------- + # backend/core/audit.py emits a tag=heartbeat line every ~15 minutes. If it + # stops, subscribers stop getting weather alerts and absolutely nothing + # else in this file would notice — the site stays up and serves 200s while + # the product quietly stops working. + # + # DATA: prod produced 1-2 beats in every single 20-minute bucket across + # the full 24h backtest. Not one empty bucket. `for: 5m` at a 5m interval + # means two consecutive empty evaluations, so roughly 25 minutes of real + # silence before it pages — about one and a half missed beats. + # + # dev is deliberately excluded: the LAN box had zero heartbeats for the + # first 10.5h of the backtest window because it was simply off. Alerting + # on dev would mean an alert that is firing more often than not. + - uid: tg_notifier_heartbeat_prod + title: NotifierHeartbeatMissingProd + condition: C + for: 5m + noDataState: OK + execErrState: OK + annotations: + summary: 'prod: no notifier heartbeat for 20 minutes' + description: >- + The subscription notifier has gone quiet on prod. Expected cadence is + ~15 minutes and it did not miss a single 20-minute window over the + backtested day. Subscribers are not being notified right now, even + though the website is probably still serving fine. Check the + thermograph_worker service and the "Notifier alive?" panel on the + Fleet Logs dashboard. + labels: + severity: critical + host: prod + class: product + isPaused: false + data: + - refId: A + relativeTimeRange: { from: 1200, to: 0 } + datasourceUid: loki + model: + refId: A + datasource: { type: loki, uid: loki } + expr: 'sum(count_over_time({job="app-json", tag="heartbeat", host="prod"} [20m])) or vector(0)' + queryType: instant + editorMode: code + intervalMs: 1000 + maxDataPoints: 43200 + - refId: B + relativeTimeRange: { from: 0, to: 0 } + datasourceUid: __expr__ + model: + refId: B + type: reduce + reducer: last + expression: A + datasource: { type: __expr__, uid: __expr__ } + - refId: C + relativeTimeRange: { from: 0, to: 0 } + datasourceUid: __expr__ + model: + refId: C + type: threshold + expression: B + datasource: { type: __expr__, uid: __expr__ } + conditions: + - evaluator: { type: lt, params: [1] } + + # -- 7 ----------------------------------------------------------------- + # Same signal on beta. Beta was equally gap-free over the backtest (1-2 + # beats every 20m, no empty buckets), but beta is UAT — worth knowing, + # not worth a page. + - uid: tg_notifier_heartbeat_beta + title: NotifierHeartbeatMissingBeta + condition: C + for: 15m + noDataState: OK + execErrState: OK + annotations: + summary: 'beta: no notifier heartbeat for 20 minutes' + description: >- + The notifier has gone quiet on beta (UAT). Same signal as the prod + rule but a longer `for`, because beta gets redeployed all day and a + short gap is usually just a release. + labels: + severity: warning + host: beta + class: product + isPaused: false + data: + - refId: A + relativeTimeRange: { from: 1200, to: 0 } + datasourceUid: loki + model: + refId: A + datasource: { type: loki, uid: loki } + expr: 'sum(count_over_time({job="app-json", tag="heartbeat", host="beta"} [20m])) or vector(0)' + queryType: instant + editorMode: code + intervalMs: 1000 + maxDataPoints: 43200 + - refId: B + relativeTimeRange: { from: 0, to: 0 } + datasourceUid: __expr__ + model: + refId: B + type: reduce + reducer: last + expression: A + datasource: { type: __expr__, uid: __expr__ } + - refId: C + relativeTimeRange: { from: 0, to: 0 } + datasourceUid: __expr__ + model: + refId: C + type: threshold + expression: B + datasource: { type: __expr__, uid: __expr__ } + conditions: + - evaluator: { type: lt, params: [1] } + + # =========================================================================== + # Per-container liveness, inferred from log volume. The fleet collects no + # container metrics, so "did this service stop producing any output at all" is + # the only death certificate available — and it turns out to be a good one, + # because every service below logs on a floor that never approaches zero. + # + # Caveat worth knowing before you trust these: a container can be running and + # logging while its lines never reach Loki, if the node's Alloy agent stops + # tailing it. That is a real observed failure on prod (see README). These + # rules cannot tell the two apart — they say "no logs", which is the honest + # claim. Either way something needs looking at. + # =========================================================================== + - orgId: 1 + name: prod-services + folder: Alerts + interval: 5m + rules: + + # -- 8 ----------------------------------------------------------------- + # DATA: 15-minute windows over 24h ranged 54 -> 954 lines. The minimum, 54, + # occurred during the outage; the quietest healthy window was ~150. A + # 30-minute window has never been below ~110. Zero means the container is + # gone. This is the app itself, so it pages. + - uid: tg_prod_web_silent + title: ProdWebContainerSilent + condition: C + for: 5m + noDataState: OK + execErrState: OK + annotations: + summary: 'prod: thermograph_web has logged nothing for 30 minutes' + description: >- + The main app container has produced zero log lines in 30 minutes. + Healthy floor is ~110 lines per 30m and it never dipped below 54 even + mid-outage, so this is a container that is gone, wedged, or no longer + being tailed by prod's Alloy agent. Run `docker service ps + thermograph_web` on prod. + labels: + severity: critical + host: prod + class: liveness + isPaused: false + data: + - refId: A + relativeTimeRange: { from: 1800, to: 0 } + datasourceUid: loki + model: + refId: A + datasource: { type: loki, uid: loki } + expr: 'sum(count_over_time({job="docker", host="prod", container=~"thermograph_web.+"} [30m])) or vector(0)' + queryType: instant + editorMode: code + intervalMs: 1000 + maxDataPoints: 43200 + - refId: B + relativeTimeRange: { from: 0, to: 0 } + datasourceUid: __expr__ + model: + refId: B + type: reduce + reducer: last + expression: A + datasource: { type: __expr__, uid: __expr__ } + - refId: C + relativeTimeRange: { from: 0, to: 0 } + datasourceUid: __expr__ + model: + refId: C + type: threshold + expression: B + datasource: { type: __expr__, uid: __expr__ } + conditions: + - evaluator: { type: lt, params: [1] } + + # -- 9 ----------------------------------------------------------------- + # DATA: the steadiest series in the fleet — 2-hour windows read 54 to 69 + # lines, all 24 hours, with no trend and no gaps. TimescaleDB chatters at a + # metronomic ~30 lines/hour. Zero over two hours is as close to a + # definitive "Postgres is not there" as logs can give. + - uid: tg_prod_db_silent + title: ProdDbContainerSilent + condition: C + for: 10m + noDataState: OK + execErrState: OK + annotations: + summary: 'prod: thermograph_db has logged nothing for 2 hours' + description: >- + The production TimescaleDB container has produced zero log lines in + 2 hours. Its healthy range is a metronomic 54-69 lines per 2h with no + observed gaps, so zero is not quiet — it is absent. Everything else + in the app depends on this. + labels: + severity: critical + host: prod + class: liveness + isPaused: false + data: + - refId: A + relativeTimeRange: { from: 7200, to: 0 } + datasourceUid: loki + model: + refId: A + datasource: { type: loki, uid: loki } + expr: 'sum(count_over_time({job="docker", host="prod", container=~"thermograph_db.+"} [2h])) or vector(0)' + queryType: instant + editorMode: code + intervalMs: 1000 + maxDataPoints: 43200 + - refId: B + relativeTimeRange: { from: 0, to: 0 } + datasourceUid: __expr__ + model: + refId: B + type: reduce + reducer: last + expression: A + datasource: { type: __expr__, uid: __expr__ } + - refId: C + relativeTimeRange: { from: 0, to: 0 } + datasourceUid: __expr__ + model: + refId: C + type: threshold + expression: B + datasource: { type: __expr__, uid: __expr__ } + conditions: + - evaluator: { type: lt, params: [1] } + + # -- 10 ---------------------------------------------------------------- + # DATA: 15-minute windows ranged 33 -> 197, never zero. Warning rather than + # critical because Caddy serves the static build and ProdEdgeDark / + # ProdHTTP5xxBurst will catch it first if users are actually affected. + - uid: tg_prod_frontend_silent + title: ProdFrontendContainerSilent + condition: C + for: 5m + noDataState: OK + execErrState: OK + annotations: + summary: 'prod: thermograph_frontend has logged nothing for 30 minutes' + description: >- + The frontend container has produced zero log lines in 30 minutes + (healthy floor ~66 per 30m). If users were visibly affected you would + expect ProdHTTP5xxBurst alongside this; on its own it more often + means a stuck redeploy or a lost Alloy tail. + labels: + severity: warning + host: prod + class: liveness + isPaused: false + data: + - refId: A + relativeTimeRange: { from: 1800, to: 0 } + datasourceUid: loki + model: + refId: A + datasource: { type: loki, uid: loki } + expr: 'sum(count_over_time({job="docker", host="prod", container=~"thermograph_frontend.+"} [30m])) or vector(0)' + queryType: instant + editorMode: code + intervalMs: 1000 + maxDataPoints: 43200 + - refId: B + relativeTimeRange: { from: 0, to: 0 } + datasourceUid: __expr__ + model: + refId: B + type: reduce + reducer: last + expression: A + datasource: { type: __expr__, uid: __expr__ } + - refId: C + relativeTimeRange: { from: 0, to: 0 } + datasourceUid: __expr__ + model: + refId: C + type: threshold + expression: B + datasource: { type: __expr__, uid: __expr__ } + conditions: + - evaluator: { type: lt, params: [1] } + + # -- 11 ---------------------------------------------------------------- + # DATA: a dead-steady 121 lines/hour (its own healthcheck polling) for the + # first 14h of the backtest, then nothing at all. + # + # HEADS UP: THIS RULE WILL BE FIRING THE MOMENT YOU DEPLOY IT, and it is + # right to. As of 2026-07-24 19:45Z the prod worker container is running + # and healthy and is writing ~19 lines per 10 minutes to its own docker + # log — but not one of those lines has reached Loki since 04:43Z. Prod's + # Alloy agent stopped tailing the container when Swarm replaced the task. + # Nobody knew, because nothing was watching. See README "Known gaps". + # + # WHY THIS IS A WARNING AND NotifierHeartbeatMissingProd IS CRITICAL: the + # two draw on *different* Alloy sources — this one on loki.source.docker + # (the container's stdout), the heartbeat on loki.source.file tailing + # /applogs. That independence is the diagnostic. Right now prod's + # heartbeats are arriving in every single 20-minute window while its + # worker docker stream has been empty for 14 hours, which localises the + # fault to Alloy's docker tail rather than to the worker: the process is + # doing its job, we just cannot see it breathe. Both rules firing together + # means the worker really is down. + - uid: tg_prod_worker_silent + title: ProdWorkerContainerSilent + condition: C + for: 10m + noDataState: OK + execErrState: OK + annotations: + summary: 'prod: thermograph_worker has logged nothing for 2 hours' + description: >- + The worker container has produced zero log lines in 2 hours against + a healthy floor of ~242 per 2h. Two different faults look identical + here: the worker is actually dead, or prod's Alloy agent has stopped + tailing it (a real, observed failure — Alloy can miss a container + after Swarm replaces the task). Check both: `docker service ps + thermograph_worker` on prod, and whether `docker logs` on the live + container is producing lines that Loki does not have. Restarting + alloy-alloy-1 on prod fixes the second case. + labels: + severity: warning + host: prod + class: liveness + isPaused: false + data: + - refId: A + relativeTimeRange: { from: 7200, to: 0 } + datasourceUid: loki + model: + refId: A + datasource: { type: loki, uid: loki } + expr: 'sum(count_over_time({job="docker", host="prod", container=~"thermograph_worker.+"} [2h])) or vector(0)' + queryType: instant + editorMode: code + intervalMs: 1000 + maxDataPoints: 43200 + - refId: B + relativeTimeRange: { from: 0, to: 0 } + datasourceUid: __expr__ + model: + refId: B + type: reduce + reducer: last + expression: A + datasource: { type: __expr__, uid: __expr__ } + - refId: C + relativeTimeRange: { from: 0, to: 0 } + datasourceUid: __expr__ + model: + refId: C + type: threshold + expression: B + datasource: { type: __expr__, uid: __expr__ } + conditions: + - evaluator: { type: lt, params: [1] } + + # =========================================================================== + # The dead-man's switch. + # =========================================================================== + - orgId: 1 + name: watchdog + folder: Alerts + interval: 5m + rules: + + # -- 12 ---------------------------------------------------------------- + # Nothing in this file can tell you that Grafana itself has stopped, or + # that beta is down, or that the Discord webhook was revoked — a monitoring + # system cannot report its own death. So this rule fires permanently and + # by design, throttled by the notification policy to exactly one Discord + # message per 24 hours. + # + # HOW TO USE IT: if #ops-alerts has been silent for more than a day, the + # alerting pipeline is broken, not the estate healthy. That one daily + # message is the only evidence you will ever get that the other eleven + # rules are still capable of reaching you. + # + # To silence it (and lose that guarantee), set isPaused: true. + - uid: tg_alerting_watchdog + title: AlertingWatchdog + condition: C + for: 0m + noDataState: Alerting + execErrState: Alerting + annotations: + summary: 'Alerting pipeline is alive (daily heartbeat — not a fault)' + description: >- + This is the dead-man's switch and it is supposed to be firing. It + proves Grafana on beta is evaluating rules, Loki is answering, and + the Discord webhook still works. Seeing it once a day is correct. + NOT seeing it for over a day means alerting itself is down and no + other rule can reach you — check observability-grafana-1 on beta. + labels: + severity: watchdog + host: beta + class: meta + isPaused: false + data: + - refId: A + relativeTimeRange: { from: 600, to: 0 } + datasourceUid: loki + model: + refId: A + datasource: { type: loki, uid: loki } + expr: 'vector(1)' + queryType: instant + editorMode: code + intervalMs: 1000 + maxDataPoints: 43200 + - refId: B + relativeTimeRange: { from: 0, to: 0 } + datasourceUid: __expr__ + model: + refId: B + type: reduce + reducer: last + expression: A + datasource: { type: __expr__, uid: __expr__ } + - refId: C + relativeTimeRange: { from: 0, to: 0 } + datasourceUid: __expr__ + model: + refId: C + type: threshold + expression: B + datasource: { type: __expr__, uid: __expr__ } + conditions: + - evaluator: { type: gt, params: [0] }