observability: add the estate's first alerting; supervise Postfix

The estate had zero alert rules. The only contact point was Grafana's factory default pointing at the literal string <example@email.com>, and beta's untracked compose override routed Grafana's SMTP at prod's Postfix — so alerts, had any existed, would have been delivered by the box most likely to be on fire. Prod served 86 5xx in 24h including five /healthz failures and nobody was told. Alerting (deploys to beta, which is where Grafana runs): - 12 Loki-based rules. There is no Prometheus in this estate and Loki is the only datasource, so every rule is log-derived. - Thresholds come from a 24h backtest that happens to contain a real ~20min prod outage at 04:20Z. Absolute counts, not ratios: prod runs ~7 req/min, so one bad request is 1.4% and a ratio alert would scream all night. The 5xx burst rule fires on the outage's 45 and 22 buckets and on nothing else in the day; the largest benign bucket all day was 5. - Routed to a new private #ops-alerts channel, not to any existing channel — #weather-events, #announcements and #prod are product surfaces that notify real subscribers. - AlertingWatchdog is a dead-man's switch; its value is its absence. - Delivery is proven, not assumed: the rules were provisioned into a throwaway Grafana against beta's live Loki and a real alert arrived in Discord. This matters because GET /api/v1/provisioning/contact-points returns [REDACTED] for the URL — a contact point holding an uninterpolated env var looks perfectly healthy and pages nobody. The only proof is a message arriving. CI gains a structural check, because the existing one only proves YAML parses: an alert rule whose condition names a missing refId is valid YAML, provisions cleanly, and never fires. It also hard-fails on a literal Discord webhook in the repo. Verified against all three breakages deliberately introduced. Postfix supervision: - The 13h outage was a boot-ordering race, not a Docker renumbering: postfix started at 08:09:49, wg0 came up at :51, postfix fataled at :52 on a missing docker_gwbridge address, and dockerd did not finish starting until 08:10:16. Stock postfix@.service is ordered only After=network-online.target and ships no Restart=, so one lost race became a permanent outage. - An ExecStartPre gate now blocks up to 60s until every inet_interfaces address actually exists, which absorbs the transient case inside a single start attempt. That makes bounded retry correct: 5 attempts in 600s, then failed — a genuinely broken config reaches a visible failed state in ~100s instead of re-fataling every 15s forever. - A 5-minute watchdog timer retries indefinitely and runs reset-failed, so "failed" still self-heals. Worst case is ~5 minutes, not 13 hours. - Wants=, not Requires=: a dockerd failure must not take down the loopback and mesh listeners that do not depend on Docker at all. Health checks must read config with `postconf -c`, never postmulti/postqueue/ postfix — those three RESOLVE inet_interfaces and so fatal precisely when an address is missing, which made the first version of this check report status=ok bound=0/0. A health check that fails open is worse than none. DEPLOY.md carries the monitoring contract, including that systemctl is-active postfix is a known-false signal: postfix.service is a wrapper whose ExecStart=/bin/true, so it reports active forever while the real postfix@- instance is failed with zero listeners. Reproduced live. Known gap, documented not fixed: Alloy ships Docker stdout, Caddy files and app JSONL, not journald — so no Postfix line reaches Loki and the mail rules cannot fire until loki.source.journal is added. Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY
2026-07-24 13:19:19 -07:00 · 2026-07-24 13:19:19 -07:00 · e4693dce58
commit e4693dce58
parent 519541c378
12 changed files with 1932 additions and 46 deletions
--- a/.forgejo/workflows/observability-validate.yml
+++ b/.forgejo/workflows/observability-validate.yml
@ -17,6 +17,13 @@ name: Validate observability stack
 # GitHub release (pinned to the same v1.9.1 the fleet runs; see
 # observability/alloy/docker-compose.agent.yml), no docker CLI needed.
 #
 # The alerting config gets a third, stricter step. A YAML parse is not enough
 # for it: an alert rule whose `condition` names a refId that does not exist is
 # perfectly valid YAML, provisions without complaint, and then never fires --
 # a paging path that looks healthy and reaches nobody, which is the exact
 # failure this whole config exists to end. That step also refuses a literal
 # Discord webhook in the repo (it must stay an env reference).
 #
 # Not validated here: docker-compose *schema* (unknown-key checks, which would
 # need the docker CLI). Add it if the churn warrants the extra tooling.
@ -91,3 +98,75 @@ jobs:
          # subcommand, not a config error, so strip that one line before checking.
          grep -v '^livedebugging ' observability/alloy/config.alloy > /tmp/config-for-validate.alloy
          /tmp/alloybin/alloy-linux-amd64 validate /tmp/config-for-validate.alloy
      - name: Alerting config is structurally sound
        run: |
          set -euo pipefail
          apt-get update -qq && apt-get install -y -qq python3-yaml >/dev/null
          python3 - <<'PY'
          import pathlib, re, sys
          import yaml
          d = pathlib.Path("observability/grafana/provisioning/alerting")
          if not d.is_dir():
              print("no alerting provisioning yet — nothing to check"); sys.exit(0)
          docs, bad = {}, []
          for f in sorted(list(d.glob("*.yml")) + list(d.glob("*.yaml"))):
              docs[f] = yaml.safe_load(f.read_text()) or {}
              print("loaded", f)
          receivers, referenced = set(), set()
          for f, doc in docs.items():
              # --- contact points ---------------------------------------------------
              for cp in doc.get("contactPoints") or []:
                  receivers.add(cp.get("name"))
                  for r in cp.get("receivers") or []:
                      url = str((r.get("settings") or {}).get("url", ""))
                      # A real webhook in the repo is a credential leak: anyone with
                      # it can post into the ops channel. It must stay an env ref.
                      if re.search(r"https://(discord|discordapp)\.com/api/webhooks/", url):
                          bad.append(f"{f}: contact point '{cp.get('name')}' has a LITERAL Discord webhook URL — use $DISCORD_ALERT_WEBHOOK_URL")
                      elif not url.startswith("$"):
                          bad.append(f"{f}: contact point '{cp.get('name')}' url is neither an env reference nor empty: {url!r}")
              # --- notification policies --------------------------------------------
              def walk(route, where):
                  if route.get("receiver"):
                      referenced.add(route["receiver"])
                  for child in route.get("routes") or []:
                      walk(child, where)
              for pol in doc.get("policies") or []:
                  walk(pol, f)
              # --- alert rules ------------------------------------------------------
              for g in doc.get("groups") or []:
                  for rule in g.get("rules") or []:
                      title = rule.get("title", "<untitled>")
                      where = f"{f}: rule '{title}'"
                      refs = {q.get("refId") for q in rule.get("data") or []}
                      if not rule.get("uid"):
                          bad.append(f"{where}: missing uid (provisioning needs a stable one)")
                      if rule.get("condition") not in refs:
                          bad.append(f"{where}: condition '{rule.get('condition')}' is not one of {sorted(refs)} — this rule can never fire")
                      if not (rule.get("labels") or {}).get("severity"):
                          bad.append(f"{where}: no severity label — the notification policy routes on it")
                      if not (rule.get("annotations") or {}).get("summary"):
                          bad.append(f"{where}: no summary annotation — the Discord message would be blank")
                      for q in rule.get("data") or []:
                          model = q.get("model") or {}
                          upstream = model.get("expression")
                          if upstream and upstream not in refs:
                              bad.append(f"{where}: query '{q.get('refId')}' reads refId '{upstream}' which does not exist")
          missing = referenced - receivers
          if missing:
              bad.append(f"notification policy routes to undefined receiver(s): {sorted(missing)}")
          for b in bad:
              print("::error::" + b)
          print(f"checked {len(docs)} file(s): {len(receivers)} contact point(s), {len(bad)} problem(s)")
          sys.exit(1 if bad else 0)
          PY
--- a/infra/DEPLOY.md
+++ b/infra/DEPLOY.md
@ -300,6 +300,115 @@ the message and sends nothing — so LAN dev and the test suite exercise the who
 signup path with no mail server and no risk of mailing a real person. Production
 opts in with `=smtp`.
 ### Postfix unit topology — and the check that lies
 Debian/Ubuntu ships Postfix as **three** systemd units, and only one of them
 does anything:
 | Unit | What it is | Does it bind sockets? |
 |---|---|---|
 | `postfix.service` | umbrella wrapper: `Type=oneshot`, `ExecStart=/bin/true`, `RemainAfterExit=yes` | **No.** Reports `active (exited)` forever. |
 | `postfix@.service` | the template all instances are built from | n/a |
 | `postfix@-.service` | the instance for `/etc/postfix` — the real MTA | **Yes.** This is the one that matters. |
 > **`systemctl is-active postfix` is not a health check.** It returned `active`
 > for the entire 13-hour mail outage of 2026-07-24 while `postfix@-.service` was
 > `failed` and nothing was listening on `:25`. Reproduced deliberately during
 > supervision testing: with the instance in `failed` and zero listeners,
 > `is-active postfix` still said `active`. **Always query `postfix@-.service`.**
 Two consequences worth internalising:
 - Restart the **instance** (`systemctl restart 'postfix@-'`), not the umbrella,
  for a `main.cf` change to take effect.
 - **Postfix's own tools lie in the one state you most need to detect.** Any
  command that *resolves* `inet_interfaces` — `postmulti`, `postqueue`,
  `postfix` — fatals when one of the listed addresses is missing from the host.
  A checker built on them returns empty and reads as "nothing wrong". Use plain
  `postconf -c /etc/postfix -h inet_interfaces`, which only reads `main.cf`.
 ### Supervision (`deploy/provision-mail-supervision.sh`)
 Installed by `provision-mail.sh`, and safe to run on its own — it touches
 nothing in `main.cf`. Everything it writes lives outside `/opt/thermograph`, so
 `deploy.sh`'s `git reset --hard` cannot erase it; re-running the script is what
 makes a rebuilt box match.
 - **Drop-in** `/etc/systemd/system/postfix@.service.d/10-wait-for-interfaces.conf`
  (a drop-in, never an edit to the packaged unit, which apt replaces):
  orders Postfix `After=docker.service wg-quick@wg0.service`, adds
  `Restart=on-failure` / `RestartSec=15s`, and bounds retries at
  `StartLimitBurst=5` / `StartLimitIntervalSec=600`.
 - **Start gate** `/usr/local/sbin/postfix-wait-interfaces` — an `ExecStartPre`
  that blocks (≤60s) until every address in `inet_interfaces` actually exists.
  Ordering makes the boot race unlikely; this makes it deterministic, and it is
  what makes a *bounded* start limit safe: a late interface costs seconds inside
  one attempt instead of burning the retry budget.
 - **Health check** `/usr/local/sbin/postfix-health` — the single definition of
  "mail works" (see below).
 - **Watchdog** `/usr/local/sbin/postfix-watchdog` + `postfix-watchdog.timer` —
  every 5 minutes. `Restart=` gives up after 5 attempts *by design*, so that a
  genuinely broken config lands in `failed` loudly instead of re-fataling every
  15s forever; the watchdog is what stops "failed" meaning "dead until a human
  notices". It runs `reset-failed` (clearing the start-limit counter, which
  otherwise makes plain `systemctl start` refuse) and retries, forever, at a
  cadence 20× slower than `RestartSec`. Net effect: **bounded fast retry for
  transients, slow unbounded retry for everything else**, worst-case recovery
  ~5 minutes instead of 13 hours.
 `sudo touch /etc/postfix/maintenance` stops the watchdog fighting a deliberate
 `systemctl stop postfix@-`. Remove it when done.
 ### Monitoring specification
 For the Grafana/Discord alerting and any future `mail_health` tool. Alerts must
 **not** route over this box's own SMTP — prod's MTA cannot be the transport for
 "prod's MTA is down".
 `postfix-health` is the contract. Exit `0` healthy, `1` unhealthy, and one
 logfmt line on stdout (LogQL: `| logfmt`):
 ```
 mail_health status=ok unit=postfix@-.service unit_state=active bound=3/3 \
  banner=ok queue=2 queue_real=0 oldest_deferred_s=0 reason=none
 ```
 It asserts, in order: the **instance** unit is active; every address in
 `inet_interfaces` is really bound on `:25`; none of them is **missing from the
 host** (the latent state that stays invisible until the next reboot and then
 kills all mail); a live `220` greeting comes back, not merely an open socket;
 **no public address is bound**; and queue depth/age with null-sender bounces
 excluded (they are undeliverable by design — this host accepts no inbound mail —
 and would otherwise hold an alert open forever).
 Alert on:
 | Severity | Condition | Why |
 |---|---|---|
 | **Page** | `mail_health status=fail` twice consecutively (≥10 min) | mail is down |
 | **Page** | any `postfix_watchdog action=RECOVERY_FAILED` | self-heal exhausted |
 | **Page** | **no `postfix_watchdog` line at all for 15 min** | see below |
 | Warn | `status=fail` where `reason` contains only `address-missing-from-host` | still serving, but the next reboot kills all mail |
 | Warn | `queue_real > 0` and `oldest_deferred_s > 3600` | mail accepted but not leaving |
 The **heartbeat-absence** rule is the one that is easy to leave out and the one
 that would have caught this outage. Every other check is an *active probe* and
 therefore only reports while something is alive to run it; a dead box, dead
 timer, or dead log shipper produces silence, and silence renders identically to
 health on a dashboard. Alert on the absence of the 5-minute heartbeat, or the
 monitoring inherits the same blind spot the umbrella unit had.
 > **Blocker for the alerting agent:** none of this reaches Loki today. Alloy
 > ships Docker container stdout, Caddy files and the app's JSONL — **not the
 > systemd journal** — and Postfix runs on the host. Until
 > `observability/alloy/config.alloy` gains a `loki.source.journal` (plus
 > `/var/log/journal` and `/etc/machine-id` mounted read-only in
 > `alloy/docker-compose.agent.yml`), no Postfix or watchdog line is queryable
 > and none of the rules above can fire. That change is untested — validate it
 > before relying on it. Until then the check is still reachable synchronously
 > via Centralis `run_on_host` running `postfix-health`, which is also the
 > natural implementation of a `mail_health` tool.
 ### Deliverability (do this before mailing real subscribers)
 Mail from a bare VPS IP is very often junked no matter how Postfix is configured,
--- a/infra/deploy/provision-mail-supervision.sh
+++ b/infra/deploy/provision-mail-supervision.sh
@ -0,0 +1,476 @@
 #!/usr/bin/env bash
 # Postfix SUPERVISION layer — restart policy, start-time interface gate, and a
 # self-heal watchdog. Called by provision-mail.sh; also safe to run standalone
 # (it touches nothing in main.cf, so it will not fight a mail-config change):
 #
 #   sudo bash infra/deploy/provision-mail-supervision.sh
 #
 # WHY THIS EXISTS — the 2026-07-24 prod outage.
 #
 # Postfix treats an `inet_interfaces` address with no matching local interface
 # as FATAL, and takes down EVERY listener with it — loopback and mesh included,
 # not just the missing one. prod lists three addresses, and two of them are
 # created by other daemons: 10.10.0.1 (wg-quick@wg0) and 172.18.0.1
 # (docker_gwbridge, created by dockerd).
 #
 # The stock postfix@.service is ordered only `After=network-online.target`,
 # which says nothing about dockerd or wg-quick, and it ships NO `Restart=`.
 # At the 08:09 reboot that cost 13 hours of mail:
 #
 #   08:09:49  postfix@-.service starts
 #   08:09:51  wg-quick@wg0 finishes  <- 10.10.0.1 appears, wins by one second
 #   08:09:52  fatal: inet_interfaces: no local interface found for 172.18.0.1
 #   08:09:53  docker.service only BEGINS starting
 #   08:10:16  docker.service active  <- docker_gwbridge finally exists
 #
 # Note this was a BOOT-ORDERING RACE, not a Docker renumbering: 172.18.0.1 is
 # the pinned docker_gwbridge gateway and has always had that address. Postfix
 # simply asked for it 24 seconds before dockerd created it.
 #
 # Nobody noticed for 13h because `systemctl is-active postfix` said "active"
 # the whole time — see the umbrella-unit note below.
 #
 # THREE LAYERS, because no single one of them is sufficient:
 #
 #   1. ORDERING (After=docker.service wg-quick@wg0.service) — makes the race
 #      very unlikely. Not a guarantee: dockerd reporting "active" is not a
 #      promise that a specific bridge is already numbered.
 #   2. A START-TIME GATE (postfix-wait-interfaces, an ExecStartPre) — makes it
 #      deterministic. It blocks until every address in inet_interfaces actually
 #      exists, so a late interface costs seconds inside one start attempt
 #      instead of burning a restart. This is what makes a BOUNDED start limit
 #      safe; see the StartLimit discussion in the drop-in below.
 #   3. A WATCHDOG TIMER (postfix-watchdog) — the backstop. It is not redundant
 #      with Restart=: postfix@.service is Type=forking with GuessMainPID=no and
 #      no PIDFile=, so systemd tracks NO main PID for it. `Restart=` therefore
 #      only covers failures during START. If the master process dies at
 #      runtime, systemd does not reliably notice, and no restart policy fires.
 #      The watchdog is the only thing that catches that.
 #
 # THE UMBRELLA-UNIT TRAP. `postfix.service` is a wrapper whose ExecStart is
 # /bin/true and which sets RemainAfterExit=yes. It reports `active (exited)`
 # forever, whether or not any mail is being handled. The unit that actually
 # binds sockets is the instance `postfix@-.service`. Never health-check the
 # umbrella — that is precisely the check that reported green for 13 hours.
 set -euo pipefail
 if [[ $EUID -ne 0 ]]; then
  echo "run as root (sudo)" >&2
  exit 1
 fi
 # How long the start-time gate waits for a missing interface before giving up
 # and letting the restart policy take over. Overridable for testing.
 POSTFIX_WAIT_TIMEOUT="${POSTFIX_WAIT_TIMEOUT:-60}"
 echo "==> installing /usr/local/sbin/postfix-wait-interfaces"
 cat > /usr/local/sbin/postfix-wait-interfaces <<'WAITSCRIPT'
 #!/usr/bin/env bash
 # Managed by deploy/provision-mail-supervision.sh -- do not hand-edit.
 #
 # ExecStartPre gate for postfix@.service. Blocks until every address listed in
 # this instance's inet_interfaces actually exists on some local interface.
 #
 # Postfix treats a missing inet_interfaces address as fatal and drops ALL its
 # listeners, so at boot it must not be allowed to ask before dockerd/wg-quick
 # have finished. Unit ordering makes that unlikely; this makes it deterministic.
 #
 # FAIL-OPEN BY DESIGN: if this script cannot introspect the config, it exits 0
 # and lets Postfix try anyway. A supervision gate must only ever DELAY a start,
 # never be the reason mail stays down.
 set -uo pipefail
 TIMEOUT="${POSTFIX_WAIT_TIMEOUT:-60}"
 INSTANCE="${1:--}"
 # Instance '-' is the default instance and always lives in /etc/postfix. A
 # second config dir can be passed explicitly; this estate runs one instance.
 CONFDIR="${2:-/etc/postfix}"
 log() { echo "postfix-wait-interfaces: $*"; }
 # MUST be plain `postconf -c`, NOT `postmulti -x postconf`.
 #
 # postmulti RESOLVES inet_interfaces, so it fatals with rc=1 and prints nothing
 # whenever an address is missing -- which is precisely the situation this gate
 # exists to detect. Using it here made the gate fail open and silently never
 # wait at all. Plain postconf only reads main.cf and returns the value happily
 # whether or not the address exists. (Same trap applies to postqueue/postfix;
 # see postfix-health.)
 addrs=$(postconf -c "$CONFDIR" -h inet_interfaces 2>/dev/null) || {
  log "cannot read inet_interfaces from $CONFDIR; not blocking startup"
  exit 0
 }
 # Only literal IP addresses can be waited for. Keywords (all, loopback-only)
 # and hostnames are left to Postfix to resolve.
 want=()
 for a in ${addrs//,/ }; do
  case "$a" in
    all|loopback-only|localhost) continue ;;
  esac
  if [[ "$a" =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ || "$a" == *:* ]]; then
    want+=("$a")
  fi
 done
 if (( ${#want[@]} == 0 )); then
  log "no literal addresses to wait for (inet_interfaces = $addrs)"
  exit 0
 fi
 deadline=$(( SECONDS + TIMEOUT ))
 missing=()
 while :; do
  present=$(ip -o addr show 2>/dev/null | awk '{print $4}' | cut -d/ -f1)
  missing=()
  for a in "${want[@]}"; do
    grep -qxF "$a" <<<"$present" || missing+=("$a")
  done
  (( ${#missing[@]} == 0 )) && break
  (( SECONDS >= deadline )) && break
  sleep 1
 done
 if (( ${#missing[@]} == 0 )); then
  log "all addresses present after ${SECONDS}s: ${want[*]}"
  exit 0
 fi
 # Deliberately non-zero: let the unit's Restart=/StartLimit policy decide what
 # happens next, and make the reason visible in the journal.
 log "TIMEOUT after ${TIMEOUT}s; still missing: ${missing[*]} (inet_interfaces = $addrs)"
 log "Postfix will fatal on this. Either the interface provider is broken, or"
 log "inet_interfaces names an address this host no longer has."
 exit 1
 WAITSCRIPT
 chmod 0755 /usr/local/sbin/postfix-wait-interfaces
 echo "==> installing /usr/local/sbin/postfix-health"
 cat > /usr/local/sbin/postfix-health <<'HEALTHSCRIPT'
 #!/usr/bin/env bash
 # Managed by deploy/provision-mail-supervision.sh -- do not hand-edit.
 #
 # THE canonical "is this box's mail actually working" check. One implementation,
 # so the watchdog, a human, a Grafana/Loki alert and a future `mail_health` tool
 # all agree on what healthy means.
 #
 #   exit 0 = healthy      exit 1 = unhealthy
 #
 # Output is one logfmt line on stdout (parseable with LogQL `| logfmt`),
 # followed by human-readable detail on stderr.
 #
 # WHAT IT DELIBERATELY DOES NOT DO: check `postfix.service`. That umbrella unit
 # is ExecStart=/bin/true + RemainAfterExit=yes and reports active(exited)
 # forever. It reported green through the entire 13h outage of 2026-07-24.
 set -uo pipefail
 export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
 INSTANCE="${1:--}"
 CONFDIR="${2:-/etc/postfix}"
 UNIT="postfix@${INSTANCE}.service"
 problems=()
 # --- 1. the INSTANCE unit, never the umbrella --------------------------------
 unit_state=$(systemctl is-active "$UNIT" 2>/dev/null || true)
 [[ "$unit_state" == "active" ]] || problems+=("unit-not-active:$unit_state")
 # --- 2. every configured address is actually bound on :25 --------------------
 #
 # MUST be plain `postconf -c`, NOT `postmulti -x postconf` / `postqueue`.
 #
 # Every Postfix command that RESOLVES inet_interfaces (postmulti, postqueue,
 # postfix) fatals when one of the addresses is missing from the host -- exactly
 # the condition this check exists to find. An earlier version of this script
 # used postmulti, got an empty string back, concluded there were 0 addresses to
 # verify, and cheerfully reported "status=ok bound=0/0" while main.cf named a
 # ghost address that would kill all mail at the next reboot. A health check that
 # fails OPEN is worse than no health check.
 #
 # Plain postconf only reads main.cf and never touches an interface.
 addrs=$(postconf -c "$CONFDIR" -h inet_interfaces 2>/dev/null || echo "")
 if [[ -z "${addrs// /}" ]]; then
  problems+=("cannot-read-inet_interfaces")
 fi
 listening=$(ss -lnt 2>/dev/null | awk '{print $4}')
 expected=(); bound=0; unbound=()
 for a in ${addrs//,/ }; do
  case "$a" in
    all)           expected+=("0.0.0.0") ;;
    loopback-only) expected+=("127.0.0.1") ;;
    localhost)     expected+=("127.0.0.1") ;;
    *)             expected+=("$a") ;;
  esac
 done
 for a in "${expected[@]}"; do
  if grep -qxF "${a}:25" <<<"$listening"; then bound=$(( bound + 1 )); else unbound+=("$a"); fi
 done
 (( ${#unbound[@]} == 0 )) || problems+=("not-bound:$(IFS=,; echo "${unbound[*]}")")
 # Never let "I could not work out what to check" read as healthy.
 (( ${#expected[@]} > 0 )) || problems+=("no-configured-addresses")
 # --- 3. LATENT FATAL: config names an address the host does not have ---------
 # This is the state that survives silently until the next reboot and then kills
 # all mail. Catching it here turns a future outage into a warning today.
 present=$(ip -o addr show 2>/dev/null | awk '{print $4}' | cut -d/ -f1)
 ghosts=()
 for a in "${expected[@]}"; do
  [[ "$a" == "0.0.0.0" ]] && continue
  grep -qxF "$a" <<<"$present" || ghosts+=("$a")
 done
 (( ${#ghosts[@]} == 0 )) || problems+=("address-missing-from-host:$(IFS=,; echo "${ghosts[*]}")")
 # --- 4. a real SMTP greeting, not just an open socket ------------------------
 banner="none"
 if command -v timeout >/dev/null 2>&1; then
  greeting=$(timeout 5 bash -c 'exec 3<>/dev/tcp/127.0.0.1/25 || exit 1; head -1 <&3; printf "QUIT\r\n" >&3' 2>/dev/null || true)
  case "$greeting" in
    220*) banner="ok" ;;
    *)    banner="bad"; problems+=("no-220-greeting") ;;
  esac
 fi
 # --- 5. SEND-ONLY INVARIANT: must NOT be bound on a public address -----------
 # A regression here is worse than an outage: it is an internet-facing MTA.
 public_ips=$(ip -4 -o addr show scope global 2>/dev/null | awk '{print $4}' | cut -d/ -f1 \
             | grep -vE '^(10\.|172\.(1[6-9]|2[0-9]|3[01])\.|192\.168\.|127\.)' || true)
 exposed=()
 for p in $public_ips; do
  grep -qxF "${p}:25" <<<"$listening" && exposed+=("$p")
 done
 grep -qxF "0.0.0.0:25" <<<"$listening" && exposed+=("0.0.0.0")
 (( ${#exposed[@]} == 0 )) || problems+=("PUBLICLY-BOUND:$(IFS=,; echo "${exposed[*]}")")
 # --- 6. queue depth and age, excluding the known-benign null-sender bounces ---
 # Read arrival_time from `postqueue -j`, NOT the queue file's mtime: Postfix
 # sets a deferred file's mtime to the NEXT RETRY time, which is in the future,
 # so mtime-based ages come out negative and meaningless.
 #
 # Null-sender bounces (postqueue reports sender "MAILER-DAEMON") addressed to
 # our own no-reply are undeliverable by design -- this host accepts no inbound
 # mail -- so they sit in the queue until they expire. They must not hold an
 # alert open forever, so they are excluded from queue_real/oldest. Both the raw
 # and filtered counts are reported so a monitor can see either.
 #
 # NOTE: postqueue RESOLVES inet_interfaces and so fatals (rc=69) in the very
 # latent-fatal state flagged above. When that happens it yields no JSON, so fall
 # back to counting spool files directly -- a count that needs no Postfix command
 # and therefore still works when Postfix itself will not run.
 queue_json=$(postqueue -j 2>/dev/null || true)
 read -r queue_total queue_real oldest_age <<<"$(
  printf '%s\n' "$queue_json" | awk -v now="$(date +%s)" '
    NF {
      total++
      sender=""; arr=0
      if (match($0, /"sender": "[^"]*"/)) {
        s = substr($0, RSTART, RLENGTH); gsub(/"sender": "|"/, "", s); sender = s
      }
      if (match($0, /"arrival_time": [0-9]+/)) {
        a = substr($0, RSTART, RLENGTH); gsub(/[^0-9]/, "", a); arr = a + 0
      }
      if (sender != "MAILER-DAEMON" && sender != "") {
        real++
        age = now - arr
        if (age > oldest) oldest = age
      }
    }
    END { printf "%d %d %d", total + 0, real + 0, oldest + 0 }
  '
 )"
 queue_total=${queue_total:-0}; queue_real=${queue_real:-0}; oldest_age=${oldest_age:-0}
 if [[ -z "${queue_json// /}" ]]; then
  queue_total=$(find /var/spool/postfix/incoming /var/spool/postfix/active \
                     /var/spool/postfix/deferred -type f 2>/dev/null | wc -l)
  queue_real=-1   # unknown: cannot classify senders without postqueue
 fi
 status=ok; rc=0
 (( ${#problems[@]} == 0 )) || { status=fail; rc=1; }
 printf 'mail_health status=%s unit=%s unit_state=%s bound=%d/%d banner=%s queue=%d queue_real=%d oldest_deferred_s=%d reason=%s\n' \
  "$status" "$UNIT" "${unit_state:-unknown}" "$bound" "${#expected[@]}" "$banner" \
  "$queue_total" "$queue_real" "$oldest_age" \
  "$( (( ${#problems[@]} )) && (IFS=,; echo "${problems[*]}") || echo none )"
 if (( rc != 0 )); then
  { echo "--- postfix-health detail ---"
    echo "inet_interfaces : $addrs"
    echo "host addresses  : $(echo "$present" | tr '\n' ' ')"
    echo "listening on 25 : $(grep ':25$' <<<"$listening" | tr '\n' ' ')"
    systemctl status "$UNIT" --no-pager -l 2>/dev/null | head -20
  } >&2
 fi
 exit $rc
 HEALTHSCRIPT
 chmod 0755 /usr/local/sbin/postfix-health
 echo "==> installing /usr/local/sbin/postfix-watchdog"
 cat > /usr/local/sbin/postfix-watchdog <<'WATCHSCRIPT'
 #!/usr/bin/env bash
 # Managed by deploy/provision-mail-supervision.sh -- do not hand-edit.
 #
 # Self-heal backstop, run every 5 minutes by postfix-watchdog.timer.
 #
 # It exists for two failure modes that `Restart=on-failure` provably cannot
 # cover:
 #
 #   1. RUNTIME DEATH. postfix@.service is Type=forking, GuessMainPID=no, and
 #      declares no PIDFile=, so systemd tracks no main PID. If the Postfix
 #      master dies while running, no restart policy fires.
 #   2. GIVING UP. The unit uses a BOUNDED start limit on purpose, so a broken
 #      config lands in `failed` instead of thrashing forever. Without a
 #      backstop, "failed" would mean "dead until a human looks" — which is the
 #      13-hour outage again. This retries every 5 minutes, forever, and clears
 #      the start-limit counter so the bound never becomes permanent death.
 #
 # Cadence is deliberately 20x slower than RestartSec: fast enough that nothing
 # is ever down for more than ~5 minutes, slow enough that a genuinely broken
 # config produces ~288 log lines a day instead of ~5,760.
 set -uo pipefail
 export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
 INSTANCE=-
 UNIT="postfix@${INSTANCE}.service"
 # Escape hatch: `sudo touch /etc/postfix/maintenance` to stop the watchdog
 # fighting a deliberate `systemctl stop postfix@-`.
 if [[ -e /etc/postfix/maintenance ]]; then
  echo "postfix_watchdog action=skipped reason=maintenance-flag"
  exit 0
 fi
 # Do not interfere with a start already in progress.
 state=$(systemctl show "$UNIT" -p ActiveState --value 2>/dev/null || echo unknown)
 if [[ "$state" == "activating" || "$state" == "deactivating" ]]; then
  echo "postfix_watchdog action=skipped reason=transitional:$state"
  exit 0
 fi
 if health=$(postfix-health "$INSTANCE" 2>/dev/null); then
  echo "postfix_watchdog action=ok ${health#mail_health }"
  exit 0
 fi
 echo "postfix_watchdog action=recovering ${health#mail_health }"
 # reset-failed clears both the failed state and the start-limit counter, so the
 # bounded StartLimitBurst can never become a permanent death sentence.
 systemctl reset-failed "$UNIT" 2>/dev/null || true
 systemctl start "$UNIT" 2>/dev/null || true
 sleep 5
 if health=$(postfix-health "$INSTANCE" 2>/dev/null); then
  echo "postfix_watchdog action=recovered ${health#mail_health }"
  exit 0
 fi
 # Distinctive, greppable line: this is the one that should page a human.
 echo "postfix_watchdog action=RECOVERY_FAILED ${health#mail_health }"
 exit 1
 WATCHSCRIPT
 chmod 0755 /usr/local/sbin/postfix-watchdog
 echo "==> installing postfix@.service drop-in"
 mkdir -p /etc/systemd/system/postfix@.service.d
 # Quoted heredoc + a placeholder, NOT shell interpolation: this block contains
 # backticks and $-signs in its prose, and an unquoted heredoc would run them.
 cat > /etc/systemd/system/postfix@.service.d/10-wait-for-interfaces.conf <<'DROPIN'
 # Managed by deploy/provision-mail-supervision.sh -- do not hand-edit.
 #
 # A DROP-IN, never an edit to /usr/lib/systemd/system/postfix@.service: the
 # packaged unit is replaced wholesale by an apt upgrade.
 #
 # ORDERING. inet_interfaces may name a Docker bridge gateway and/or a WireGuard
 # mesh address; neither exists until dockerd / wg-quick has run, and Postfix
 # treats a missing address as fatal for ALL listeners. Stock ordering is only
 # After=network-online.target, which says nothing about either daemon.
 #
 # Wants=, not Requires=, on purpose. Requires= would mean a dockerd failure
 # takes mail down entirely — including the loopback and mesh listeners, which
 # do not depend on Docker at all. That escalates someone else's outage into a
 # total mail outage. Wants= keeps the ordering when both are present without
 # making mail's availability a hostage to Docker's.
 #
 # THE GATE. ExecStartPre blocks until every configured address actually exists
 # (bounded by POSTFIX_WAIT_TIMEOUT). Ordering alone is a strong heuristic —
 # dockerd reporting "active" is not a promise that a specific bridge is
 # numbered — and this turns the residual race into a deterministic wait.
 # It is what makes the bounded start limit below safe.
 #
 # THE START LIMIT — the deliberate tradeoff.
 #
 # StartLimitIntervalSec=0 (retry forever) was considered and rejected. With the
 # gate in place, the transient case is absorbed INSIDE a single start attempt,
 # so restarts now overwhelmingly mean "genuinely broken", not "too early". For
 # that case, retry-forever is the worse option: the unit never reaches `failed`
 # (so the strongest local signal never exists), and a malformed main.cf
 # re-fatals every RestartSec forever.
 #
 #   * 5 attempts / 600s. A broken config reaches `failed` in ~100s (fast
 #     failure: ~5s start + 15s wait) or ~5min (slow failure: the full 60s gate
 #     + 15s wait, 5 times) — either way, loudly, and then it stops.
 #   * Normal operation never approaches this: an isolated crash is 1 restart.
 #   * "Failed" does NOT mean "dead until a human looks" — postfix-watchdog.timer
 #     retries every 5 minutes forever and clears the counter. Bounded fast
 #     retry for transients, slow unbounded retry for everything else.
 #
 # TimeoutStartSec covers the worst case: the full @WAIT_TIMEOUT@s gate plus
 # instance configure plus master start, with headroom. The default 90s would be
 # uncomfortably tight against a @WAIT_TIMEOUT@s gate.
 [Unit]
 After=docker.service wg-quick@wg0.service
 Wants=docker.service wg-quick@wg0.service
 StartLimitIntervalSec=600
 StartLimitBurst=5
 [Service]
 Environment=POSTFIX_WAIT_TIMEOUT=@WAIT_TIMEOUT@
 ExecStartPre=/usr/local/sbin/postfix-wait-interfaces %i
 TimeoutStartSec=180s
 Restart=on-failure
 RestartSec=15s
 DROPIN
 sed -i "s/@WAIT_TIMEOUT@/${POSTFIX_WAIT_TIMEOUT}/g" \
  /etc/systemd/system/postfix@.service.d/10-wait-for-interfaces.conf
 echo "==> installing postfix-watchdog.service / .timer"
 cat > /etc/systemd/system/postfix-watchdog.service <<'WDSERVICE'
 # Managed by deploy/provision-mail-supervision.sh -- do not hand-edit.
 [Unit]
 Description=Postfix health watchdog (self-heal backstop)
 Documentation=file:///usr/local/sbin/postfix-watchdog
 # Never race the thing we are supervising.
 After=postfix@-.service
 [Service]
 Type=oneshot
 ExecStart=/usr/local/sbin/postfix-watchdog
 # The watchdog failing is itself a signal worth seeing, but it must never be
 # able to wedge the timer.
 SuccessExitStatus=0 1
 WDSERVICE
 cat > /etc/systemd/system/postfix-watchdog.timer <<'WDTIMER'
 # Managed by deploy/provision-mail-supervision.sh -- do not hand-edit.
 #
 # Every 5 minutes: the recovery-latency ceiling for mail on this box. The
 # 2026-07-24 outage ran 13h; this bounds the same failure at ~5min even with
 # no alerting configured at all.
 [Unit]
 Description=Run the Postfix health watchdog every 5 minutes
 [Timer]
 OnBootSec=5min
 OnUnitActiveSec=5min
 # Spread the load a little so it never lands exactly on other cron-ish work.
 RandomizedDelaySec=20s
 AccuracySec=10s
 [Install]
 WantedBy=timers.target
 WDTIMER
 systemctl daemon-reload
 systemctl enable --now postfix-watchdog.timer
 echo "==> supervision layer installed"
 echo "    drop-in : /etc/systemd/system/postfix@.service.d/10-wait-for-interfaces.conf"
 echo "    gate    : /usr/local/sbin/postfix-wait-interfaces"
 echo "    health  : /usr/local/sbin/postfix-health"
 echo "    watchdog: /usr/local/sbin/postfix-watchdog  (postfix-watchdog.timer)"
--- a/infra/deploy/provision-mail.sh
+++ b/infra/deploy/provision-mail.sh
@ -79,6 +79,19 @@ postconf -e "myorigin = ${MAIL_DOMAIN}"
 # stack cutover while still listed in inet_interfaces. Also note: postfix on
 # this distro is an umbrella unit; restart `postfix@-`, not `postfix`, for
 # inet_interfaces changes to take effect.
 #
 # THE SAME TRAP FIRES AT BOOT, NOT JUST ON RENUMBERING (prod outage 2026-07-24).
 # A Docker bridge address does not exist until dockerd creates it, and the stock
 # postfix@.service is only ordered `After=network-online.target` -- which says
 # nothing about dockerd or wg-quick. At the 08:09 reboot Postfix started at
 # 08:09:49 and fataled at 08:09:52 on "no local interface found for 172.18.0.1";
 # dockerd did not even begin starting until 08:09:53. wg0 (10.10.0.1) won the
 # same race by one second. Because postfix@.service ships no Restart=, that
 # single lost race killed ALL mail for 13h -- loopback and mesh included.
 # install_postfix_ordering_dropin below is what makes this survive a reboot:
 # it orders postfix@ after docker.service and wg-quick@wg0.service and retries
 # on failure. If you add an address here that some other daemon creates, add
 # that daemon to the drop-in too.
 DOCKER_MAIL_GATEWAY="${DOCKER_MAIL_GATEWAY-172.19.0.1}"
 DOCKER_MAIL_SUBNET="${DOCKER_MAIL_SUBNET-172.19.0.0/16}"
 # Optional WireGuard-mesh listener: other mesh nodes (e.g. beta's Forgejo, whose
@ -139,11 +152,36 @@ else
  postconf -e "relayhost ="
 fi
-systemctl enable postfix
+# Postfix fatals if ANY inet_interfaces address is missing when it starts, and
-systemctl restart postfix
+# takes every listener down with it. Docker bridge and WireGuard addresses are
 # created by other daemons, so order Postfix after them, gate the start on the
 # addresses actually existing, and supervise the result. See the long comment
 # above the inet_interfaces block, and the header of the script itself for the
 # full incident write-up and the reasoning behind each number.
 bash "$(dirname "${BASH_SOURCE[0]}")/provision-mail-supervision.sh"
-echo "==> verifying it listens on loopback only"
+systemctl enable postfix
-ss -lntp | grep ':25 ' || true
+# postfix.service is an umbrella whose ExecStart is /bin/true; the instance
 # postfix@- is what actually binds. Restarting the umbrella propagates via
 # PartOf=, but restart the instance directly so a failure surfaces here.
 systemctl restart 'postfix@-'
 # Assert, don't hope. postfix-health is the ONE definition of "mail works" on
 # this estate -- the same check the watchdog, the Grafana alert and any
 # mail_health tool use, so provisioning cannot pass on a laxer standard than
 # monitoring. It checks the INSTANCE unit (never the active(exited) umbrella,
 # which is the check that reported green through the whole 13h 2026-07-24
 # outage), that every configured address is really bound, that a live 220
 # greeting comes back, that no PUBLIC address is bound, and that nothing in
 # inet_interfaces is missing from the host -- the latent state that stays
 # invisible until the next reboot and then kills all mail.
 echo "==> verifying postfix is actually up and bound"
 /usr/local/sbin/postfix-health || {
  echo "FATAL: postfix-health failed -- see detail above" >&2
  exit 1
 }
 echo "==> listeners (must NOT include a public address):"
 ss -lntp | grep ':25 '
 cat <<'NOTE'
--- a/infra/deploy/thermograph.env.example
+++ b/infra/deploy/thermograph.env.example
@ -83,35 +83,12 @@ WORKERS=4
 # Read once at process start; changing it needs a restart, not a live toggle.
 #THERMOGRAPH_ROLE=all
-# The recurring jobs (city warming, IndexNow). Hours between runs; both are
+# The worker's own recurring jobs (city warming, IndexNow), gated by the same
-# cheap no-op skips when there's nothing to do, so the defaults rarely need
+# leader election as the notifier above. Hours between runs; both are cheap
-# changing. The daemon service (below) owns these timers and fires the work
+# no-op skips when there's nothing to do, so the defaults rarely need changing.
 # through the backend's internal routes.
 #THERMOGRAPH_WARM_CITIES_INTERVAL_HOURS=24
 #THERMOGRAPH_INDEXNOW_INTERVAL_HOURS=6
 # --- Internal daemon<->backend API ------------------------------------------------
 # Shared secret authenticating the daemon service's calls to the backend's
 # /internal/* routes (X-Thermograph-Internal-Token header). A credential:
 # OPTIONAL — leave unset unless you specifically want to override the derivation.
 #
 # When unset, the backend and the daemon each DERIVE this token from
 # THERMOGRAPH_AUTH_SECRET (HMAC-SHA256 under a fixed domain-separation label).
 # That secret is already required and already in every vault, so the daemon
 # stands up with NO new key to add, rotate, or forget on a new host. HMAC under a
 # distinct label rather than reusing the auth secret directly, so a leak of this
 # token can't be replayed as the session-signing key.
 #
 # Set it explicitly to rotate this surface independently of the auth secret
 # (e.g. openssl rand -hex 32). On a SOPS host that means `sops edit` on
 # deploy/secrets/*.yaml, never hand-editing /etc/thermograph.env — it is a
 # rendered artifact. Both ends must agree, so set it on both or neither.
 #
 # With neither this nor THERMOGRAPH_AUTH_SECRET set, both ends fail closed: the
 # backend disables the internal routes and the daemon refuses to start. Defence
 # in depth: Caddy never routes /internal/* to the backend anyway.
 #THERMOGRAPH_INTERNAL_TOKEN=
 # Base path the app is served under.
 #   /            -> app at the domain root (Thermograph owns the whole domain —
 #                   this is what the Caddyfile expects: thermograph.org proxies "/")
@ -159,6 +136,17 @@ THERMOGRAPH_BASE_URL=https://thermograph.org
 # can't reach a compose bridge gateway; leave this file's value as the compose
 # default.
 #
 # WARNING -- this host's rendered value must name an address Postfix ACTUALLY
 # LISTENS ON (`postconf -h inet_interfaces`), or mail fails at send time with
 # connection-refused. prod's SOPS-rendered /etc/thermograph.env currently says
 # 172.19.0.1, which is the *compose* bridge and is NOT in prod's
 # inet_interfaces. It is inert today only because prod runs in stack mode, where
 # thermograph-stack.yml sets THERMOGRAPH_SMTP_HOST per-service and nothing on
 # the box reads this variable from here. It becomes live breakage the moment
 # prod falls back to compose mode. Fix it in deploy/secrets/prod.yaml (sops)
 # rather than on the box; 10.10.0.1 (the mesh address) is the durable choice,
 # since it depends on wg0 rather than on Docker.
 #
 # Backends: console (log it, send nothing — the default, right for dev),
 #           smtp (actually send), disabled (drop silently).
 # Leave unset until Postfix is provisioned: signups are still collected either way.
@ -211,13 +199,12 @@ THERMOGRAPH_BASE_URL=https://thermograph.org
 # alone). Leave unset to disable. In the Thermograph.org server:
 #   weather-events = 1529274516746932307
 #THERMOGRAPH_DISCORD_WEATHER_CHANNEL=
-# Discord gateway bot (@mention/DM grading): runs in the daemon service (the
+# Discord gateway bot (@mention/DM grading): runs INSIDE the backend leader
-# Go thermograph-daemon container — no longer inside the backend leader
+# process (same singleton election as the notifier — see web/app.py's lifespan),
-# process). Opt-in: any truthy value here starts it, provided BOT_TOKEN above
+# riding its event loop; no separate bot process. Opt-in: any truthy value here
-# is also set (flag alone is a silent no-op). Discord allows ONE gateway
+# starts it, provided BOT_TOKEN above is also set (flag alone is a silent no-op).
-# connection per bot token, so enable this in exactly one environment — prod,
+# Discord allows ONE gateway connection per bot token, so enable this in exactly
-# the only vault with the token (the stack pins the daemon to 1 replica for
+# one environment — prod, the only vault with the token.
 # the same reason).
 #THERMOGRAPH_DISCORD_BOT=
 # Account linking (OAuth2 "identify"): lets a signed-in user connect their Discord
 # account, storing their Discord user id for DM alerts. CLIENT_ID is the same App
@ -226,9 +213,8 @@ THERMOGRAPH_BASE_URL=https://thermograph.org
 #THERMOGRAPH_DISCORD_CLIENT_SECRET=
 # Gateway bot (opt-in): holds a live websocket so the bot replies to messages that
 # @mention it (or DM it) with a city grade — e.g. "@Thermograph Phoenix". Reuses
-# THERMOGRAPH_DISCORD_BOT_TOKEN above. Runs in the single-replica daemon service
+# THERMOGRAPH_DISCORD_BOT_TOKEN above. Runs on the single notifier leader only, so
-# (which also needs THERMOGRAPH_INTERNAL_TOKEN below — it grades via the
+# it needs THERMOGRAPH_ENABLE_NOTIFIER on (the default). No privileged intent
-# backend's internal API). No privileged intent required — Discord delivers
+# required — Discord delivers content for mentions/DMs. Unset/0 => no gateway
-# content for mentions/DMs. Unset/0 => no gateway connection.
+# connection. (Code: thermograph-backend notifications/discord_bot.py.)
 # (Code: backend/daemon/, calling backend's /internal/discord/grade.)
 #THERMOGRAPH_DISCORD_BOT=1
--- a/observability/.env.example
+++ b/observability/.env.example
@ -18,3 +18,14 @@ GF_SECURITY_ADMIN_PASSWORD=change-me-to-something-strong
 OAUTH_ENABLED=false
 GOOGLE_CLIENT_ID=
 GOOGLE_CLIENT_SECRET=
 # --- Alerting (Discord) ----------------------------------------------------------
 # Where every alert in grafana/provisioning/alerting/ is delivered: a webhook on
 # the private #ops-alerts channel (Owners category) of the Thermograph.org
 # Discord. NOT email — beta's Grafana relays SMTP through prod's Postfix, so
 # email alerts die exactly when prod does.
 #
 # To mint one: Discord -> #ops-alerts -> Edit Channel -> Integrations ->
 # Webhooks -> New Webhook -> Copy Webhook URL. Treat it like a password; anyone
 # with it can post to the channel. Grafana refuses to start without it.
 DISCORD_ALERT_WEBHOOK_URL=https://discord.com/api/webhooks/<id>/<token>
--- a/observability/CLAUDE.md
+++ b/observability/CLAUDE.md
@ -16,6 +16,14 @@ the full per-node procedure.
 - `loki/config.yml` — Loki config (mesh-only, filesystem storage).
 - `grafana/provisioning/` — datasource (Loki) + dashboard provider, auto-loaded
  at startup. `grafana/dashboards/*.json` — the dashboards themselves.
 - `grafana/provisioning/alerting/` — the alert rules, the Discord contact point
  and the notification policy, also auto-loaded at startup. Same rule as the
  dashboards: **the repo is the only durable path**, UI edits get overwritten.
  Alerts go to Discord `#ops-alerts`, never email — beta's Grafana relays SMTP
  through prod's Postfix, so email dies exactly when prod does. Every rule is
  LogQL (there is no Prometheus anywhere in the fleet). Thresholds were derived
  from real Loki data and the working is in the comments beside each rule —
  re-derive before changing a number rather than guessing.
 - `alloy/config.alloy` + `alloy/docker-compose.agent.yml` — the per-node log
  shipper. The node name comes from `ALLOY_NODE` (set per host).
 - `caddy-grafana.conf` — the Caddy vhost for `dashboard.thermograph.org` (lives
@ -25,10 +33,16 @@ the full per-node procedure.
 ## Conventions
- **Every artifact that ships to a node is CI-validated** (`.forgejo/workflows/validate.yml`):
+- **Every artifact that ships to a node is CI-validated**
-  both compose files parse, all dashboard JSON is valid, and the Loki/provisioning
+  (`.forgejo/workflows/observability-validate.yml`): both compose files parse, all
-  YAML parses. Keep new dashboards as valid JSON and new config as valid YAML or
+  dashboard JSON is valid, and the Loki/provisioning YAML parses. Keep new
-  CI fails. The Alloy config is not yet CI-validated (needs the `alloy` binary).
+  dashboards as valid JSON and new config as valid YAML or CI fails. The alerting
  config gets a stricter third step — every rule's `condition` must name a refId
  that exists, every policy must route to a receiver that exists, and a literal
  Discord webhook URL in the repo is a hard failure. (A rule pointing at a missing
  refId is valid YAML, provisions cleanly, and then never fires; that is precisely
  the silent-no-op this whole domain exists to prevent.) The Alloy config is still
  not CI-validated (needs the `alloy` binary).
 - The public hostname is **`dashboard.thermograph.org`** everywhere — not
  `grafana.thermograph.org`. Match it in any new comment/config.
 - Secrets (OAuth client id/secret, admin password) live only in the host `.env`,
--- a/observability/README.md
+++ b/observability/README.md
@ -31,6 +31,7 @@ persistent, queryable, fleet-wide log store and UI.
 | `docker-compose.yml` | the central Loki + Grafana stack (runs on beta) |
 | `loki/config.yml` | Loki config (filesystem, retention) |
 | `grafana/provisioning/` | auto-wired Loki datasource + dashboard provider |
 | `grafana/provisioning/alerting/` | contact point, notification policy and alert rules — see [Alerting](#alerting) |
 | `grafana/dashboards/thermograph-logs.json` | the fleet-logs dashboard |
 | `alloy/config.alloy` | the per-node collector config |
 | `alloy/docker-compose.agent.yml` | runs the Alloy agent on a node |
@ -38,6 +39,22 @@ persistent, queryable, fleet-wide log store and UI.
 ## Deploy
 > ### ⚠️ Merging is not deploying
 >
 > `/opt/observability` on beta (and prod) is a checkout of the **archived**
 > `emi/thermograph-observability` repo. It can never `git pull` again — the
 > content now lives here, in the mono repo, under `observability/`. A previous
 > observability PR merged and had **zero live effect** for exactly this reason.
 >
 > Until someone re-points those checkouts at the mono repo, every change here
 > ships by hand: back up the live file with a UTC-timestamped suffix, `scp` the
 > new one up, `sudo cp` it into place, restart the specific container. Use
 > `sudo docker restart <container>` — **not `docker compose`**, which on prod
 > demands a `GF_SECURITY_ADMIN_PASSWORD` it has no `.env` to interpolate from.
 > (Beta's stack does have an `.env`, so compose works there — and is required
 > when a change adds a new environment variable, since `docker restart` alone
 > will not pick one up.)
 ### 1. Central stack (on beta)
 ```bash
@ -113,6 +130,161 @@ log volume by service, app error rate by tag, upstream 429 count, Caddy 5xx
 count, a per-node notifier-liveness indicator (from the heartbeat log), recent app errors, and a live all-container
 tail. Edit it in Grafana and re-export the JSON here to version a change.
 ## Alerting
 Provisioned from `grafana/provisioning/alerting/`, mounted read-only into Grafana
 by the compose file, so **the repo is the only durable path** — as with the
 dashboards, edits made in the Grafana UI are overwritten on the next provision.
 Every rule is **LogQL**, because there is no Prometheus in the fleet: Loki is the
 only datasource, so every signal is derived from log lines.
 ### Where alerts go: Discord, not email
 `#ops-alerts` (Owners category, private to Emi and Jin) via Grafana's native
 Discord contact point. Deliberately **not email**:
 - Grafana's factory default pointed at the literal string `<example@email.com>` —
  a paging path that had never delivered a message to anyone.
 - beta's Grafana relays SMTP through **prod's** Postfix at `10.10.0.1:25` (see
  `docker-compose.override.yml` on beta). Email alerts therefore travel through
  the box most likely to be on fire, and vanish exactly when they matter.
 - Discord is off-estate, works when prod is dead, and reaches a phone.
 `#ops-alerts` is a new dedicated channel, not one of the existing ones: the
 `#dev` / `#uat` / `#prod` channels are the *product's* alert-subscription output
 (real deliveries — routing ops noise there would corrupt the evidence), and
 `#notes` / `#staff-chat` are for humans.
 The webhook URL is a secret and is **not in this repo**: it comes from
 `DISCORD_ALERT_WEBHOOK_URL` in beta's gitignored `.env`, which compose passes to
 Grafana and which Grafana interpolates when reading the provisioning file.
 Grafana refuses to start if it is unset.
 ### The rules
 | Rule | Fires when | Severity |
 |------|-----------|----------|
 | `ProdEdgeDark` | zero prod Caddy log lines in 15m — site dark, or the log pipeline is broken | critical |
 | `ProdHealthzFailing` | any non-200 on `/healthz` in 10m | critical |
 | `ProdHTTP5xxBurst` | ≥10 prod 5xx in 10m | critical |
 | `ProdHTTP5xxElevated` | ≥25 prod 5xx accumulated over 6h (the error-budget alarm) | warning |
 | `ProdUnhandledExceptions` | ≥3 app `tag=http.error` in 30m | warning |
 | `NotifierHeartbeatMissingProd` | no `tag=heartbeat` line from prod in 20m | critical |
 | `NotifierHeartbeatMissingBeta` | same, on beta | warning |
 | `ProdWebContainerSilent` | `thermograph_web` logged nothing in 30m | critical |
 | `ProdDbContainerSilent` | `thermograph_db` logged nothing in 2h | critical |
 | `ProdFrontendContainerSilent` | `thermograph_frontend` logged nothing in 30m | warning |
 | `ProdWorkerContainerSilent` | `thermograph_worker` logged nothing in 2h | warning |
 | `AlertingWatchdog` | always — one Discord message per day | watchdog |
 Every threshold was backtested against real Loki data; the working is written out
 in the comments of `rules-prod.yml`, next to the rule it justifies. Read those
 before changing a number.
 `AlertingWatchdog` is the dead-man's switch and fires permanently by design. **Its
 value is its absence**: nothing can report its own death, so if `#ops-alerts` has
 been silent for more than a day, alerting is broken rather than the estate
 healthy. Set `isPaused: true` on it to opt out of that guarantee.
 ### Deploying an alerting change to beta
 Grafana already mounts `./grafana/provisioning` read-only, so new files under
 `alerting/` need no compose change — but the **first** deploy does, because it
 introduces `DISCORD_ALERT_WEBHOOK_URL`, and `docker restart` will not pick up a
 new environment variable.
 ```bash
 # run from the mono-repo root
 BETA="agent@75.119.132.91"
 SSH="ssh -i ~/.ssh/thermograph_agent_ed25519"
 TS=$(date -u +%Y%m%dT%H%M%SZ)
 # 1. put the secret in beta's .env (once), then confirm it took:
 $SSH $BETA "grep -q '^DISCORD_ALERT_WEBHOOK_URL=' /opt/observability/.env && echo present || echo MISSING"
 # 2. back up the live compose file, stage the new files, move them into place:
 $SSH $BETA "sudo cp -a /opt/observability/docker-compose.yml /opt/observability/docker-compose.yml.$TS \
            && rm -rf /tmp/obs-stage && mkdir -p /tmp/obs-stage/alerting"
 scp -i ~/.ssh/thermograph_agent_ed25519 observability/docker-compose.yml $BETA:/tmp/obs-stage/
 scp -i ~/.ssh/thermograph_agent_ed25519 observability/grafana/provisioning/alerting/*.yml $BETA:/tmp/obs-stage/alerting/
 $SSH $BETA "sudo mkdir -p /opt/observability/grafana/provisioning/alerting \
            && sudo cp /tmp/obs-stage/docker-compose.yml /opt/observability/ \
            && sudo cp /tmp/obs-stage/alerting/*.yml /opt/observability/grafana/provisioning/alerting/"
 # 3. recreate Grafana so it reads the new env var AND the new provisioning.
 #    `docker restart` is NOT enough here — it will not pick up a new env var.
 $SSH $BETA "cd /opt/observability && docker compose up -d grafana"
 ```
 Subsequent rule-only changes are just step 2 plus
 `sudo docker restart observability-grafana-1` — no compose needed, because no
 new environment variable is involved.
 **Rollback:** the provisioning files are additive. To back the whole thing out,
 `sudo rm -rf /opt/observability/grafana/provisioning/alerting`, restore
 `docker-compose.yml.<timestamp>`, and `docker compose up -d grafana`. Grafana
 falls back to its default (email → `<example@email.com>`) root policy — i.e. no
 alerting, which is where this started.
 Alerting lives on **beta only** — prod and dev run Alloy agents, not Grafana, so
 nothing in `alerting/` ever ships to them.
 ### Verify alerting
 ```bash
 # on beta, after deploying and restarting Grafana:
 GFP=$(grep '^GF_SECURITY_ADMIN_PASSWORD=' /opt/observability/.env | cut -d= -f2-)
 WH=$(grep '^DISCORD_ALERT_WEBHOOK_URL=' /opt/observability/.env | cut -d= -f2-)
 # 1. all 12 rules loaded and healthy (health should be "ok", never "error"):
 curl -s -u admin:"$GFP" localhost:3000/api/prometheus/grafana/api/v1/rules \
  | python3 -c 'import json,sys; [print(r["name"], r["state"], r["health"]) for g in json.load(sys.stdin)["data"]["groups"] for r in g["rules"]]'
 # 2. the root policy points at Discord, NOT grafana-default-email:
 curl -s -u admin:"$GFP" localhost:3000/api/v1/provisioning/policies
 # 3. force a real message into #ops-alerts (does not persist anything):
 curl -s -u admin:"$GFP" -H 'Content-Type: application/json' \
  -X POST localhost:3000/api/alertmanager/grafana/config/api/v1/receivers/test \
  -d '{"receivers":[{"name":"t","grafana_managed_receiver_configs":[{"name":"t","type":"discord","settings":{"url":"'"$WH"'"}}]}]}'
 #    -> {"receivers":[{... "status":"ok"}]} and a message in #ops-alerts.
 ```
 The webhook URL is redacted (`[REDACTED]`) by
 `GET /api/v1/provisioning/contact-points`, so you **cannot** confirm
 interpolation by reading it back — the only proof is a message actually arriving
 in `#ops-alerts`. A contact point holding the literal string
 `$DISCORD_ALERT_WEBHOOK_URL` looks completely healthy from the API and pages
 nobody.
 ### The leftover default contact point
 Grafana's built-in `grafana-default-email` → `<example@email.com>` receiver is
 created by Grafana's *default Alertmanager config*, not by provisioning, and its
 uid is the empty string. `deleteContactPoints` matches on uid, so **provisioning
 cannot remove it** (`DELETE /api/v1/provisioning/contact-points/` with an empty
 uid returns 404 — verified).
 Provisioning the notification policy already makes it inert: nothing routes there
 any more, so it cannot receive anything. Actually deleting the row is a one-time
 manual step, and only works *after* this config is live (Grafana refuses to
 delete a receiver a policy still references). Either use the UI — Alerting →
 Contact points → `grafana-default-email` → Delete — or:
 ```bash
 # on beta. Reads the running Alertmanager config, drops the one receiver,
 # writes it back. Verified against Grafana 11.6.1 (returns 202).
 GFP=$(grep '^GF_SECURITY_ADMIN_PASSWORD=' /opt/observability/.env | cut -d= -f2-)
 AM=localhost:3000/api/alertmanager/grafana/config/api/v1/alerts
 curl -s -u admin:"$GFP" $AM > /tmp/am.json
 python3 -c "
 import json; d=json.load(open('/tmp/am.json')); c=d['alertmanager_config']
 c['receivers']=[r for r in c['receivers'] if r['name']!='grafana-default-email']
 json.dump(d, open('/tmp/am2.json','w'))"
 curl -s -u admin:"$GFP" -H 'Content-Type: application/json' -X POST --data @/tmp/am2.json $AM
 rm -f /tmp/am.json /tmp/am2.json
 ```
 This is cosmetic — leaving it costs nothing but a confusing row in the UI.
 ## Notes / follow-ups
 - **Loki is mesh-only** by design; only Grafana (with auth) is public.
@ -121,3 +293,32 @@ tail. Edit it in Grafana and re-export the JSON here to version a change.
 - The app emits a `tag=heartbeat` line for the subscription notifier every
  ~15m; the dashboard's "Notifier alive?" panel turns red if a node misses it.
  Extend the same `audit.log_heartbeat()` to any future background daemon.
 ### Known gaps
 - **Alloy can silently stop tailing a container after Swarm replaces the task.**
  Observed on prod on 2026-07-24: `thermograph_worker` was `1/1`, healthy, and
  writing ~19 lines per 10 minutes to its own docker log, yet not one line had
  reached Loki since 04:43Z — a 13½-hour hole nobody could see, because nothing
  was watching. `ProdWorkerContainerSilent` now catches exactly this.
  What localises it to Alloy rather than the app: prod's `tag=heartbeat` lines
  kept arriving in *every* 20-minute window throughout, and those come from a
  different Alloy source (`loki.source.file` on `/applogs`) than the container
  stdout (`loki.source.docker`). One source went deaf, the other did not.
  The workaround is `sudo docker restart alloy-alloy-1` on the affected node;
  the fix is probably an explicit `refresh_interval` on `discovery.docker` plus
  a look at whether `loki.source.docker` re-resolves targets. Not root-caused.
 - **Some services are too quiet to alert on.** `thermograph_daemon` (23 log
  lines/24h), `thermograph_autoscaler` (1) and `thermograph_autoscaler-lake` (2)
  have a healthy floor indistinguishable from dead, and `thermograph_lake` is
  bursty ETL with legitimate idle gaps. Giving each one an
  `audit.log_heartbeat()` line would make all four alertable with the same
  pattern the notifier already uses. That is the single highest-value follow-up
  here.
 - **Grafana cannot alert on its own death** — that is what `AlertingWatchdog`
  is for, but it is a *manual* dead-man's switch: it relies on somebody noticing
  that the daily message stopped. A real external watchdog (an uptime pinger
  hitting a Grafana endpoint from off-estate) would close this properly.
 - **beta runs a `docker-compose.override.yml` that is not in this repo**, wiring
  Grafana's SMTP to prod's Postfix. It should be mirrored into the tracked
  compose file or deleted; alerting no longer depends on it.
--- a/observability/docker-compose.yml
+++ b/observability/docker-compose.yml
@ -58,6 +58,14 @@ services:
      # verifies email ownership and it's the only OAuth provider, so there's no
      # cross-provider takeover vector the "insecure" name warns about.
      GF_AUTH_OAUTH_ALLOW_INSECURE_EMAIL_LOOKUP: "true"
      # --- Alerting --------------------------------------------------------------
      # The Discord webhook that grafana/provisioning/alerting/contact-points.yml
      # interpolates as $DISCORD_ALERT_WEBHOOK_URL. It is a secret (holding it is
      # enough to post in the channel), so it lives only in beta's .env.
      # Required, not defaulted: a Grafana that comes up with an empty webhook
      # looks perfectly healthy and pages nobody, which is the failure mode this
      # whole config exists to end. Better to refuse to start.
      DISCORD_ALERT_WEBHOOK_URL: ${DISCORD_ALERT_WEBHOOK_URL:?set DISCORD_ALERT_WEBHOOK_URL — see .env.example}
    volumes:
      - ./grafana/provisioning:/etc/grafana/provisioning:ro
      - ./grafana/dashboards:/var/lib/grafana/dashboards:ro
--- a/observability/grafana/provisioning/alerting/contact-points.yml
+++ b/observability/grafana/provisioning/alerting/contact-points.yml
@ -0,0 +1,64 @@
 # Where alerts go. ONE destination: a Discord webhook posting into the private
 # #ops-alerts channel (Owners category) of the Thermograph.org server.
 #
 # WHY NOT EMAIL. The Grafana factory default routes to the literal string
 # "<example@email.com>", i.e. nowhere. Fixing it by pointing at a real mailbox
 # would still be wrong: beta's Grafana relays SMTP through prod's Postfix at
 # 10.10.0.1:25 (see the live docker-compose.override.yml on beta), so email
 # alerts travel *through the box most likely to be on fire* and are silently
 # lost whenever Postfix is down — which is exactly when you need them. Discord
 # is off-estate: it works when prod is dead, and it works from a phone.
 #
 # THE WEBHOOK URL IS A SECRET. Anyone holding it can post into the channel, so
 # it is NOT in this repo. It is read from the Grafana process environment, which
 # docker-compose.yml feeds from beta's gitignored .env (see .env.example).
 # Grafana expands $VAR / $__env{VAR} when it reads provisioning files; if the
 # variable is unset the contact point ends up with a literal "$DISCORD_..."
 # string and every notification fails — see README "Verify alerting".
 #
 # To rotate: make a new webhook on the #ops-alerts channel, replace the value in
 # beta's .env, `docker restart observability-grafana-1`, delete the old webhook
 # in Discord.
 apiVersion: 1
 contactPoints:
  - orgId: 1
    name: thermograph-ops-discord
    receivers:
      - uid: tg_ops_discord
        type: discord
        # Send a green "resolved" message too — a page you never see close is a
        # page you stop trusting.
        disableResolveMessage: false
        settings:
          url: $DISCORD_ALERT_WEBHOOK_URL
          use_discord_username: false
          title: '{{ if eq .Status "firing" }}[FIRING]{{ else }}[RESOLVED]{{ end }} {{ .CommonLabels.alertname }}'
          # Deliberately plain. A template error here breaks EVERY notification
          # silently, so this uses only functions verified against Grafana
          # 11.6.1 via /api/alertmanager/grafana/config/api/v1/receivers/test.
          message: |-
            {{ range .Alerts }}**severity:** {{ .Labels.severity }} · **host:** {{ .Labels.host }}
            {{ .Annotations.summary }}
            {{ .Annotations.description }}
            `value: {{ .ValueString }}`
            {{ end }}
            <https://dashboard.thermograph.org/alerting/list>
 # --- The factory-default contact point -------------------------------------------
 # Grafana ships a built-in receiver "grafana-default-email" whose only integration
 # is `email receiver` -> "<example@email.com>". It is created by Grafana's default
 # Alertmanager config, NOT by provisioning, and its uid is the empty string:
 #
 #   GET /api/v1/provisioning/contact-points
 #   -> [{"name":"email receiver","type":"email","settings":{...},"uid":""}]
 #
 # Because `deleteContactPoints` matches on uid, provisioning CANNOT remove it —
 # DELETE /api/v1/provisioning/contact-points/ with an empty uid 404s (verified
 # against 11.6.1). What notification-policies.yml *does* do is stop anything ever
 # routing to it, which makes it inert: it can no longer receive anything.
 #
 # Deleting the row itself is cosmetic and manual, and only works once no policy
 # references it (i.e. after this config is live) — either the UI (Alerting ->
 # Contact points -> grafana-default-email -> Delete) or the Alertmanager config
 # API. The README's "The leftover default contact point" has the exact command.
--- a/observability/grafana/provisioning/alerting/notification-policies.yml
+++ b/observability/grafana/provisioning/alerting/notification-policies.yml
@ -0,0 +1,64 @@
 # The routing tree. Provisioning `policies:` REPLACES Grafana's root policy, so
 # applying this file is what actually severs the factory default
 # (`receiver: grafana-default-email` -> "<example@email.com>") and points the
 # whole estate at Discord. Everything lands in the same #ops-alerts channel; the
 # severity branches differ only in how *insistent* they are.
 #
 # Timing rationale, for a solo operator on a phone:
 #   critical — 10s wait so a burst arrives as one message, re-nag hourly while
 #              it is still broken. You want to be pestered about a dead prod.
 #   warning  — 2m wait (lets a flapping thing settle), re-nag twice a day.
 #              Not worth waking up for; worth not forgetting.
 #   watchdog — the dead-man's-switch. Fires forever by design, so it is throttled
 #              to one message per day. Its VALUE IS ITS ABSENCE: if #ops-alerts
 #              is silent for >24h, Grafana/Loki/the mesh is down and no other
 #              alert in this file can reach you either.
 #
 # There is deliberately no catch-all branch: an alert whose `severity` label is
 # missing or unrecognised matches none of the routes and falls through to the
 # ROOT receiver, which is also Discord. Nothing can be silently swallowed by a
 # typo in a label — it just arrives on the default (4h) cadence instead.
 apiVersion: 1
 policies:
  - orgId: 1
    receiver: thermograph-ops-discord
    # Collapse by rule + host so a whole-node outage (every prod rule firing at
    # once) still arrives as a handful of messages, not thirty.
    group_by:
      - alertname
      - host
    group_wait: 30s
    group_interval: 5m
    repeat_interval: 4h
    routes:
      - receiver: thermograph-ops-discord
        object_matchers:
          - ['severity', '=', 'watchdog']
        group_wait: 0s
        group_interval: 24h
        repeat_interval: 24h
        continue: false
      - receiver: thermograph-ops-discord
        object_matchers:
          - ['severity', '=', 'critical']
        group_by:
          - alertname
          - host
        group_wait: 10s
        group_interval: 5m
        repeat_interval: 1h
        continue: false
      - receiver: thermograph-ops-discord
        object_matchers:
          - ['severity', '=', 'warning']
        group_by:
          - alertname
          - host
        group_wait: 2m
        group_interval: 10m
        repeat_interval: 12h
        continue: false
--- a/observability/grafana/provisioning/alerting/rules-prod.yml
+++ b/observability/grafana/provisioning/alerting/rules-prod.yml
@ -0,0 +1,836 @@
 # Thermograph alert rules — the estate's first. All Loki/LogQL, because there is
 # no Prometheus anywhere in the fleet: Loki is the only datasource, so every
 # signal here is derived from log lines rather than metrics.
 #
 # ---------------------------------------------------------------------------
 # HOW THE THRESHOLDS WERE CHOSEN
 # ---------------------------------------------------------------------------
 # Every number below was backtested against the real 24h ending 2026-07-24
 # ~19:45Z. That window contains a genuine ~20-minute prod outage starting
 # 2026-07-24 04:20Z, which is the event these rules must catch:
 #
 #   10-min buckets of prod Caddy 5xx over 24h (only non-zero buckets shown):
 #     2, 45, 22, 3, 1, 1, 1, 1, 1, 1, 2, 1, 5, 1        (total 86)
 #              ^^^^^^  the outage: 64% and 56% of all requests 5xx
 #   All five /healthz failures in the whole day fell inside those two buckets.
 #   Prod Caddy request volume: 24-170 per 10 min (~7 req/min). Low traffic —
 #   which is why absolute counts beat error *ratios* here: at 7 req/min a single
 #   bad request is 1.4% and a ratio alert would scream all night.
 #
 # The design goal was: fire on the outage, stay silent for the other 142 of 144
 # buckets, and still surface the slow drip of ~20 scattered errors that made up
 # the rest of the day's 86 (that is what ProdHTTP5xxElevated is for).
 #
 # ---------------------------------------------------------------------------
 # TWO CONVENTIONS WORTH KNOWING
 # ---------------------------------------------------------------------------
 # 1. `or vector(0)`. An absent Loki stream returns *no series*, not zero — so a
 #    naive "count < 1" silence alert goes to NoData instead of firing, which is
 #    the exact failure mode of every homemade log alert ever written. Appending
 #    `or vector(0)` makes an empty result evaluate to a real 0. Verified working
 #    on Loki 3.5.1.
 # 2. Prod addressing. Prod is Swarm, and its docker streams carry NO `service`
 #    label — only `container`, which includes a per-task suffix that changes on
 #    every redeploy (thermograph_web.1.<taskid>). So prod services are matched as
 #    container=~"thermograph_<name>.+" with job="docker". Do not use `service=`
 #    here; it only works on the compose hosts (beta, dev).
 #
 # ---------------------------------------------------------------------------
 # NOT ALERTABLE, ON PURPOSE
 # ---------------------------------------------------------------------------
 # thermograph_daemon (23 log lines/24h), thermograph_autoscaler (1) and
 # thermograph_autoscaler-lake (2) are too quiet for a log-silence rule — their
 # healthy floor is indistinguishable from dead. thermograph_lake is bursty ETL
 # with long legitimate idle gaps. Catching those needs either a heartbeat line
 # (extend backend/core/audit.py log_heartbeat(), as the notifier already does)
 # or container metrics, which the fleet does not collect. Tracked as a gap.
 #
 # Error budget / failure isolation: only ProdEdgeDark escalates on NoData or
 # datasource errors, so a Loki or WireGuard blip produces ONE page rather than
 # twelve. Every other rule degrades to OK if the query cannot run.
 apiVersion: 1
 groups:
  # ===========================================================================
  # Is the product up? Evaluated every minute; all of these page.
  # ===========================================================================
  - orgId: 1
    name: prod-availability
    folder: Alerts
    interval: 1m
    rules:
      # -- 1 -----------------------------------------------------------------
      # The top-level "is anything alive" check, and deliberately the only rule
      # that also owns NoData/error: Caddy fronts every request to
      # thermograph.org, so zero access-log lines means the site is dark, the
      # box is gone, Alloy died, or the mesh dropped. Any of those is a page.
      #
      # DATA: 15-minute windows over 24h ranged 35 -> 301 lines, minimum 35,
      # and never once reached zero — including *during* the outage (72), when
      # Caddy was happily logging 503s. Zero is unambiguous.
      - uid: tg_prod_edge_dark
        title: ProdEdgeDark
        condition: C
        for: 5m
        noDataState: Alerting
        execErrState: Alerting
        annotations:
          summary: 'prod: no Caddy access-log lines for 15 minutes'
          description: >-
            Nothing has reached thermograph.org's reverse proxy in 15 minutes
            (healthy floor is 35 lines/15m, and it stayed at 72 even during the
            04:20Z outage). Either prod is down, Caddy is down, prod's Alloy
            agent stopped shipping, or the WireGuard mesh to beta is broken.
            This rule also fires on Loki NoData/errors, so it doubles as the
            "we have lost observability" alarm — if it is the only thing firing,
            suspect the pipeline before suspecting the app.
        labels:
          severity: critical
          host: prod
          class: availability
        isPaused: false
        data:
          - refId: A
            relativeTimeRange: { from: 900, to: 0 }
            datasourceUid: loki
            model:
              refId: A
              datasource: { type: loki, uid: loki }
              expr: 'sum(count_over_time({job="caddy", host="prod"} [15m])) or vector(0)'
              queryType: instant
              editorMode: code
              intervalMs: 1000
              maxDataPoints: 43200
          - refId: B
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: B
              type: reduce
              reducer: last
              expression: A
              datasource: { type: __expr__, uid: __expr__ }
          - refId: C
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: C
              type: threshold
              expression: B
              datasource: { type: __expr__, uid: __expr__ }
              conditions:
                - evaluator: { type: lt, params: [1] }
      # -- 2 -----------------------------------------------------------------
      # /healthz is the app's own statement that it is well. When it 503s, it is
      # not an opinion.
      #
      # DATA: 57 /healthz requests in 24h (52x 200, 5x 503) — roughly 2.4/hour,
      # so this endpoint is polled sparsely and irregularly. That is why `for`
      # is 0m: waiting for a second consecutive bad sample could take 25
      # minutes. All 5 failures landed inside the two outage buckets and nowhere
      # else in the day, so one failure is already a real signal, not noise.
      - uid: tg_prod_healthz_failing
        title: ProdHealthzFailing
        condition: C
        for: 0m
        noDataState: OK
        execErrState: OK
        annotations:
          summary: 'prod: /healthz returned a non-200'
          description: >-
            The app's own liveness endpoint failed at the edge in the last 10
            minutes. Over the backtested 24h this happened 5 times, all of them
            inside the 04:20Z outage — /healthz failing has a 100% correlation
            with prod actually being broken. Check `docker service ps
            thermograph_web` on prod first.
        labels:
          severity: critical
          host: prod
          class: availability
        isPaused: false
        data:
          - refId: A
            relativeTimeRange: { from: 600, to: 0 }
            datasourceUid: loki
            model:
              refId: A
              datasource: { type: loki, uid: loki }
              expr: 'sum(count_over_time({job="caddy", host="prod"} | json | request_uri="/healthz" | status!="200" [10m])) or vector(0)'
              queryType: instant
              editorMode: code
              intervalMs: 1000
              maxDataPoints: 43200
          - refId: B
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: B
              type: reduce
              reducer: last
              expression: A
              datasource: { type: __expr__, uid: __expr__ }
          - refId: C
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: C
              type: threshold
              expression: B
              datasource: { type: __expr__, uid: __expr__ }
              conditions:
                - evaluator: { type: gt, params: [0] }
      # -- 3 -----------------------------------------------------------------
      # The acute "users are seeing errors right now" page.
      #
      # DATA: of 144 ten-minute buckets in the backtest, 130 had zero 5xx and
      # the largest *benign* bucket had 5. The outage produced 45 and 22.
      # Threshold 10 (evaluator gt 9, integer counts) sits at 2x the worst
      # benign bucket and 4.5x below the incident peak — it fires on the outage
      # and on nothing else in the entire day. It catches 67 of that day's 86
      # errors as a single incident; the remaining scattered ~20 are picked up
      # by ProdHTTP5xxElevated below rather than by pushing this threshold down
      # into the noise.
      - uid: tg_prod_5xx_burst
        title: ProdHTTP5xxBurst
        condition: C
        for: 2m
        noDataState: OK
        execErrState: OK
        annotations:
          summary: 'prod: 10+ HTTP 5xx in the last 10 minutes'
          description: >-
            Prod is serving server errors at an outage-grade rate. Baseline is
            zero in 90% of 10-minute windows and never above 5; the reference
            incident hit 45. Look at the "Thermograph — Traffic & Domains"
            dashboard, panel "HTTP errors by status", and at
            {job="app-json", host="prod", filename=~".+/errors/.+"} for the
            exceptions behind them.
        labels:
          severity: critical
          host: prod
          class: errors
        isPaused: false
        data:
          - refId: A
            relativeTimeRange: { from: 600, to: 0 }
            datasourceUid: loki
            model:
              refId: A
              datasource: { type: loki, uid: loki }
              expr: 'sum(count_over_time({job="caddy", host="prod"} | json | status>=500 [10m])) or vector(0)'
              queryType: instant
              editorMode: code
              intervalMs: 1000
              maxDataPoints: 43200
          - refId: B
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: B
              type: reduce
              reducer: last
              expression: A
              datasource: { type: __expr__, uid: __expr__ }
          - refId: C
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: C
              type: threshold
              expression: B
              datasource: { type: __expr__, uid: __expr__ }
              conditions:
                - evaluator: { type: gt, params: [9] }
  # ===========================================================================
  # Slow-burn error signals. Five-minute evaluation is plenty; these are tickets,
  # not pages.
  # ===========================================================================
  - orgId: 1
    name: prod-errors
    folder: Alerts
    interval: 5m
    rules:
      # -- 4 -----------------------------------------------------------------
      # The rule that would have broken the silence on "84 5xx in 24h and nobody
      # was told". ProdHTTP5xxBurst only sees spikes; this sees accumulation.
      #
      # DATA: 6-hour rolling windows sampled hourly across the backtest read
      # 1, 2, 2, 2, 2, 2, 2, 67, 72, 73, 74, 74, 74, 7, 2, 1, 1, 4, 11.
      # The largest window containing NO incident was 11. Threshold 25
      # (evaluator gt 24) is 2.3x that — quiet on the drip alone, loud on any
      # day that looks like this one. Deliberately a long window: it will stay
      # firing for ~6h after an incident, which is the point. That is the daily
      # error budget talking, not a flap.
      - uid: tg_prod_5xx_elevated
        title: ProdHTTP5xxElevated
        condition: C
        for: 10m
        noDataState: OK
        execErrState: OK
        annotations:
          summary: 'prod: 25+ HTTP 5xx accumulated over 6 hours'
          description: >-
            Sustained server errors. This is the error-budget alarm, not an
            outage alarm — it catches the scattered drip that never trips the
            burst rule but still added up to 86 errors (1.7% of requests) on
            2026-07-24. Expect it to stay firing for up to 6h after an incident
            clears; that is the window emptying, not a new fault.
        labels:
          severity: warning
          host: prod
          class: errors
        isPaused: false
        data:
          - refId: A
            relativeTimeRange: { from: 21600, to: 0 }
            datasourceUid: loki
            model:
              refId: A
              datasource: { type: loki, uid: loki }
              expr: 'sum(count_over_time({job="caddy", host="prod"} | json | status>=500 [6h])) or vector(0)'
              queryType: instant
              editorMode: code
              intervalMs: 1000
              maxDataPoints: 43200
          - refId: B
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: B
              type: reduce
              reducer: last
              expression: A
              datasource: { type: __expr__, uid: __expr__ }
          - refId: C
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: C
              type: threshold
              expression: B
              datasource: { type: __expr__, uid: __expr__ }
              conditions:
                - evaluator: { type: gt, params: [24] }
      # -- 5 -----------------------------------------------------------------
      # A different signal from the Caddy 5xx rules, not a duplicate: Caddy sees
      # 502/503 (the app is unreachable), while tag="http.error" is the app
      # catching an unhandled exception and still answering. A bad deploy shows
      # up here first, and can do so without moving the Caddy numbers at all.
      #
      # DATA: exactly 2 unhandled 500s in 24h, never more than 1 in any hour.
      # Threshold 3-per-30min (evaluator gt 2) is 3x the observed daily peak:
      # silent today, loud on an exception storm.
      - uid: tg_prod_unhandled_exceptions
        title: ProdUnhandledExceptions
        condition: C
        for: 0m
        noDataState: OK
        execErrState: OK
        annotations:
          summary: 'prod: 3+ unhandled exceptions in 30 minutes'
          description: >-
            The app is throwing. Normal is under 2 per DAY. A cluster this tight
            almost always means a regression just shipped — check the app.start
            lines on the "App Activity & Delivery" dashboard for a recent deploy,
            then {job="app-json", host="prod", filename=~".+/errors/.+"} |
            tag="http.error" for the traceback and route.
        labels:
          severity: warning
          host: prod
          class: errors
        isPaused: false
        data:
          - refId: A
            relativeTimeRange: { from: 1800, to: 0 }
            datasourceUid: loki
            model:
              refId: A
              datasource: { type: loki, uid: loki }
              expr: 'sum(count_over_time({job="app-json", host="prod", filename=~".+/errors/.+"} | tag="http.error" [30m])) or vector(0)'
              queryType: instant
              editorMode: code
              intervalMs: 1000
              maxDataPoints: 43200
          - refId: B
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: B
              type: reduce
              reducer: last
              expression: A
              datasource: { type: __expr__, uid: __expr__ }
          - refId: C
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: C
              type: threshold
              expression: B
              datasource: { type: __expr__, uid: __expr__ }
              conditions:
                - evaluator: { type: gt, params: [2] }
  # ===========================================================================
  # The subscription notifier — the thing that actually delivers the product.
  # ===========================================================================
  - orgId: 1
    name: notifier
    folder: Alerts
    interval: 5m
    rules:
      # -- 6 -----------------------------------------------------------------
      # backend/core/audit.py emits a tag=heartbeat line every ~15 minutes. If it
      # stops, subscribers stop getting weather alerts and absolutely nothing
      # else in this file would notice — the site stays up and serves 200s while
      # the product quietly stops working.
      #
      # DATA: prod produced 1-2 beats in every single 20-minute bucket across
      # the full 24h backtest. Not one empty bucket. `for: 5m` at a 5m interval
      # means two consecutive empty evaluations, so roughly 25 minutes of real
      # silence before it pages — about one and a half missed beats.
      #
      # dev is deliberately excluded: the LAN box had zero heartbeats for the
      # first 10.5h of the backtest window because it was simply off. Alerting
      # on dev would mean an alert that is firing more often than not.
      - uid: tg_notifier_heartbeat_prod
        title: NotifierHeartbeatMissingProd
        condition: C
        for: 5m
        noDataState: OK
        execErrState: OK
        annotations:
          summary: 'prod: no notifier heartbeat for 20 minutes'
          description: >-
            The subscription notifier has gone quiet on prod. Expected cadence is
            ~15 minutes and it did not miss a single 20-minute window over the
            backtested day. Subscribers are not being notified right now, even
            though the website is probably still serving fine. Check the
            thermograph_worker service and the "Notifier alive?" panel on the
            Fleet Logs dashboard.
        labels:
          severity: critical
          host: prod
          class: product
        isPaused: false
        data:
          - refId: A
            relativeTimeRange: { from: 1200, to: 0 }
            datasourceUid: loki
            model:
              refId: A
              datasource: { type: loki, uid: loki }
              expr: 'sum(count_over_time({job="app-json", tag="heartbeat", host="prod"} [20m])) or vector(0)'
              queryType: instant
              editorMode: code
              intervalMs: 1000
              maxDataPoints: 43200
          - refId: B
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: B
              type: reduce
              reducer: last
              expression: A
              datasource: { type: __expr__, uid: __expr__ }
          - refId: C
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: C
              type: threshold
              expression: B
              datasource: { type: __expr__, uid: __expr__ }
              conditions:
                - evaluator: { type: lt, params: [1] }
      # -- 7 -----------------------------------------------------------------
      # Same signal on beta. Beta was equally gap-free over the backtest (1-2
      # beats every 20m, no empty buckets), but beta is UAT — worth knowing,
      # not worth a page.
      - uid: tg_notifier_heartbeat_beta
        title: NotifierHeartbeatMissingBeta
        condition: C
        for: 15m
        noDataState: OK
        execErrState: OK
        annotations:
          summary: 'beta: no notifier heartbeat for 20 minutes'
          description: >-
            The notifier has gone quiet on beta (UAT). Same signal as the prod
            rule but a longer `for`, because beta gets redeployed all day and a
            short gap is usually just a release.
        labels:
          severity: warning
          host: beta
          class: product
        isPaused: false
        data:
          - refId: A
            relativeTimeRange: { from: 1200, to: 0 }
            datasourceUid: loki
            model:
              refId: A
              datasource: { type: loki, uid: loki }
              expr: 'sum(count_over_time({job="app-json", tag="heartbeat", host="beta"} [20m])) or vector(0)'
              queryType: instant
              editorMode: code
              intervalMs: 1000
              maxDataPoints: 43200
          - refId: B
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: B
              type: reduce
              reducer: last
              expression: A
              datasource: { type: __expr__, uid: __expr__ }
          - refId: C
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: C
              type: threshold
              expression: B
              datasource: { type: __expr__, uid: __expr__ }
              conditions:
                - evaluator: { type: lt, params: [1] }
  # ===========================================================================
  # Per-container liveness, inferred from log volume. The fleet collects no
  # container metrics, so "did this service stop producing any output at all" is
  # the only death certificate available — and it turns out to be a good one,
  # because every service below logs on a floor that never approaches zero.
  #
  # Caveat worth knowing before you trust these: a container can be running and
  # logging while its lines never reach Loki, if the node's Alloy agent stops
  # tailing it. That is a real observed failure on prod (see README). These
  # rules cannot tell the two apart — they say "no logs", which is the honest
  # claim. Either way something needs looking at.
  # ===========================================================================
  - orgId: 1
    name: prod-services
    folder: Alerts
    interval: 5m
    rules:
      # -- 8 -----------------------------------------------------------------
      # DATA: 15-minute windows over 24h ranged 54 -> 954 lines. The minimum, 54,
      # occurred during the outage; the quietest healthy window was ~150. A
      # 30-minute window has never been below ~110. Zero means the container is
      # gone. This is the app itself, so it pages.
      - uid: tg_prod_web_silent
        title: ProdWebContainerSilent
        condition: C
        for: 5m
        noDataState: OK
        execErrState: OK
        annotations:
          summary: 'prod: thermograph_web has logged nothing for 30 minutes'
          description: >-
            The main app container has produced zero log lines in 30 minutes.
            Healthy floor is ~110 lines per 30m and it never dipped below 54 even
            mid-outage, so this is a container that is gone, wedged, or no longer
            being tailed by prod's Alloy agent. Run `docker service ps
            thermograph_web` on prod.
        labels:
          severity: critical
          host: prod
          class: liveness
        isPaused: false
        data:
          - refId: A
            relativeTimeRange: { from: 1800, to: 0 }
            datasourceUid: loki
            model:
              refId: A
              datasource: { type: loki, uid: loki }
              expr: 'sum(count_over_time({job="docker", host="prod", container=~"thermograph_web.+"} [30m])) or vector(0)'
              queryType: instant
              editorMode: code
              intervalMs: 1000
              maxDataPoints: 43200
          - refId: B
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: B
              type: reduce
              reducer: last
              expression: A
              datasource: { type: __expr__, uid: __expr__ }
          - refId: C
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: C
              type: threshold
              expression: B
              datasource: { type: __expr__, uid: __expr__ }
              conditions:
                - evaluator: { type: lt, params: [1] }
      # -- 9 -----------------------------------------------------------------
      # DATA: the steadiest series in the fleet — 2-hour windows read 54 to 69
      # lines, all 24 hours, with no trend and no gaps. TimescaleDB chatters at a
      # metronomic ~30 lines/hour. Zero over two hours is as close to a
      # definitive "Postgres is not there" as logs can give.
      - uid: tg_prod_db_silent
        title: ProdDbContainerSilent
        condition: C
        for: 10m
        noDataState: OK
        execErrState: OK
        annotations:
          summary: 'prod: thermograph_db has logged nothing for 2 hours'
          description: >-
            The production TimescaleDB container has produced zero log lines in
            2 hours. Its healthy range is a metronomic 54-69 lines per 2h with no
            observed gaps, so zero is not quiet — it is absent. Everything else
            in the app depends on this.
        labels:
          severity: critical
          host: prod
          class: liveness
        isPaused: false
        data:
          - refId: A
            relativeTimeRange: { from: 7200, to: 0 }
            datasourceUid: loki
            model:
              refId: A
              datasource: { type: loki, uid: loki }
              expr: 'sum(count_over_time({job="docker", host="prod", container=~"thermograph_db.+"} [2h])) or vector(0)'
              queryType: instant
              editorMode: code
              intervalMs: 1000
              maxDataPoints: 43200
          - refId: B
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: B
              type: reduce
              reducer: last
              expression: A
              datasource: { type: __expr__, uid: __expr__ }
          - refId: C
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: C
              type: threshold
              expression: B
              datasource: { type: __expr__, uid: __expr__ }
              conditions:
                - evaluator: { type: lt, params: [1] }
      # -- 10 ----------------------------------------------------------------
      # DATA: 15-minute windows ranged 33 -> 197, never zero. Warning rather than
      # critical because Caddy serves the static build and ProdEdgeDark /
      # ProdHTTP5xxBurst will catch it first if users are actually affected.
      - uid: tg_prod_frontend_silent
        title: ProdFrontendContainerSilent
        condition: C
        for: 5m
        noDataState: OK
        execErrState: OK
        annotations:
          summary: 'prod: thermograph_frontend has logged nothing for 30 minutes'
          description: >-
            The frontend container has produced zero log lines in 30 minutes
            (healthy floor ~66 per 30m). If users were visibly affected you would
            expect ProdHTTP5xxBurst alongside this; on its own it more often
            means a stuck redeploy or a lost Alloy tail.
        labels:
          severity: warning
          host: prod
          class: liveness
        isPaused: false
        data:
          - refId: A
            relativeTimeRange: { from: 1800, to: 0 }
            datasourceUid: loki
            model:
              refId: A
              datasource: { type: loki, uid: loki }
              expr: 'sum(count_over_time({job="docker", host="prod", container=~"thermograph_frontend.+"} [30m])) or vector(0)'
              queryType: instant
              editorMode: code
              intervalMs: 1000
              maxDataPoints: 43200
          - refId: B
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: B
              type: reduce
              reducer: last
              expression: A
              datasource: { type: __expr__, uid: __expr__ }
          - refId: C
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: C
              type: threshold
              expression: B
              datasource: { type: __expr__, uid: __expr__ }
              conditions:
                - evaluator: { type: lt, params: [1] }
      # -- 11 ----------------------------------------------------------------
      # DATA: a dead-steady 121 lines/hour (its own healthcheck polling) for the
      # first 14h of the backtest, then nothing at all.
      #
      # HEADS UP: THIS RULE WILL BE FIRING THE MOMENT YOU DEPLOY IT, and it is
      # right to. As of 2026-07-24 19:45Z the prod worker container is running
      # and healthy and is writing ~19 lines per 10 minutes to its own docker
      # log — but not one of those lines has reached Loki since 04:43Z. Prod's
      # Alloy agent stopped tailing the container when Swarm replaced the task.
      # Nobody knew, because nothing was watching. See README "Known gaps".
      #
      # WHY THIS IS A WARNING AND NotifierHeartbeatMissingProd IS CRITICAL: the
      # two draw on *different* Alloy sources — this one on loki.source.docker
      # (the container's stdout), the heartbeat on loki.source.file tailing
      # /applogs. That independence is the diagnostic. Right now prod's
      # heartbeats are arriving in every single 20-minute window while its
      # worker docker stream has been empty for 14 hours, which localises the
      # fault to Alloy's docker tail rather than to the worker: the process is
      # doing its job, we just cannot see it breathe. Both rules firing together
      # means the worker really is down.
      - uid: tg_prod_worker_silent
        title: ProdWorkerContainerSilent
        condition: C
        for: 10m
        noDataState: OK
        execErrState: OK
        annotations:
          summary: 'prod: thermograph_worker has logged nothing for 2 hours'
          description: >-
            The worker container has produced zero log lines in 2 hours against
            a healthy floor of ~242 per 2h. Two different faults look identical
            here: the worker is actually dead, or prod's Alloy agent has stopped
            tailing it (a real, observed failure — Alloy can miss a container
            after Swarm replaces the task). Check both: `docker service ps
            thermograph_worker` on prod, and whether `docker logs` on the live
            container is producing lines that Loki does not have. Restarting
            alloy-alloy-1 on prod fixes the second case.
        labels:
          severity: warning
          host: prod
          class: liveness
        isPaused: false
        data:
          - refId: A
            relativeTimeRange: { from: 7200, to: 0 }
            datasourceUid: loki
            model:
              refId: A
              datasource: { type: loki, uid: loki }
              expr: 'sum(count_over_time({job="docker", host="prod", container=~"thermograph_worker.+"} [2h])) or vector(0)'
              queryType: instant
              editorMode: code
              intervalMs: 1000
              maxDataPoints: 43200
          - refId: B
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: B
              type: reduce
              reducer: last
              expression: A
              datasource: { type: __expr__, uid: __expr__ }
          - refId: C
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: C
              type: threshold
              expression: B
              datasource: { type: __expr__, uid: __expr__ }
              conditions:
                - evaluator: { type: lt, params: [1] }
  # ===========================================================================
  # The dead-man's switch.
  # ===========================================================================
  - orgId: 1
    name: watchdog
    folder: Alerts
    interval: 5m
    rules:
      # -- 12 ----------------------------------------------------------------
      # Nothing in this file can tell you that Grafana itself has stopped, or
      # that beta is down, or that the Discord webhook was revoked — a monitoring
      # system cannot report its own death. So this rule fires permanently and
      # by design, throttled by the notification policy to exactly one Discord
      # message per 24 hours.
      #
      # HOW TO USE IT: if #ops-alerts has been silent for more than a day, the
      # alerting pipeline is broken, not the estate healthy. That one daily
      # message is the only evidence you will ever get that the other eleven
      # rules are still capable of reaching you.
      #
      # To silence it (and lose that guarantee), set isPaused: true.
      - uid: tg_alerting_watchdog
        title: AlertingWatchdog
        condition: C
        for: 0m
        noDataState: Alerting
        execErrState: Alerting
        annotations:
          summary: 'Alerting pipeline is alive (daily heartbeat — not a fault)'
          description: >-
            This is the dead-man's switch and it is supposed to be firing. It
            proves Grafana on beta is evaluating rules, Loki is answering, and
            the Discord webhook still works. Seeing it once a day is correct.
            NOT seeing it for over a day means alerting itself is down and no
            other rule can reach you — check observability-grafana-1 on beta.
        labels:
          severity: watchdog
          host: beta
          class: meta
        isPaused: false
        data:
          - refId: A
            relativeTimeRange: { from: 600, to: 0 }
            datasourceUid: loki
            model:
              refId: A
              datasource: { type: loki, uid: loki }
              expr: 'vector(1)'
              queryType: instant
              editorMode: code
              intervalMs: 1000
              maxDataPoints: 43200
          - refId: B
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: B
              type: reduce
              reducer: last
              expression: A
              datasource: { type: __expr__, uid: __expr__ }
          - refId: C
            relativeTimeRange: { from: 0, to: 0 }
            datasourceUid: __expr__
            model:
              refId: C
              type: threshold
              expression: B
              datasource: { type: __expr__, uid: __expr__ }
              conditions:
                - evaluator: { type: gt, params: [0] }