observability: add the estate's first alerting; supervise Postfix #58

Merged

admin_emi merged 1 commit from ops/mail-supervision-and-alerting into dev

2026-07-24 20:44:22 +00:00

admin_emi commented

2026-07-24 20:19:53 +00:00

Owner

The estate had no alerting at all

Zero alert rules. The only contact point was Grafana's factory default pointing at the literal string <example@email.com>. And beta runs an untracked docker-compose.override.yml pointing Grafana's SMTP at 10.10.0.1:25 — prod's Postfix — so alerts, had any existed, would have been delivered by the box most likely to be on fire.

Meanwhile prod served 86 5xx in 24h including five /healthz failures, and nobody was told.

Thresholds came from a backtest, not from taste

The 24h window used happens to contain a real ~20-minute prod outage at 04:20Z. 10-minute buckets of prod 5xx (non-zero only): 2, 45, 22, 3, 1, 1, 1, 1, 1, 1, 2, 1, 5, 1.

Absolute counts, not ratios — prod runs ~7 req/min, so one bad request is 1.4% and a ratio alert screams all night. ProdHTTP5xxBurst at ≥10/10m fires on the 45 and 22 buckets and on nothing else in the day; the largest benign bucket was 5. ProdHealthzFailing uses for: 0m because there are only ~2.4 probes an hour, so waiting for a second sample costs 25 minutes.

AlertingWatchdog is a dead-man's switch — one message a day, whose value is its absence.

Delivery is proven, not assumed

The rules were provisioned into a throwaway Grafana 11.6.1 against beta's live Loki, and a real alert arrived in Discord. That verification is not optional here: GET /api/v1/provisioning/contact-points returns [REDACTED] for the URL, so a contact point holding an uninterpolated $DISCORD_ALERT_WEBHOOK_URL looks perfectly healthy and pages nobody. The only proof is a message arriving.

Routed to a new private #ops-alerts channel — deliberately not to #weather-events, #announcements or #prod, which are product surfaces that notify real subscribers.

It immediately found a real fault

ProdWorkerContainerSilent fired 10 minutes after first evaluation, correctly: the prod worker's logs have not reached Loki since 04:43Z while the container is up, healthy, and writing ~19 lines per 10 minutes to its own docker log. Prod's Alloy stopped tailing it after Swarm replaced the task. Localised to Alloy rather than the app because tag=heartbeat lines kept arriving from a different Alloy source throughout. This rule will be firing the moment you deploy, and it should be.

CI gains a structural check

The existing check only proves YAML parses. An alert rule whose condition names a refId that does not exist is valid YAML, provisions without complaint, and never fires — a paging path that looks healthy and reaches nobody, which is the exact failure this config exists to end. The new step also hard-fails on a literal Discord webhook in the repo. Verified by deliberately introducing all three breakages.

Postfix supervision

The 13-hour outage was a boot-ordering race, not a Docker renumbering. From the journal: postfix started 08:09:49, wg0 came up at :51 (winning by one second), postfix fataled at :52 on the missing docker_gwbridge address — and dockerd did not finish starting until 08:10:16. Stock postfix@.service is ordered only After=network-online.target and ships no Restart=, so one lost race became permanent.

An ExecStartPre gate now blocks up to 60s until every inet_interfaces address actually exists, absorbing the transient case inside a single start attempt. That is what makes bounded retry correct: 5 attempts in 600s, then failed — a genuinely broken config reaches a visible failed state in ~100s instead of re-fataling every 15s forever. A 5-minute watchdog timer then retries indefinitely and runs reset-failed, so "failed" still self-heals. Worst case ~5 minutes rather than 13 hours.

Wants=, not Requires= — a dockerd failure must not take down the loopback and mesh listeners that have nothing to do with Docker.

Tested on live prod, all reversed: kill -9 on master restored all three listeners in ~21s; a deliberately broken config reached failed at ~100s and stopped retrying; and the original lie was reproduced — zero listeners while systemctl is-active postfix reported active.

One gotcha worth reading before writing any mail tooling

postmulti, postqueue and postfix all resolve inet_interfaces, so they fatal precisely when an address is missing — the state you most need to detect. The first version of this health check used postmulti, got an empty string, concluded "0 addresses to verify", and reported status=ok bound=0/0. It now uses postconf -c, which only reads the file, and correctly reports status=fail reason=address-missing-from-host while Postfix is still serving — catching the next-reboot outage with zero downtime.

main.cf is untouched and byte-identical to its pre-work backup.

Known gaps, documented rather than papered over

Alloy ships Docker stdout, Caddy files and app JSONL — not journald. No Postfix line reaches Loki today, so the mail rules cannot fire until loki.source.journal is added. Specified, deliberately not implemented here.
mynetworks on prod has drifted from what provision-mail.sh generates: live config includes 172.19.0.0/16, which is not dead — Postfix has accepted real mail from 172.19.0.3, the bridge Centralis runs on. Re-running the provisioning script would silently drop it. Left alone; flagged.
thermograph_daemon, autoscaler, autoscaler-lake and lake are not alertable by log silence — their healthy floor is indistinguishable from dead. A heartbeat line each is the highest-value follow-up.

Deploy note

The first beta deploy needs docker compose up -d grafana, not docker restart — it introduces a new env var and restart will not pick it up. Beta has an .env so compose is safe there; the "never compose" rule is a prod constraint. DISCORD_ALERT_WEBHOOK_URL is a secret, deliberately not in the repo, and must be added to beta's /opt/observability/.env.

## The estate had no alerting at all Zero alert rules. The only contact point was Grafana's factory default pointing at the literal string `<example@email.com>`. And beta runs an **untracked `docker-compose.override.yml`** pointing Grafana's SMTP at `10.10.0.1:25` — prod's Postfix — so alerts, had any existed, would have been delivered by the box most likely to be on fire. Meanwhile prod served **86 5xx in 24h including five `/healthz` failures**, and nobody was told. ### Thresholds came from a backtest, not from taste The 24h window used happens to contain a real ~20-minute prod outage at 04:20Z. 10-minute buckets of prod 5xx (non-zero only): `2, 45, 22, 3, 1, 1, 1, 1, 1, 1, 2, 1, 5, 1`. Absolute counts, not ratios — prod runs ~7 req/min, so one bad request is 1.4% and a ratio alert screams all night. `ProdHTTP5xxBurst` at ≥10/10m fires on the `45` and `22` buckets and on **nothing else in the day**; the largest benign bucket was 5. `ProdHealthzFailing` uses `for: 0m` because there are only ~2.4 probes an hour, so waiting for a second sample costs 25 minutes. `AlertingWatchdog` is a dead-man's switch — one message a day, whose value is its absence. ### Delivery is proven, not assumed The rules were provisioned into a throwaway Grafana 11.6.1 against beta's live Loki, and a real alert arrived in Discord. That verification is not optional here: `GET /api/v1/provisioning/contact-points` returns `[REDACTED]` for the URL, so **a contact point holding an uninterpolated `$DISCORD_ALERT_WEBHOOK_URL` looks perfectly healthy and pages nobody.** The only proof is a message arriving. Routed to a new private `#ops-alerts` channel — deliberately not to `#weather-events`, `#announcements` or `#prod`, which are product surfaces that notify real subscribers. ### It immediately found a real fault `ProdWorkerContainerSilent` fired 10 minutes after first evaluation, correctly: **the prod worker's logs have not reached Loki since 04:43Z while the container is up, healthy, and writing ~19 lines per 10 minutes to its own docker log.** Prod's Alloy stopped tailing it after Swarm replaced the task. Localised to Alloy rather than the app because `tag=heartbeat` lines kept arriving from a different Alloy source throughout. **This rule will be firing the moment you deploy, and it should be.** ### CI gains a structural check The existing check only proves YAML parses. An alert rule whose `condition` names a refId that does not exist is valid YAML, provisions without complaint, and never fires — a paging path that looks healthy and reaches nobody, which is the exact failure this config exists to end. The new step also hard-fails on a literal Discord webhook in the repo. Verified by deliberately introducing all three breakages. ## Postfix supervision **The 13-hour outage was a boot-ordering race, not a Docker renumbering.** From the journal: postfix started 08:09:49, `wg0` came up at :51 (winning by one second), postfix fataled at :52 on the missing `docker_gwbridge` address — and dockerd did not finish starting until 08:10:16. Stock `postfix@.service` is ordered only `After=network-online.target` and ships no `Restart=`, so one lost race became permanent. An `ExecStartPre` gate now blocks up to 60s until every `inet_interfaces` address actually exists, absorbing the transient case *inside a single start attempt*. That is what makes bounded retry correct: 5 attempts in 600s, then `failed` — a genuinely broken config reaches a visible failed state in ~100s instead of re-fataling every 15s forever. A 5-minute watchdog timer then retries indefinitely and runs `reset-failed`, so "failed" still self-heals. Worst case ~5 minutes rather than 13 hours. `Wants=`, not `Requires=` — a dockerd failure must not take down the loopback and mesh listeners that have nothing to do with Docker. **Tested on live prod, all reversed:** `kill -9` on master restored all three listeners in ~21s; a deliberately broken config reached `failed` at ~100s and stopped retrying; and the original lie was reproduced — zero listeners while `systemctl is-active postfix` reported `active`. ### One gotcha worth reading before writing any mail tooling `postmulti`, `postqueue` and `postfix` all **resolve** `inet_interfaces`, so they fatal precisely when an address is missing — the state you most need to detect. The first version of this health check used `postmulti`, got an empty string, concluded "0 addresses to verify", and reported `status=ok bound=0/0`. It now uses `postconf -c`, which only reads the file, and correctly reports `status=fail reason=address-missing-from-host` while Postfix is still serving — catching the next-reboot outage with zero downtime. `main.cf` is untouched and byte-identical to its pre-work backup. ## Known gaps, documented rather than papered over - **Alloy ships Docker stdout, Caddy files and app JSONL — not journald.** No Postfix line reaches Loki today, so the mail rules cannot fire until `loki.source.journal` is added. Specified, deliberately not implemented here. - `mynetworks` on prod has drifted from what `provision-mail.sh` generates: live config includes `172.19.0.0/16`, which is **not dead** — Postfix has accepted real mail from `172.19.0.3`, the bridge Centralis runs on. Re-running the provisioning script would silently drop it. Left alone; flagged. - `thermograph_daemon`, `autoscaler`, `autoscaler-lake` and `lake` are not alertable by log silence — their healthy floor is indistinguishable from dead. A heartbeat line each is the highest-value follow-up. ## Deploy note The first beta deploy needs `docker compose up -d grafana`, not `docker restart` — it introduces a new env var and restart will not pick it up. Beta has an `.env` so compose is safe there; the "never compose" rule is a prod constraint. `DISCORD_ALERT_WEBHOOK_URL` is a secret, deliberately not in the repo, and must be added to beta's `/opt/observability/.env`.

admin_emi added 1 commit 2026-07-24 20:19:54 +00:00

observability: add the estate's first alerting; supervise Postfix

secrets-guard / encrypted (pull_request) Successful in 8s

Details

shell-lint / shellcheck (pull_request) Successful in 10s

Details

PR build (required check) / changes (pull_request) Successful in 16s

Details

PR build (required check) / validate-observability (pull_request) Successful in 18s

Details

PR build (required check) / gate (pull_request) Successful in 2s

Details

PR build (required check) / build-backend (pull_request) Has been skipped

Details

PR build (required check) / build-frontend (pull_request) Has been skipped

Details

e4693dce58

The estate had zero alert rules. The only contact point was Grafana's factory
default pointing at the literal string <example@email.com>, and beta's
untracked compose override routed Grafana's SMTP at prod's Postfix — so
alerts, had any existed, would have been delivered by the box most likely to
be on fire. Prod served 86 5xx in 24h including five /healthz failures and
nobody was told.

Alerting (deploys to beta, which is where Grafana runs):
- 12 Loki-based rules. There is no Prometheus in this estate and Loki is the
  only datasource, so every rule is log-derived.
- Thresholds come from a 24h backtest that happens to contain a real ~20min
  prod outage at 04:20Z. Absolute counts, not ratios: prod runs ~7 req/min, so
  one bad request is 1.4% and a ratio alert would scream all night. The 5xx
  burst rule fires on the outage's 45 and 22 buckets and on nothing else in
  the day; the largest benign bucket all day was 5.
- Routed to a new private #ops-alerts channel, not to any existing channel —
  #weather-events, #announcements and #prod are product surfaces that notify
  real subscribers.
- AlertingWatchdog is a dead-man's switch; its value is its absence.
- Delivery is proven, not assumed: the rules were provisioned into a throwaway
  Grafana against beta's live Loki and a real alert arrived in Discord. This
  matters because GET /api/v1/provisioning/contact-points returns [REDACTED]
  for the URL — a contact point holding an uninterpolated env var looks
  perfectly healthy and pages nobody. The only proof is a message arriving.

CI gains a structural check, because the existing one only proves YAML parses:
an alert rule whose condition names a missing refId is valid YAML, provisions
cleanly, and never fires. It also hard-fails on a literal Discord webhook in
the repo. Verified against all three breakages deliberately introduced.

Postfix supervision:
- The 13h outage was a boot-ordering race, not a Docker renumbering: postfix
  started at 08:09:49, wg0 came up at :51, postfix fataled at :52 on a missing
  docker_gwbridge address, and dockerd did not finish starting until 08:10:16.
  Stock postfix@.service is ordered only After=network-online.target and ships
  no Restart=, so one lost race became a permanent outage.
- An ExecStartPre gate now blocks up to 60s until every inet_interfaces
  address actually exists, which absorbs the transient case inside a single
  start attempt. That makes bounded retry correct: 5 attempts in 600s, then
  failed — a genuinely broken config reaches a visible failed state in ~100s
  instead of re-fataling every 15s forever.
- A 5-minute watchdog timer retries indefinitely and runs reset-failed, so
  "failed" still self-heals. Worst case is ~5 minutes, not 13 hours.
- Wants=, not Requires=: a dockerd failure must not take down the loopback and
  mesh listeners that do not depend on Docker at all.

Health checks must read config with `postconf -c`, never postmulti/postqueue/
postfix — those three RESOLVE inet_interfaces and so fatal precisely when an
address is missing, which made the first version of this check report
status=ok bound=0/0. A health check that fails open is worse than none.

DEPLOY.md carries the monitoring contract, including that systemctl is-active
postfix is a known-false signal: postfix.service is a wrapper whose
ExecStart=/bin/true, so it reports active forever while the real postfix@-
instance is failed with zero listeners. Reproduced live.

Known gap, documented not fixed: Alloy ships Docker stdout, Caddy files and
app JSONL, not journald — so no Postfix line reaches Loki and the mail rules
cannot fire until loki.source.journal is added.

Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY