Promote dev to main: first alerting + Postfix supervision #60

Merged

admin_emi merged 2 commits from dev into main

2026-07-24 20:49:10 +00:00

admin_emi commented

2026-07-24 20:46:42 +00:00

Owner

Promotes the first alerting this estate has ever had, plus Postfix supervision, to beta.

Full detail is in #58. What matters for this hop:

This does not deploy the alerting by itself. /opt/observability on both boxes is a checkout of the archived thermograph-observability repo, so it can never pull monorepo content. Merging here deploys beta's app; the alerting config has to be hand-shipped (backup → scp → docker compose up -d grafana). That mechanism is exactly why a previous observability PR merged with zero live effect until a manual cutover. The deploy will be done immediately after this merge and verified by an alert actually arriving in Discord — reading the provisioning API back proves nothing, since it returns [REDACTED] for the webhook URL.

docker compose up -d, not docker restart: this introduces DISCORD_ALERT_WEBHOOK_URL and a restart will not pick up a new env var. Beta has an .env, so compose is safe there — the "never compose" rule is a prod constraint.

Expect ProdWorkerContainerSilent to fire immediately, and correctly. Prod worker logs stopped reaching Loki at 04:43Z while the container is up, healthy, and writing to its own docker log; prod's Alloy stopped tailing it after Swarm replaced the task. The alert is right and the underlying gap is real.

Postfix supervision is already live on prod — installed and tested there directly (kill -9 recovered all three listeners in ~21s; a deliberately broken config reached failed at ~100s and stopped retrying). The repo change is what makes a rebuilt box reproduce it.

Not safe to run during any deploy: provision-mail.sh. Live mynetworks on prod has drifted to include 172.19.0.0/16, which is not dead — Postfix has accepted real mail from 172.19.0.3, the bridge Centralis runs on — and the script derives mynetworks from a variable that would silently drop it. The new supervision script is separate and does not touch main.cf.

All 7 checks passed on #58, including the new structural alerting check.

Promotes the first alerting this estate has ever had, plus Postfix supervision, to **beta**. Full detail is in #58. What matters for this hop: **This does not deploy the alerting by itself.** `/opt/observability` on both boxes is a checkout of the *archived* `thermograph-observability` repo, so it can never pull monorepo content. Merging here deploys beta's app; the alerting config has to be hand-shipped (backup → scp → `docker compose up -d grafana`). That mechanism is exactly why a previous observability PR merged with **zero live effect** until a manual cutover. The deploy will be done immediately after this merge and verified by an alert actually arriving in Discord — reading the provisioning API back proves nothing, since it returns `[REDACTED]` for the webhook URL. `docker compose up -d`, not `docker restart`: this introduces `DISCORD_ALERT_WEBHOOK_URL` and a restart will not pick up a new env var. Beta has an `.env`, so compose is safe there — the "never compose" rule is a prod constraint. **Expect `ProdWorkerContainerSilent` to fire immediately, and correctly.** Prod worker logs stopped reaching Loki at 04:43Z while the container is up, healthy, and writing to its own docker log; prod's Alloy stopped tailing it after Swarm replaced the task. The alert is right and the underlying gap is real. **Postfix supervision is already live on prod** — installed and tested there directly (`kill -9` recovered all three listeners in ~21s; a deliberately broken config reached `failed` at ~100s and stopped retrying). The repo change is what makes a rebuilt box reproduce it. **Not safe to run during any deploy:** `provision-mail.sh`. Live `mynetworks` on prod has drifted to include `172.19.0.0/16`, which is *not* dead — Postfix has accepted real mail from `172.19.0.3`, the bridge Centralis runs on — and the script derives `mynetworks` from a variable that would silently drop it. The new supervision script is separate and does not touch `main.cf`. All 7 checks passed on #58, including the new structural alerting check.

admin_emi added 2 commits 2026-07-24 20:46:44 +00:00

observability: add the estate's first alerting; supervise Postfix

secrets-guard / encrypted (pull_request) Successful in 8s

Details

shell-lint / shellcheck (pull_request) Successful in 10s

Details

PR build (required check) / changes (pull_request) Successful in 16s

Details

PR build (required check) / validate-observability (pull_request) Successful in 18s

Details

PR build (required check) / gate (pull_request) Successful in 2s

Details

PR build (required check) / build-backend (pull_request) Has been skipped

Details

PR build (required check) / build-frontend (pull_request) Has been skipped

Details

e4693dce58

The estate had zero alert rules. The only contact point was Grafana's factory
default pointing at the literal string <example@email.com>, and beta's
untracked compose override routed Grafana's SMTP at prod's Postfix — so
alerts, had any existed, would have been delivered by the box most likely to
be on fire. Prod served 86 5xx in 24h including five /healthz failures and
nobody was told.

Alerting (deploys to beta, which is where Grafana runs):
- 12 Loki-based rules. There is no Prometheus in this estate and Loki is the
  only datasource, so every rule is log-derived.
- Thresholds come from a 24h backtest that happens to contain a real ~20min
  prod outage at 04:20Z. Absolute counts, not ratios: prod runs ~7 req/min, so
  one bad request is 1.4% and a ratio alert would scream all night. The 5xx
  burst rule fires on the outage's 45 and 22 buckets and on nothing else in
  the day; the largest benign bucket all day was 5.
- Routed to a new private #ops-alerts channel, not to any existing channel —
  #weather-events, #announcements and #prod are product surfaces that notify
  real subscribers.
- AlertingWatchdog is a dead-man's switch; its value is its absence.
- Delivery is proven, not assumed: the rules were provisioned into a throwaway
  Grafana against beta's live Loki and a real alert arrived in Discord. This
  matters because GET /api/v1/provisioning/contact-points returns [REDACTED]
  for the URL — a contact point holding an uninterpolated env var looks
  perfectly healthy and pages nobody. The only proof is a message arriving.

CI gains a structural check, because the existing one only proves YAML parses:
an alert rule whose condition names a missing refId is valid YAML, provisions
cleanly, and never fires. It also hard-fails on a literal Discord webhook in
the repo. Verified against all three breakages deliberately introduced.

Postfix supervision:
- The 13h outage was a boot-ordering race, not a Docker renumbering: postfix
  started at 08:09:49, wg0 came up at :51, postfix fataled at :52 on a missing
  docker_gwbridge address, and dockerd did not finish starting until 08:10:16.
  Stock postfix@.service is ordered only After=network-online.target and ships
  no Restart=, so one lost race became a permanent outage.
- An ExecStartPre gate now blocks up to 60s until every inet_interfaces
  address actually exists, which absorbs the transient case inside a single
  start attempt. That makes bounded retry correct: 5 attempts in 600s, then
  failed — a genuinely broken config reaches a visible failed state in ~100s
  instead of re-fataling every 15s forever.
- A 5-minute watchdog timer retries indefinitely and runs reset-failed, so
  "failed" still self-heals. Worst case is ~5 minutes, not 13 hours.
- Wants=, not Requires=: a dockerd failure must not take down the loopback and
  mesh listeners that do not depend on Docker at all.

Health checks must read config with `postconf -c`, never postmulti/postqueue/
postfix — those three RESOLVE inet_interfaces and so fatal precisely when an
address is missing, which made the first version of this check report
status=ok bound=0/0. A health check that fails open is worse than none.

DEPLOY.md carries the monitoring contract, including that systemctl is-active
postfix is a known-false signal: postfix.service is a wrapper whose
ExecStart=/bin/true, so it reports active forever while the real postfix@-
instance is failed with zero listeners. Reproduced live.

Known gap, documented not fixed: Alloy ships Docker stdout, Caddy files and
app JSONL, not journald — so no Postfix line reaches Loki and the mail rules
cannot fire until loki.source.journal is added.

Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY