Promote main to release: first alerting + Postfix supervision #61

Merged

admin_emi merged 7 commits from main into release 2026-07-24 20:58:15 +00:00

Conversation 0 Commits 7 Files changed 16 +2151 -108

admin_emi commented 2026-07-24 20:53:40 +00:00

Owner

Copy link

Promotes to prod. Detail in #58 and #60.

Verified on beta first, as the rehearsal:

• Grafana recreated cleanly with docker compose up -d grafana; health 200
• 12 alert rules provisioned, contact point thermograph-ops-discord live
• Delivery proven end to end — a verification alert arrived in #ops-alerts at 20:51:49Z. This is the only check that counts: the provisioning API returns [REDACTED] for the webhook, so a contact point holding an uninterpolated env var reads as perfectly healthy and pages nobody.
• Two real alerts also arrived: the daily watchdog heartbeat, and ProdWorkerContainerSilent firing correctly on the Alloy gap.

What this hop actually changes on prod. Prod runs Alloy, not Grafana, so no alerting config ships here — the rules live on beta and watch prod remotely. What reaches prod is infra/: the Postfix supervision script, DEPLOY.md's monitoring contract, and the env example.

Postfix supervision is already live on prod and was tested there directly — kill -9 on master restored all three listeners in ~21s; a deliberately broken config reached failed at ~100s and stopped retrying rather than looping. This merge makes a rebuilt box reproduce that, since the installed artifacts live in /etc and /usr/local/sbin, outside the checkout deploy.sh hard-resets.

Deploy discipline carried over from beta: ship specific files, never git pull into /opt/observability. Beta's checkout carries local modifications and two untracked dashboards that a pull would have destroyed.

Still not safe to run: provision-mail.sh on prod. Live mynetworks has drifted to include 172.19.0.0/16 — not dead, it is the bridge Centralis sends mail from — and the script would silently drop it. The supervision script is separate and never touches main.cf.

Promotes to **prod**. Detail in #58 and #60. **Verified on beta first, as the rehearsal:** - Grafana recreated cleanly with `docker compose up -d grafana`; health 200 - **12 alert rules provisioned**, contact point `thermograph-ops-discord` live - **Delivery proven end to end** — a verification alert arrived in `#ops-alerts` at 20:51:49Z. This is the only check that counts: the provisioning API returns `[REDACTED]` for the webhook, so a contact point holding an uninterpolated env var reads as perfectly healthy and pages nobody. - Two real alerts also arrived: the daily watchdog heartbeat, and `ProdWorkerContainerSilent` firing correctly on the Alloy gap. **What this hop actually changes on prod.** Prod runs Alloy, not Grafana, so no alerting config ships here — the rules live on beta and watch prod remotely. What reaches prod is `infra/`: the Postfix supervision script, `DEPLOY.md`'s monitoring contract, and the env example. **Postfix supervision is already live on prod** and was tested there directly — `kill -9` on master restored all three listeners in ~21s; a deliberately broken config reached `failed` at ~100s and stopped retrying rather than looping. This merge makes a rebuilt box reproduce that, since the installed artifacts live in `/etc` and `/usr/local/sbin`, outside the checkout `deploy.sh` hard-resets. **Deploy discipline carried over from beta:** ship specific files, never `git pull` into `/opt/observability`. Beta's checkout carries local modifications and two untracked dashboards that a pull would have destroyed. **Still not safe to run:** `provision-mail.sh` on prod. Live `mynetworks` has drifted to include `172.19.0.0/16` — not dead, it is the bridge Centralis sends mail from — and the script would silently drop it. The supervision script is separate and never touches `main.cf`.

admin_emi added 7 commits 2026-07-24 20:53:41 +00:00

Add off-box encrypted backups to Contabo Object Storage ... 6086b40fe6

Prod DB dumps and Forgejo (db + data volume) are now copied off-box to the
era5-thermograph S3 bucket, age-encrypted to the vault recipient so the host
age key at /etc/thermograph/age.key decrypts them for restore. Nothing plaintext
leaves the box; no local intermediate (streamed via age | rclone rcat).

ops-cron: the prod pg_dump job gains an off-box push; a new forgejo-backup job
dumps Forgejo's Postgres + tars its data volume on beta (the git host had NO
backup at all). 30-day off-box retention on both. S3 creds are Forgejo Actions
secrets (S3_ENDPOINT/BUCKET/ACCESS_KEY/SECRET_KEY), mirrored into the SOPS vault
(prod.yaml/beta.yaml) for host-side use. Contabo needs path-style addressing.

Adds deploy/backup/README.md with the DR restore runbook.

Merge remote-tracking branch 'origin/dev' into feature/storage-backups-merged ... e91b49a000

# Conflicts:
#	infra/deploy/secrets/beta.yaml
#	infra/deploy/secrets/prod.yaml

Merge pull request 'Off-box encrypted backups to Contabo Object Storage' (#14) from feature/storage-backups into dev 519541c378

Merge pull request 'Promote dev to main: off-box encrypted backups' (#57) from dev into main 60d5af41dd

observability: add the estate's first alerting; supervise Postfix ... e4693dce58

The estate had zero alert rules. The only contact point was Grafana's factory
default pointing at the literal string <example@email.com>, and beta's
untracked compose override routed Grafana's SMTP at prod's Postfix — so
alerts, had any existed, would have been delivered by the box most likely to
be on fire. Prod served 86 5xx in 24h including five /healthz failures and
nobody was told.

Alerting (deploys to beta, which is where Grafana runs):
- 12 Loki-based rules. There is no Prometheus in this estate and Loki is the
  only datasource, so every rule is log-derived.
- Thresholds come from a 24h backtest that happens to contain a real ~20min
  prod outage at 04:20Z. Absolute counts, not ratios: prod runs ~7 req/min, so
  one bad request is 1.4% and a ratio alert would scream all night. The 5xx
  burst rule fires on the outage's 45 and 22 buckets and on nothing else in
  the day; the largest benign bucket all day was 5.
- Routed to a new private #ops-alerts channel, not to any existing channel —
  #weather-events, #announcements and #prod are product surfaces that notify
  real subscribers.
- AlertingWatchdog is a dead-man's switch; its value is its absence.
- Delivery is proven, not assumed: the rules were provisioned into a throwaway
  Grafana against beta's live Loki and a real alert arrived in Discord. This
  matters because GET /api/v1/provisioning/contact-points returns [REDACTED]
  for the URL — a contact point holding an uninterpolated env var looks
  perfectly healthy and pages nobody. The only proof is a message arriving.

CI gains a structural check, because the existing one only proves YAML parses:
an alert rule whose condition names a missing refId is valid YAML, provisions
cleanly, and never fires. It also hard-fails on a literal Discord webhook in
the repo. Verified against all three breakages deliberately introduced.

Postfix supervision:
- The 13h outage was a boot-ordering race, not a Docker renumbering: postfix
  started at 08:09:49, wg0 came up at :51, postfix fataled at :52 on a missing
  docker_gwbridge address, and dockerd did not finish starting until 08:10:16.
  Stock postfix@.service is ordered only After=network-online.target and ships
  no Restart=, so one lost race became a permanent outage.
- An ExecStartPre gate now blocks up to 60s until every inet_interfaces
  address actually exists, which absorbs the transient case inside a single
  start attempt. That makes bounded retry correct: 5 attempts in 600s, then
  failed — a genuinely broken config reaches a visible failed state in ~100s
  instead of re-fataling every 15s forever.
- A 5-minute watchdog timer retries indefinitely and runs reset-failed, so
  "failed" still self-heals. Worst case is ~5 minutes, not 13 hours.
- Wants=, not Requires=: a dockerd failure must not take down the loopback and
  mesh listeners that do not depend on Docker at all.

Health checks must read config with `postconf -c`, never postmulti/postqueue/
postfix — those three RESOLVE inet_interfaces and so fatal precisely when an
address is missing, which made the first version of this check report
status=ok bound=0/0. A health check that fails open is worse than none.

DEPLOY.md carries the monitoring contract, including that systemctl is-active
postfix is a known-false signal: postfix.service is a wrapper whose
ExecStart=/bin/true, so it reports active forever while the real postfix@-
instance is failed with zero listeners. Reproduced live.

Known gap, documented not fixed: Alloy ships Docker stdout, Caddy files and
app JSONL, not journald — so no Postfix line reaches Loki and the mail rules
cannot fire until loki.source.journal is added.

Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY

Merge pull request 'observability: add the estate's first alerting; supervise Postfix' (#58) from ops/mail-supervision-and-alerting into dev 008052b412

Merge pull request 'Promote dev to main: first alerting + Postfix supervision' (#60) from dev into main 633c6a5ace

admin_emi merged commit 67996d5b77 into release 2026-07-24 20:58:15 +00:00

admin_emi referenced this pull request from a commit 2026-07-24 20:58:17 +00:00

Merge pull request 'Promote main to release: first alerting + Postfix supervision' (#61) from main into release

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

  

Compat/Breaking

Breaking change that won't be backward compatible

  

Kind/Bug

Something is not working

  

Kind/Documentation

Documentation changes

  

Kind/Enhancement

Improve existing functionality

  

Kind/Feature

New functionality

  

Kind/Security

This is security issue

  

Kind/Testing

Issue or pull request related to testing

  

Priority

Critical

The priority is critical

  

Priority

High

The priority is high

  

Priority

Low

The priority is low

  

Priority

Medium

The priority is medium

  

Reviewed

Confirmed

Issue has been confirmed

  

Reviewed

Duplicate

This issue or pull request already exists

  

Reviewed

Invalid

Invalid issue

  

Reviewed

Won't Fix

This issue won't be fixed

  

Status

Abandoned

Somebody has started to work on this but abandoned work

  

Status

Blocked

Something is blocking this issue or pull request

  

Status

Need More Info

Feedback is required to reproduce issue or to continue work

No labels

Compat/Breaking

Kind/Bug

Kind/Documentation

Kind/Enhancement

Kind/Feature

Kind/Security

Kind/Testing

Priority

Critical

Priority

High

Priority

Low

Priority

Medium

Reviewed

Confirmed

Reviewed

Duplicate

Reviewed

Invalid

Reviewed

Won't Fix

Status

Abandoned

Status

Blocked

Status

Need More Info

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Jinemi/thermograph#61

No description provided.