2026-07-25 07:08:54 +00:00
|
|
|
# observability/ — agent instructions
|
2026-07-22 18:47:21 +00:00
|
|
|
|
2026-07-26 06:56:38 +00:00
|
|
|
The **logging stack** for the fleet: Loki + Grafana on **vps1** — the monitoring
|
|
|
|
|
host, mesh `10.10.0.2`, and NOT the beta *environment* (that runs on vps2 now;
|
|
|
|
|
any doc that still says "beta" meaning this box is stale) — with a Grafana
|
|
|
|
|
Alloy agent on every node shipping container and app logs over the WireGuard
|
|
|
|
|
mesh. vps1 also hosts Forgejo, the portfolio site, and the **dev** environment
|
|
|
|
|
(mesh-only, no public DNS). vps2 runs **prod and beta** as two Swarm stacks.
|
|
|
|
|
Grafana is fronted by vps1's Caddy at **`dashboard.thermograph.org`** (Google
|
|
|
|
|
SSO, pre-provisioned users only) — use that hostname everywhere, never
|
|
|
|
|
`grafana.thermograph.org`.
|
2026-07-22 18:47:21 +00:00
|
|
|
|
2026-07-25 07:08:54 +00:00
|
|
|
This is operational config, not application code: there is **no build** and no
|
2026-07-26 06:56:38 +00:00
|
|
|
deploy automation. It ships by hand — `docker compose up -d` on vps1 for
|
2026-07-25 07:08:54 +00:00
|
|
|
Loki+Grafana, and `alloy/docker-compose.agent.yml` on each node. Read the root
|
|
|
|
|
`CLAUDE.md` first; the branch model there applies here too.
|
2026-07-22 18:47:21 +00:00
|
|
|
|
|
|
|
|
## Layout
|
|
|
|
|
|
2026-07-26 06:56:38 +00:00
|
|
|
- `docker-compose.yml` — the Loki + Grafana stack (vps1).
|
2026-07-25 07:08:54 +00:00
|
|
|
- `loki/config.yml` — mesh-only, filesystem storage.
|
|
|
|
|
- `grafana/provisioning/` — datasource + dashboard provider, auto-loaded at
|
|
|
|
|
startup. `grafana/dashboards/*.json` — the dashboards.
|
|
|
|
|
- `grafana/provisioning/alerting/` — alert rules, the Discord contact point, and
|
|
|
|
|
the notification policy.
|
2026-07-26 06:56:38 +00:00
|
|
|
- `alloy/config.alloy` + `alloy/docker-compose.agent.yml` (+
|
|
|
|
|
`alloy/docker-compose.agent.beta.yml`, a vps2-only overlay adding beta's
|
|
|
|
|
`applogs` mount) — the per-node shipper. `ALLOY_NODE` is the MACHINE
|
|
|
|
|
(`vps1`|`vps2`|`desktop`) — that's a semantic change from when the fleet had
|
|
|
|
|
one environment per node; `host` (`prod`|`beta`|`dev`), the ENVIRONMENT
|
|
|
|
|
label every dashboard/alert slices on, is now derived per source instead,
|
|
|
|
|
since vps2 alone carries two of them.
|
|
|
|
|
- `caddy-grafana.conf` — the vps1 Caddy vhost, kept here for reference.
|
|
|
|
|
- `.env.example` — copy to `.env` on vps1. `.env` is gitignored; never commit
|
2026-07-25 07:08:54 +00:00
|
|
|
OAuth secrets or the admin password.
|
|
|
|
|
|
|
|
|
|
## Rules
|
|
|
|
|
|
|
|
|
|
- **The repo is the only durable path.** Dashboards and alerting are provisioned
|
|
|
|
|
from this directory at startup; edits made through Grafana's UI or API are
|
|
|
|
|
overwritten on the next provision. Changes go through a PR.
|
observability: add the estate's first alerting; supervise Postfix
The estate had zero alert rules. The only contact point was Grafana's factory
default pointing at the literal string <example@email.com>, and beta's
untracked compose override routed Grafana's SMTP at prod's Postfix — so
alerts, had any existed, would have been delivered by the box most likely to
be on fire. Prod served 86 5xx in 24h including five /healthz failures and
nobody was told.
Alerting (deploys to beta, which is where Grafana runs):
- 12 Loki-based rules. There is no Prometheus in this estate and Loki is the
only datasource, so every rule is log-derived.
- Thresholds come from a 24h backtest that happens to contain a real ~20min
prod outage at 04:20Z. Absolute counts, not ratios: prod runs ~7 req/min, so
one bad request is 1.4% and a ratio alert would scream all night. The 5xx
burst rule fires on the outage's 45 and 22 buckets and on nothing else in
the day; the largest benign bucket all day was 5.
- Routed to a new private #ops-alerts channel, not to any existing channel —
#weather-events, #announcements and #prod are product surfaces that notify
real subscribers.
- AlertingWatchdog is a dead-man's switch; its value is its absence.
- Delivery is proven, not assumed: the rules were provisioned into a throwaway
Grafana against beta's live Loki and a real alert arrived in Discord. This
matters because GET /api/v1/provisioning/contact-points returns [REDACTED]
for the URL — a contact point holding an uninterpolated env var looks
perfectly healthy and pages nobody. The only proof is a message arriving.
CI gains a structural check, because the existing one only proves YAML parses:
an alert rule whose condition names a missing refId is valid YAML, provisions
cleanly, and never fires. It also hard-fails on a literal Discord webhook in
the repo. Verified against all three breakages deliberately introduced.
Postfix supervision:
- The 13h outage was a boot-ordering race, not a Docker renumbering: postfix
started at 08:09:49, wg0 came up at :51, postfix fataled at :52 on a missing
docker_gwbridge address, and dockerd did not finish starting until 08:10:16.
Stock postfix@.service is ordered only After=network-online.target and ships
no Restart=, so one lost race became a permanent outage.
- An ExecStartPre gate now blocks up to 60s until every inet_interfaces
address actually exists, which absorbs the transient case inside a single
start attempt. That makes bounded retry correct: 5 attempts in 600s, then
failed — a genuinely broken config reaches a visible failed state in ~100s
instead of re-fataling every 15s forever.
- A 5-minute watchdog timer retries indefinitely and runs reset-failed, so
"failed" still self-heals. Worst case is ~5 minutes, not 13 hours.
- Wants=, not Requires=: a dockerd failure must not take down the loopback and
mesh listeners that do not depend on Docker at all.
Health checks must read config with `postconf -c`, never postmulti/postqueue/
postfix — those three RESOLVE inet_interfaces and so fatal precisely when an
address is missing, which made the first version of this check report
status=ok bound=0/0. A health check that fails open is worse than none.
DEPLOY.md carries the monitoring contract, including that systemctl is-active
postfix is a known-false signal: postfix.service is a wrapper whose
ExecStart=/bin/true, so it reports active forever while the real postfix@-
instance is failed with zero listeners. Reproduced live.
Known gap, documented not fixed: Alloy ships Docker stdout, Caddy files and
app JSONL, not journald — so no Postfix line reaches Loki and the mail rules
cannot fire until loki.source.journal is added.
Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY
2026-07-24 20:19:19 +00:00
|
|
|
- **Every artifact that ships to a node is CI-validated**
|
2026-07-25 07:08:54 +00:00
|
|
|
(`.forgejo/workflows/observability-validate.yml`, at the repo root): both
|
|
|
|
|
compose files parse, all dashboard JSON is valid, the Loki and provisioning
|
|
|
|
|
YAML parse, and the Alloy config is checked with the pinned `alloy` binary
|
|
|
|
|
(v1.9.1, matching what the fleet runs).
|
|
|
|
|
- **Alerting gets a stricter third step.** Every rule's `condition` must name a
|
|
|
|
|
refId that exists, every policy must route to a receiver that exists, and a
|
|
|
|
|
literal Discord webhook URL in the repo is a hard failure. A rule pointing at a
|
|
|
|
|
missing refId is valid YAML, provisions cleanly, and then never fires — that
|
|
|
|
|
silent no-op is what this check exists to prevent.
|
2026-07-26 06:56:38 +00:00
|
|
|
- **Alerts go to Discord `#ops-alerts`, never email.** vps1's Grafana relays
|
|
|
|
|
SMTP through vps2's Postfix, so email dies exactly when prod (on vps2) does.
|
2026-07-25 07:08:54 +00:00
|
|
|
- **Every rule is LogQL** — there is no Prometheus anywhere in the fleet.
|
|
|
|
|
Thresholds were derived from real Loki data and the working is in the comments
|
|
|
|
|
beside each rule; re-derive before changing a number rather than guessing.
|
|
|
|
|
- Secrets (OAuth client id/secret, admin password) live only in the host `.env`.
|
|
|
|
|
|
|
|
|
|
## Commits & PRs
|
|
|
|
|
|
|
|
|
|
Concise and technical. Never mention AI, assistants or automated authorship.
|
