# 9. Observability Fleet-wide log aggregation: **Loki + Grafana on beta**, fed by a **Grafana Alloy** agent on every node, all over the WireGuard mesh. ``` prod (10.10.0.1) beta (10.10.0.2) desktop / LAN dev (10.10.0.3) ┌────────────┐ ┌──────────────────┐ ┌────────────┐ │ Alloy agent│──wg0──┐ │ Loki ◀── Alloy │ ┌───│ Alloy agent│ └────────────┘ └──▶│ Grafana (Caddy) │◀─┘ └────────────┘ docker+caddy+app └──────────────────┘ docker+caddy+app ``` - **Loki** — filesystem storage, **30-day retention**, listening on the mesh IP `10.10.0.2:3100`. Never public. - **Grafana** — the UI, fronted by beta's Caddy at **`dashboard.thermograph.org`** (Google SSO, pre-provisioned accounts only, `allow_sign_up=false`, plus a break-glass local admin). Use that hostname everywhere; **never** `grafana.thermograph.org`. - **Alloy** — on each node, shipping three sources: every Docker container's stdout/stderr, Caddy's host access logs, and the app's structured JSON logs (`errors`/`access`/`audit` `*.jsonl`, parsed so `level`/`tag`/`phase` become labels). Every line is tagged `host = prod|beta|dev`. There is **no build and no deploy automation** for this domain — it ships by hand. But `observability-validate.yml` parses every artifact that ships to a node, so a malformed dashboard or an unparseable Alloy config fails in CI instead of on the host. ## The rule that matters most **The repo is the only durable path.** Dashboards and alerting are provisioned from `observability/grafana/provisioning/` at startup, so **edits made through Grafana's UI or API are overwritten on the next provision**. A change goes through a PR. Centralis's `dashboard_write` opens one for you. ## Reading the fleet Start wide, then narrow: 1. `fleet_status` — HTTP reachability plus what's running on each host. 2. `logs_overview since="1h"` — where the volume is, and what has gone quiet. 3. `logs_query service=… host=…` — the actual lines. ### ⚠ The prod trap Prod runs a Swarm stack, and its `job="docker"` streams carry **no `service` label**. A raw `{host="prod", service="backend"}` query matches nothing and reads as *"no logs"* when it actually means *"wrong selector"*. Use Centralis's `logs_query` `service` argument, which rewrites it to a container name-regex with the churning task suffix wildcarded. Prod service names are `thermograph_web`, `_worker`, `_frontend`, `_db`, `_autoscaler`, `_lake`. Related: **prod container names change on every redeploy** (the Swarm task suffix). Never hardcode one; resolve via `docker ps --filter name=` at call time. ## The dashboard `Thermograph — Fleet Logs` (auto-provisioned), with a `Node` selector: log volume by service, app error rate by tag, upstream 429 count, Caddy 5xx count, a per-node notifier-liveness indicator, recent app errors, and a live all-container tail. ## Alerting Provisioned from `grafana/provisioning/alerting/`, mounted read-only into Grafana. **Every rule is LogQL** — there is no Prometheus anywhere in the fleet. Loki is the only datasource, so every signal is derived from log lines. Thresholds were backtested against real Loki data and the working is written out in the comments beside each rule; **re-derive before changing a number rather than guessing**. ### Alerts go to Discord `#ops-alerts`, never email Three reasons, all learned the hard way: - Grafana's factory default pointed at the literal string `` — a paging path that had never delivered a message to anyone. - Beta's Grafana relays SMTP through **prod's** Postfix. Email alerts therefore travel through the box most likely to be on fire, and vanish exactly when they matter. - Discord is off-estate, works when prod is dead, and reaches a phone. `#ops-alerts` is dedicated. `#dev` / `#uat` / `#prod` are the *product's* alert subscription output — real deliveries to real subscribers — so routing ops noise there would corrupt the evidence. The webhook URL is a secret and lives only in beta's gitignored `.env` as `DISCORD_ALERT_WEBHOOK_URL`. Grafana refuses to start if it's unset. A literal webhook URL committed to the repo is a **hard CI failure**. ### The twelve rules | Rule | Fires when | Severity | |---|---|---| | `ProdEdgeDark` | zero prod Caddy log lines in 15m — site dark, or the pipeline is broken | critical | | `ProdHealthzFailing` | any non-200 on `/healthz` in 10m | critical | | `ProdHTTP5xxBurst` | ≥10 prod 5xx in 10m | critical | | `ProdHTTP5xxElevated` | ≥25 prod 5xx over 6h (the error-budget alarm) | warning | | `ProdUnhandledExceptions` | ≥3 app `tag=http.error` in 30m | warning | | `NotifierHeartbeatMissingProd` | no `tag=heartbeat` from prod in 20m | critical | | `NotifierHeartbeatMissingBeta` | same, on beta | warning | | `ProdWebContainerSilent` | `thermograph_web` logged nothing in 30m | critical | | `ProdDbContainerSilent` | `thermograph_db` logged nothing in 2h | critical | | `ProdFrontendContainerSilent` | `thermograph_frontend` logged nothing in 30m | warning | | `ProdWorkerContainerSilent` | `thermograph_worker` logged nothing in 2h | warning | | `AlertingWatchdog` | always — one Discord message per day | watchdog | `AlertingWatchdog` is the dead-man's switch and fires permanently by design. **Its value is its absence:** nothing can report its own death, so if `#ops-alerts` has been silent for more than a day, alerting is broken rather than the estate healthy. ### The strict CI check, and why A rule whose `condition` names a refId that does not exist is **perfectly valid YAML, provisions without complaint, and then never fires** — a paging path that looks healthy and reaches nobody. `observability-validate.yml` checks that every condition names a real refId and every policy routes to a receiver that exists, precisely because nothing else would catch it. The same asymmetry shows up in verification: `GET /api/v1/provisioning/contact-points` **redacts** the webhook URL, so you cannot confirm interpolation by reading it back. A contact point holding the literal string `$DISCORD_ALERT_WEBHOOK_URL` looks completely healthy from the API and pages nobody. **The only proof is a message actually arriving in `#ops-alerts`** — force one with the receivers-test endpoint (the command is in `observability/README.md`). ## Deploying an observability change > ⚠️ **Merging is not deploying — and this one has already bitten.** > `/opt/observability` on beta and prod is a checkout of the **archived** > `emi/thermograph-observability` repo. It can never `git pull` again; the > content now lives here under `observability/`. A previous observability PR > merged and had **zero live effect** for exactly this reason. Until someone re-points those checkouts at the monorepo, every change ships by hand: back up the live file with a UTC-timestamped suffix, `scp` the new one up, `sudo cp` it into place, restart the specific container. Two gotchas in that sequence: - Use `sudo docker restart ` on **prod** — not `docker compose`, which there demands a `GF_SECURITY_ADMIN_PASSWORD` it has no `.env` to interpolate from. Beta's stack *does* have an `.env`, so compose works there. - **`docker restart` will not pick up a new environment variable.** Any change that introduces one needs `docker compose up -d grafana` on beta. Alerting lives on **beta only** — prod and dev run Alloy agents, not Grafana. Full step-by-step, including rollback, is in [`observability/README.md`](../../observability/README.md). ## Known gaps (as documented in the repo) - **Alloy can silently stop tailing a container after Swarm replaces the task.** Observed on prod 2026-07-24: `thermograph_worker` was `1/1`, healthy, writing ~19 lines per 10 minutes to its own docker log, yet not one line reached Loki for 13½ hours. What localises it to Alloy: prod's `tag=heartbeat` lines kept arriving throughout, and those come from a *different* Alloy source (`loki.source.file` on `/applogs`) than container stdout (`loki.source.docker`). One source went deaf, the other didn't. Workaround: `sudo docker restart alloy-alloy-1` on the node. **Not root-caused.** `ProdWorkerContainerSilent` now catches this class of failure. - **Some services are too quiet to alert on.** `thermograph_daemon` (23 lines / 24h), `thermograph_autoscaler` (1) and `_autoscaler-lake` (2) have a healthy floor indistinguishable from dead; `thermograph_lake` is bursty ETL with legitimate idle gaps. Giving each an `audit.log_heartbeat()` line would make all four alertable with the pattern the notifier already uses. **The single highest-value follow-up in this domain.** - **Grafana cannot alert on its own death.** `AlertingWatchdog` is a *manual* dead-man's switch — it relies on somebody noticing the daily message stopped. An external uptime pinger from off-estate would close this properly. - **Beta runs a `docker-compose.override.yml` that is not in this repo**, wiring Grafana's SMTP to prod's Postfix. It should be mirrored into the tracked compose file or deleted; alerting no longer depends on it. ## When something breaks 1. `fleet_status` — is it up at all? 2. `logs_query host=prod contains="(?i)error|traceback"` since the incident. 3. `deployed_version env=prod` — did something ship just before it started? 4. `forge_prs` / `forge_ci` — what landed recently, and did CI actually pass? 5. **Write down what you found with `notes_write` *before* fixing it.** The reconstruction is worth more than the fix, and it will be gone by tomorrow. You are authorised to fix production roadblocks directly — restart or roll a wedged service, clear a full disk, kill a runaway process, bounce a crash-looping container. What you must not do is make the fix *durable* on the box: writes to code, docs and dashboards go through a PR, and `deploy.sh` hard-resets the checkout anyway. The on-box action is stabilisation; the PR is the fix. Next: [Recipes](10-recipes.md).