thermograph/docs/onboarding/09-observability.md

# 9. Observability

Fleet-wide log aggregation: **Loki + Grafana on beta**, fed by a **Grafana
Alloy** agent on every node, all over the WireGuard mesh.

```
  prod (10.10.0.1)         beta (10.10.0.2)          desktop / LAN dev (10.10.0.3)
  ┌────────────┐           ┌──────────────────┐      ┌────────────┐
  │ Alloy agent│──wg0──┐   │ Loki  ◀── Alloy  │  ┌───│ Alloy agent│
  └────────────┘       └──▶│ Grafana (Caddy)  │◀─┘   └────────────┘
   docker+caddy+app        └──────────────────┘       docker+caddy+app
```

- **Loki** — filesystem storage, **30-day retention**, listening on the mesh IP
  `10.10.0.2:3100`. Never public.
- **Grafana** — the UI, fronted by beta's Caddy at
  **`dashboard.thermograph.org`** (Google SSO, pre-provisioned accounts only,
  `allow_sign_up=false`, plus a break-glass local admin). Use that hostname
  everywhere; **never** `grafana.thermograph.org`.
- **Alloy** — on each node, shipping three sources: every Docker container's
  stdout/stderr, Caddy's host access logs, and the app's structured JSON logs
  (`errors`/`access`/`audit` `*.jsonl`, parsed so `level`/`tag`/`phase` become
  labels). Every line is tagged `host = prod|beta|dev`.

There is **no build and no deploy automation** for this domain — it ships by
hand. But `observability-validate.yml` parses every artifact that ships to a
node, so a malformed dashboard or an unparseable Alloy config fails in CI
instead of on the host.

## The rule that matters most

**The repo is the only durable path.** Dashboards and alerting are provisioned
from `observability/grafana/provisioning/` at startup, so **edits made through
Grafana's UI or API are overwritten on the next provision**. A change goes
through a PR. Centralis's `dashboard_write` opens one for you.

## Reading the fleet

Start wide, then narrow:

1. `fleet_status` — HTTP reachability plus what's running on each host.
2. `logs_overview since="1h"` — where the volume is, and what has gone quiet.
3. `logs_query service=… host=…` — the actual lines.

### ⚠ The prod trap

Prod runs a Swarm stack, and its `job="docker"` streams carry **no `service`
label**. A raw `{host="prod", service="backend"}` query matches nothing and
reads as *"no logs"* when it actually means *"wrong selector"*.

Use Centralis's `logs_query` `service` argument, which rewrites it to a
container name-regex with the churning task suffix wildcarded. Prod service
names are `thermograph_web`, `_worker`, `_frontend`, `_db`, `_autoscaler`,
`_lake`.

Related: **prod container names change on every redeploy** (the Swarm task
suffix). Never hardcode one; resolve via `docker ps --filter name=` at call
time.

## The dashboard

`Thermograph — Fleet Logs` (auto-provisioned), with a `Node` selector: log
volume by service, app error rate by tag, upstream 429 count, Caddy 5xx count, a
per-node notifier-liveness indicator, recent app errors, and a live
all-container tail.

## Alerting

Provisioned from `grafana/provisioning/alerting/`, mounted read-only into
Grafana.

**Every rule is LogQL** — there is no Prometheus anywhere in the fleet. Loki is
the only datasource, so every signal is derived from log lines. Thresholds were
backtested against real Loki data and the working is written out in the comments
beside each rule; **re-derive before changing a number rather than guessing**.

### Alerts go to Discord `#ops-alerts`, never email

Three reasons, all learned the hard way:

- Grafana's factory default pointed at the literal string
  `<example@email.com>` — a paging path that had never delivered a message to
  anyone.
- Beta's Grafana relays SMTP through **prod's** Postfix. Email alerts therefore
  travel through the box most likely to be on fire, and vanish exactly when they
  matter.
- Discord is off-estate, works when prod is dead, and reaches a phone.

`#ops-alerts` is dedicated. `#dev` / `#uat` / `#prod` are the *product's* alert
subscription output — real deliveries to real subscribers — so routing ops noise
there would corrupt the evidence.

The webhook URL is a secret and lives only in beta's gitignored `.env` as
`DISCORD_ALERT_WEBHOOK_URL`. Grafana refuses to start if it's unset. A literal
webhook URL committed to the repo is a **hard CI failure**.

### The twelve rules

| Rule | Fires when | Severity |
|---|---|---|
| `ProdEdgeDark` | zero prod Caddy log lines in 15m — site dark, or the pipeline is broken | critical |
| `ProdHealthzFailing` | any non-200 on `/healthz` in 10m | critical |
| `ProdHTTP5xxBurst` | ≥10 prod 5xx in 10m | critical |
| `ProdHTTP5xxElevated` | ≥25 prod 5xx over 6h (the error-budget alarm) | warning |
| `ProdUnhandledExceptions` | ≥3 app `tag=http.error` in 30m | warning |
| `NotifierHeartbeatMissingProd` | no `tag=heartbeat` from prod in 20m | critical |
| `NotifierHeartbeatMissingBeta` | same, on beta | warning |
| `ProdWebContainerSilent` | `thermograph_web` logged nothing in 30m | critical |
| `ProdDbContainerSilent` | `thermograph_db` logged nothing in 2h | critical |
| `ProdFrontendContainerSilent` | `thermograph_frontend` logged nothing in 30m | warning |
| `ProdWorkerContainerSilent` | `thermograph_worker` logged nothing in 2h | warning |
| `AlertingWatchdog` | always — one Discord message per day | watchdog |

`AlertingWatchdog` is the dead-man's switch and fires permanently by design.
**Its value is its absence:** nothing can report its own death, so if
`#ops-alerts` has been silent for more than a day, alerting is broken rather
than the estate healthy.

### The strict CI check, and why

A rule whose `condition` names a refId that does not exist is **perfectly valid
YAML, provisions without complaint, and then never fires** — a paging path that
looks healthy and reaches nobody. `observability-validate.yml` checks that every
condition names a real refId and every policy routes to a receiver that exists,
precisely because nothing else would catch it.

The same asymmetry shows up in verification: `GET
/api/v1/provisioning/contact-points` **redacts** the webhook URL, so you cannot
confirm interpolation by reading it back. A contact point holding the literal
string `$DISCORD_ALERT_WEBHOOK_URL` looks completely healthy from the API and
pages nobody. **The only proof is a message actually arriving in
`#ops-alerts`** — force one with the receivers-test endpoint (the command is in
`observability/README.md`).

## Deploying an observability change

> ⚠️ **Merging is not deploying — and this one has already bitten.**
> `/opt/observability` on beta and prod is a checkout of the **archived**
> `emi/thermograph-observability` repo. It can never `git pull` again; the
> content now lives here under `observability/`. A previous observability PR
> merged and had **zero live effect** for exactly this reason.

Until someone re-points those checkouts at the monorepo, every change ships by
hand: back up the live file with a UTC-timestamped suffix, `scp` the new one up,
`sudo cp` it into place, restart the specific container.

Two gotchas in that sequence:

- Use `sudo docker restart <container>` on **prod** — not `docker compose`,
  which there demands a `GF_SECURITY_ADMIN_PASSWORD` it has no `.env` to
  interpolate from. Beta's stack *does* have an `.env`, so compose works there.
- **`docker restart` will not pick up a new environment variable.** Any change
  that introduces one needs `docker compose up -d grafana` on beta.

Alerting lives on **beta only** — prod and dev run Alloy agents, not Grafana.

Full step-by-step, including rollback, is in
[`observability/README.md`](../../observability/README.md).

## Known gaps (as documented in the repo)

- **Alloy can silently stop tailing a container after Swarm replaces the task.**
  Observed on prod 2026-07-24: `thermograph_worker` was `1/1`, healthy, writing
  ~19 lines per 10 minutes to its own docker log, yet not one line reached Loki
  for 13½ hours. What localises it to Alloy: prod's `tag=heartbeat` lines kept
  arriving throughout, and those come from a *different* Alloy source
  (`loki.source.file` on `/applogs`) than container stdout
  (`loki.source.docker`). One source went deaf, the other didn't. Workaround:
  `sudo docker restart alloy-alloy-1` on the node. **Not root-caused.**
  `ProdWorkerContainerSilent` now catches this class of failure.
- **Some services are too quiet to alert on.** `thermograph_daemon` (23 lines /
  24h), `thermograph_autoscaler` (1) and `_autoscaler-lake` (2) have a healthy
  floor indistinguishable from dead; `thermograph_lake` is bursty ETL with
  legitimate idle gaps. Giving each an `audit.log_heartbeat()` line would make
  all four alertable with the pattern the notifier already uses. **The single
  highest-value follow-up in this domain.**
- **Grafana cannot alert on its own death.** `AlertingWatchdog` is a *manual*
  dead-man's switch — it relies on somebody noticing the daily message stopped.
  An external uptime pinger from off-estate would close this properly.
- **Beta runs a `docker-compose.override.yml` that is not in this repo**, wiring
  Grafana's SMTP to prod's Postfix. It should be mirrored into the tracked
  compose file or deleted; alerting no longer depends on it.

## When something breaks

1. `fleet_status` — is it up at all?
2. `logs_query host=prod contains="(?i)error|traceback"` since the incident.
3. `deployed_version env=prod` — did something ship just before it started?
4. `forge_prs` / `forge_ci` — what landed recently, and did CI actually pass?
5. **Write down what you found with `notes_write` *before* fixing it.** The
   reconstruction is worth more than the fix, and it will be gone by tomorrow.

You are authorised to fix production roadblocks directly — restart or roll a
wedged service, clear a full disk, kill a runaway process, bounce a crash-looping
container. What you must not do is make the fix *durable* on the box: writes to
code, docs and dashboards go through a PR, and `deploy.sh` hard-resets the
checkout anyway. The on-box action is stabilisation; the PR is the fix.

Next: [Recipes](10-recipes.md).