thermograph/docs/onboarding/09-observability.md

# 9. Observability

Fleet-wide log aggregation: **Loki + Grafana on vps1**, fed by a **Grafana
Alloy** agent on every node, all over the WireGuard mesh. vps2 runs two
environments (beta, prod) behind that one Alloy agent; vps1 runs Alloy
alongside the things it's shipping logs *from* (Forgejo, Grafana itself, dev).

```
  vps2 (10.10.0.1)          vps1 (10.10.0.2)          desktop (10.10.0.3)
  prod + beta               Loki, Grafana, dev
  ┌────────────┐           ┌──────────────────┐      ┌────────────┐
  │ Alloy agent│──wg0──┐   │ Loki  ◀── Alloy  │  ┌───│ Alloy agent│
  └────────────┘       └──▶│ Grafana (Caddy)  │◀─┘   └────────────┘
   docker+caddy+app        └──────────────────┘       docker (AI models)
```

- **Loki** — filesystem storage, **30-day retention**, listening on the mesh IP
  `10.10.0.2:3100`. Never public.
- **Grafana** — the UI, fronted by vps1's Caddy at
  **`dashboard.thermograph.org`** (Google SSO, pre-provisioned accounts only,
  `allow_sign_up=false`, plus a break-glass local admin). Use that hostname
  everywhere; **never** `grafana.thermograph.org`.
- **Alloy** — one agent per *host*, not per environment: vps2's single agent
  ships logs for both beta and prod, vps1's ships Forgejo/Grafana/dev, and the
  desktop's ships whatever runs there now. Each ships three sources: every
  Docker container's stdout/stderr, Caddy's host access logs, and the app's
  structured JSON logs (`errors`/`access`/`audit` `*.jsonl`, parsed so
  `level`/`tag`/`phase` become labels). Every app log line is tagged
  `host = prod|beta|dev` — that label is the *environment*, not the box, which
  matters on vps2 where one Alloy agent ships lines carrying two different
  `host` values.

There is **no build and no deploy automation** for this domain — it ships by
hand. But `observability-validate.yml` parses every artifact that ships to a
node, so a malformed dashboard or an unparseable Alloy config fails in CI
instead of on the host.

## The rule that matters most

**The repo is the only durable path.** Dashboards and alerting are provisioned
from `observability/grafana/provisioning/` at startup, so **edits made through
Grafana's UI or API are overwritten on the next provision**. A change goes
through a PR. Centralis's `dashboard_write` opens one for you.

## Reading the fleet

Start wide, then narrow:

1. `fleet_status` — HTTP reachability plus what's running on each host.
2. `logs_overview since="1h"` — where the volume is, and what has gone quiet.
3. `logs_query service=… host=…` — the actual lines.

### ⚠ The prod (and now beta) trap

Prod and beta both run Swarm stacks — co-resident on vps2 — and their
`job="docker"` streams carry **no `service` label**. A raw
`{host="prod", service="backend"}` query matches nothing and reads as *"no
logs"* when it actually means *"wrong selector"*. This used to be a prod-only
trap when beta ran compose; it now applies to both environments on vps2, and
dev (compose, on vps1) is the one environment this does *not* affect.

Use Centralis's `logs_query` `service` argument, which rewrites it to a
container name-regex with the churning task suffix wildcarded. Prod service
names are `thermograph_web`, `_worker`, `_frontend`, `_db`, `_autoscaler`,
`_lake`; beta's are the same names prefixed `beta-` (`beta-web`, `beta-worker`,
…) with no `_db` or autoscaler equivalents.

Related: **prod and beta container names both change on every redeploy** (the
Swarm task suffix). Never hardcode one; resolve via `docker ps --filter name=`
at call time.

## The dashboard

`Thermograph — Fleet Logs` (auto-provisioned), with a `Node` selector: log
volume by service, app error rate by tag, upstream 429 count, Caddy 5xx count, a
per-node notifier-liveness indicator, recent app errors, and a live
all-container tail.

## Alerting

Provisioned from `grafana/provisioning/alerting/`, mounted read-only into
Grafana.

**Every rule is LogQL** — there is no Prometheus anywhere in the fleet. Loki is
the only datasource, so every signal is derived from log lines. Thresholds were
backtested against real Loki data and the working is written out in the comments
beside each rule; **re-derive before changing a number rather than guessing**.

### Alerts go to Discord `#ops-alerts`, never email

Three reasons, all learned the hard way:

- Grafana's factory default pointed at the literal string
  `<example@email.com>` — a paging path that had never delivered a message to
  anyone.
- vps1's Grafana relays SMTP through **vps2's** Postfix (prod's mail service).
  Email alerts therefore travel through the box most likely to be on fire, and
  vanish exactly when they matter.
- Discord is off-estate, works when prod is dead, and reaches a phone.

`#ops-alerts` is dedicated. `#dev` / `#uat` / `#prod` are the *product's* alert
subscription output — real deliveries to real subscribers — so routing ops noise
there would corrupt the evidence.

The webhook URL is a secret and lives only in vps1's gitignored `.env` as
`DISCORD_ALERT_WEBHOOK_URL`. Grafana refuses to start if it's unset. A literal
webhook URL committed to the repo is a **hard CI failure**.

### The twelve rules

| Rule | Fires when | Severity |
|---|---|---|
| `ProdEdgeDark` | zero prod Caddy log lines in 15m — site dark, or the pipeline is broken | critical |
| `ProdHealthzFailing` | any non-200 on `/healthz` in 10m | critical |
| `ProdHTTP5xxBurst` | ≥10 prod 5xx in 10m | critical |
| `ProdHTTP5xxElevated` | ≥25 prod 5xx over 6h (the error-budget alarm) | warning |
| `ProdUnhandledExceptions` | ≥3 app `tag=http.error` in 30m | warning |
| `NotifierHeartbeatMissingProd` | no `tag=heartbeat` from prod in 20m | critical |
| `NotifierHeartbeatMissingBeta` | same, on beta | warning |
| `ProdWebContainerSilent` | `thermograph_web` logged nothing in 30m | critical |
| `ProdDbContainerSilent` | `thermograph_db` logged nothing in 2h | critical |
| `ProdFrontendContainerSilent` | `thermograph_frontend` logged nothing in 30m | warning |
| `ProdWorkerContainerSilent` | `thermograph_worker` logged nothing in 2h | warning |
| `AlertingWatchdog` | always — one Discord message per day | watchdog |

`AlertingWatchdog` is the dead-man's switch and fires permanently by design.
**Its value is its absence:** nothing can report its own death, so if
`#ops-alerts` has been silent for more than a day, alerting is broken rather
than the estate healthy.

### The strict CI check, and why

A rule whose `condition` names a refId that does not exist is **perfectly valid
YAML, provisions without complaint, and then never fires** — a paging path that
looks healthy and reaches nobody. `observability-validate.yml` checks that every
condition names a real refId and every policy routes to a receiver that exists,
precisely because nothing else would catch it.

The same asymmetry shows up in verification: `GET
/api/v1/provisioning/contact-points` **redacts** the webhook URL, so you cannot
confirm interpolation by reading it back. A contact point holding the literal
string `$DISCORD_ALERT_WEBHOOK_URL` looks completely healthy from the API and
pages nobody. **The only proof is a message actually arriving in
`#ops-alerts`** — force one with the receivers-test endpoint (the command is in
`observability/README.md`).

## Deploying an observability change

> ⚠️ **Merging is not deploying — and this one has already bitten.**
> `/opt/observability` on vps1 and vps2 is a checkout of the **archived**
> `admin_emi/thermograph-observability` repo. It can never `git pull` again; the
> content now lives here under `observability/`. A previous observability PR
> merged and had **zero live effect** for exactly this reason.

Until someone re-points those checkouts at the monorepo, every change ships by
hand: back up the live file with a UTC-timestamped suffix, `scp` the new one up,
`sudo cp` it into place, restart the specific container.

Two gotchas in that sequence:

- Use `sudo docker restart <container>` on **vps2** — not `docker compose`,
  which there demands a `GF_SECURITY_ADMIN_PASSWORD` it has no `.env` to
  interpolate from. vps1's stack *does* have an `.env`, so compose works there.
- **`docker restart` will not pick up a new environment variable.** Any change
  that introduces one needs `docker compose up -d grafana` on vps1.

Alerting lives on **vps1 only** — vps2 (prod and beta alike) and dev run Alloy
agents, not Grafana.

Full step-by-step, including rollback, is in
[`observability/README.md`](../../observability/README.md).

## Known gaps (as documented in the repo)

- **Alloy can silently stop tailing a container after Swarm replaces the task.**
  Observed on prod 2026-07-24: `thermograph_worker` was `1/1`, healthy, writing
  ~19 lines per 10 minutes to its own docker log, yet not one line reached Loki
  for 13½ hours. What localises it to Alloy: prod's `tag=heartbeat` lines kept
  arriving throughout, and those come from a *different* Alloy source
  (`loki.source.file` on `/applogs`) than container stdout
  (`loki.source.docker`). One source went deaf, the other didn't. Workaround:
  `sudo docker restart alloy-alloy-1` on the node. **Not root-caused.**
  `ProdWorkerContainerSilent` now catches this class of failure.
- **Some services are too quiet to alert on.** `thermograph_daemon` (23 lines /
  24h), `thermograph_autoscaler` (1) and `_autoscaler-lake` (2) have a healthy
  floor indistinguishable from dead; `thermograph_lake` is bursty ETL with
  legitimate idle gaps. Giving each an `audit.log_heartbeat()` line would make
  all four alertable with the pattern the notifier already uses. **The single
  highest-value follow-up in this domain.**
- **Grafana cannot alert on its own death.** `AlertingWatchdog` is a *manual*
  dead-man's switch — it relies on somebody noticing the daily message stopped.
  An external uptime pinger from off-estate would close this properly.
- **vps1 runs a `docker-compose.override.yml` that is not in this repo**, wiring
  Grafana's SMTP to vps2's Postfix (prod's). It should be mirrored into the
  tracked compose file or deleted; alerting no longer depends on it.

## When something breaks

1. `fleet_status` — is it up at all?
2. `logs_query host=prod contains="(?i)error|traceback"` since the incident.
3. `deployed_version env=prod` — did something ship just before it started?
4. `forge_prs` / `forge_ci` — what landed recently, and did CI actually pass?
5. **Write down what you found with `notes_write` *before* fixing it.** The
   reconstruction is worth more than the fix, and it will be gone by tomorrow.

You are authorised to fix production roadblocks directly — restart or roll a
wedged service, clear a full disk, kill a runaway process, bounce a crash-looping
container. What you must not do is make the fix *durable* on the box: writes to
code, docs and dashboards go through a PR, and `deploy.sh` hard-resets the
checkout anyway. The on-box action is stabilisation; the PR is the fix.

Next: [Recipes](10-recipes.md).