# thermograph-observability — agent instructions

The **logging/metrics stack** for the Thermograph fleet: Loki + Grafana on beta,
with a Grafana Alloy agent on every node (prod, beta, dev) shipping container and
app logs over the WireGuard mesh. Grafana is fronted by beta's Caddy at
`dashboard.thermograph.org` (Google SSO, pre-provisioned users only).

This is **operational infra config, not application code** — there is no build.
It deploys by hand: `docker compose up -d` on beta for Loki+Grafana, and the
Alloy agent (`alloy/docker-compose.agent.yml`) on each node. See the README for
the full per-node procedure.

## Layout

- `docker-compose.yml` — the Loki + Grafana stack (runs on beta).
- `loki/config.yml` — Loki config (mesh-only, filesystem storage).
- `grafana/provisioning/` — datasource (Loki) + dashboard provider, auto-loaded
  at startup. `grafana/dashboards/*.json` — the dashboards themselves.
- `grafana/provisioning/alerting/` — the alert rules, the Discord contact point
  and the notification policy, also auto-loaded at startup. Same rule as the
  dashboards: **the repo is the only durable path**, UI edits get overwritten.
  Alerts go to Discord `#ops-alerts`, never email — beta's Grafana relays SMTP
  through prod's Postfix, so email dies exactly when prod does. Every rule is
  LogQL (there is no Prometheus anywhere in the fleet). Thresholds were derived
  from real Loki data and the working is in the comments beside each rule —
  re-derive before changing a number rather than guessing.
- `alloy/config.alloy` + `alloy/docker-compose.agent.yml` — the per-node log
  shipper. The node name comes from `ALLOY_NODE` (set per host).
- `caddy-grafana.conf` — the Caddy vhost for `dashboard.thermograph.org` (lives
  in beta's Caddy config, kept here for reference).
- `.env.example` — copy to `.env` on beta before `docker compose up`. `.env` is
  gitignored; never commit real OAuth secrets or the admin password.

## Conventions

- **Every artifact that ships to a node is CI-validated**
  (`.forgejo/workflows/observability-validate.yml`): both compose files parse, all
  dashboard JSON is valid, and the Loki/provisioning YAML parses. Keep new
  dashboards as valid JSON and new config as valid YAML or CI fails. The alerting
  config gets a stricter third step — every rule's `condition` must name a refId
  that exists, every policy must route to a receiver that exists, and a literal
  Discord webhook URL in the repo is a hard failure. (A rule pointing at a missing
  refId is valid YAML, provisions cleanly, and then never fires; that is precisely
  the silent-no-op this whole domain exists to prevent.) The Alloy config is still
  not CI-validated (needs the `alloy` binary).
- The public hostname is **`dashboard.thermograph.org`** everywhere — not
  `grafana.thermograph.org`. Match it in any new comment/config.
- Secrets (OAuth client id/secret, admin password) live only in the host `.env`,
  never in the repo.

## Branching

Single `main` branch, no protection. Open a PR against `main`; the validator
gates it. Deploying the change to beta / the nodes is a separate manual step
(this repo has no deploy automation — a known gap).