# thermograph-observability — agent instructions

The **logging/metrics stack** for the Thermograph fleet: Loki + Grafana on beta,
with a Grafana Alloy agent on every node (prod, beta, dev) shipping container and
app logs over the WireGuard mesh. Grafana is fronted by beta's Caddy at
`dashboard.thermograph.org` (Google SSO, pre-provisioned users only).

This is **operational infra config, not application code** — there is no build.
It deploys by hand: `docker compose up -d` on beta for Loki+Grafana, and the
Alloy agent (`alloy/docker-compose.agent.yml`) on each node. See the README for
the full per-node procedure.

## Layout

- `docker-compose.yml` — the Loki + Grafana stack (runs on beta).
- `loki/config.yml` — Loki config (mesh-only, filesystem storage).
- `grafana/provisioning/` — datasource (Loki) + dashboard provider, auto-loaded
  at startup. `grafana/dashboards/*.json` — the dashboards themselves.
- `alloy/config.alloy` + `alloy/docker-compose.agent.yml` — the per-node log
  shipper. The node name comes from `ALLOY_NODE` (set per host).
- `caddy-grafana.conf` — the Caddy vhost for `dashboard.thermograph.org` (lives
  in beta's Caddy config, kept here for reference).
- `.env.example` — copy to `.env` on beta before `docker compose up`. `.env` is
  gitignored; never commit real OAuth secrets or the admin password.

## Conventions

- **Every artifact that ships to a node is CI-validated** (`.forgejo/workflows/validate.yml`):
  both compose files parse, all dashboard JSON is valid, and the Loki/provisioning
  YAML parses. Keep new dashboards as valid JSON and new config as valid YAML or
  CI fails. The Alloy config is not yet CI-validated (needs the `alloy` binary).
- The public hostname is **`dashboard.thermograph.org`** everywhere — not
  `grafana.thermograph.org`. Match it in any new comment/config.
- Secrets (OAuth client id/secret, admin password) live only in the host `.env`,
  never in the repo.

## Branching

Single `main` branch, no protection. Open a PR against `main`; the validator
gates it. Deploying the change to beta / the nodes is a separate manual step
(this repo has no deploy automation — a known gap).