Emi Griffith
46edb0bca0
A central Grafana + Loki stack (on beta) fed by a Grafana Alloy agent on every node, replacing the old SSH-tailed single-host scripts/dashboard.py. - docker-compose.yml: central Loki (mesh-only :3100) + Grafana (Caddy-fronted) - loki/config.yml: single-binary, filesystem storage, 30-day retention - alloy/config.alloy + docker-compose.agent.yml: per-node collector — every container's stdout/stderr via the Docker socket, Caddy host logs, and the app's structured JSON logs (errors/access/audit), each line tagged by node - grafana/: auto-provisioned Loki datasource + a fleet-logs dashboard (volume by service, error rate, upstream 429s, Caddy 5xx, notifier liveness, live tail), with a per-node selector - caddy-grafana.conf, README, .env.example
92 lines
4.4 KiB
Markdown
92 lines
4.4 KiB
Markdown
# Thermograph — observability
|
|
|
|
Fleet-wide log aggregation for [Thermograph](https://thermograph.org): a central
|
|
**Grafana + Loki** stack, fed by a **Grafana Alloy** agent on every node, all
|
|
over the private WireGuard mesh. This replaces the old in-repo
|
|
`scripts/dashboard.py` (an SSH-tailed, single-host, metrics-endpoint view) with a
|
|
persistent, queryable, fleet-wide log store and UI.
|
|
|
|
```
|
|
prod (10.10.0.1) beta (10.10.0.2) desktop / LAN dev (10.10.0.3)
|
|
┌────────────┐ ┌──────────────────┐ ┌────────────┐
|
|
│ Alloy agent│──wg0──┐ │ Loki ◀── Alloy │ ┌──wg0│ Alloy agent│
|
|
└────────────┘ └───▶│ Grafana (Caddy) │◀──┘ └────────────┘
|
|
docker+caddy+app └──────────────────┘ docker+caddy+app
|
|
logs central store + UI logs
|
|
```
|
|
|
|
- **Loki** stores logs (filesystem, 30-day retention). Listens on the mesh IP
|
|
`10.10.0.2:3100` — never public.
|
|
- **Grafana** is the UI, fronted by beta's Caddy at `grafana.thermograph.org`
|
|
(its own login; Loki is not exposed).
|
|
- **Alloy** runs on each node and ships three sources to Loki: every Docker
|
|
container's stdout/stderr, Caddy's host access logs, and the app's structured
|
|
JSON logs (`errors`/`access`/`audit` `*.jsonl`, parsed so `level`/`tag`/`phase`
|
|
become labels). Every line is tagged with its node (`host = prod|beta|dev`).
|
|
|
|
## Layout
|
|
|
|
| Path | What |
|
|
|------|------|
|
|
| `docker-compose.yml` | the central Loki + Grafana stack (runs on beta) |
|
|
| `loki/config.yml` | Loki config (filesystem, retention) |
|
|
| `grafana/provisioning/` | auto-wired Loki datasource + dashboard provider |
|
|
| `grafana/dashboards/thermograph-logs.json` | the fleet-logs dashboard |
|
|
| `alloy/config.alloy` | the per-node collector config |
|
|
| `alloy/docker-compose.agent.yml` | runs the Alloy agent on a node |
|
|
| `caddy-grafana.conf` | beta Caddy site block for the Grafana UI |
|
|
|
|
## Deploy
|
|
|
|
### 1. Central stack (on beta)
|
|
|
|
```bash
|
|
# on beta (75.119.132.91):
|
|
git clone <this repo> thermograph-observability && cd thermograph-observability
|
|
cp .env.example .env # set GF_SECURITY_ADMIN_PASSWORD
|
|
docker compose up -d # Loki on 10.10.0.2:3100 + 127.0.0.1:3100; Grafana on 127.0.0.1:3000
|
|
```
|
|
|
|
Expose the Grafana UI: point `grafana.thermograph.org` at beta, append
|
|
`caddy-grafana.conf` to `/etc/caddy/Caddyfile`, `systemctl reload caddy`.
|
|
|
|
### 2. Alloy agent (on every node — prod, beta, desktop)
|
|
|
|
```bash
|
|
cd thermograph-observability/alloy
|
|
# beta pushes to local Loki; prod & desktop push over the mesh:
|
|
# beta: LOKI_URL=http://127.0.0.1:3100/loki/api/v1/push ALLOY_NODE=beta
|
|
# prod: LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push ALLOY_NODE=prod
|
|
# desktop: LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push ALLOY_NODE=dev
|
|
ALLOY_NODE=prod LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push \
|
|
docker compose -f docker-compose.agent.yml up -d
|
|
```
|
|
|
|
On the LAN dev node the app's log volume is `thermograph-dev_applogs` (not
|
|
`thermograph_applogs`) — edit the two references in `docker-compose.agent.yml`
|
|
there. The mesh must already be up (see the main repo's `deploy/swarm/` /
|
|
`INFRA.md`); Swarm is **not** required — the agents talk plain HTTP over wg0.
|
|
|
|
## Verify
|
|
|
|
```bash
|
|
# From any mesh node, confirm Loki is receiving from every host:
|
|
curl -s 'http://10.10.0.2:3100/loki/api/v1/label/host/values' # -> {"data":["beta","dev","prod"]}
|
|
# Then open Grafana → the "Thermograph — Fleet Logs" dashboard.
|
|
```
|
|
|
|
## The dashboard
|
|
|
|
`Thermograph — Fleet Logs` (auto-provisioned), with a `Node` selector:
|
|
log volume by service, app error rate by tag, upstream 429 count, Caddy 5xx
|
|
count, a notifier-liveness log panel, recent app errors, and a live all-container
|
|
tail. Edit it in Grafana and re-export the JSON here to version a change.
|
|
|
|
## Notes / follow-ups
|
|
|
|
- **Loki is mesh-only** by design; only Grafana (with auth) is public.
|
|
- Alloy reads the Docker socket read-only to discover containers.
|
|
- Retention is 30 days on beta's disk — bump `loki/config.yml` if you want more.
|
|
- Possible next step: have the app emit a periodic `notifier heartbeat` log line
|
|
so notifier liveness is a first-class log-derived signal, not just container
|
|
chatter.
|