Emi Griffith
6137a08c23
Replace the grep-docker-logs notifier panel with a per-node UP/DOWN stat driven by the app's new tag=heartbeat log line (count_over_time over 20m; 0 beats => DOWN/red). Update the README now that the app emits heartbeats.
123 lines
5.9 KiB
Markdown
123 lines
5.9 KiB
Markdown
# Thermograph — observability
|
|
|
|
Fleet-wide log aggregation for [Thermograph](https://thermograph.org): a central
|
|
**Grafana + Loki** stack, fed by a **Grafana Alloy** agent on every node, all
|
|
over the private WireGuard mesh. This replaces the old in-repo
|
|
`scripts/dashboard.py` (an SSH-tailed, single-host, metrics-endpoint view) with a
|
|
persistent, queryable, fleet-wide log store and UI.
|
|
|
|
```
|
|
prod (10.10.0.1) beta (10.10.0.2) desktop / LAN dev (10.10.0.3)
|
|
┌────────────┐ ┌──────────────────┐ ┌────────────┐
|
|
│ Alloy agent│──wg0──┐ │ Loki ◀── Alloy │ ┌──wg0│ Alloy agent│
|
|
└────────────┘ └───▶│ Grafana (Caddy) │◀──┘ └────────────┘
|
|
docker+caddy+app └──────────────────┘ docker+caddy+app
|
|
logs central store + UI logs
|
|
```
|
|
|
|
- **Loki** stores logs (filesystem, 30-day retention). Listens on the mesh IP
|
|
`10.10.0.2:3100` — never public.
|
|
- **Grafana** is the UI, fronted by beta's Caddy at `dashboard.thermograph.org`,
|
|
login via **Google SSO** (with a break-glass local admin); Loki is not exposed.
|
|
- **Alloy** runs on each node and ships three sources to Loki: every Docker
|
|
container's stdout/stderr, Caddy's host access logs, and the app's structured
|
|
JSON logs (`errors`/`access`/`audit` `*.jsonl`, parsed so `level`/`tag`/`phase`
|
|
become labels). Every line is tagged with its node (`host = prod|beta|dev`).
|
|
|
|
## Layout
|
|
|
|
| Path | What |
|
|
|------|------|
|
|
| `docker-compose.yml` | the central Loki + Grafana stack (runs on beta) |
|
|
| `loki/config.yml` | Loki config (filesystem, retention) |
|
|
| `grafana/provisioning/` | auto-wired Loki datasource + dashboard provider |
|
|
| `grafana/dashboards/thermograph-logs.json` | the fleet-logs dashboard |
|
|
| `alloy/config.alloy` | the per-node collector config |
|
|
| `alloy/docker-compose.agent.yml` | runs the Alloy agent on a node |
|
|
| `caddy-grafana.conf` | beta Caddy site block for the Grafana UI |
|
|
|
|
## Deploy
|
|
|
|
### 1. Central stack (on beta)
|
|
|
|
```bash
|
|
# on beta (75.119.132.91):
|
|
git clone <this repo> thermograph-observability && cd thermograph-observability
|
|
cp .env.example .env # set GF_SECURITY_ADMIN_PASSWORD
|
|
docker compose up -d # Loki on 10.10.0.2:3100 + 127.0.0.1:3100; Grafana on 127.0.0.1:3000
|
|
```
|
|
|
|
Expose the Grafana UI: point `dashboard.thermograph.org` at beta, append
|
|
`caddy-grafana.conf` to `/etc/caddy/Caddyfile`, `systemctl reload caddy`.
|
|
|
|
### Google SSO (Grafana + Forgejo)
|
|
|
|
Both the dashboard and Forgejo log in with Google. Create **one** Google Cloud
|
|
OAuth 2.0 Client (type: *Web application*) and register both redirect URIs:
|
|
|
|
- Grafana: `https://dashboard.thermograph.org/login/google`
|
|
- Forgejo: `https://git.thermograph.org/user/oauth2/google/callback`
|
|
|
|
**Grafana:** put the client id/secret in `.env`, set `OAUTH_ENABLED=true`, and
|
|
`docker compose up -d`. Login is locked to pre-provisioned accounts
|
|
(`allow_sign_up=false`), so provision your email as an admin once:
|
|
|
|
```bash
|
|
# on beta, after the stack is up (uses the break-glass admin creds from .env):
|
|
curl -s -u admin:"$GF_SECURITY_ADMIN_PASSWORD" -H 'Content-Type: application/json' \
|
|
-X POST http://127.0.0.1:3000/api/admin/users \
|
|
-d '{"name":"you","login":"you@gmail.com","email":"you@gmail.com","password":"'"$(openssl rand -base64 24)"'"}'
|
|
# then make that user a Grafana admin (org role Admin + server admin) via the UI
|
|
# or /api/org/users; the Google login matches on email.
|
|
```
|
|
|
|
**Forgejo:** add the Google auth source (once, as a Forgejo admin):
|
|
|
|
```bash
|
|
docker exec -u git <forgejo-container> forgejo admin auth add-oauth \
|
|
--name google --provider openidConnect \
|
|
--key "<CLIENT_ID>" --secret "<CLIENT_SECRET>" \
|
|
--auto-discover-url https://accounts.google.com/.well-known/openid-configuration
|
|
```
|
|
|
|
### 2. Alloy agent (on every node — prod, beta, desktop)
|
|
|
|
```bash
|
|
cd thermograph-observability/alloy
|
|
# Every node (beta included) pushes to Loki over the mesh IP — a containerized
|
|
# agent can't reach the host's 127.0.0.1, but it can reach beta's wg0 address:
|
|
# beta: LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push ALLOY_NODE=beta
|
|
# prod: LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push ALLOY_NODE=prod
|
|
# desktop: LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push ALLOY_NODE=dev
|
|
ALLOY_NODE=prod LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push \
|
|
docker compose -f docker-compose.agent.yml up -d
|
|
```
|
|
|
|
On the LAN dev node the app's log volume is `thermograph-dev_applogs` (not
|
|
`thermograph_applogs`) — edit the two references in `docker-compose.agent.yml`
|
|
there. The mesh must already be up (see the main repo's `deploy/swarm/` /
|
|
`INFRA.md`); Swarm is **not** required — the agents talk plain HTTP over wg0.
|
|
|
|
## Verify
|
|
|
|
```bash
|
|
# From any mesh node, confirm Loki is receiving from every host:
|
|
curl -s 'http://10.10.0.2:3100/loki/api/v1/label/host/values' # -> {"data":["beta","dev","prod"]}
|
|
# Then open Grafana → the "Thermograph — Fleet Logs" dashboard.
|
|
```
|
|
|
|
## The dashboard
|
|
|
|
`Thermograph — Fleet Logs` (auto-provisioned), with a `Node` selector:
|
|
log volume by service, app error rate by tag, upstream 429 count, Caddy 5xx
|
|
count, a per-node notifier-liveness indicator (from the heartbeat log), recent app errors, and a live all-container
|
|
tail. Edit it in Grafana and re-export the JSON here to version a change.
|
|
|
|
## Notes / follow-ups
|
|
|
|
- **Loki is mesh-only** by design; only Grafana (with auth) is public.
|
|
- Alloy reads the Docker socket read-only to discover containers.
|
|
- Retention is 30 days on beta's disk — bump `loki/config.yml` if you want more.
|
|
- The app emits a `tag=heartbeat` line for the subscription notifier every
|
|
~15m; the dashboard's "Notifier alive?" panel turns red if a node misses it.
|
|
Extend the same `audit.log_heartbeat()` to any future background daemon.
|