commit 46edb0bca0043ac91eab355a56342a5ad36579f8 Author: Emi Griffith Date: Tue Jul 21 09:11:52 2026 -0700 Fleet log aggregation: Grafana + Loki + Alloy over the WireGuard mesh A central Grafana + Loki stack (on beta) fed by a Grafana Alloy agent on every node, replacing the old SSH-tailed single-host scripts/dashboard.py. - docker-compose.yml: central Loki (mesh-only :3100) + Grafana (Caddy-fronted) - loki/config.yml: single-binary, filesystem storage, 30-day retention - alloy/config.alloy + docker-compose.agent.yml: per-node collector — every container's stdout/stderr via the Docker socket, Caddy host logs, and the app's structured JSON logs (errors/access/audit), each line tagged by node - grafana/: auto-provisioned Loki datasource + a fleet-logs dashboard (volume by service, error rate, upstream 429s, Caddy 5xx, notifier liveness, live tail), with a per-node selector - caddy-grafana.conf, README, .env.example diff --git a/.env.example b/.env.example new file mode 100644 index 0000000..7ff6852 --- /dev/null +++ b/.env.example @@ -0,0 +1,9 @@ +# Copy to .env on the monitoring host (beta) before `docker compose up`. +# .env is gitignored — never commit real secrets. + +# Grafana admin login (the UI's own auth; Grafana is the only thing exposed). +GF_ADMIN_USER=admin +GF_SECURITY_ADMIN_PASSWORD=change-me-to-something-strong + +# The public hostname beta's Caddy serves Grafana at (see caddy-grafana.conf). +GRAFANA_DOMAIN=grafana.thermograph.org diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..845a438 --- /dev/null +++ b/.gitignore @@ -0,0 +1,3 @@ +.env +alloy/.env +*.log diff --git a/README.md b/README.md new file mode 100644 index 0000000..cee80ec --- /dev/null +++ b/README.md @@ -0,0 +1,92 @@ +# Thermograph — observability + +Fleet-wide log aggregation for [Thermograph](https://thermograph.org): a central +**Grafana + Loki** stack, fed by a **Grafana Alloy** agent on every node, all +over the private WireGuard mesh. This replaces the old in-repo +`scripts/dashboard.py` (an SSH-tailed, single-host, metrics-endpoint view) with a +persistent, queryable, fleet-wide log store and UI. + +``` + prod (10.10.0.1) beta (10.10.0.2) desktop / LAN dev (10.10.0.3) + ┌────────────┐ ┌──────────────────┐ ┌────────────┐ + │ Alloy agent│──wg0──┐ │ Loki ◀── Alloy │ ┌──wg0│ Alloy agent│ + └────────────┘ └───▶│ Grafana (Caddy) │◀──┘ └────────────┘ + docker+caddy+app └──────────────────┘ docker+caddy+app + logs central store + UI logs +``` + +- **Loki** stores logs (filesystem, 30-day retention). Listens on the mesh IP + `10.10.0.2:3100` — never public. +- **Grafana** is the UI, fronted by beta's Caddy at `grafana.thermograph.org` + (its own login; Loki is not exposed). +- **Alloy** runs on each node and ships three sources to Loki: every Docker + container's stdout/stderr, Caddy's host access logs, and the app's structured + JSON logs (`errors`/`access`/`audit` `*.jsonl`, parsed so `level`/`tag`/`phase` + become labels). Every line is tagged with its node (`host = prod|beta|dev`). + +## Layout + +| Path | What | +|------|------| +| `docker-compose.yml` | the central Loki + Grafana stack (runs on beta) | +| `loki/config.yml` | Loki config (filesystem, retention) | +| `grafana/provisioning/` | auto-wired Loki datasource + dashboard provider | +| `grafana/dashboards/thermograph-logs.json` | the fleet-logs dashboard | +| `alloy/config.alloy` | the per-node collector config | +| `alloy/docker-compose.agent.yml` | runs the Alloy agent on a node | +| `caddy-grafana.conf` | beta Caddy site block for the Grafana UI | + +## Deploy + +### 1. Central stack (on beta) + +```bash +# on beta (75.119.132.91): +git clone thermograph-observability && cd thermograph-observability +cp .env.example .env # set GF_SECURITY_ADMIN_PASSWORD +docker compose up -d # Loki on 10.10.0.2:3100 + 127.0.0.1:3100; Grafana on 127.0.0.1:3000 +``` + +Expose the Grafana UI: point `grafana.thermograph.org` at beta, append +`caddy-grafana.conf` to `/etc/caddy/Caddyfile`, `systemctl reload caddy`. + +### 2. Alloy agent (on every node — prod, beta, desktop) + +```bash +cd thermograph-observability/alloy +# beta pushes to local Loki; prod & desktop push over the mesh: +# beta: LOKI_URL=http://127.0.0.1:3100/loki/api/v1/push ALLOY_NODE=beta +# prod: LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push ALLOY_NODE=prod +# desktop: LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push ALLOY_NODE=dev +ALLOY_NODE=prod LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push \ + docker compose -f docker-compose.agent.yml up -d +``` + +On the LAN dev node the app's log volume is `thermograph-dev_applogs` (not +`thermograph_applogs`) — edit the two references in `docker-compose.agent.yml` +there. The mesh must already be up (see the main repo's `deploy/swarm/` / +`INFRA.md`); Swarm is **not** required — the agents talk plain HTTP over wg0. + +## Verify + +```bash +# From any mesh node, confirm Loki is receiving from every host: +curl -s 'http://10.10.0.2:3100/loki/api/v1/label/host/values' # -> {"data":["beta","dev","prod"]} +# Then open Grafana → the "Thermograph — Fleet Logs" dashboard. +``` + +## The dashboard + +`Thermograph — Fleet Logs` (auto-provisioned), with a `Node` selector: +log volume by service, app error rate by tag, upstream 429 count, Caddy 5xx +count, a notifier-liveness log panel, recent app errors, and a live all-container +tail. Edit it in Grafana and re-export the JSON here to version a change. + +## Notes / follow-ups + +- **Loki is mesh-only** by design; only Grafana (with auth) is public. +- Alloy reads the Docker socket read-only to discover containers. +- Retention is 30 days on beta's disk — bump `loki/config.yml` if you want more. +- Possible next step: have the app emit a periodic `notifier heartbeat` log line + so notifier liveness is a first-class log-derived signal, not just container + chatter. diff --git a/alloy/config.alloy b/alloy/config.alloy new file mode 100644 index 0000000..7ddbc29 --- /dev/null +++ b/alloy/config.alloy @@ -0,0 +1,96 @@ +// Grafana Alloy — the log collector that runs on EVERY node (prod, beta, LAN +// dev). It gathers three sources and ships them to the central Loki on beta: +// +// 1. Every Docker container's stdout/stderr (app, db, and on beta forgejo) +// via the Docker socket. +// 2. Caddy's host access logs (/var/log/caddy/*.log) — the reverse proxy runs +// on the host, not in a container, so its logs aren't in Docker. +// 3. The app's structured JSON logs (errors/access/audit *.jsonl) from the +// `applogs` Docker volume, parsed so `level`/`tag`/`phase` become labels. +// +// Per-node settings come from the environment (see docker-compose.agent.yml): +// ALLOY_NODE — this node's name label (prod | beta | dev) +// LOKI_URL — where to push (http://10.10.0.2:3100/loki/api/v1/push over wg0) + +livedebugging { enabled = false } + +// --- 1. All Docker container logs ------------------------------------------------ +discovery.docker "containers" { + host = "unix:///var/run/docker.sock" +} + +// Turn Docker metadata into tidy labels: `container` (short name) and `service` +// (the compose service, e.g. app/db). Drop Alloy's own container to avoid a loop. +discovery.relabel "containers" { + targets = discovery.docker.containers.targets + + rule { + source_labels = ["__meta_docker_container_name"] + regex = "/(.*)" + target_label = "container" + } + rule { + source_labels = ["__meta_docker_container_label_com_docker_compose_service"] + target_label = "service" + } + rule { + source_labels = ["container"] + regex = ".*alloy.*" + action = "drop" + } +} + +loki.source.docker "containers" { + host = "unix:///var/run/docker.sock" + targets = discovery.relabel.containers.output + forward_to = [loki.write.central.receiver] + relabel_rules = discovery.relabel.containers.rules + labels = { job = "docker" } +} + +// --- 2. Caddy host access logs --------------------------------------------------- +local.file_match "caddy" { + path_targets = [{ __path__ = "/var/log/caddy/*.log", job = "caddy" }] +} + +loki.source.file "caddy" { + targets = local.file_match.caddy.targets + forward_to = [loki.write.central.receiver] +} + +// --- 3. App structured JSON logs (errors / access / audit) ----------------------- +// Mounted read-only from the app's `applogs` volume at /applogs (see the agent +// compose). Lift `level`/`tag`/`phase` out of the JSON so they're queryable. +local.file_match "app_jsonl" { + path_targets = [{ __path__ = "/applogs/**/*.jsonl", job = "app-json" }] +} + +loki.source.file "app_jsonl" { + targets = local.file_match.app_jsonl.targets + forward_to = [loki.process.app_jsonl.receiver] +} + +loki.process "app_jsonl" { + forward_to = [loki.write.central.receiver] + + stage.json { + expressions = { level = "level", tag = "tag", phase = "phase", status = "status" } + } + // A record with no explicit level: an error-folder line is an error, else info. + stage.static_labels { + values = { source = "app" } + } + stage.labels { + values = { level = "", tag = "", phase = "" } + } +} + +// --- Ship to the central Loki over the WireGuard mesh ---------------------------- +loki.write "central" { + endpoint { + url = sys.env("LOKI_URL") + } + // Every line from this node is stamped with its node name, so one Grafana + // view can slice prod vs beta vs dev. + external_labels = { host = sys.env("ALLOY_NODE") } +} diff --git a/alloy/docker-compose.agent.yml b/alloy/docker-compose.agent.yml new file mode 100644 index 0000000..9e3efdd --- /dev/null +++ b/alloy/docker-compose.agent.yml @@ -0,0 +1,48 @@ +# The Alloy log-collector agent. Runs on EACH node (prod, beta, LAN dev) — one +# per node — reading that node's Docker containers, Caddy logs, and the app's +# JSON logs, and shipping them to the central Loki on beta over the mesh. +# +# Per-node config comes from the environment (an .env file next to this, or +# exported before `up`): +# ALLOY_NODE this node's label: prod | beta | dev +# LOKI_URL push endpoint. On beta: http://127.0.0.1:3100/loki/api/v1/push +# Elsewhere (over wg0): http://10.10.0.2:3100/loki/api/v1/push +# +# Deploy on a node: +# ALLOY_NODE=prod LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push \ +# docker compose -f docker-compose.agent.yml up -d +# +# The app's JSON logs are read from its `applogs` Docker volume, named +# `_applogs`. prod and beta run the app under the compose project +# `thermograph`, so the external volume below is `thermograph_applogs`. The LAN +# dev stack uses the project `thermograph-dev` — on that node, change the two +# `thermograph_applogs` references below to `thermograph-dev_applogs`. + +services: + alloy: + image: grafana/alloy:v1.9.1 + command: + - run + - /etc/alloy/config.alloy + - --storage.path=/var/lib/alloy/data + - --server.http.listen-addr=0.0.0.0:12345 + environment: + ALLOY_NODE: ${ALLOY_NODE:?set ALLOY_NODE (prod|beta|dev)} + LOKI_URL: ${LOKI_URL:?set LOKI_URL} + volumes: + - ./config.alloy:/etc/alloy/config.alloy:ro + # Read-only Docker socket: discover + tail every container's stdout/stderr. + - /var/run/docker.sock:/var/run/docker.sock:ro + # Caddy's host access logs (runs on the host, not a container). + - /var/log/caddy:/var/log/caddy:ro + # The app's structured JSON logs, straight off its named volume. + - thermograph_applogs:/applogs:ro + - alloy_data:/var/lib/alloy/data + restart: unless-stopped + +volumes: + # The app's existing log volume (created by the app's own stack). external:true + # means compose references it, never creates or deletes it. + thermograph_applogs: + external: true + alloy_data: {} diff --git a/caddy-grafana.conf b/caddy-grafana.conf new file mode 100644 index 0000000..641bf5f --- /dev/null +++ b/caddy-grafana.conf @@ -0,0 +1,15 @@ +# Append to beta's /etc/caddy/Caddyfile (beta is the monitoring host), then +# `systemctl reload caddy`. Grafana publishes to 127.0.0.1:3000; this fronts it +# with TLS, same pattern as beta's other site blocks. Grafana owns its own login, +# so the UI is safe to expose publicly — but Loki (the log store, on +# 10.10.0.2:3100) is NOT proxied here and stays mesh-only. +# +# Point grafana.thermograph.org's A record at beta (75.119.132.91) first. + +grafana.thermograph.org { + encode zstd gzip + reverse_proxy 127.0.0.1:3000 + log { + output file /var/log/caddy/grafana.log + } +} diff --git a/docker-compose.yml b/docker-compose.yml new file mode 100644 index 0000000..57a6d41 --- /dev/null +++ b/docker-compose.yml @@ -0,0 +1,51 @@ +# The central observability stack: Loki (log store) + Grafana (UI). Runs on ONE +# node — the monitoring host, `beta` (75.119.132.91 / mesh 10.10.0.2) — because +# beta is an always-on VPS with a public Caddy already fronting it. Every node's +# Alloy agent (see alloy/) ships logs here; agents on prod and the desktop reach +# Loki over the WireGuard mesh, so Loki must listen on the mesh interface. +# +# Deploy (on beta): docker compose up -d +# Grafana is proxied by beta's Caddy at grafana.thermograph.org (see README); +# Loki is NOT public — it binds the mesh IP only, reachable to agents over wg0. + +services: + loki: + image: grafana/loki:3.5.1 + command: -config.file=/etc/loki/config.yml + volumes: + - ./loki/config.yml:/etc/loki/config.yml:ro + - loki_data:/loki + ports: + # Mesh-only: agents on prod (10.10.0.1) and desktop (10.10.0.3) push here + # over wg0. Never published on the public interface. + - "10.10.0.2:3100:3100" + # Also localhost, so the beta-local Alloy agent and curl checks can reach it. + - "127.0.0.1:3100:3100" + restart: unless-stopped + + grafana: + image: grafana/grafana:11.6.1 + depends_on: [loki] + environment: + # Grafana owns its own auth. Admin password is injected from the host env + # (set GF_SECURITY_ADMIN_PASSWORD before `up`, or in an .env file — see + # .env.example); never bake a credential into the compose file. + GF_SECURITY_ADMIN_USER: ${GF_ADMIN_USER:-admin} + GF_SECURITY_ADMIN_PASSWORD: ${GF_SECURITY_ADMIN_PASSWORD:?set GF_SECURITY_ADMIN_PASSWORD} + GF_USERS_ALLOW_SIGN_UP: "false" + GF_ANALYTICS_REPORTING_ENABLED: "false" + GF_ANALYTICS_CHECK_FOR_UPDATES: "false" + # Served behind Caddy at this external URL (sub-path-safe cookie/redirects). + GF_SERVER_ROOT_URL: "https://${GRAFANA_DOMAIN:-grafana.thermograph.org}/" + volumes: + - ./grafana/provisioning:/etc/grafana/provisioning:ro + - ./grafana/dashboards:/var/lib/grafana/dashboards:ro + - grafana_data:/var/lib/grafana + ports: + # Host-local; beta's Caddy reverse-proxies to it. Not public directly. + - "127.0.0.1:3000:3000" + restart: unless-stopped + +volumes: + loki_data: {} + grafana_data: {} diff --git a/grafana/dashboards/thermograph-logs.json b/grafana/dashboards/thermograph-logs.json new file mode 100644 index 0000000..8587568 --- /dev/null +++ b/grafana/dashboards/thermograph-logs.json @@ -0,0 +1,132 @@ +{ + "annotations": { "list": [] }, + "editable": true, + "graphTooltip": 1, + "title": "Thermograph — Fleet Logs", + "uid": "thermograph-logs", + "tags": ["thermograph"], + "timezone": "", + "schemaVersion": 39, + "refresh": "30s", + "time": { "from": "now-6h", "to": "now" }, + "templating": { + "list": [ + { + "name": "host", + "type": "custom", + "label": "Node", + "query": "prod,beta,dev", + "current": { "text": "All", "value": "$__all" }, + "includeAll": true, + "allValue": ".+", + "multi": true + } + ] + }, + "panels": [ + { + "type": "timeseries", + "title": "Log volume by service", + "description": "Lines/sec from every container, split by compose service (app, db, forgejo…).", + "gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 }, + "datasource": { "type": "loki", "uid": "loki" }, + "fieldConfig": { "defaults": { "custom": { "drawStyle": "line", "fillOpacity": 15, "stacking": { "mode": "normal" } } } }, + "targets": [ + { + "datasource": { "type": "loki", "uid": "loki" }, + "expr": "sum by (service) (count_over_time({job=\"docker\", host=~\"$host\"} [$__auto]))", + "legendFormat": "{{service}}" + } + ] + }, + { + "type": "timeseries", + "title": "App errors/sec (structured logs)", + "description": "count_over_time of the app's errors-*.jsonl lines, by tag.", + "gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 }, + "datasource": { "type": "loki", "uid": "loki" }, + "fieldConfig": { "defaults": { "custom": { "drawStyle": "bars", "fillOpacity": 60 }, "color": { "mode": "palette-classic" } } }, + "targets": [ + { + "datasource": { "type": "loki", "uid": "loki" }, + "expr": "sum by (tag) (count_over_time({job=\"app-json\", source=\"app\", host=~\"$host\"} | level=~\"error|warn\" [$__auto]))", + "legendFormat": "{{tag}}" + } + ] + }, + { + "type": "stat", + "title": "Upstream rate-limits (429) — last 1h", + "description": "Open-Meteo / upstream 429s pulled from the app's error logs.", + "gridPos": { "h": 4, "w": 6, "x": 0, "y": 8 }, + "datasource": { "type": "loki", "uid": "loki" }, + "fieldConfig": { "defaults": { "color": { "mode": "thresholds" }, "thresholds": { "steps": [ { "color": "green", "value": null }, { "color": "yellow", "value": 1 }, { "color": "red", "value": 50 } ] } } }, + "options": { "reduceOptions": { "calcs": ["sum"] }, "colorMode": "background" }, + "targets": [ + { + "datasource": { "type": "loki", "uid": "loki" }, + "expr": "sum (count_over_time({job=\"app-json\", host=~\"$host\"} | status=\"429\" [1h]))", + "instant": true + } + ] + }, + { + "type": "stat", + "title": "Caddy 5xx — last 1h", + "description": "Server errors seen by the reverse proxy across all sites.", + "gridPos": { "h": 4, "w": 6, "x": 6, "y": 8 }, + "datasource": { "type": "loki", "uid": "loki" }, + "fieldConfig": { "defaults": { "color": { "mode": "thresholds" }, "thresholds": { "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 1 } ] } } }, + "options": { "reduceOptions": { "calcs": ["sum"] }, "colorMode": "background" }, + "targets": [ + { + "datasource": { "type": "loki", "uid": "loki" }, + "expr": "sum (count_over_time({job=\"caddy\", host=~\"$host\"} | json | status>=500 [1h]))", + "instant": true + } + ] + }, + { + "type": "logs", + "title": "Subscription notifier", + "description": "Everything the app logs mentioning the notifier — its liveness at a glance.", + "gridPos": { "h": 4, "w": 12, "x": 12, "y": 8 }, + "datasource": { "type": "loki", "uid": "loki" }, + "options": { "showTime": true, "wrapLogMessage": true, "dedupStrategy": "exact", "sortOrder": "Descending" }, + "targets": [ + { + "datasource": { "type": "loki", "uid": "loki" }, + "expr": "{job=\"docker\", service=\"app\", host=~\"$host\"} |~ \"(?i)notifier\"" + } + ] + }, + { + "type": "logs", + "title": "Recent app errors", + "description": "The app's errors-*.jsonl, newest first.", + "gridPos": { "h": 9, "w": 12, "x": 0, "y": 12 }, + "datasource": { "type": "loki", "uid": "loki" }, + "options": { "showTime": true, "wrapLogMessage": true, "sortOrder": "Descending", "enableLogDetails": true }, + "targets": [ + { + "datasource": { "type": "loki", "uid": "loki" }, + "expr": "{job=\"app-json\", host=~\"$host\"} | level=~\"error|warn\"" + } + ] + }, + { + "type": "logs", + "title": "Live tail — all containers", + "description": "Raw stdout/stderr from every container on the selected node(s).", + "gridPos": { "h": 9, "w": 12, "x": 12, "y": 12 }, + "datasource": { "type": "loki", "uid": "loki" }, + "options": { "showTime": true, "wrapLogMessage": false, "sortOrder": "Descending", "enableLogDetails": true }, + "targets": [ + { + "datasource": { "type": "loki", "uid": "loki" }, + "expr": "{job=\"docker\", host=~\"$host\"}" + } + ] + } + ] +} diff --git a/grafana/provisioning/dashboards/provider.yml b/grafana/provisioning/dashboards/provider.yml new file mode 100644 index 0000000..fdabaad --- /dev/null +++ b/grafana/provisioning/dashboards/provider.yml @@ -0,0 +1,16 @@ +# Load any dashboard JSON dropped in /var/lib/grafana/dashboards (mounted from +# ./grafana/dashboards) at startup, so the fleet overview is version-controlled +# in this repo rather than hand-built in the UI. +apiVersion: 1 + +providers: + - name: thermograph + orgId: 1 + folder: Thermograph + type: file + disableDeletion: false + editable: true + updateIntervalSeconds: 30 + options: + path: /var/lib/grafana/dashboards + foldersFromFilesStructure: false diff --git a/grafana/provisioning/datasources/loki.yml b/grafana/provisioning/datasources/loki.yml new file mode 100644 index 0000000..c2029f9 --- /dev/null +++ b/grafana/provisioning/datasources/loki.yml @@ -0,0 +1,13 @@ +# Auto-provision the Loki datasource so Grafana comes up already wired to it — +# no click-ops. Loki runs in the same compose stack, reachable by service name. +apiVersion: 1 + +datasources: + - name: Loki + type: loki + access: proxy + url: http://loki:3100 + uid: loki # stable UID so the provisioned dashboards can bind to it + isDefault: true + jsonData: + maxLines: 2000 diff --git a/loki/config.yml b/loki/config.yml new file mode 100644 index 0000000..d17a37f --- /dev/null +++ b/loki/config.yml @@ -0,0 +1,51 @@ +# Loki — the single log store for the whole Thermograph fleet (prod, beta, LAN +# dev). Every node's Alloy agent pushes here over the WireGuard mesh; Grafana +# reads from here. Single-binary, filesystem-backed: no object storage, no +# clustering — right for a three-node hobby fleet, and everything lives on one +# volume you can back up or blow away. +auth_enabled: false + +server: + http_listen_port: 3100 + grpc_listen_port: 9096 + log_level: warn + +common: + instance_addr: 127.0.0.1 + path_prefix: /loki + storage: + filesystem: + chunks_directory: /loki/chunks + rules_directory: /loki/rules + replication_factor: 1 + ring: + kvstore: + store: inmemory + +schema_config: + configs: + - from: 2024-01-01 + store: tsdb + object_store: filesystem + schema: v13 + index: + prefix: index_ + period: 24h + +limits_config: + # A hobby fleet's volume is tiny; keep 30 days and cap ingestion generously. + retention_period: 720h + reject_old_samples: true + reject_old_samples_max_age: 168h + max_query_series: 5000 + allow_structured_metadata: true + volume_enabled: true + +compactor: + working_directory: /loki/compactor + retention_enabled: true + delete_request_store: filesystem + +# No analytics phone-home from a private box. +analytics: + reporting_enabled: false