Fleet log aggregation: Grafana + Loki + Alloy over the WireGuard mesh
A central Grafana + Loki stack (on beta) fed by a Grafana Alloy agent on
every node, replacing the old SSH-tailed single-host scripts/dashboard.py.
- docker-compose.yml: central Loki (mesh-only :3100) + Grafana (Caddy-fronted)
- loki/config.yml: single-binary, filesystem storage, 30-day retention
- alloy/config.alloy + docker-compose.agent.yml: per-node collector — every
container's stdout/stderr via the Docker socket, Caddy host logs, and the
app's structured JSON logs (errors/access/audit), each line tagged by node
- grafana/: auto-provisioned Loki datasource + a fleet-logs dashboard
(volume by service, error rate, upstream 429s, Caddy 5xx, notifier liveness,
live tail), with a per-node selector
- caddy-grafana.conf, README, .env.example
2026-07-21 16:11:52 +00:00
|
|
|
# The Alloy log-collector agent. Runs on EACH node (prod, beta, LAN dev) — one
|
|
|
|
|
# per node — reading that node's Docker containers, Caddy logs, and the app's
|
|
|
|
|
# JSON logs, and shipping them to the central Loki on beta over the mesh.
|
|
|
|
|
#
|
|
|
|
|
# Per-node config comes from the environment (an .env file next to this, or
|
|
|
|
|
# exported before `up`):
|
|
|
|
|
# ALLOY_NODE this node's label: prod | beta | dev
|
2026-07-21 16:24:26 +00:00
|
|
|
# LOKI_URL push endpoint. All nodes push over the mesh:
|
|
|
|
|
# http://10.10.0.2:3100/loki/api/v1/push
|
Fleet log aggregation: Grafana + Loki + Alloy over the WireGuard mesh
A central Grafana + Loki stack (on beta) fed by a Grafana Alloy agent on
every node, replacing the old SSH-tailed single-host scripts/dashboard.py.
- docker-compose.yml: central Loki (mesh-only :3100) + Grafana (Caddy-fronted)
- loki/config.yml: single-binary, filesystem storage, 30-day retention
- alloy/config.alloy + docker-compose.agent.yml: per-node collector — every
container's stdout/stderr via the Docker socket, Caddy host logs, and the
app's structured JSON logs (errors/access/audit), each line tagged by node
- grafana/: auto-provisioned Loki datasource + a fleet-logs dashboard
(volume by service, error rate, upstream 429s, Caddy 5xx, notifier liveness,
live tail), with a per-node selector
- caddy-grafana.conf, README, .env.example
2026-07-21 16:11:52 +00:00
|
|
|
#
|
|
|
|
|
# Deploy on a node:
|
|
|
|
|
# ALLOY_NODE=prod LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push \
|
|
|
|
|
# docker compose -f docker-compose.agent.yml up -d
|
|
|
|
|
#
|
|
|
|
|
# The app's JSON logs are read from its `applogs` Docker volume, named
|
|
|
|
|
# `<project>_applogs`. prod and beta run the app under the compose project
|
|
|
|
|
# `thermograph`, so the external volume below is `thermograph_applogs`. The LAN
|
|
|
|
|
# dev stack uses the project `thermograph-dev` — on that node, change the two
|
|
|
|
|
# `thermograph_applogs` references below to `thermograph-dev_applogs`.
|
|
|
|
|
|
|
|
|
|
services:
|
|
|
|
|
alloy:
|
|
|
|
|
image: grafana/alloy:v1.9.1
|
|
|
|
|
command:
|
|
|
|
|
- run
|
|
|
|
|
- /etc/alloy/config.alloy
|
|
|
|
|
- --storage.path=/var/lib/alloy/data
|
|
|
|
|
- --server.http.listen-addr=0.0.0.0:12345
|
|
|
|
|
environment:
|
|
|
|
|
ALLOY_NODE: ${ALLOY_NODE:?set ALLOY_NODE (prod|beta|dev)}
|
|
|
|
|
LOKI_URL: ${LOKI_URL:?set LOKI_URL}
|
|
|
|
|
volumes:
|
|
|
|
|
- ./config.alloy:/etc/alloy/config.alloy:ro
|
|
|
|
|
# Read-only Docker socket: discover + tail every container's stdout/stderr.
|
|
|
|
|
- /var/run/docker.sock:/var/run/docker.sock:ro
|
|
|
|
|
# Caddy's host access logs (runs on the host, not a container).
|
|
|
|
|
- /var/log/caddy:/var/log/caddy:ro
|
|
|
|
|
# The app's structured JSON logs, straight off its named volume.
|
|
|
|
|
- thermograph_applogs:/applogs:ro
|
|
|
|
|
- alloy_data:/var/lib/alloy/data
|
|
|
|
|
restart: unless-stopped
|
|
|
|
|
|
|
|
|
|
volumes:
|
|
|
|
|
# The app's existing log volume (created by the app's own stack). external:true
|
|
|
|
|
# means compose references it, never creates or deletes it.
|
|
|
|
|
thermograph_applogs:
|
|
|
|
|
external: true
|
|
|
|
|
alloy_data: {}
|