Fleet log aggregation: Grafana + Loki + Alloy over the WireGuard mesh
A central Grafana + Loki stack (on beta) fed by a Grafana Alloy agent on every node, replacing the old SSH-tailed single-host scripts/dashboard.py. - docker-compose.yml: central Loki (mesh-only :3100) + Grafana (Caddy-fronted) - loki/config.yml: single-binary, filesystem storage, 30-day retention - alloy/config.alloy + docker-compose.agent.yml: per-node collector — every container's stdout/stderr via the Docker socket, Caddy host logs, and the app's structured JSON logs (errors/access/audit), each line tagged by node - grafana/: auto-provisioned Loki datasource + a fleet-logs dashboard (volume by service, error rate, upstream 429s, Caddy 5xx, notifier liveness, live tail), with a per-node selector - caddy-grafana.conf, README, .env.example
This commit is contained in:
commit
46edb0bca0
11 changed files with 526 additions and 0 deletions
9
.env.example
Normal file
9
.env.example
Normal file
|
|
@ -0,0 +1,9 @@
|
||||||
|
# Copy to .env on the monitoring host (beta) before `docker compose up`.
|
||||||
|
# .env is gitignored — never commit real secrets.
|
||||||
|
|
||||||
|
# Grafana admin login (the UI's own auth; Grafana is the only thing exposed).
|
||||||
|
GF_ADMIN_USER=admin
|
||||||
|
GF_SECURITY_ADMIN_PASSWORD=change-me-to-something-strong
|
||||||
|
|
||||||
|
# The public hostname beta's Caddy serves Grafana at (see caddy-grafana.conf).
|
||||||
|
GRAFANA_DOMAIN=grafana.thermograph.org
|
||||||
3
.gitignore
vendored
Normal file
3
.gitignore
vendored
Normal file
|
|
@ -0,0 +1,3 @@
|
||||||
|
.env
|
||||||
|
alloy/.env
|
||||||
|
*.log
|
||||||
92
README.md
Normal file
92
README.md
Normal file
|
|
@ -0,0 +1,92 @@
|
||||||
|
# Thermograph — observability
|
||||||
|
|
||||||
|
Fleet-wide log aggregation for [Thermograph](https://thermograph.org): a central
|
||||||
|
**Grafana + Loki** stack, fed by a **Grafana Alloy** agent on every node, all
|
||||||
|
over the private WireGuard mesh. This replaces the old in-repo
|
||||||
|
`scripts/dashboard.py` (an SSH-tailed, single-host, metrics-endpoint view) with a
|
||||||
|
persistent, queryable, fleet-wide log store and UI.
|
||||||
|
|
||||||
|
```
|
||||||
|
prod (10.10.0.1) beta (10.10.0.2) desktop / LAN dev (10.10.0.3)
|
||||||
|
┌────────────┐ ┌──────────────────┐ ┌────────────┐
|
||||||
|
│ Alloy agent│──wg0──┐ │ Loki ◀── Alloy │ ┌──wg0│ Alloy agent│
|
||||||
|
└────────────┘ └───▶│ Grafana (Caddy) │◀──┘ └────────────┘
|
||||||
|
docker+caddy+app └──────────────────┘ docker+caddy+app
|
||||||
|
logs central store + UI logs
|
||||||
|
```
|
||||||
|
|
||||||
|
- **Loki** stores logs (filesystem, 30-day retention). Listens on the mesh IP
|
||||||
|
`10.10.0.2:3100` — never public.
|
||||||
|
- **Grafana** is the UI, fronted by beta's Caddy at `grafana.thermograph.org`
|
||||||
|
(its own login; Loki is not exposed).
|
||||||
|
- **Alloy** runs on each node and ships three sources to Loki: every Docker
|
||||||
|
container's stdout/stderr, Caddy's host access logs, and the app's structured
|
||||||
|
JSON logs (`errors`/`access`/`audit` `*.jsonl`, parsed so `level`/`tag`/`phase`
|
||||||
|
become labels). Every line is tagged with its node (`host = prod|beta|dev`).
|
||||||
|
|
||||||
|
## Layout
|
||||||
|
|
||||||
|
| Path | What |
|
||||||
|
|------|------|
|
||||||
|
| `docker-compose.yml` | the central Loki + Grafana stack (runs on beta) |
|
||||||
|
| `loki/config.yml` | Loki config (filesystem, retention) |
|
||||||
|
| `grafana/provisioning/` | auto-wired Loki datasource + dashboard provider |
|
||||||
|
| `grafana/dashboards/thermograph-logs.json` | the fleet-logs dashboard |
|
||||||
|
| `alloy/config.alloy` | the per-node collector config |
|
||||||
|
| `alloy/docker-compose.agent.yml` | runs the Alloy agent on a node |
|
||||||
|
| `caddy-grafana.conf` | beta Caddy site block for the Grafana UI |
|
||||||
|
|
||||||
|
## Deploy
|
||||||
|
|
||||||
|
### 1. Central stack (on beta)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# on beta (75.119.132.91):
|
||||||
|
git clone <this repo> thermograph-observability && cd thermograph-observability
|
||||||
|
cp .env.example .env # set GF_SECURITY_ADMIN_PASSWORD
|
||||||
|
docker compose up -d # Loki on 10.10.0.2:3100 + 127.0.0.1:3100; Grafana on 127.0.0.1:3000
|
||||||
|
```
|
||||||
|
|
||||||
|
Expose the Grafana UI: point `grafana.thermograph.org` at beta, append
|
||||||
|
`caddy-grafana.conf` to `/etc/caddy/Caddyfile`, `systemctl reload caddy`.
|
||||||
|
|
||||||
|
### 2. Alloy agent (on every node — prod, beta, desktop)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd thermograph-observability/alloy
|
||||||
|
# beta pushes to local Loki; prod & desktop push over the mesh:
|
||||||
|
# beta: LOKI_URL=http://127.0.0.1:3100/loki/api/v1/push ALLOY_NODE=beta
|
||||||
|
# prod: LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push ALLOY_NODE=prod
|
||||||
|
# desktop: LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push ALLOY_NODE=dev
|
||||||
|
ALLOY_NODE=prod LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push \
|
||||||
|
docker compose -f docker-compose.agent.yml up -d
|
||||||
|
```
|
||||||
|
|
||||||
|
On the LAN dev node the app's log volume is `thermograph-dev_applogs` (not
|
||||||
|
`thermograph_applogs`) — edit the two references in `docker-compose.agent.yml`
|
||||||
|
there. The mesh must already be up (see the main repo's `deploy/swarm/` /
|
||||||
|
`INFRA.md`); Swarm is **not** required — the agents talk plain HTTP over wg0.
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# From any mesh node, confirm Loki is receiving from every host:
|
||||||
|
curl -s 'http://10.10.0.2:3100/loki/api/v1/label/host/values' # -> {"data":["beta","dev","prod"]}
|
||||||
|
# Then open Grafana → the "Thermograph — Fleet Logs" dashboard.
|
||||||
|
```
|
||||||
|
|
||||||
|
## The dashboard
|
||||||
|
|
||||||
|
`Thermograph — Fleet Logs` (auto-provisioned), with a `Node` selector:
|
||||||
|
log volume by service, app error rate by tag, upstream 429 count, Caddy 5xx
|
||||||
|
count, a notifier-liveness log panel, recent app errors, and a live all-container
|
||||||
|
tail. Edit it in Grafana and re-export the JSON here to version a change.
|
||||||
|
|
||||||
|
## Notes / follow-ups
|
||||||
|
|
||||||
|
- **Loki is mesh-only** by design; only Grafana (with auth) is public.
|
||||||
|
- Alloy reads the Docker socket read-only to discover containers.
|
||||||
|
- Retention is 30 days on beta's disk — bump `loki/config.yml` if you want more.
|
||||||
|
- Possible next step: have the app emit a periodic `notifier heartbeat` log line
|
||||||
|
so notifier liveness is a first-class log-derived signal, not just container
|
||||||
|
chatter.
|
||||||
96
alloy/config.alloy
Normal file
96
alloy/config.alloy
Normal file
|
|
@ -0,0 +1,96 @@
|
||||||
|
// Grafana Alloy — the log collector that runs on EVERY node (prod, beta, LAN
|
||||||
|
// dev). It gathers three sources and ships them to the central Loki on beta:
|
||||||
|
//
|
||||||
|
// 1. Every Docker container's stdout/stderr (app, db, and on beta forgejo)
|
||||||
|
// via the Docker socket.
|
||||||
|
// 2. Caddy's host access logs (/var/log/caddy/*.log) — the reverse proxy runs
|
||||||
|
// on the host, not in a container, so its logs aren't in Docker.
|
||||||
|
// 3. The app's structured JSON logs (errors/access/audit *.jsonl) from the
|
||||||
|
// `applogs` Docker volume, parsed so `level`/`tag`/`phase` become labels.
|
||||||
|
//
|
||||||
|
// Per-node settings come from the environment (see docker-compose.agent.yml):
|
||||||
|
// ALLOY_NODE — this node's name label (prod | beta | dev)
|
||||||
|
// LOKI_URL — where to push (http://10.10.0.2:3100/loki/api/v1/push over wg0)
|
||||||
|
|
||||||
|
livedebugging { enabled = false }
|
||||||
|
|
||||||
|
// --- 1. All Docker container logs ------------------------------------------------
|
||||||
|
discovery.docker "containers" {
|
||||||
|
host = "unix:///var/run/docker.sock"
|
||||||
|
}
|
||||||
|
|
||||||
|
// Turn Docker metadata into tidy labels: `container` (short name) and `service`
|
||||||
|
// (the compose service, e.g. app/db). Drop Alloy's own container to avoid a loop.
|
||||||
|
discovery.relabel "containers" {
|
||||||
|
targets = discovery.docker.containers.targets
|
||||||
|
|
||||||
|
rule {
|
||||||
|
source_labels = ["__meta_docker_container_name"]
|
||||||
|
regex = "/(.*)"
|
||||||
|
target_label = "container"
|
||||||
|
}
|
||||||
|
rule {
|
||||||
|
source_labels = ["__meta_docker_container_label_com_docker_compose_service"]
|
||||||
|
target_label = "service"
|
||||||
|
}
|
||||||
|
rule {
|
||||||
|
source_labels = ["container"]
|
||||||
|
regex = ".*alloy.*"
|
||||||
|
action = "drop"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
loki.source.docker "containers" {
|
||||||
|
host = "unix:///var/run/docker.sock"
|
||||||
|
targets = discovery.relabel.containers.output
|
||||||
|
forward_to = [loki.write.central.receiver]
|
||||||
|
relabel_rules = discovery.relabel.containers.rules
|
||||||
|
labels = { job = "docker" }
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- 2. Caddy host access logs ---------------------------------------------------
|
||||||
|
local.file_match "caddy" {
|
||||||
|
path_targets = [{ __path__ = "/var/log/caddy/*.log", job = "caddy" }]
|
||||||
|
}
|
||||||
|
|
||||||
|
loki.source.file "caddy" {
|
||||||
|
targets = local.file_match.caddy.targets
|
||||||
|
forward_to = [loki.write.central.receiver]
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- 3. App structured JSON logs (errors / access / audit) -----------------------
|
||||||
|
// Mounted read-only from the app's `applogs` volume at /applogs (see the agent
|
||||||
|
// compose). Lift `level`/`tag`/`phase` out of the JSON so they're queryable.
|
||||||
|
local.file_match "app_jsonl" {
|
||||||
|
path_targets = [{ __path__ = "/applogs/**/*.jsonl", job = "app-json" }]
|
||||||
|
}
|
||||||
|
|
||||||
|
loki.source.file "app_jsonl" {
|
||||||
|
targets = local.file_match.app_jsonl.targets
|
||||||
|
forward_to = [loki.process.app_jsonl.receiver]
|
||||||
|
}
|
||||||
|
|
||||||
|
loki.process "app_jsonl" {
|
||||||
|
forward_to = [loki.write.central.receiver]
|
||||||
|
|
||||||
|
stage.json {
|
||||||
|
expressions = { level = "level", tag = "tag", phase = "phase", status = "status" }
|
||||||
|
}
|
||||||
|
// A record with no explicit level: an error-folder line is an error, else info.
|
||||||
|
stage.static_labels {
|
||||||
|
values = { source = "app" }
|
||||||
|
}
|
||||||
|
stage.labels {
|
||||||
|
values = { level = "", tag = "", phase = "" }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- Ship to the central Loki over the WireGuard mesh ----------------------------
|
||||||
|
loki.write "central" {
|
||||||
|
endpoint {
|
||||||
|
url = sys.env("LOKI_URL")
|
||||||
|
}
|
||||||
|
// Every line from this node is stamped with its node name, so one Grafana
|
||||||
|
// view can slice prod vs beta vs dev.
|
||||||
|
external_labels = { host = sys.env("ALLOY_NODE") }
|
||||||
|
}
|
||||||
48
alloy/docker-compose.agent.yml
Normal file
48
alloy/docker-compose.agent.yml
Normal file
|
|
@ -0,0 +1,48 @@
|
||||||
|
# The Alloy log-collector agent. Runs on EACH node (prod, beta, LAN dev) — one
|
||||||
|
# per node — reading that node's Docker containers, Caddy logs, and the app's
|
||||||
|
# JSON logs, and shipping them to the central Loki on beta over the mesh.
|
||||||
|
#
|
||||||
|
# Per-node config comes from the environment (an .env file next to this, or
|
||||||
|
# exported before `up`):
|
||||||
|
# ALLOY_NODE this node's label: prod | beta | dev
|
||||||
|
# LOKI_URL push endpoint. On beta: http://127.0.0.1:3100/loki/api/v1/push
|
||||||
|
# Elsewhere (over wg0): http://10.10.0.2:3100/loki/api/v1/push
|
||||||
|
#
|
||||||
|
# Deploy on a node:
|
||||||
|
# ALLOY_NODE=prod LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push \
|
||||||
|
# docker compose -f docker-compose.agent.yml up -d
|
||||||
|
#
|
||||||
|
# The app's JSON logs are read from its `applogs` Docker volume, named
|
||||||
|
# `<project>_applogs`. prod and beta run the app under the compose project
|
||||||
|
# `thermograph`, so the external volume below is `thermograph_applogs`. The LAN
|
||||||
|
# dev stack uses the project `thermograph-dev` — on that node, change the two
|
||||||
|
# `thermograph_applogs` references below to `thermograph-dev_applogs`.
|
||||||
|
|
||||||
|
services:
|
||||||
|
alloy:
|
||||||
|
image: grafana/alloy:v1.9.1
|
||||||
|
command:
|
||||||
|
- run
|
||||||
|
- /etc/alloy/config.alloy
|
||||||
|
- --storage.path=/var/lib/alloy/data
|
||||||
|
- --server.http.listen-addr=0.0.0.0:12345
|
||||||
|
environment:
|
||||||
|
ALLOY_NODE: ${ALLOY_NODE:?set ALLOY_NODE (prod|beta|dev)}
|
||||||
|
LOKI_URL: ${LOKI_URL:?set LOKI_URL}
|
||||||
|
volumes:
|
||||||
|
- ./config.alloy:/etc/alloy/config.alloy:ro
|
||||||
|
# Read-only Docker socket: discover + tail every container's stdout/stderr.
|
||||||
|
- /var/run/docker.sock:/var/run/docker.sock:ro
|
||||||
|
# Caddy's host access logs (runs on the host, not a container).
|
||||||
|
- /var/log/caddy:/var/log/caddy:ro
|
||||||
|
# The app's structured JSON logs, straight off its named volume.
|
||||||
|
- thermograph_applogs:/applogs:ro
|
||||||
|
- alloy_data:/var/lib/alloy/data
|
||||||
|
restart: unless-stopped
|
||||||
|
|
||||||
|
volumes:
|
||||||
|
# The app's existing log volume (created by the app's own stack). external:true
|
||||||
|
# means compose references it, never creates or deletes it.
|
||||||
|
thermograph_applogs:
|
||||||
|
external: true
|
||||||
|
alloy_data: {}
|
||||||
15
caddy-grafana.conf
Normal file
15
caddy-grafana.conf
Normal file
|
|
@ -0,0 +1,15 @@
|
||||||
|
# Append to beta's /etc/caddy/Caddyfile (beta is the monitoring host), then
|
||||||
|
# `systemctl reload caddy`. Grafana publishes to 127.0.0.1:3000; this fronts it
|
||||||
|
# with TLS, same pattern as beta's other site blocks. Grafana owns its own login,
|
||||||
|
# so the UI is safe to expose publicly — but Loki (the log store, on
|
||||||
|
# 10.10.0.2:3100) is NOT proxied here and stays mesh-only.
|
||||||
|
#
|
||||||
|
# Point grafana.thermograph.org's A record at beta (75.119.132.91) first.
|
||||||
|
|
||||||
|
grafana.thermograph.org {
|
||||||
|
encode zstd gzip
|
||||||
|
reverse_proxy 127.0.0.1:3000
|
||||||
|
log {
|
||||||
|
output file /var/log/caddy/grafana.log
|
||||||
|
}
|
||||||
|
}
|
||||||
51
docker-compose.yml
Normal file
51
docker-compose.yml
Normal file
|
|
@ -0,0 +1,51 @@
|
||||||
|
# The central observability stack: Loki (log store) + Grafana (UI). Runs on ONE
|
||||||
|
# node — the monitoring host, `beta` (75.119.132.91 / mesh 10.10.0.2) — because
|
||||||
|
# beta is an always-on VPS with a public Caddy already fronting it. Every node's
|
||||||
|
# Alloy agent (see alloy/) ships logs here; agents on prod and the desktop reach
|
||||||
|
# Loki over the WireGuard mesh, so Loki must listen on the mesh interface.
|
||||||
|
#
|
||||||
|
# Deploy (on beta): docker compose up -d
|
||||||
|
# Grafana is proxied by beta's Caddy at grafana.thermograph.org (see README);
|
||||||
|
# Loki is NOT public — it binds the mesh IP only, reachable to agents over wg0.
|
||||||
|
|
||||||
|
services:
|
||||||
|
loki:
|
||||||
|
image: grafana/loki:3.5.1
|
||||||
|
command: -config.file=/etc/loki/config.yml
|
||||||
|
volumes:
|
||||||
|
- ./loki/config.yml:/etc/loki/config.yml:ro
|
||||||
|
- loki_data:/loki
|
||||||
|
ports:
|
||||||
|
# Mesh-only: agents on prod (10.10.0.1) and desktop (10.10.0.3) push here
|
||||||
|
# over wg0. Never published on the public interface.
|
||||||
|
- "10.10.0.2:3100:3100"
|
||||||
|
# Also localhost, so the beta-local Alloy agent and curl checks can reach it.
|
||||||
|
- "127.0.0.1:3100:3100"
|
||||||
|
restart: unless-stopped
|
||||||
|
|
||||||
|
grafana:
|
||||||
|
image: grafana/grafana:11.6.1
|
||||||
|
depends_on: [loki]
|
||||||
|
environment:
|
||||||
|
# Grafana owns its own auth. Admin password is injected from the host env
|
||||||
|
# (set GF_SECURITY_ADMIN_PASSWORD before `up`, or in an .env file — see
|
||||||
|
# .env.example); never bake a credential into the compose file.
|
||||||
|
GF_SECURITY_ADMIN_USER: ${GF_ADMIN_USER:-admin}
|
||||||
|
GF_SECURITY_ADMIN_PASSWORD: ${GF_SECURITY_ADMIN_PASSWORD:?set GF_SECURITY_ADMIN_PASSWORD}
|
||||||
|
GF_USERS_ALLOW_SIGN_UP: "false"
|
||||||
|
GF_ANALYTICS_REPORTING_ENABLED: "false"
|
||||||
|
GF_ANALYTICS_CHECK_FOR_UPDATES: "false"
|
||||||
|
# Served behind Caddy at this external URL (sub-path-safe cookie/redirects).
|
||||||
|
GF_SERVER_ROOT_URL: "https://${GRAFANA_DOMAIN:-grafana.thermograph.org}/"
|
||||||
|
volumes:
|
||||||
|
- ./grafana/provisioning:/etc/grafana/provisioning:ro
|
||||||
|
- ./grafana/dashboards:/var/lib/grafana/dashboards:ro
|
||||||
|
- grafana_data:/var/lib/grafana
|
||||||
|
ports:
|
||||||
|
# Host-local; beta's Caddy reverse-proxies to it. Not public directly.
|
||||||
|
- "127.0.0.1:3000:3000"
|
||||||
|
restart: unless-stopped
|
||||||
|
|
||||||
|
volumes:
|
||||||
|
loki_data: {}
|
||||||
|
grafana_data: {}
|
||||||
132
grafana/dashboards/thermograph-logs.json
Normal file
132
grafana/dashboards/thermograph-logs.json
Normal file
|
|
@ -0,0 +1,132 @@
|
||||||
|
{
|
||||||
|
"annotations": { "list": [] },
|
||||||
|
"editable": true,
|
||||||
|
"graphTooltip": 1,
|
||||||
|
"title": "Thermograph — Fleet Logs",
|
||||||
|
"uid": "thermograph-logs",
|
||||||
|
"tags": ["thermograph"],
|
||||||
|
"timezone": "",
|
||||||
|
"schemaVersion": 39,
|
||||||
|
"refresh": "30s",
|
||||||
|
"time": { "from": "now-6h", "to": "now" },
|
||||||
|
"templating": {
|
||||||
|
"list": [
|
||||||
|
{
|
||||||
|
"name": "host",
|
||||||
|
"type": "custom",
|
||||||
|
"label": "Node",
|
||||||
|
"query": "prod,beta,dev",
|
||||||
|
"current": { "text": "All", "value": "$__all" },
|
||||||
|
"includeAll": true,
|
||||||
|
"allValue": ".+",
|
||||||
|
"multi": true
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"panels": [
|
||||||
|
{
|
||||||
|
"type": "timeseries",
|
||||||
|
"title": "Log volume by service",
|
||||||
|
"description": "Lines/sec from every container, split by compose service (app, db, forgejo…).",
|
||||||
|
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 },
|
||||||
|
"datasource": { "type": "loki", "uid": "loki" },
|
||||||
|
"fieldConfig": { "defaults": { "custom": { "drawStyle": "line", "fillOpacity": 15, "stacking": { "mode": "normal" } } } },
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"datasource": { "type": "loki", "uid": "loki" },
|
||||||
|
"expr": "sum by (service) (count_over_time({job=\"docker\", host=~\"$host\"} [$__auto]))",
|
||||||
|
"legendFormat": "{{service}}"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "timeseries",
|
||||||
|
"title": "App errors/sec (structured logs)",
|
||||||
|
"description": "count_over_time of the app's errors-*.jsonl lines, by tag.",
|
||||||
|
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 },
|
||||||
|
"datasource": { "type": "loki", "uid": "loki" },
|
||||||
|
"fieldConfig": { "defaults": { "custom": { "drawStyle": "bars", "fillOpacity": 60 }, "color": { "mode": "palette-classic" } } },
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"datasource": { "type": "loki", "uid": "loki" },
|
||||||
|
"expr": "sum by (tag) (count_over_time({job=\"app-json\", source=\"app\", host=~\"$host\"} | level=~\"error|warn\" [$__auto]))",
|
||||||
|
"legendFormat": "{{tag}}"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "stat",
|
||||||
|
"title": "Upstream rate-limits (429) — last 1h",
|
||||||
|
"description": "Open-Meteo / upstream 429s pulled from the app's error logs.",
|
||||||
|
"gridPos": { "h": 4, "w": 6, "x": 0, "y": 8 },
|
||||||
|
"datasource": { "type": "loki", "uid": "loki" },
|
||||||
|
"fieldConfig": { "defaults": { "color": { "mode": "thresholds" }, "thresholds": { "steps": [ { "color": "green", "value": null }, { "color": "yellow", "value": 1 }, { "color": "red", "value": 50 } ] } } },
|
||||||
|
"options": { "reduceOptions": { "calcs": ["sum"] }, "colorMode": "background" },
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"datasource": { "type": "loki", "uid": "loki" },
|
||||||
|
"expr": "sum (count_over_time({job=\"app-json\", host=~\"$host\"} | status=\"429\" [1h]))",
|
||||||
|
"instant": true
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "stat",
|
||||||
|
"title": "Caddy 5xx — last 1h",
|
||||||
|
"description": "Server errors seen by the reverse proxy across all sites.",
|
||||||
|
"gridPos": { "h": 4, "w": 6, "x": 6, "y": 8 },
|
||||||
|
"datasource": { "type": "loki", "uid": "loki" },
|
||||||
|
"fieldConfig": { "defaults": { "color": { "mode": "thresholds" }, "thresholds": { "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 1 } ] } } },
|
||||||
|
"options": { "reduceOptions": { "calcs": ["sum"] }, "colorMode": "background" },
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"datasource": { "type": "loki", "uid": "loki" },
|
||||||
|
"expr": "sum (count_over_time({job=\"caddy\", host=~\"$host\"} | json | status>=500 [1h]))",
|
||||||
|
"instant": true
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "logs",
|
||||||
|
"title": "Subscription notifier",
|
||||||
|
"description": "Everything the app logs mentioning the notifier — its liveness at a glance.",
|
||||||
|
"gridPos": { "h": 4, "w": 12, "x": 12, "y": 8 },
|
||||||
|
"datasource": { "type": "loki", "uid": "loki" },
|
||||||
|
"options": { "showTime": true, "wrapLogMessage": true, "dedupStrategy": "exact", "sortOrder": "Descending" },
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"datasource": { "type": "loki", "uid": "loki" },
|
||||||
|
"expr": "{job=\"docker\", service=\"app\", host=~\"$host\"} |~ \"(?i)notifier\""
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "logs",
|
||||||
|
"title": "Recent app errors",
|
||||||
|
"description": "The app's errors-*.jsonl, newest first.",
|
||||||
|
"gridPos": { "h": 9, "w": 12, "x": 0, "y": 12 },
|
||||||
|
"datasource": { "type": "loki", "uid": "loki" },
|
||||||
|
"options": { "showTime": true, "wrapLogMessage": true, "sortOrder": "Descending", "enableLogDetails": true },
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"datasource": { "type": "loki", "uid": "loki" },
|
||||||
|
"expr": "{job=\"app-json\", host=~\"$host\"} | level=~\"error|warn\""
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "logs",
|
||||||
|
"title": "Live tail — all containers",
|
||||||
|
"description": "Raw stdout/stderr from every container on the selected node(s).",
|
||||||
|
"gridPos": { "h": 9, "w": 12, "x": 12, "y": 12 },
|
||||||
|
"datasource": { "type": "loki", "uid": "loki" },
|
||||||
|
"options": { "showTime": true, "wrapLogMessage": false, "sortOrder": "Descending", "enableLogDetails": true },
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"datasource": { "type": "loki", "uid": "loki" },
|
||||||
|
"expr": "{job=\"docker\", host=~\"$host\"}"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
16
grafana/provisioning/dashboards/provider.yml
Normal file
16
grafana/provisioning/dashboards/provider.yml
Normal file
|
|
@ -0,0 +1,16 @@
|
||||||
|
# Load any dashboard JSON dropped in /var/lib/grafana/dashboards (mounted from
|
||||||
|
# ./grafana/dashboards) at startup, so the fleet overview is version-controlled
|
||||||
|
# in this repo rather than hand-built in the UI.
|
||||||
|
apiVersion: 1
|
||||||
|
|
||||||
|
providers:
|
||||||
|
- name: thermograph
|
||||||
|
orgId: 1
|
||||||
|
folder: Thermograph
|
||||||
|
type: file
|
||||||
|
disableDeletion: false
|
||||||
|
editable: true
|
||||||
|
updateIntervalSeconds: 30
|
||||||
|
options:
|
||||||
|
path: /var/lib/grafana/dashboards
|
||||||
|
foldersFromFilesStructure: false
|
||||||
13
grafana/provisioning/datasources/loki.yml
Normal file
13
grafana/provisioning/datasources/loki.yml
Normal file
|
|
@ -0,0 +1,13 @@
|
||||||
|
# Auto-provision the Loki datasource so Grafana comes up already wired to it —
|
||||||
|
# no click-ops. Loki runs in the same compose stack, reachable by service name.
|
||||||
|
apiVersion: 1
|
||||||
|
|
||||||
|
datasources:
|
||||||
|
- name: Loki
|
||||||
|
type: loki
|
||||||
|
access: proxy
|
||||||
|
url: http://loki:3100
|
||||||
|
uid: loki # stable UID so the provisioned dashboards can bind to it
|
||||||
|
isDefault: true
|
||||||
|
jsonData:
|
||||||
|
maxLines: 2000
|
||||||
51
loki/config.yml
Normal file
51
loki/config.yml
Normal file
|
|
@ -0,0 +1,51 @@
|
||||||
|
# Loki — the single log store for the whole Thermograph fleet (prod, beta, LAN
|
||||||
|
# dev). Every node's Alloy agent pushes here over the WireGuard mesh; Grafana
|
||||||
|
# reads from here. Single-binary, filesystem-backed: no object storage, no
|
||||||
|
# clustering — right for a three-node hobby fleet, and everything lives on one
|
||||||
|
# volume you can back up or blow away.
|
||||||
|
auth_enabled: false
|
||||||
|
|
||||||
|
server:
|
||||||
|
http_listen_port: 3100
|
||||||
|
grpc_listen_port: 9096
|
||||||
|
log_level: warn
|
||||||
|
|
||||||
|
common:
|
||||||
|
instance_addr: 127.0.0.1
|
||||||
|
path_prefix: /loki
|
||||||
|
storage:
|
||||||
|
filesystem:
|
||||||
|
chunks_directory: /loki/chunks
|
||||||
|
rules_directory: /loki/rules
|
||||||
|
replication_factor: 1
|
||||||
|
ring:
|
||||||
|
kvstore:
|
||||||
|
store: inmemory
|
||||||
|
|
||||||
|
schema_config:
|
||||||
|
configs:
|
||||||
|
- from: 2024-01-01
|
||||||
|
store: tsdb
|
||||||
|
object_store: filesystem
|
||||||
|
schema: v13
|
||||||
|
index:
|
||||||
|
prefix: index_
|
||||||
|
period: 24h
|
||||||
|
|
||||||
|
limits_config:
|
||||||
|
# A hobby fleet's volume is tiny; keep 30 days and cap ingestion generously.
|
||||||
|
retention_period: 720h
|
||||||
|
reject_old_samples: true
|
||||||
|
reject_old_samples_max_age: 168h
|
||||||
|
max_query_series: 5000
|
||||||
|
allow_structured_metadata: true
|
||||||
|
volume_enabled: true
|
||||||
|
|
||||||
|
compactor:
|
||||||
|
working_directory: /loki/compactor
|
||||||
|
retention_enabled: true
|
||||||
|
delete_request_store: filesystem
|
||||||
|
|
||||||
|
# No analytics phone-home from a private box.
|
||||||
|
analytics:
|
||||||
|
reporting_enabled: false
|
||||||
Loading…
Reference in a new issue