From 46edb0bca0043ac91eab355a56342a5ad36579f8 Mon Sep 17 00:00:00 2001 From: Emi Griffith Date: Tue, 21 Jul 2026 09:11:52 -0700 Subject: [PATCH 1/8] Fleet log aggregation: Grafana + Loki + Alloy over the WireGuard mesh MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A central Grafana + Loki stack (on beta) fed by a Grafana Alloy agent on every node, replacing the old SSH-tailed single-host scripts/dashboard.py. - docker-compose.yml: central Loki (mesh-only :3100) + Grafana (Caddy-fronted) - loki/config.yml: single-binary, filesystem storage, 30-day retention - alloy/config.alloy + docker-compose.agent.yml: per-node collector — every container's stdout/stderr via the Docker socket, Caddy host logs, and the app's structured JSON logs (errors/access/audit), each line tagged by node - grafana/: auto-provisioned Loki datasource + a fleet-logs dashboard (volume by service, error rate, upstream 429s, Caddy 5xx, notifier liveness, live tail), with a per-node selector - caddy-grafana.conf, README, .env.example --- .env.example | 9 ++ .gitignore | 3 + README.md | 92 +++++++++++++ alloy/config.alloy | 96 ++++++++++++++ alloy/docker-compose.agent.yml | 48 +++++++ caddy-grafana.conf | 15 +++ docker-compose.yml | 51 +++++++ grafana/dashboards/thermograph-logs.json | 132 +++++++++++++++++++ grafana/provisioning/dashboards/provider.yml | 16 +++ grafana/provisioning/datasources/loki.yml | 13 ++ loki/config.yml | 51 +++++++ 11 files changed, 526 insertions(+) create mode 100644 .env.example create mode 100644 .gitignore create mode 100644 README.md create mode 100644 alloy/config.alloy create mode 100644 alloy/docker-compose.agent.yml create mode 100644 caddy-grafana.conf create mode 100644 docker-compose.yml create mode 100644 grafana/dashboards/thermograph-logs.json create mode 100644 grafana/provisioning/dashboards/provider.yml create mode 100644 grafana/provisioning/datasources/loki.yml create mode 100644 loki/config.yml diff --git a/.env.example b/.env.example new file mode 100644 index 0000000..7ff6852 --- /dev/null +++ b/.env.example @@ -0,0 +1,9 @@ +# Copy to .env on the monitoring host (beta) before `docker compose up`. +# .env is gitignored — never commit real secrets. + +# Grafana admin login (the UI's own auth; Grafana is the only thing exposed). +GF_ADMIN_USER=admin +GF_SECURITY_ADMIN_PASSWORD=change-me-to-something-strong + +# The public hostname beta's Caddy serves Grafana at (see caddy-grafana.conf). +GRAFANA_DOMAIN=grafana.thermograph.org diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..845a438 --- /dev/null +++ b/.gitignore @@ -0,0 +1,3 @@ +.env +alloy/.env +*.log diff --git a/README.md b/README.md new file mode 100644 index 0000000..cee80ec --- /dev/null +++ b/README.md @@ -0,0 +1,92 @@ +# Thermograph — observability + +Fleet-wide log aggregation for [Thermograph](https://thermograph.org): a central +**Grafana + Loki** stack, fed by a **Grafana Alloy** agent on every node, all +over the private WireGuard mesh. This replaces the old in-repo +`scripts/dashboard.py` (an SSH-tailed, single-host, metrics-endpoint view) with a +persistent, queryable, fleet-wide log store and UI. + +``` + prod (10.10.0.1) beta (10.10.0.2) desktop / LAN dev (10.10.0.3) + ┌────────────┐ ┌──────────────────┐ ┌────────────┐ + │ Alloy agent│──wg0──┐ │ Loki ◀── Alloy │ ┌──wg0│ Alloy agent│ + └────────────┘ └───▶│ Grafana (Caddy) │◀──┘ └────────────┘ + docker+caddy+app └──────────────────┘ docker+caddy+app + logs central store + UI logs +``` + +- **Loki** stores logs (filesystem, 30-day retention). Listens on the mesh IP + `10.10.0.2:3100` — never public. +- **Grafana** is the UI, fronted by beta's Caddy at `grafana.thermograph.org` + (its own login; Loki is not exposed). +- **Alloy** runs on each node and ships three sources to Loki: every Docker + container's stdout/stderr, Caddy's host access logs, and the app's structured + JSON logs (`errors`/`access`/`audit` `*.jsonl`, parsed so `level`/`tag`/`phase` + become labels). Every line is tagged with its node (`host = prod|beta|dev`). + +## Layout + +| Path | What | +|------|------| +| `docker-compose.yml` | the central Loki + Grafana stack (runs on beta) | +| `loki/config.yml` | Loki config (filesystem, retention) | +| `grafana/provisioning/` | auto-wired Loki datasource + dashboard provider | +| `grafana/dashboards/thermograph-logs.json` | the fleet-logs dashboard | +| `alloy/config.alloy` | the per-node collector config | +| `alloy/docker-compose.agent.yml` | runs the Alloy agent on a node | +| `caddy-grafana.conf` | beta Caddy site block for the Grafana UI | + +## Deploy + +### 1. Central stack (on beta) + +```bash +# on beta (75.119.132.91): +git clone thermograph-observability && cd thermograph-observability +cp .env.example .env # set GF_SECURITY_ADMIN_PASSWORD +docker compose up -d # Loki on 10.10.0.2:3100 + 127.0.0.1:3100; Grafana on 127.0.0.1:3000 +``` + +Expose the Grafana UI: point `grafana.thermograph.org` at beta, append +`caddy-grafana.conf` to `/etc/caddy/Caddyfile`, `systemctl reload caddy`. + +### 2. Alloy agent (on every node — prod, beta, desktop) + +```bash +cd thermograph-observability/alloy +# beta pushes to local Loki; prod & desktop push over the mesh: +# beta: LOKI_URL=http://127.0.0.1:3100/loki/api/v1/push ALLOY_NODE=beta +# prod: LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push ALLOY_NODE=prod +# desktop: LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push ALLOY_NODE=dev +ALLOY_NODE=prod LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push \ + docker compose -f docker-compose.agent.yml up -d +``` + +On the LAN dev node the app's log volume is `thermograph-dev_applogs` (not +`thermograph_applogs`) — edit the two references in `docker-compose.agent.yml` +there. The mesh must already be up (see the main repo's `deploy/swarm/` / +`INFRA.md`); Swarm is **not** required — the agents talk plain HTTP over wg0. + +## Verify + +```bash +# From any mesh node, confirm Loki is receiving from every host: +curl -s 'http://10.10.0.2:3100/loki/api/v1/label/host/values' # -> {"data":["beta","dev","prod"]} +# Then open Grafana → the "Thermograph — Fleet Logs" dashboard. +``` + +## The dashboard + +`Thermograph — Fleet Logs` (auto-provisioned), with a `Node` selector: +log volume by service, app error rate by tag, upstream 429 count, Caddy 5xx +count, a notifier-liveness log panel, recent app errors, and a live all-container +tail. Edit it in Grafana and re-export the JSON here to version a change. + +## Notes / follow-ups + +- **Loki is mesh-only** by design; only Grafana (with auth) is public. +- Alloy reads the Docker socket read-only to discover containers. +- Retention is 30 days on beta's disk — bump `loki/config.yml` if you want more. +- Possible next step: have the app emit a periodic `notifier heartbeat` log line + so notifier liveness is a first-class log-derived signal, not just container + chatter. diff --git a/alloy/config.alloy b/alloy/config.alloy new file mode 100644 index 0000000..7ddbc29 --- /dev/null +++ b/alloy/config.alloy @@ -0,0 +1,96 @@ +// Grafana Alloy — the log collector that runs on EVERY node (prod, beta, LAN +// dev). It gathers three sources and ships them to the central Loki on beta: +// +// 1. Every Docker container's stdout/stderr (app, db, and on beta forgejo) +// via the Docker socket. +// 2. Caddy's host access logs (/var/log/caddy/*.log) — the reverse proxy runs +// on the host, not in a container, so its logs aren't in Docker. +// 3. The app's structured JSON logs (errors/access/audit *.jsonl) from the +// `applogs` Docker volume, parsed so `level`/`tag`/`phase` become labels. +// +// Per-node settings come from the environment (see docker-compose.agent.yml): +// ALLOY_NODE — this node's name label (prod | beta | dev) +// LOKI_URL — where to push (http://10.10.0.2:3100/loki/api/v1/push over wg0) + +livedebugging { enabled = false } + +// --- 1. All Docker container logs ------------------------------------------------ +discovery.docker "containers" { + host = "unix:///var/run/docker.sock" +} + +// Turn Docker metadata into tidy labels: `container` (short name) and `service` +// (the compose service, e.g. app/db). Drop Alloy's own container to avoid a loop. +discovery.relabel "containers" { + targets = discovery.docker.containers.targets + + rule { + source_labels = ["__meta_docker_container_name"] + regex = "/(.*)" + target_label = "container" + } + rule { + source_labels = ["__meta_docker_container_label_com_docker_compose_service"] + target_label = "service" + } + rule { + source_labels = ["container"] + regex = ".*alloy.*" + action = "drop" + } +} + +loki.source.docker "containers" { + host = "unix:///var/run/docker.sock" + targets = discovery.relabel.containers.output + forward_to = [loki.write.central.receiver] + relabel_rules = discovery.relabel.containers.rules + labels = { job = "docker" } +} + +// --- 2. Caddy host access logs --------------------------------------------------- +local.file_match "caddy" { + path_targets = [{ __path__ = "/var/log/caddy/*.log", job = "caddy" }] +} + +loki.source.file "caddy" { + targets = local.file_match.caddy.targets + forward_to = [loki.write.central.receiver] +} + +// --- 3. App structured JSON logs (errors / access / audit) ----------------------- +// Mounted read-only from the app's `applogs` volume at /applogs (see the agent +// compose). Lift `level`/`tag`/`phase` out of the JSON so they're queryable. +local.file_match "app_jsonl" { + path_targets = [{ __path__ = "/applogs/**/*.jsonl", job = "app-json" }] +} + +loki.source.file "app_jsonl" { + targets = local.file_match.app_jsonl.targets + forward_to = [loki.process.app_jsonl.receiver] +} + +loki.process "app_jsonl" { + forward_to = [loki.write.central.receiver] + + stage.json { + expressions = { level = "level", tag = "tag", phase = "phase", status = "status" } + } + // A record with no explicit level: an error-folder line is an error, else info. + stage.static_labels { + values = { source = "app" } + } + stage.labels { + values = { level = "", tag = "", phase = "" } + } +} + +// --- Ship to the central Loki over the WireGuard mesh ---------------------------- +loki.write "central" { + endpoint { + url = sys.env("LOKI_URL") + } + // Every line from this node is stamped with its node name, so one Grafana + // view can slice prod vs beta vs dev. + external_labels = { host = sys.env("ALLOY_NODE") } +} diff --git a/alloy/docker-compose.agent.yml b/alloy/docker-compose.agent.yml new file mode 100644 index 0000000..9e3efdd --- /dev/null +++ b/alloy/docker-compose.agent.yml @@ -0,0 +1,48 @@ +# The Alloy log-collector agent. Runs on EACH node (prod, beta, LAN dev) — one +# per node — reading that node's Docker containers, Caddy logs, and the app's +# JSON logs, and shipping them to the central Loki on beta over the mesh. +# +# Per-node config comes from the environment (an .env file next to this, or +# exported before `up`): +# ALLOY_NODE this node's label: prod | beta | dev +# LOKI_URL push endpoint. On beta: http://127.0.0.1:3100/loki/api/v1/push +# Elsewhere (over wg0): http://10.10.0.2:3100/loki/api/v1/push +# +# Deploy on a node: +# ALLOY_NODE=prod LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push \ +# docker compose -f docker-compose.agent.yml up -d +# +# The app's JSON logs are read from its `applogs` Docker volume, named +# `_applogs`. prod and beta run the app under the compose project +# `thermograph`, so the external volume below is `thermograph_applogs`. The LAN +# dev stack uses the project `thermograph-dev` — on that node, change the two +# `thermograph_applogs` references below to `thermograph-dev_applogs`. + +services: + alloy: + image: grafana/alloy:v1.9.1 + command: + - run + - /etc/alloy/config.alloy + - --storage.path=/var/lib/alloy/data + - --server.http.listen-addr=0.0.0.0:12345 + environment: + ALLOY_NODE: ${ALLOY_NODE:?set ALLOY_NODE (prod|beta|dev)} + LOKI_URL: ${LOKI_URL:?set LOKI_URL} + volumes: + - ./config.alloy:/etc/alloy/config.alloy:ro + # Read-only Docker socket: discover + tail every container's stdout/stderr. + - /var/run/docker.sock:/var/run/docker.sock:ro + # Caddy's host access logs (runs on the host, not a container). + - /var/log/caddy:/var/log/caddy:ro + # The app's structured JSON logs, straight off its named volume. + - thermograph_applogs:/applogs:ro + - alloy_data:/var/lib/alloy/data + restart: unless-stopped + +volumes: + # The app's existing log volume (created by the app's own stack). external:true + # means compose references it, never creates or deletes it. + thermograph_applogs: + external: true + alloy_data: {} diff --git a/caddy-grafana.conf b/caddy-grafana.conf new file mode 100644 index 0000000..641bf5f --- /dev/null +++ b/caddy-grafana.conf @@ -0,0 +1,15 @@ +# Append to beta's /etc/caddy/Caddyfile (beta is the monitoring host), then +# `systemctl reload caddy`. Grafana publishes to 127.0.0.1:3000; this fronts it +# with TLS, same pattern as beta's other site blocks. Grafana owns its own login, +# so the UI is safe to expose publicly — but Loki (the log store, on +# 10.10.0.2:3100) is NOT proxied here and stays mesh-only. +# +# Point grafana.thermograph.org's A record at beta (75.119.132.91) first. + +grafana.thermograph.org { + encode zstd gzip + reverse_proxy 127.0.0.1:3000 + log { + output file /var/log/caddy/grafana.log + } +} diff --git a/docker-compose.yml b/docker-compose.yml new file mode 100644 index 0000000..57a6d41 --- /dev/null +++ b/docker-compose.yml @@ -0,0 +1,51 @@ +# The central observability stack: Loki (log store) + Grafana (UI). Runs on ONE +# node — the monitoring host, `beta` (75.119.132.91 / mesh 10.10.0.2) — because +# beta is an always-on VPS with a public Caddy already fronting it. Every node's +# Alloy agent (see alloy/) ships logs here; agents on prod and the desktop reach +# Loki over the WireGuard mesh, so Loki must listen on the mesh interface. +# +# Deploy (on beta): docker compose up -d +# Grafana is proxied by beta's Caddy at grafana.thermograph.org (see README); +# Loki is NOT public — it binds the mesh IP only, reachable to agents over wg0. + +services: + loki: + image: grafana/loki:3.5.1 + command: -config.file=/etc/loki/config.yml + volumes: + - ./loki/config.yml:/etc/loki/config.yml:ro + - loki_data:/loki + ports: + # Mesh-only: agents on prod (10.10.0.1) and desktop (10.10.0.3) push here + # over wg0. Never published on the public interface. + - "10.10.0.2:3100:3100" + # Also localhost, so the beta-local Alloy agent and curl checks can reach it. + - "127.0.0.1:3100:3100" + restart: unless-stopped + + grafana: + image: grafana/grafana:11.6.1 + depends_on: [loki] + environment: + # Grafana owns its own auth. Admin password is injected from the host env + # (set GF_SECURITY_ADMIN_PASSWORD before `up`, or in an .env file — see + # .env.example); never bake a credential into the compose file. + GF_SECURITY_ADMIN_USER: ${GF_ADMIN_USER:-admin} + GF_SECURITY_ADMIN_PASSWORD: ${GF_SECURITY_ADMIN_PASSWORD:?set GF_SECURITY_ADMIN_PASSWORD} + GF_USERS_ALLOW_SIGN_UP: "false" + GF_ANALYTICS_REPORTING_ENABLED: "false" + GF_ANALYTICS_CHECK_FOR_UPDATES: "false" + # Served behind Caddy at this external URL (sub-path-safe cookie/redirects). + GF_SERVER_ROOT_URL: "https://${GRAFANA_DOMAIN:-grafana.thermograph.org}/" + volumes: + - ./grafana/provisioning:/etc/grafana/provisioning:ro + - ./grafana/dashboards:/var/lib/grafana/dashboards:ro + - grafana_data:/var/lib/grafana + ports: + # Host-local; beta's Caddy reverse-proxies to it. Not public directly. + - "127.0.0.1:3000:3000" + restart: unless-stopped + +volumes: + loki_data: {} + grafana_data: {} diff --git a/grafana/dashboards/thermograph-logs.json b/grafana/dashboards/thermograph-logs.json new file mode 100644 index 0000000..8587568 --- /dev/null +++ b/grafana/dashboards/thermograph-logs.json @@ -0,0 +1,132 @@ +{ + "annotations": { "list": [] }, + "editable": true, + "graphTooltip": 1, + "title": "Thermograph — Fleet Logs", + "uid": "thermograph-logs", + "tags": ["thermograph"], + "timezone": "", + "schemaVersion": 39, + "refresh": "30s", + "time": { "from": "now-6h", "to": "now" }, + "templating": { + "list": [ + { + "name": "host", + "type": "custom", + "label": "Node", + "query": "prod,beta,dev", + "current": { "text": "All", "value": "$__all" }, + "includeAll": true, + "allValue": ".+", + "multi": true + } + ] + }, + "panels": [ + { + "type": "timeseries", + "title": "Log volume by service", + "description": "Lines/sec from every container, split by compose service (app, db, forgejo…).", + "gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 }, + "datasource": { "type": "loki", "uid": "loki" }, + "fieldConfig": { "defaults": { "custom": { "drawStyle": "line", "fillOpacity": 15, "stacking": { "mode": "normal" } } } }, + "targets": [ + { + "datasource": { "type": "loki", "uid": "loki" }, + "expr": "sum by (service) (count_over_time({job=\"docker\", host=~\"$host\"} [$__auto]))", + "legendFormat": "{{service}}" + } + ] + }, + { + "type": "timeseries", + "title": "App errors/sec (structured logs)", + "description": "count_over_time of the app's errors-*.jsonl lines, by tag.", + "gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 }, + "datasource": { "type": "loki", "uid": "loki" }, + "fieldConfig": { "defaults": { "custom": { "drawStyle": "bars", "fillOpacity": 60 }, "color": { "mode": "palette-classic" } } }, + "targets": [ + { + "datasource": { "type": "loki", "uid": "loki" }, + "expr": "sum by (tag) (count_over_time({job=\"app-json\", source=\"app\", host=~\"$host\"} | level=~\"error|warn\" [$__auto]))", + "legendFormat": "{{tag}}" + } + ] + }, + { + "type": "stat", + "title": "Upstream rate-limits (429) — last 1h", + "description": "Open-Meteo / upstream 429s pulled from the app's error logs.", + "gridPos": { "h": 4, "w": 6, "x": 0, "y": 8 }, + "datasource": { "type": "loki", "uid": "loki" }, + "fieldConfig": { "defaults": { "color": { "mode": "thresholds" }, "thresholds": { "steps": [ { "color": "green", "value": null }, { "color": "yellow", "value": 1 }, { "color": "red", "value": 50 } ] } } }, + "options": { "reduceOptions": { "calcs": ["sum"] }, "colorMode": "background" }, + "targets": [ + { + "datasource": { "type": "loki", "uid": "loki" }, + "expr": "sum (count_over_time({job=\"app-json\", host=~\"$host\"} | status=\"429\" [1h]))", + "instant": true + } + ] + }, + { + "type": "stat", + "title": "Caddy 5xx — last 1h", + "description": "Server errors seen by the reverse proxy across all sites.", + "gridPos": { "h": 4, "w": 6, "x": 6, "y": 8 }, + "datasource": { "type": "loki", "uid": "loki" }, + "fieldConfig": { "defaults": { "color": { "mode": "thresholds" }, "thresholds": { "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 1 } ] } } }, + "options": { "reduceOptions": { "calcs": ["sum"] }, "colorMode": "background" }, + "targets": [ + { + "datasource": { "type": "loki", "uid": "loki" }, + "expr": "sum (count_over_time({job=\"caddy\", host=~\"$host\"} | json | status>=500 [1h]))", + "instant": true + } + ] + }, + { + "type": "logs", + "title": "Subscription notifier", + "description": "Everything the app logs mentioning the notifier — its liveness at a glance.", + "gridPos": { "h": 4, "w": 12, "x": 12, "y": 8 }, + "datasource": { "type": "loki", "uid": "loki" }, + "options": { "showTime": true, "wrapLogMessage": true, "dedupStrategy": "exact", "sortOrder": "Descending" }, + "targets": [ + { + "datasource": { "type": "loki", "uid": "loki" }, + "expr": "{job=\"docker\", service=\"app\", host=~\"$host\"} |~ \"(?i)notifier\"" + } + ] + }, + { + "type": "logs", + "title": "Recent app errors", + "description": "The app's errors-*.jsonl, newest first.", + "gridPos": { "h": 9, "w": 12, "x": 0, "y": 12 }, + "datasource": { "type": "loki", "uid": "loki" }, + "options": { "showTime": true, "wrapLogMessage": true, "sortOrder": "Descending", "enableLogDetails": true }, + "targets": [ + { + "datasource": { "type": "loki", "uid": "loki" }, + "expr": "{job=\"app-json\", host=~\"$host\"} | level=~\"error|warn\"" + } + ] + }, + { + "type": "logs", + "title": "Live tail — all containers", + "description": "Raw stdout/stderr from every container on the selected node(s).", + "gridPos": { "h": 9, "w": 12, "x": 12, "y": 12 }, + "datasource": { "type": "loki", "uid": "loki" }, + "options": { "showTime": true, "wrapLogMessage": false, "sortOrder": "Descending", "enableLogDetails": true }, + "targets": [ + { + "datasource": { "type": "loki", "uid": "loki" }, + "expr": "{job=\"docker\", host=~\"$host\"}" + } + ] + } + ] +} diff --git a/grafana/provisioning/dashboards/provider.yml b/grafana/provisioning/dashboards/provider.yml new file mode 100644 index 0000000..fdabaad --- /dev/null +++ b/grafana/provisioning/dashboards/provider.yml @@ -0,0 +1,16 @@ +# Load any dashboard JSON dropped in /var/lib/grafana/dashboards (mounted from +# ./grafana/dashboards) at startup, so the fleet overview is version-controlled +# in this repo rather than hand-built in the UI. +apiVersion: 1 + +providers: + - name: thermograph + orgId: 1 + folder: Thermograph + type: file + disableDeletion: false + editable: true + updateIntervalSeconds: 30 + options: + path: /var/lib/grafana/dashboards + foldersFromFilesStructure: false diff --git a/grafana/provisioning/datasources/loki.yml b/grafana/provisioning/datasources/loki.yml new file mode 100644 index 0000000..c2029f9 --- /dev/null +++ b/grafana/provisioning/datasources/loki.yml @@ -0,0 +1,13 @@ +# Auto-provision the Loki datasource so Grafana comes up already wired to it — +# no click-ops. Loki runs in the same compose stack, reachable by service name. +apiVersion: 1 + +datasources: + - name: Loki + type: loki + access: proxy + url: http://loki:3100 + uid: loki # stable UID so the provisioned dashboards can bind to it + isDefault: true + jsonData: + maxLines: 2000 diff --git a/loki/config.yml b/loki/config.yml new file mode 100644 index 0000000..d17a37f --- /dev/null +++ b/loki/config.yml @@ -0,0 +1,51 @@ +# Loki — the single log store for the whole Thermograph fleet (prod, beta, LAN +# dev). Every node's Alloy agent pushes here over the WireGuard mesh; Grafana +# reads from here. Single-binary, filesystem-backed: no object storage, no +# clustering — right for a three-node hobby fleet, and everything lives on one +# volume you can back up or blow away. +auth_enabled: false + +server: + http_listen_port: 3100 + grpc_listen_port: 9096 + log_level: warn + +common: + instance_addr: 127.0.0.1 + path_prefix: /loki + storage: + filesystem: + chunks_directory: /loki/chunks + rules_directory: /loki/rules + replication_factor: 1 + ring: + kvstore: + store: inmemory + +schema_config: + configs: + - from: 2024-01-01 + store: tsdb + object_store: filesystem + schema: v13 + index: + prefix: index_ + period: 24h + +limits_config: + # A hobby fleet's volume is tiny; keep 30 days and cap ingestion generously. + retention_period: 720h + reject_old_samples: true + reject_old_samples_max_age: 168h + max_query_series: 5000 + allow_structured_metadata: true + volume_enabled: true + +compactor: + working_directory: /loki/compactor + retention_enabled: true + delete_request_store: filesystem + +# No analytics phone-home from a private box. +analytics: + reporting_enabled: false From 7e8a484ad51ebb28498fa5773b0b44cd37b45f57 Mon Sep 17 00:00:00 2001 From: Emi Griffith Date: Tue, 21 Jul 2026 09:18:29 -0700 Subject: [PATCH 2/8] Use dashboard.thermograph.org + Google SSO for the dashboard - Grafana served at dashboard.thermograph.org (the record now points at beta) - Google OAuth (OIDC) login, env-gated (OAUTH_ENABLED + GOOGLE_CLIENT_ID/SECRET); allow_sign_up=false so access is locked to pre-provisioned accounts, with a break-glass local admin fallback - README: Google Cloud OAuth client setup with the redirect URIs for both Grafana and Forgejo, plus the Forgejo add-oauth command --- .env.example | 17 ++++++++++++++--- README.md | 36 +++++++++++++++++++++++++++++++++--- caddy-grafana.conf | 12 ++++++------ docker-compose.yml | 22 ++++++++++++++++++---- 4 files changed, 71 insertions(+), 16 deletions(-) diff --git a/.env.example b/.env.example index 7ff6852..60c6633 100644 --- a/.env.example +++ b/.env.example @@ -1,9 +1,20 @@ # Copy to .env on the monitoring host (beta) before `docker compose up`. # .env is gitignored — never commit real secrets. -# Grafana admin login (the UI's own auth; Grafana is the only thing exposed). +# The public hostname beta's Caddy serves Grafana at (see caddy-grafana.conf). +GRAFANA_DOMAIN=dashboard.thermograph.org + +# Break-glass local admin (Google SSO below is the primary login). Keep this +# strong; it's the fallback if OAuth ever breaks. GF_ADMIN_USER=admin GF_SECURITY_ADMIN_PASSWORD=change-me-to-something-strong -# The public hostname beta's Caddy serves Grafana at (see caddy-grafana.conf). -GRAFANA_DOMAIN=grafana.thermograph.org +# --- Google SSO ------------------------------------------------------------------ +# Create a Google Cloud "OAuth 2.0 Client ID" (type: Web application) and set: +# Authorized redirect URI: https://dashboard.thermograph.org/login/google +# Then paste the client id/secret here and flip OAUTH_ENABLED=true. +# Login is locked to pre-provisioned accounts (allow_sign_up=false) — provision +# your email as an admin (see README) so only you can sign in. +OAUTH_ENABLED=false +GOOGLE_CLIENT_ID= +GOOGLE_CLIENT_SECRET= diff --git a/README.md b/README.md index cee80ec..5841582 100644 --- a/README.md +++ b/README.md @@ -17,8 +17,8 @@ persistent, queryable, fleet-wide log store and UI. - **Loki** stores logs (filesystem, 30-day retention). Listens on the mesh IP `10.10.0.2:3100` — never public. -- **Grafana** is the UI, fronted by beta's Caddy at `grafana.thermograph.org` - (its own login; Loki is not exposed). +- **Grafana** is the UI, fronted by beta's Caddy at `dashboard.thermograph.org`, + login via **Google SSO** (with a break-glass local admin); Loki is not exposed. - **Alloy** runs on each node and ships three sources to Loki: every Docker container's stdout/stderr, Caddy's host access logs, and the app's structured JSON logs (`errors`/`access`/`audit` `*.jsonl`, parsed so `level`/`tag`/`phase` @@ -47,9 +47,39 @@ cp .env.example .env # set GF_SECURITY_ADMIN_PASSWORD docker compose up -d # Loki on 10.10.0.2:3100 + 127.0.0.1:3100; Grafana on 127.0.0.1:3000 ``` -Expose the Grafana UI: point `grafana.thermograph.org` at beta, append +Expose the Grafana UI: point `dashboard.thermograph.org` at beta, append `caddy-grafana.conf` to `/etc/caddy/Caddyfile`, `systemctl reload caddy`. +### Google SSO (Grafana + Forgejo) + +Both the dashboard and Forgejo log in with Google. Create **one** Google Cloud +OAuth 2.0 Client (type: *Web application*) and register both redirect URIs: + +- Grafana: `https://dashboard.thermograph.org/login/google` +- Forgejo: `https://git.thermograph.org/user/oauth2/google/callback` + +**Grafana:** put the client id/secret in `.env`, set `OAUTH_ENABLED=true`, and +`docker compose up -d`. Login is locked to pre-provisioned accounts +(`allow_sign_up=false`), so provision your email as an admin once: + +```bash +# on beta, after the stack is up (uses the break-glass admin creds from .env): +curl -s -u admin:"$GF_SECURITY_ADMIN_PASSWORD" -H 'Content-Type: application/json' \ + -X POST http://127.0.0.1:3000/api/admin/users \ + -d '{"name":"you","login":"you@gmail.com","email":"you@gmail.com","password":"'"$(openssl rand -base64 24)"'"}' +# then make that user a Grafana admin (org role Admin + server admin) via the UI +# or /api/org/users; the Google login matches on email. +``` + +**Forgejo:** add the Google auth source (once, as a Forgejo admin): + +```bash +docker exec -u git forgejo admin auth add-oauth \ + --name google --provider openidConnect \ + --key "" --secret "" \ + --auto-discover-url https://accounts.google.com/.well-known/openid-configuration +``` + ### 2. Alloy agent (on every node — prod, beta, desktop) ```bash diff --git a/caddy-grafana.conf b/caddy-grafana.conf index 641bf5f..787b816 100644 --- a/caddy-grafana.conf +++ b/caddy-grafana.conf @@ -1,15 +1,15 @@ # Append to beta's /etc/caddy/Caddyfile (beta is the monitoring host), then # `systemctl reload caddy`. Grafana publishes to 127.0.0.1:3000; this fronts it -# with TLS, same pattern as beta's other site blocks. Grafana owns its own login, -# so the UI is safe to expose publicly — but Loki (the log store, on -# 10.10.0.2:3100) is NOT proxied here and stays mesh-only. +# with TLS, same pattern as beta's other site blocks. Login is Google SSO (with a +# break-glass local admin); Loki (the log store, on 10.10.0.2:3100) is NOT proxied +# here and stays mesh-only. # -# Point grafana.thermograph.org's A record at beta (75.119.132.91) first. +# dashboard.thermograph.org's A record must already point at beta (75.119.132.91). -grafana.thermograph.org { +dashboard.thermograph.org { encode zstd gzip reverse_proxy 127.0.0.1:3000 log { - output file /var/log/caddy/grafana.log + output file /var/log/caddy/dashboard.log } } diff --git a/docker-compose.yml b/docker-compose.yml index 57a6d41..e8298cf 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -27,16 +27,30 @@ services: image: grafana/grafana:11.6.1 depends_on: [loki] environment: - # Grafana owns its own auth. Admin password is injected from the host env - # (set GF_SECURITY_ADMIN_PASSWORD before `up`, or in an .env file — see - # .env.example); never bake a credential into the compose file. + # Primary login is Google SSO (below). The admin user is kept as a + # break-glass local fallback; its password comes from the host env + # (.env — see .env.example), never baked into the compose file. GF_SECURITY_ADMIN_USER: ${GF_ADMIN_USER:-admin} GF_SECURITY_ADMIN_PASSWORD: ${GF_SECURITY_ADMIN_PASSWORD:?set GF_SECURITY_ADMIN_PASSWORD} GF_USERS_ALLOW_SIGN_UP: "false" GF_ANALYTICS_REPORTING_ENABLED: "false" GF_ANALYTICS_CHECK_FOR_UPDATES: "false" # Served behind Caddy at this external URL (sub-path-safe cookie/redirects). - GF_SERVER_ROOT_URL: "https://${GRAFANA_DOMAIN:-grafana.thermograph.org}/" + GF_SERVER_ROOT_URL: "https://${GRAFANA_DOMAIN:-dashboard.thermograph.org}/" + # --- Google SSO (OIDC) ----------------------------------------------------- + # Enabled once GOOGLE_CLIENT_ID/SECRET are set in .env and OAUTH_ENABLED=true. + # allow_sign_up is FALSE: a Google login only succeeds if a Grafana user with + # that email already exists — so this dashboard is locked to pre-provisioned + # accounts (see README), not "any Google user". Redirect URI to register in + # Google Cloud: https://dashboard.thermograph.org/login/google + GF_AUTH_GOOGLE_ENABLED: ${OAUTH_ENABLED:-false} + GF_AUTH_GOOGLE_CLIENT_ID: ${GOOGLE_CLIENT_ID:-} + GF_AUTH_GOOGLE_CLIENT_SECRET: ${GOOGLE_CLIENT_SECRET:-} + GF_AUTH_GOOGLE_SCOPES: "openid email profile" + GF_AUTH_GOOGLE_AUTH_URL: "https://accounts.google.com/o/oauth2/v2/auth" + GF_AUTH_GOOGLE_TOKEN_URL: "https://oauth2.googleapis.com/token" + GF_AUTH_GOOGLE_API_URL: "https://openidconnect.googleapis.com/v1/userinfo" + GF_AUTH_GOOGLE_ALLOW_SIGN_UP: "false" volumes: - ./grafana/provisioning:/etc/grafana/provisioning:ro - ./grafana/dashboards:/var/lib/grafana/dashboards:ro From 710d9d4fecb2517c93095ee819ff9f7712368d15 Mon Sep 17 00:00:00 2001 From: Emi Griffith Date: Tue, 21 Jul 2026 09:24:26 -0700 Subject: [PATCH 3/8] Agents push to Loki over the mesh IP, not host loopback A containerized Alloy agent can't reach the host's 127.0.0.1, so even the beta-local agent must push to beta's wg0 address (10.10.0.2:3100), not 127.0.0.1. Fix the beta example in the README and the agent compose header. --- README.md | 5 +++-- alloy/docker-compose.agent.yml | 4 ++-- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 5841582..53c3878 100644 --- a/README.md +++ b/README.md @@ -84,8 +84,9 @@ docker exec -u git forgejo admin auth add-oauth \ ```bash cd thermograph-observability/alloy -# beta pushes to local Loki; prod & desktop push over the mesh: -# beta: LOKI_URL=http://127.0.0.1:3100/loki/api/v1/push ALLOY_NODE=beta +# Every node (beta included) pushes to Loki over the mesh IP — a containerized +# agent can't reach the host's 127.0.0.1, but it can reach beta's wg0 address: +# beta: LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push ALLOY_NODE=beta # prod: LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push ALLOY_NODE=prod # desktop: LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push ALLOY_NODE=dev ALLOY_NODE=prod LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push \ diff --git a/alloy/docker-compose.agent.yml b/alloy/docker-compose.agent.yml index 9e3efdd..341f9ca 100644 --- a/alloy/docker-compose.agent.yml +++ b/alloy/docker-compose.agent.yml @@ -5,8 +5,8 @@ # Per-node config comes from the environment (an .env file next to this, or # exported before `up`): # ALLOY_NODE this node's label: prod | beta | dev -# LOKI_URL push endpoint. On beta: http://127.0.0.1:3100/loki/api/v1/push -# Elsewhere (over wg0): http://10.10.0.2:3100/loki/api/v1/push +# LOKI_URL push endpoint. All nodes push over the mesh: +# http://10.10.0.2:3100/loki/api/v1/push # # Deploy on a node: # ALLOY_NODE=prod LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push \ From 6137a08c2387a2a4633f76cdba1149a85927dc04 Mon Sep 17 00:00:00 2001 From: Emi Griffith Date: Tue, 21 Jul 2026 09:51:14 -0700 Subject: [PATCH 4/8] Dashboard: notifier liveness from the heartbeat log Replace the grep-docker-logs notifier panel with a per-node UP/DOWN stat driven by the app's new tag=heartbeat log line (count_over_time over 20m; 0 beats => DOWN/red). Update the README now that the app emits heartbeats. --- README.md | 8 ++++---- grafana/dashboards/thermograph-logs.json | 19 ++++++++++++++----- 2 files changed, 18 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index 53c3878..c27b427 100644 --- a/README.md +++ b/README.md @@ -110,7 +110,7 @@ curl -s 'http://10.10.0.2:3100/loki/api/v1/label/host/values' # -> {"data":["b `Thermograph — Fleet Logs` (auto-provisioned), with a `Node` selector: log volume by service, app error rate by tag, upstream 429 count, Caddy 5xx -count, a notifier-liveness log panel, recent app errors, and a live all-container +count, a per-node notifier-liveness indicator (from the heartbeat log), recent app errors, and a live all-container tail. Edit it in Grafana and re-export the JSON here to version a change. ## Notes / follow-ups @@ -118,6 +118,6 @@ tail. Edit it in Grafana and re-export the JSON here to version a change. - **Loki is mesh-only** by design; only Grafana (with auth) is public. - Alloy reads the Docker socket read-only to discover containers. - Retention is 30 days on beta's disk — bump `loki/config.yml` if you want more. -- Possible next step: have the app emit a periodic `notifier heartbeat` log line - so notifier liveness is a first-class log-derived signal, not just container - chatter. +- The app emits a `tag=heartbeat` line for the subscription notifier every + ~15m; the dashboard's "Notifier alive?" panel turns red if a node misses it. + Extend the same `audit.log_heartbeat()` to any future background daemon. diff --git a/grafana/dashboards/thermograph-logs.json b/grafana/dashboards/thermograph-logs.json index 8587568..c417c8a 100644 --- a/grafana/dashboards/thermograph-logs.json +++ b/grafana/dashboards/thermograph-logs.json @@ -87,16 +87,25 @@ ] }, { - "type": "logs", - "title": "Subscription notifier", - "description": "Everything the app logs mentioning the notifier — its liveness at a glance.", + "type": "stat", + "title": "Notifier alive? (heartbeats / 20m, per node)", + "description": "The subscription notifier logs a heartbeat every ~15m (tag=heartbeat). 0 beats in the last 20m on a node means its notifier is down.", "gridPos": { "h": 4, "w": 12, "x": 12, "y": 8 }, "datasource": { "type": "loki", "uid": "loki" }, - "options": { "showTime": true, "wrapLogMessage": true, "dedupStrategy": "exact", "sortOrder": "Descending" }, + "fieldConfig": { + "defaults": { + "color": { "mode": "thresholds" }, + "thresholds": { "mode": "absolute", "steps": [ { "color": "red", "value": null }, { "color": "green", "value": 1 } ] }, + "mappings": [ { "type": "value", "options": { "0": { "text": "DOWN" } } } ] + } + }, + "options": { "reduceOptions": { "calcs": ["lastNotNull"] }, "colorMode": "background", "textMode": "value_and_name", "graphMode": "none" }, "targets": [ { "datasource": { "type": "loki", "uid": "loki" }, - "expr": "{job=\"docker\", service=\"app\", host=~\"$host\"} |~ \"(?i)notifier\"" + "expr": "sum by (host) (count_over_time({job=\"app-json\", tag=\"heartbeat\", host=~\"$host\"} [20m]))", + "legendFormat": "{{host}}", + "instant": true } ] }, From dafed81f8859918501ed385551dd89bdeef5adb2 Mon Sep 17 00:00:00 2001 From: Emi Griffith Date: Tue, 21 Jul 2026 13:24:37 -0700 Subject: [PATCH 5/8] Grafana: match Google logins to pre-provisioned users by email Grafana 9.3+ won't look up an OAuth login by email, so a Google sign-in for the API-created admin (no prior Google-auth linkage) tried to auto-create the user and hit allow_sign_up=false ("signup is disabled"). Enable oauth_allow_insecure_email_lookup so the login matches the existing user by its Google-verified email. Single provider + verified email = no takeover risk. --- docker-compose.yml | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docker-compose.yml b/docker-compose.yml index e8298cf..90f610f 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -51,6 +51,13 @@ services: GF_AUTH_GOOGLE_TOKEN_URL: "https://oauth2.googleapis.com/token" GF_AUTH_GOOGLE_API_URL: "https://openidconnect.googleapis.com/v1/userinfo" GF_AUTH_GOOGLE_ALLOW_SIGN_UP: "false" + # Match a Google login to a PRE-PROVISIONED user by email (the admin created + # via the API has no prior Google-auth linkage). Without this, Grafana 9.3+ + # won't look up by email and instead tries to auto-create the user — which + # allow_sign_up=false then rejects ("signup is disabled"). Safe here: Google + # verifies email ownership and it's the only OAuth provider, so there's no + # cross-provider takeover vector the "insecure" name warns about. + GF_AUTH_OAUTH_ALLOW_INSECURE_EMAIL_LOOKUP: "true" volumes: - ./grafana/provisioning:/etc/grafana/provisioning:ro - ./grafana/dashboards:/var/lib/grafana/dashboards:ro From 147b1c51a50210d6dc20b9547c775c8517cf86d6 Mon Sep 17 00:00:00 2001 From: Emi Griffith Date: Wed, 22 Jul 2026 02:23:16 +0000 Subject: [PATCH 6/8] Add Traffic & Domains dashboard (domain-grouped, visual) (#1) Co-authored-by: Emi Griffith Co-committed-by: Emi Griffith --- grafana/dashboards/thermograph-traffic.json | 1328 +++++++++++++++++++ 1 file changed, 1328 insertions(+) create mode 100644 grafana/dashboards/thermograph-traffic.json diff --git a/grafana/dashboards/thermograph-traffic.json b/grafana/dashboards/thermograph-traffic.json new file mode 100644 index 0000000..1087aa7 --- /dev/null +++ b/grafana/dashboards/thermograph-traffic.json @@ -0,0 +1,1328 @@ +{ + "uid": "tg-traffic-domains", + "title": "Thermograph \u2014 Traffic & Domains", + "tags": [ + "thermograph", + "traffic", + "domains" + ], + "schemaVersion": 39, + "editable": true, + "refresh": "30s", + "time": { + "from": "now-6h", + "to": "now" + }, + "timezone": "browser", + "templating": { + "list": [ + { + "name": "host", + "type": "query", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "query": { + "label": "host", + "stream": "", + "type": 1 + }, + "definition": "label_values(host)", + "refresh": 2, + "includeAll": true, + "multi": true, + "allValue": ".+", + "current": { + "text": "All", + "value": "$__all" + }, + "sort": 1 + } + ] + }, + "annotations": { + "list": [ + { + "builtIn": 1, + "datasource": { + "type": "grafana", + "uid": "-- Grafana --" + }, + "enable": true, + "hide": true, + "name": "Annotations & Alerts", + "type": "dashboard" + } + ] + }, + "panels": [ + { + "id": 1, + "type": "row", + "title": "Overview", + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 0 + } + }, + { + "id": 2, + "type": "stat", + "title": "App requests", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "h": 5, + "w": 4, + "x": 0, + "y": 1 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "expr": "sum(count_over_time({job=\"app-json\", host=~\"$host\", filename=~\".+/access/.+\"} | json [$__range]))", + "refId": "A", + "queryType": "range" + } + ], + "fieldConfig": { + "defaults": { + "unit": "short", + "custom": {}, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "color": { + "mode": "fixed", + "fixedColor": "text" + } + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "colorMode": "value", + "graphMode": "area", + "justifyMode": "auto", + "textMode": "auto", + "orientation": "auto" + } + }, + { + "id": 3, + "type": "stat", + "title": "HTTP errors (4xx/5xx)", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "h": 5, + "w": 4, + "x": 4, + "y": 1 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "expr": "sum(count_over_time({job=\"app-json\", host=~\"$host\", filename=~\".+/access/.+\"} | json | status=~\"4..|5..\" [$__range]))", + "refId": "A", + "queryType": "range" + } + ], + "fieldConfig": { + "defaults": { + "unit": "short", + "custom": {}, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "orange", + "value": 1 + }, + { + "color": "red", + "value": 25 + } + ] + }, + "color": { + "mode": "thresholds", + "fixedColor": "text" + } + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "colorMode": "background", + "graphMode": "area", + "justifyMode": "auto", + "textMode": "auto", + "orientation": "auto" + } + }, + { + "id": 4, + "type": "stat", + "title": "Upstream failures", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "h": 5, + "w": 4, + "x": 8, + "y": 1 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "expr": "sum(count_over_time({job=\"app-json\", host=~\"$host\", filename=~\".+/errors/.+\"} [$__range]))", + "refId": "A", + "queryType": "range" + } + ], + "fieldConfig": { + "defaults": { + "unit": "short", + "custom": {}, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "orange", + "value": 1 + }, + { + "color": "red", + "value": 20 + } + ] + }, + "color": { + "mode": "thresholds", + "fixedColor": "text" + } + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "colorMode": "background", + "graphMode": "area", + "justifyMode": "auto", + "textMode": "auto", + "orientation": "auto" + } + }, + { + "id": 5, + "type": "stat", + "title": "Rate-limited (429)", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "h": 5, + "w": 4, + "x": 12, + "y": 1 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "expr": "sum(count_over_time({job=\"app-json\", host=~\"$host\", filename=~\".+/errors/.+\"} | json | status=\"429\" [$__range]))", + "refId": "A", + "queryType": "range" + } + ], + "fieldConfig": { + "defaults": { + "unit": "short", + "custom": {}, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 1 + } + ] + }, + "color": { + "mode": "thresholds", + "fixedColor": "text" + } + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "colorMode": "background", + "graphMode": "area", + "justifyMode": "auto", + "textMode": "auto", + "orientation": "auto" + } + }, + { + "id": 6, + "type": "stat", + "title": "Notifier beats / 20m", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "h": 5, + "w": 4, + "x": 16, + "y": 1 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "expr": "sum by (host) (count_over_time({job=\"app-json\", host=~\"$host\"} | tag=\"heartbeat\" [20m]))", + "refId": "A", + "queryType": "range", + "legendFormat": "{{host}}" + } + ], + "fieldConfig": { + "defaults": { + "unit": "short", + "custom": {}, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "red", + "value": null + }, + { + "color": "green", + "value": 1 + } + ] + }, + "color": { + "mode": "thresholds", + "fixedColor": "text" + } + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "colorMode": "background", + "graphMode": "area", + "justifyMode": "auto", + "textMode": "auto", + "orientation": "auto" + } + }, + { + "id": 7, + "type": "stat", + "title": "Caddy 5xx / 1h", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "h": 5, + "w": 4, + "x": 20, + "y": 1 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "expr": "sum(count_over_time({job=\"caddy\", host=~\"$host\"} | json | status>=500 [1h]))", + "refId": "A", + "queryType": "range" + } + ], + "fieldConfig": { + "defaults": { + "unit": "short", + "custom": {}, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 1 + } + ] + }, + "color": { + "mode": "thresholds", + "fixedColor": "text" + } + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "colorMode": "background", + "graphMode": "area", + "justifyMode": "auto", + "textMode": "auto", + "orientation": "auto" + } + }, + { + "id": 8, + "type": "piechart", + "title": "Traffic mix by domain (category)", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "h": 9, + "w": 12, + "x": 0, + "y": 6 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "expr": "sum by (cat) (count_over_time({job=\"app-json\", host=~\"$host\", filename=~\".+/access/.+\"} | json [$__range]))", + "refId": "A", + "queryType": "range", + "legendFormat": "{{cat}}" + } + ], + "fieldConfig": { + "defaults": { + "unit": "short", + "custom": {} + }, + "overrides": [] + }, + "options": { + "legend": { + "displayMode": "table", + "placement": "right", + "values": [ + "value", + "percent" + ] + }, + "pieType": "donut", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "values": true + }, + "displayLabels": [ + "percent" + ] + } + }, + { + "id": 9, + "type": "timeseries", + "title": "Requests/sec by domain", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "h": 9, + "w": 12, + "x": 12, + "y": 6 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "expr": "sum by (cat) (rate({job=\"app-json\", host=~\"$host\", filename=~\".+/access/.+\"} | json [$__auto]))", + "refId": "A", + "queryType": "range", + "legendFormat": "{{cat}}" + } + ], + "fieldConfig": { + "defaults": { + "unit": "reqps", + "custom": { + "drawStyle": "line", + "lineWidth": 2, + "fillOpacity": 18, + "showPoints": "never", + "spanNulls": true, + "stacking": { + "mode": "normal", + "group": "A" + } + } + }, + "overrides": [] + }, + "options": { + "legend": { + "displayMode": "table", + "placement": "bottom", + "calcs": [ + "mean", + "max" + ] + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + } + }, + { + "id": 10, + "type": "row", + "title": "Weather API \u2014 core product", + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 15 + } + }, + { + "id": 11, + "type": "timeseries", + "title": "Weather endpoints \u2014 req/s", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 16 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "expr": "sum by (cat) (rate({job=\"app-json\", host=~\"$host\", filename=~\".+/access/.+\"} | json | cat=~\"api:.+\" [$__auto]))", + "refId": "A", + "queryType": "range", + "legendFormat": "{{cat}}" + } + ], + "fieldConfig": { + "defaults": { + "unit": "reqps", + "custom": { + "drawStyle": "line", + "lineWidth": 2, + "fillOpacity": 18, + "showPoints": "never", + "spanNulls": true, + "stacking": { + "mode": "normal", + "group": "A" + } + } + }, + "overrides": [] + }, + "options": { + "legend": { + "displayMode": "table", + "placement": "bottom", + "calcs": [ + "mean", + "max" + ] + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + } + }, + { + "id": 12, + "type": "timeseries", + "title": "Grade run latency (ms)", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 16 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "expr": "quantile_over_time(0.95, {job=\"app-json\", host=~\"$host\", filename=~\".+/audit/.+\"} | json | unwrap duration_ms [$__auto]) by ()", + "refId": "A", + "queryType": "range", + "legendFormat": "p95" + }, + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "expr": "quantile_over_time(0.50, {job=\"app-json\", host=~\"$host\", filename=~\".+/audit/.+\"} | json | unwrap duration_ms [$__auto]) by ()", + "refId": "B", + "queryType": "range", + "legendFormat": "p50" + } + ], + "fieldConfig": { + "defaults": { + "unit": "ms", + "custom": { + "drawStyle": "line", + "lineWidth": 2, + "fillOpacity": 0, + "showPoints": "never", + "spanNulls": true, + "stacking": { + "mode": "none", + "group": "A" + } + } + }, + "overrides": [] + }, + "options": { + "legend": { + "displayMode": "table", + "placement": "bottom", + "calcs": [ + "mean", + "max" + ] + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + } + }, + { + "id": 13, + "type": "table", + "title": "Top endpoints (by volume)", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 0, + "y": 24 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "expr": "topk(12, sum by (cat) (count_over_time({job=\"app-json\", host=~\"$host\", filename=~\".+/access/.+\"} | json | cat=~\"api:.+\" [$__range])))", + "refId": "A", + "queryType": "instant", + "legendFormat": "{{cat}}", + "instant": true, + "format": "table" + } + ], + "fieldConfig": { + "defaults": { + "unit": "short", + "custom": {} + }, + "overrides": [] + }, + "options": { + "showHeader": true, + "cellHeight": "sm", + "sortBy": [ + { + "displayName": "Value", + "desc": true + } + ] + } + }, + { + "id": 14, + "type": "stat", + "title": "Cache-served runs (partial %)", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 8, + "y": 24 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "expr": "sum(count_over_time({job=\"app-json\", host=~\"$host\", filename=~\".+/audit/.+\"} | json | run_type=\"partial\" [$__range])) / sum(count_over_time({job=\"app-json\", host=~\"$host\", filename=~\".+/audit/.+\"} | json | run_type=~\".+\" [$__range])) * 100", + "refId": "A", + "queryType": "range" + } + ], + "fieldConfig": { + "defaults": { + "unit": "percent", + "custom": {}, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "color": { + "mode": "fixed", + "fixedColor": "text" + } + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "colorMode": "value", + "graphMode": "area", + "justifyMode": "auto", + "textMode": "auto", + "orientation": "auto" + } + }, + { + "id": 15, + "type": "timeseries", + "title": "Fresh (full) vs cached (partial) runs", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 16, + "y": 24 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "expr": "sum by (run_type) (count_over_time({job=\"app-json\", host=~\"$host\", filename=~\".+/audit/.+\"} | json | run_type=~\".+\" [$__auto]))", + "refId": "A", + "queryType": "range", + "legendFormat": "{{run_type}}" + } + ], + "fieldConfig": { + "defaults": { + "unit": "short", + "custom": { + "drawStyle": "line", + "lineWidth": 2, + "fillOpacity": 18, + "showPoints": "never", + "spanNulls": true, + "stacking": { + "mode": "normal", + "group": "A" + } + } + }, + "overrides": [] + }, + "options": { + "legend": { + "displayMode": "table", + "placement": "bottom", + "calcs": [ + "mean", + "max" + ] + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + } + }, + { + "id": 16, + "type": "row", + "title": "Accounts, Auth & Content", + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 32 + } + }, + { + "id": 17, + "type": "timeseries", + "title": "Auth & Accounts \u2014 req/s", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 33 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "expr": "sum by (cat) (rate({job=\"app-json\", host=~\"$host\", filename=~\".+/access/.+\"} | json | cat=~\"auth|accounts\" [$__auto]))", + "refId": "A", + "queryType": "range", + "legendFormat": "{{cat}}" + } + ], + "fieldConfig": { + "defaults": { + "unit": "reqps", + "custom": { + "drawStyle": "line", + "lineWidth": 2, + "fillOpacity": 18, + "showPoints": "never", + "spanNulls": true, + "stacking": { + "mode": "none", + "group": "A" + } + } + }, + "overrides": [] + }, + "options": { + "legend": { + "displayMode": "table", + "placement": "bottom", + "calcs": [ + "mean", + "max" + ] + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + } + }, + { + "id": 18, + "type": "timeseries", + "title": "Pages & SEO \u2014 req/s", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 33 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "expr": "sum by (cat) (rate({job=\"app-json\", host=~\"$host\", filename=~\".+/access/.+\"} | json | cat=~\"page|seo\" [$__auto]))", + "refId": "A", + "queryType": "range", + "legendFormat": "{{cat}}" + } + ], + "fieldConfig": { + "defaults": { + "unit": "reqps", + "custom": { + "drawStyle": "line", + "lineWidth": 2, + "fillOpacity": 18, + "showPoints": "never", + "spanNulls": true, + "stacking": { + "mode": "none", + "group": "A" + } + } + }, + "overrides": [] + }, + "options": { + "legend": { + "displayMode": "table", + "placement": "bottom", + "calcs": [ + "mean", + "max" + ] + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + } + }, + { + "id": 19, + "type": "row", + "title": "Upstream fetches & Reliability", + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 41 + } + }, + { + "id": 20, + "type": "timeseries", + "title": "Upstream failures by phase", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 42 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "expr": "sum by (phase) (count_over_time({job=\"app-json\", host=~\"$host\", filename=~\".+/errors/.+\"} [$__auto]))", + "refId": "A", + "queryType": "range", + "legendFormat": "{{phase}}" + } + ], + "fieldConfig": { + "defaults": { + "unit": "short", + "custom": { + "drawStyle": "line", + "lineWidth": 2, + "fillOpacity": 18, + "showPoints": "never", + "spanNulls": true, + "stacking": { + "mode": "normal", + "group": "A" + } + } + }, + "overrides": [] + }, + "options": { + "legend": { + "displayMode": "table", + "placement": "bottom", + "calcs": [ + "mean", + "max" + ] + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + } + }, + { + "id": 21, + "type": "timeseries", + "title": "HTTP errors by status", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 42 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "expr": "sum by (status) (count_over_time({job=\"app-json\", host=~\"$host\", filename=~\".+/access/.+\"} | json | status=~\"4..|5..\" [$__auto]))", + "refId": "A", + "queryType": "range", + "legendFormat": "{{status}}" + } + ], + "fieldConfig": { + "defaults": { + "unit": "short", + "custom": { + "drawStyle": "line", + "lineWidth": 2, + "fillOpacity": 18, + "showPoints": "never", + "spanNulls": true, + "stacking": { + "mode": "normal", + "group": "A" + } + } + }, + "overrides": [] + }, + "options": { + "legend": { + "displayMode": "table", + "placement": "bottom", + "calcs": [ + "mean", + "max" + ] + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + } + }, + { + "id": 22, + "type": "logs", + "title": "Recent errors & retries (app)", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "h": 9, + "w": 24, + "x": 0, + "y": 50 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "expr": "{job=\"app-json\", host=~\"$host\", filename=~\".+/errors/.+\"}", + "refId": "A", + "queryType": "range" + } + ], + "options": { + "showTime": true, + "wrapLogMessage": true, + "prettifyLogMessage": true, + "enableLogDetails": true, + "dedupStrategy": "none", + "sortOrder": "Descending" + } + }, + { + "id": 23, + "type": "row", + "title": "By site (Caddy) & Fleet", + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 59 + } + }, + { + "id": 24, + "type": "timeseries", + "title": "Requests by site (Host header)", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 60 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "expr": "sum by (request_host) (rate({job=\"caddy\", host=~\"$host\"} | json [$__auto]))", + "refId": "A", + "queryType": "range", + "legendFormat": "{{request_host}}" + } + ], + "fieldConfig": { + "defaults": { + "unit": "reqps", + "custom": { + "drawStyle": "line", + "lineWidth": 2, + "fillOpacity": 18, + "showPoints": "never", + "spanNulls": true, + "stacking": { + "mode": "normal", + "group": "A" + } + } + }, + "overrides": [] + }, + "options": { + "legend": { + "displayMode": "table", + "placement": "bottom", + "calcs": [ + "mean", + "max" + ] + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + } + }, + { + "id": 25, + "type": "timeseries", + "title": "Log volume by service", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 60 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "expr": "sum by (service) (rate({job=\"docker\", host=~\"$host\"} [$__auto]))", + "refId": "A", + "queryType": "range", + "legendFormat": "{{service}}" + } + ], + "fieldConfig": { + "defaults": { + "unit": "logs", + "custom": { + "drawStyle": "line", + "lineWidth": 2, + "fillOpacity": 18, + "showPoints": "never", + "spanNulls": true, + "stacking": { + "mode": "none", + "group": "A" + } + } + }, + "overrides": [] + }, + "options": { + "legend": { + "displayMode": "table", + "placement": "bottom", + "calcs": [ + "mean", + "max" + ] + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + } + }, + { + "id": 26, + "type": "logs", + "title": "Live tail \u2014 all containers", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "h": 9, + "w": 24, + "x": 0, + "y": 68 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "expr": "{job=\"docker\", host=~\"$host\"}", + "refId": "A", + "queryType": "range" + } + ], + "options": { + "showTime": true, + "wrapLogMessage": true, + "prettifyLogMessage": true, + "enableLogDetails": true, + "dedupStrategy": "none", + "sortOrder": "Descending" + } + } + ] +} \ No newline at end of file From 9de12534a5f58907598b7bbb6802b070ad6df19e Mon Sep 17 00:00:00 2001 From: Emi Griffith Date: Wed, 22 Jul 2026 03:20:20 +0000 Subject: [PATCH 7/8] Add App Activity & Delivery dashboard (#2) Co-authored-by: Emi Griffith Co-committed-by: Emi Griffith --- grafana/dashboards/thermograph-activity.json | 1368 ++++++++++++++++++ 1 file changed, 1368 insertions(+) create mode 100644 grafana/dashboards/thermograph-activity.json diff --git a/grafana/dashboards/thermograph-activity.json b/grafana/dashboards/thermograph-activity.json new file mode 100644 index 0000000..daf6cc1 --- /dev/null +++ b/grafana/dashboards/thermograph-activity.json @@ -0,0 +1,1368 @@ +{ + "uid": "tg-app-activity", + "title": "Thermograph \u2014 App Activity & Delivery", + "tags": [ + "thermograph", + "activity", + "notifier", + "auth" + ], + "schemaVersion": 39, + "editable": true, + "refresh": "1m", + "time": { + "from": "now-24h", + "to": "now" + }, + "timezone": "browser", + "templating": { + "list": [ + { + "name": "host", + "type": "query", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "query": { + "label": "host", + "stream": "", + "type": 1 + }, + "definition": "label_values(host)", + "refresh": 2, + "includeAll": true, + "multi": true, + "allValue": ".+", + "current": { + "text": "All", + "value": "$__all" + }, + "sort": 1 + } + ] + }, + "annotations": { + "list": [ + { + "builtIn": 1, + "datasource": { + "type": "grafana", + "uid": "-- Grafana --" + }, + "enable": true, + "hide": true, + "name": "Annotations & Alerts", + "type": "dashboard" + } + ] + }, + "panels": [ + { + "id": 1, + "type": "row", + "title": "Notifier throughput", + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 0 + } + }, + { + "id": 2, + "type": "stat", + "title": "Alerts fired", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "h": 5, + "w": 4, + "x": 0, + "y": 1 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "expr": "sum(count_over_time({job=\"app-json\", host=~\"$host\", filename=~\".+/activity/.+\"} | tag=\"notif.emit\" [$__range]))", + "refId": "A", + "queryType": "range" + } + ], + "fieldConfig": { + "defaults": { + "unit": "short", + "custom": {}, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "color": { + "mode": "fixed", + "fixedColor": "text" + } + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "colorMode": "value", + "graphMode": "area", + "justifyMode": "auto", + "textMode": "auto" + } + }, + { + "id": 3, + "type": "stat", + "title": "Sweeps run", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "h": 5, + "w": 4, + "x": 4, + "y": 1 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "expr": "sum(count_over_time({job=\"app-json\", host=~\"$host\", filename=~\".+/activity/.+\"} | tag=\"notify.pass\" [$__range]))", + "refId": "A", + "queryType": "range" + } + ], + "fieldConfig": { + "defaults": { + "unit": "short", + "custom": {}, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "color": { + "mode": "fixed", + "fixedColor": "text" + } + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "colorMode": "value", + "graphMode": "area", + "justifyMode": "auto", + "textMode": "auto" + } + }, + { + "id": 4, + "type": "stat", + "title": "Registrations", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "h": 5, + "w": 4, + "x": 8, + "y": 1 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "expr": "sum(count_over_time({job=\"app-json\", host=~\"$host\", filename=~\".+/activity/.+\"} | tag=\"auth.register\" [$__range]))", + "refId": "A", + "queryType": "range" + } + ], + "fieldConfig": { + "defaults": { + "unit": "short", + "custom": {}, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "color": { + "mode": "fixed", + "fixedColor": "text" + } + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "colorMode": "value", + "graphMode": "area", + "justifyMode": "auto", + "textMode": "auto" + } + }, + { + "id": 5, + "type": "stat", + "title": "Logins", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "h": 5, + "w": 4, + "x": 12, + "y": 1 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "expr": "sum(count_over_time({job=\"app-json\", host=~\"$host\", filename=~\".+/activity/.+\"} | tag=\"auth.login\" [$__range]))", + "refId": "A", + "queryType": "range" + } + ], + "fieldConfig": { + "defaults": { + "unit": "short", + "custom": {}, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "color": { + "mode": "fixed", + "fixedColor": "text" + } + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "colorMode": "value", + "graphMode": "area", + "justifyMode": "auto", + "textMode": "auto" + } + }, + { + "id": 6, + "type": "stat", + "title": "Email failures", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "h": 5, + "w": 4, + "x": 16, + "y": 1 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "expr": "sum(count_over_time({job=\"app-json\", host=~\"$host\", filename=~\".+/activity/.+\"} | tag=\"mail.send\" | json | ok=\"false\" [$__range]))", + "refId": "A", + "queryType": "range" + } + ], + "fieldConfig": { + "defaults": { + "unit": "short", + "custom": {}, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 1 + } + ] + }, + "color": { + "mode": "thresholds", + "fixedColor": "text" + } + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "colorMode": "background", + "graphMode": "area", + "justifyMode": "auto", + "textMode": "auto" + } + }, + { + "id": 7, + "type": "stat", + "title": "Discord DM failures", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "h": 5, + "w": 4, + "x": 20, + "y": 1 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "expr": "sum(count_over_time({job=\"app-json\", host=~\"$host\", filename=~\".+/activity/.+\"} | tag=\"notif.discord\" | json | ok=\"false\" [$__range]))", + "refId": "A", + "queryType": "range" + } + ], + "fieldConfig": { + "defaults": { + "unit": "short", + "custom": {}, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 1 + } + ] + }, + "color": { + "mode": "thresholds", + "fixedColor": "text" + } + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "colorMode": "background", + "graphMode": "area", + "justifyMode": "auto", + "textMode": "auto" + } + }, + { + "id": 8, + "type": "timeseries", + "title": "Notifier sweep \u2014 subscriptions evaluated vs alerts created", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 6 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "expr": "max_over_time({job=\"app-json\", host=~\"$host\", filename=~\".+/activity/.+\"} | tag=\"notify.pass\" | json | unwrap subs_evaluated [$__auto])", + "refId": "A", + "queryType": "range", + "legendFormat": "subs evaluated" + }, + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "expr": "max_over_time({job=\"app-json\", host=~\"$host\", filename=~\".+/activity/.+\"} | tag=\"notify.pass\" | json | unwrap created [$__auto])", + "refId": "B", + "queryType": "range", + "legendFormat": "alerts created" + } + ], + "fieldConfig": { + "defaults": { + "unit": "short", + "custom": { + "drawStyle": "line", + "lineWidth": 2, + "fillOpacity": 18, + "showPoints": "auto", + "spanNulls": true, + "stacking": { + "mode": "none", + "group": "A" + } + } + }, + "overrides": [] + }, + "options": { + "legend": { + "displayMode": "table", + "placement": "bottom", + "calcs": [ + "mean", + "max" + ] + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + } + }, + { + "id": 9, + "type": "timeseries", + "title": "Notifier sweep duration (ms)", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 6 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "expr": "max_over_time({job=\"app-json\", host=~\"$host\", filename=~\".+/activity/.+\"} | tag=\"notify.pass\" | json | unwrap duration_ms [$__auto])", + "refId": "A", + "queryType": "range", + "legendFormat": "duration" + } + ], + "fieldConfig": { + "defaults": { + "unit": "ms", + "custom": { + "drawStyle": "line", + "lineWidth": 2, + "fillOpacity": 0, + "showPoints": "auto", + "spanNulls": true, + "stacking": { + "mode": "none", + "group": "A" + } + } + }, + "overrides": [] + }, + "options": { + "legend": { + "displayMode": "table", + "placement": "bottom", + "calcs": [ + "mean", + "max" + ] + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + } + }, + { + "id": 10, + "type": "timeseries", + "title": "Alerts fired \u2014 by metric", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 14 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "expr": "sum by (metric) (count_over_time({job=\"app-json\", host=~\"$host\", filename=~\".+/activity/.+\"} | tag=\"notif.emit\" | json [$__auto]))", + "refId": "A", + "queryType": "range", + "legendFormat": "{{metric}}" + } + ], + "fieldConfig": { + "defaults": { + "unit": "short", + "custom": { + "drawStyle": "bars", + "lineWidth": 2, + "fillOpacity": 60, + "showPoints": "auto", + "spanNulls": true, + "stacking": { + "mode": "normal", + "group": "A" + } + } + }, + "overrides": [] + }, + "options": { + "legend": { + "displayMode": "table", + "placement": "bottom", + "calcs": [ + "mean", + "max" + ] + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + } + }, + { + "id": 11, + "type": "timeseries", + "title": "Alerts fired \u2014 by kind & direction", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 14 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "expr": "sum by (kind, direction) (count_over_time({job=\"app-json\", host=~\"$host\", filename=~\".+/activity/.+\"} | tag=\"notif.emit\" | json [$__auto]))", + "refId": "A", + "queryType": "range", + "legendFormat": "{{kind}}/{{direction}}" + } + ], + "fieldConfig": { + "defaults": { + "unit": "short", + "custom": { + "drawStyle": "bars", + "lineWidth": 2, + "fillOpacity": 60, + "showPoints": "auto", + "spanNulls": true, + "stacking": { + "mode": "normal", + "group": "A" + } + } + }, + "overrides": [] + }, + "options": { + "legend": { + "displayMode": "table", + "placement": "bottom", + "calcs": [ + "mean", + "max" + ] + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + } + }, + { + "id": 12, + "type": "row", + "title": "Signup funnel & account lifecycle", + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 22 + } + }, + { + "id": 13, + "type": "timeseries", + "title": "Auth funnel \u2014 register \u2192 verify \u2192 login", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 23 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "expr": "sum by (tag) (count_over_time({job=\"app-json\", host=~\"$host\", filename=~\".+/activity/.+\"} | tag=~\"auth.+\" [$__auto]))", + "refId": "A", + "queryType": "range", + "legendFormat": "{{tag}}" + } + ], + "fieldConfig": { + "defaults": { + "unit": "short", + "custom": { + "drawStyle": "bars", + "lineWidth": 2, + "fillOpacity": 60, + "showPoints": "auto", + "spanNulls": true, + "stacking": { + "mode": "none", + "group": "A" + } + } + }, + "overrides": [] + }, + "options": { + "legend": { + "displayMode": "table", + "placement": "bottom", + "calcs": [ + "mean", + "max" + ] + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + } + }, + { + "id": 14, + "type": "timeseries", + "title": "Account lifecycle \u2014 subscriptions & push devices", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 23 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "expr": "sum by (tag) (count_over_time({job=\"app-json\", host=~\"$host\", filename=~\".+/activity/.+\"} | tag=~\"sub.+|push.+\" [$__auto]))", + "refId": "A", + "queryType": "range", + "legendFormat": "{{tag}}" + } + ], + "fieldConfig": { + "defaults": { + "unit": "short", + "custom": { + "drawStyle": "bars", + "lineWidth": 2, + "fillOpacity": 60, + "showPoints": "auto", + "spanNulls": true, + "stacking": { + "mode": "normal", + "group": "A" + } + } + }, + "overrides": [] + }, + "options": { + "legend": { + "displayMode": "table", + "placement": "bottom", + "calcs": [ + "mean", + "max" + ] + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + } + }, + { + "id": 15, + "type": "row", + "title": "Delivery health", + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 31 + } + }, + { + "id": 16, + "type": "timeseries", + "title": "Email sends \u2014 ok vs fail", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 0, + "y": 32 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "expr": "sum by (ok) (count_over_time({job=\"app-json\", host=~\"$host\", filename=~\".+/activity/.+\"} | tag=\"mail.send\" | json [$__auto]))", + "refId": "A", + "queryType": "range", + "legendFormat": "ok={{ok}}" + } + ], + "fieldConfig": { + "defaults": { + "unit": "short", + "custom": { + "drawStyle": "bars", + "lineWidth": 2, + "fillOpacity": 60, + "showPoints": "auto", + "spanNulls": true, + "stacking": { + "mode": "normal", + "group": "A" + } + } + }, + "overrides": [] + }, + "options": { + "legend": { + "displayMode": "table", + "placement": "bottom", + "calcs": [ + "mean", + "max" + ] + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + } + }, + { + "id": 17, + "type": "timeseries", + "title": "Discord DMs \u2014 ok vs fail", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 8, + "y": 32 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "expr": "sum by (ok) (count_over_time({job=\"app-json\", host=~\"$host\", filename=~\".+/activity/.+\"} | tag=\"notif.discord\" | json [$__auto]))", + "refId": "A", + "queryType": "range", + "legendFormat": "ok={{ok}}" + } + ], + "fieldConfig": { + "defaults": { + "unit": "short", + "custom": { + "drawStyle": "bars", + "lineWidth": 2, + "fillOpacity": 60, + "showPoints": "auto", + "spanNulls": true, + "stacking": { + "mode": "normal", + "group": "A" + } + } + }, + "overrides": [] + }, + "options": { + "legend": { + "displayMode": "table", + "placement": "bottom", + "calcs": [ + "mean", + "max" + ] + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + } + }, + { + "id": 18, + "type": "timeseries", + "title": "Email sends by backend", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 16, + "y": 32 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "expr": "sum by (backend) (count_over_time({job=\"app-json\", host=~\"$host\", filename=~\".+/activity/.+\"} | tag=\"mail.send\" | json [$__auto]))", + "refId": "A", + "queryType": "range", + "legendFormat": "{{backend}}" + } + ], + "fieldConfig": { + "defaults": { + "unit": "short", + "custom": { + "drawStyle": "bars", + "lineWidth": 2, + "fillOpacity": 60, + "showPoints": "auto", + "spanNulls": true, + "stacking": { + "mode": "normal", + "group": "A" + } + } + }, + "overrides": [] + }, + "options": { + "legend": { + "displayMode": "table", + "placement": "bottom", + "calcs": [ + "mean", + "max" + ] + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + } + }, + { + "id": 19, + "type": "row", + "title": "Unhandled 500s & slow requests", + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 40 + } + }, + { + "id": 20, + "type": "stat", + "title": "Unhandled 500s (1h)", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "h": 5, + "w": 6, + "x": 0, + "y": 41 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "expr": "sum(count_over_time({job=\"app-json\", host=~\"$host\", filename=~\".+/errors/.+\"} | tag=\"http.error\" [1h]))", + "refId": "A", + "queryType": "range" + } + ], + "fieldConfig": { + "defaults": { + "unit": "short", + "custom": {}, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "orange", + "value": 1 + }, + { + "color": "red", + "value": 5 + } + ] + }, + "color": { + "mode": "thresholds", + "fixedColor": "text" + } + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "colorMode": "background", + "graphMode": "area", + "justifyMode": "auto", + "textMode": "auto" + } + }, + { + "id": 21, + "type": "stat", + "title": "Slow requests >2s (1h)", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "h": 5, + "w": 6, + "x": 6, + "y": 41 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "expr": "sum(count_over_time({job=\"app-json\", host=~\"$host\", filename=~\".+/activity/.+\"} | tag=\"http.slow\" [1h]))", + "refId": "A", + "queryType": "range" + } + ], + "fieldConfig": { + "defaults": { + "unit": "short", + "custom": {}, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "orange", + "value": 1 + }, + { + "color": "red", + "value": 20 + } + ] + }, + "color": { + "mode": "thresholds", + "fixedColor": "text" + } + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "colorMode": "background", + "graphMode": "area", + "justifyMode": "auto", + "textMode": "auto" + } + }, + { + "id": 22, + "type": "timeseries", + "title": "500s by route (category)", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 41 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "expr": "sum by (cat) (count_over_time({job=\"app-json\", host=~\"$host\", filename=~\".+/errors/.+\"} | tag=\"http.error\" | json [$__auto]))", + "refId": "A", + "queryType": "range", + "legendFormat": "{{cat}}" + } + ], + "fieldConfig": { + "defaults": { + "unit": "short", + "custom": { + "drawStyle": "bars", + "lineWidth": 2, + "fillOpacity": 60, + "showPoints": "auto", + "spanNulls": true, + "stacking": { + "mode": "normal", + "group": "A" + } + } + }, + "overrides": [] + }, + "options": { + "legend": { + "displayMode": "table", + "placement": "bottom", + "calcs": [ + "mean", + "max" + ] + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + } + }, + { + "id": 23, + "type": "timeseries", + "title": "Slow requests (>2s) by route (category)", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 49 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "expr": "sum by (cat) (count_over_time({job=\"app-json\", host=~\"$host\", filename=~\".+/activity/.+\"} | tag=\"http.slow\" | json [$__auto]))", + "refId": "A", + "queryType": "range", + "legendFormat": "{{cat}}" + } + ], + "fieldConfig": { + "defaults": { + "unit": "short", + "custom": { + "drawStyle": "bars", + "lineWidth": 2, + "fillOpacity": 60, + "showPoints": "auto", + "spanNulls": true, + "stacking": { + "mode": "normal", + "group": "A" + } + } + }, + "overrides": [] + }, + "options": { + "legend": { + "displayMode": "table", + "placement": "bottom", + "calcs": [ + "mean", + "max" + ] + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + } + }, + { + "id": 24, + "type": "logs", + "title": "Recent unhandled 500s (exception + path)", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 49 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "expr": "{job=\"app-json\", host=~\"$host\", filename=~\".+/errors/.+\"} | tag=\"http.error\"", + "refId": "A", + "queryType": "range" + } + ], + "options": { + "showTime": true, + "wrapLogMessage": true, + "prettifyLogMessage": true, + "enableLogDetails": true, + "dedupStrategy": "none", + "sortOrder": "Descending" + } + }, + { + "id": 25, + "type": "logs", + "title": "Recent slow requests (>2s)", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 57 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "expr": "{job=\"app-json\", host=~\"$host\", filename=~\".+/activity/.+\"} | tag=\"http.slow\"", + "refId": "A", + "queryType": "range" + } + ], + "options": { + "showTime": true, + "wrapLogMessage": true, + "prettifyLogMessage": true, + "enableLogDetails": true, + "dedupStrategy": "none", + "sortOrder": "Descending" + } + }, + { + "id": 26, + "type": "row", + "title": "Deploys & lifecycle", + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 65 + } + }, + { + "id": 27, + "type": "logs", + "title": "Process starts / stops (version, role)", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "h": 8, + "w": 24, + "x": 0, + "y": 66 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "expr": "{job=\"app-json\", host=~\"$host\", filename=~\".+/activity/.+\"} | tag=~\"app.start|app.stop\"", + "refId": "A", + "queryType": "range" + } + ], + "options": { + "showTime": true, + "wrapLogMessage": true, + "prettifyLogMessage": true, + "enableLogDetails": true, + "dedupStrategy": "none", + "sortOrder": "Descending" + } + } + ] +} \ No newline at end of file From 19e74af9caaa2aafef19257f75dacfa0f0963c39 Mon Sep 17 00:00:00 2001 From: emi Date: Wed, 22 Jul 2026 18:47:21 +0000 Subject: [PATCH 8/8] Add validator CI + CLAUDE.md; fix dashboard-domain typo (#3) --- .forgejo/workflows/validate.yml | 66 +++++++++++++++++++++++++++++++++ CLAUDE.md | 41 ++++++++++++++++++++ docker-compose.yml | 2 +- 3 files changed, 108 insertions(+), 1 deletion(-) create mode 100644 .forgejo/workflows/validate.yml create mode 100644 CLAUDE.md diff --git a/.forgejo/workflows/validate.yml b/.forgejo/workflows/validate.yml new file mode 100644 index 0000000..084a861 --- /dev/null +++ b/.forgejo/workflows/validate.yml @@ -0,0 +1,66 @@ +name: Validate observability stack + +# This repo deploys by hand (`docker compose up -d` on beta + the Alloy agent on +# each node) with no build step, so nothing caught a malformed compose file, a +# broken dashboard JSON, or an unparseable config until it failed live on the +# host. This is that missing guard: it parses every YAML/JSON artifact that ships +# to a node. It does NOT deploy, and deliberately needs no docker CLI (the +# node:20-bookworm job image doesn't ship one) — a YAML parse catches the real +# breakage (bad indent / unparseable file) without it. +# +# Not validated here: the Alloy config (alloy/config.alloy, an HCL-like format, +# needs the `alloy` binary) and docker-compose *schema* (unknown-key checks, which +# would need the docker CLI). Add either if the churn warrants the extra tooling. + +on: + push: + branches: [main] + pull_request: + branches: [main] + +jobs: + validate: + runs-on: docker + steps: + - uses: actions/checkout@v4 + + - name: Dashboards are valid JSON + run: | + set -euo pipefail + python3 - <<'PY' + import json, pathlib, sys + bad = False + files = sorted(pathlib.Path("grafana/dashboards").glob("*.json")) + if not files: + print("no dashboards found"); sys.exit(1) + for f in files: + try: + json.loads(f.read_text()); print("OK", f) + except Exception as e: # noqa: BLE001 + print("INVALID JSON", f, "-", e); bad = True + sys.exit(1 if bad else 0) + PY + + - name: YAML artifacts parse (compose + loki + grafana provisioning) + run: | + set -euo pipefail + apt-get update -qq && apt-get install -y -qq python3-yaml >/dev/null + python3 - <<'PY' + import pathlib, sys + import yaml + bad = False + targets = [ + pathlib.Path("docker-compose.yml"), + pathlib.Path("alloy/docker-compose.agent.yml"), + pathlib.Path("loki/config.yml"), + *sorted(pathlib.Path("grafana/provisioning").rglob("*.yml")), + ] + for f in targets: + if not f.exists(): + print("MISSING", f); bad = True; continue + try: + yaml.safe_load(f.read_text()); print("OK", f) + except Exception as e: # noqa: BLE001 + print("INVALID YAML", f, "-", e); bad = True + sys.exit(1 if bad else 0) + PY diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..b3e4898 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,41 @@ +# thermograph-observability — agent instructions + +The **logging/metrics stack** for the Thermograph fleet: Loki + Grafana on beta, +with a Grafana Alloy agent on every node (prod, beta, dev) shipping container and +app logs over the WireGuard mesh. Grafana is fronted by beta's Caddy at +`dashboard.thermograph.org` (Google SSO, pre-provisioned users only). + +This is **operational infra config, not application code** — there is no build. +It deploys by hand: `docker compose up -d` on beta for Loki+Grafana, and the +Alloy agent (`alloy/docker-compose.agent.yml`) on each node. See the README for +the full per-node procedure. + +## Layout + +- `docker-compose.yml` — the Loki + Grafana stack (runs on beta). +- `loki/config.yml` — Loki config (mesh-only, filesystem storage). +- `grafana/provisioning/` — datasource (Loki) + dashboard provider, auto-loaded + at startup. `grafana/dashboards/*.json` — the dashboards themselves. +- `alloy/config.alloy` + `alloy/docker-compose.agent.yml` — the per-node log + shipper. The node name comes from `ALLOY_NODE` (set per host). +- `caddy-grafana.conf` — the Caddy vhost for `dashboard.thermograph.org` (lives + in beta's Caddy config, kept here for reference). +- `.env.example` — copy to `.env` on beta before `docker compose up`. `.env` is + gitignored; never commit real OAuth secrets or the admin password. + +## Conventions + +- **Every artifact that ships to a node is CI-validated** (`.forgejo/workflows/validate.yml`): + both compose files parse, all dashboard JSON is valid, and the Loki/provisioning + YAML parses. Keep new dashboards as valid JSON and new config as valid YAML or + CI fails. The Alloy config is not yet CI-validated (needs the `alloy` binary). +- The public hostname is **`dashboard.thermograph.org`** everywhere — not + `grafana.thermograph.org`. Match it in any new comment/config. +- Secrets (OAuth client id/secret, admin password) live only in the host `.env`, + never in the repo. + +## Branching + +Single `main` branch, no protection. Open a PR against `main`; the validator +gates it. Deploying the change to beta / the nodes is a separate manual step +(this repo has no deploy automation — a known gap). diff --git a/docker-compose.yml b/docker-compose.yml index 90f610f..ff9dd6b 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -5,7 +5,7 @@ # Loki over the WireGuard mesh, so Loki must listen on the mesh interface. # # Deploy (on beta): docker compose up -d -# Grafana is proxied by beta's Caddy at grafana.thermograph.org (see README); +# Grafana is proxied by beta's Caddy at dashboard.thermograph.org (see README); # Loki is NOT public — it binds the mesh IP only, reachable to agents over wg0. services: