Fleet log aggregation: Grafana + Loki + Alloy over the WireGuard mesh

A central Grafana + Loki stack (on beta) fed by a Grafana Alloy agent on every node, replacing the old SSH-tailed single-host scripts/dashboard.py. - docker-compose.yml: central Loki (mesh-only :3100) + Grafana (Caddy-fronted) - loki/config.yml: single-binary, filesystem storage, 30-day retention - alloy/config.alloy + docker-compose.agent.yml: per-node collector — every container's stdout/stderr via the Docker socket, Caddy host logs, and the app's structured JSON logs (errors/access/audit), each line tagged by node - grafana/: auto-provisioned Loki datasource + a fleet-logs dashboard (volume by service, error rate, upstream 429s, Caddy 5xx, notifier liveness, live tail), with a per-node selector - caddy-grafana.conf, README, .env.example
2026-07-21 09:11:52 -07:00 · 2026-07-21 09:11:52 -07:00 · 46edb0bca0
commit 46edb0bca0
11 changed files with 526 additions and 0 deletions
--- a/.env.example
+++ b/.env.example
@ -0,0 +1,9 @@
+# Copy to .env on the monitoring host (beta) before `docker compose up`.
+# .env is gitignored — never commit real secrets.
+
+# Grafana admin login (the UI's own auth; Grafana is the only thing exposed).
+GF_ADMIN_USER=admin
+GF_SECURITY_ADMIN_PASSWORD=change-me-to-something-strong
+
+# The public hostname beta's Caddy serves Grafana at (see caddy-grafana.conf).
+GRAFANA_DOMAIN=grafana.thermograph.org
--- a/.gitignore
+++ b/.gitignore
@ -0,0 +1,3 @@
+.env
+alloy/.env
+*.log
--- a/README.md
+++ b/README.md
@ -0,0 +1,92 @@
+# Thermograph — observability
+
+Fleet-wide log aggregation for [Thermograph](https://thermograph.org): a central
+**Grafana + Loki** stack, fed by a **Grafana Alloy** agent on every node, all
+over the private WireGuard mesh. This replaces the old in-repo
+`scripts/dashboard.py` (an SSH-tailed, single-host, metrics-endpoint view) with a
+persistent, queryable, fleet-wide log store and UI.
+
+```
+  prod (10.10.0.1)          beta (10.10.0.2)            desktop / LAN dev (10.10.0.3)
+  ┌────────────┐            ┌──────────────────┐        ┌────────────┐
+  │ Alloy agent│──wg0──┐    │ Loki  ◀── Alloy  │   ┌──wg0│ Alloy agent│
+  └────────────┘       └───▶│ Grafana (Caddy)  │◀──┘    └────────────┘
+   docker+caddy+app         └──────────────────┘         docker+caddy+app
+     logs                    central store + UI            logs
+```
+
+- **Loki** stores logs (filesystem, 30-day retention). Listens on the mesh IP
+  `10.10.0.2:3100` — never public.
+- **Grafana** is the UI, fronted by beta's Caddy at `grafana.thermograph.org`
+  (its own login; Loki is not exposed).
+- **Alloy** runs on each node and ships three sources to Loki: every Docker
+  container's stdout/stderr, Caddy's host access logs, and the app's structured
+  JSON logs (`errors`/`access`/`audit` `*.jsonl`, parsed so `level`/`tag`/`phase`
+  become labels). Every line is tagged with its node (`host = prod|beta|dev`).
+
+## Layout
+
+| Path | What |
+|------|------|
+| `docker-compose.yml` | the central Loki + Grafana stack (runs on beta) |
+| `loki/config.yml` | Loki config (filesystem, retention) |
+| `grafana/provisioning/` | auto-wired Loki datasource + dashboard provider |
+| `grafana/dashboards/thermograph-logs.json` | the fleet-logs dashboard |
+| `alloy/config.alloy` | the per-node collector config |
+| `alloy/docker-compose.agent.yml` | runs the Alloy agent on a node |
+| `caddy-grafana.conf` | beta Caddy site block for the Grafana UI |
+
+## Deploy
+
+### 1. Central stack (on beta)
+
+```bash
+# on beta (75.119.132.91):
+git clone <this repo> thermograph-observability && cd thermograph-observability
+cp .env.example .env      # set GF_SECURITY_ADMIN_PASSWORD
+docker compose up -d      # Loki on 10.10.0.2:3100 + 127.0.0.1:3100; Grafana on 127.0.0.1:3000
+```
+
+Expose the Grafana UI: point `grafana.thermograph.org` at beta, append
+`caddy-grafana.conf` to `/etc/caddy/Caddyfile`, `systemctl reload caddy`.
+
+### 2. Alloy agent (on every node — prod, beta, desktop)
+
+```bash
+cd thermograph-observability/alloy
+# beta pushes to local Loki; prod & desktop push over the mesh:
+#   beta:    LOKI_URL=http://127.0.0.1:3100/loki/api/v1/push  ALLOY_NODE=beta
+#   prod:    LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push   ALLOY_NODE=prod
+#   desktop: LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push   ALLOY_NODE=dev
+ALLOY_NODE=prod LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push \
+  docker compose -f docker-compose.agent.yml up -d
+```
+
+On the LAN dev node the app's log volume is `thermograph-dev_applogs` (not
+`thermograph_applogs`) — edit the two references in `docker-compose.agent.yml`
+there. The mesh must already be up (see the main repo's `deploy/swarm/` /
+`INFRA.md`); Swarm is **not** required — the agents talk plain HTTP over wg0.
+
+## Verify
+
+```bash
+# From any mesh node, confirm Loki is receiving from every host:
+curl -s 'http://10.10.0.2:3100/loki/api/v1/label/host/values'   # -> {"data":["beta","dev","prod"]}
+# Then open Grafana → the "Thermograph — Fleet Logs" dashboard.
+```
+
+## The dashboard
+
+`Thermograph — Fleet Logs` (auto-provisioned), with a `Node` selector:
+log volume by service, app error rate by tag, upstream 429 count, Caddy 5xx
+count, a notifier-liveness log panel, recent app errors, and a live all-container
+tail. Edit it in Grafana and re-export the JSON here to version a change.
+
+## Notes / follow-ups
+
+- **Loki is mesh-only** by design; only Grafana (with auth) is public.
+- Alloy reads the Docker socket read-only to discover containers.
+- Retention is 30 days on beta's disk — bump `loki/config.yml` if you want more.
+- Possible next step: have the app emit a periodic `notifier heartbeat` log line
+  so notifier liveness is a first-class log-derived signal, not just container
+  chatter.
--- a/alloy/config.alloy
+++ b/alloy/config.alloy
@ -0,0 +1,96 @@
+// Grafana Alloy — the log collector that runs on EVERY node (prod, beta, LAN
+// dev). It gathers three sources and ships them to the central Loki on beta:
+//
+//   1. Every Docker container's stdout/stderr (app, db, and on beta forgejo)
+//      via the Docker socket.
+//   2. Caddy's host access logs (/var/log/caddy/*.log) — the reverse proxy runs
+//      on the host, not in a container, so its logs aren't in Docker.
+//   3. The app's structured JSON logs (errors/access/audit *.jsonl) from the
+//      `applogs` Docker volume, parsed so `level`/`tag`/`phase` become labels.
+//
+// Per-node settings come from the environment (see docker-compose.agent.yml):
+//   ALLOY_NODE  — this node's name label (prod | beta | dev)
+//   LOKI_URL    — where to push (http://10.10.0.2:3100/loki/api/v1/push over wg0)
+
+livedebugging { enabled = false }
+
+// --- 1. All Docker container logs ------------------------------------------------
+discovery.docker "containers" {
+	host = "unix:///var/run/docker.sock"
+}
+
+// Turn Docker metadata into tidy labels: `container` (short name) and `service`
+// (the compose service, e.g. app/db). Drop Alloy's own container to avoid a loop.
+discovery.relabel "containers" {
+	targets = discovery.docker.containers.targets
+
+	rule {
+		source_labels = ["__meta_docker_container_name"]
+		regex         = "/(.*)"
+		target_label  = "container"
+	}
+	rule {
+		source_labels = ["__meta_docker_container_label_com_docker_compose_service"]
+		target_label  = "service"
+	}
+	rule {
+		source_labels = ["container"]
+		regex         = ".*alloy.*"
+		action        = "drop"
+	}
+}
+
+loki.source.docker "containers" {
+	host          = "unix:///var/run/docker.sock"
+	targets       = discovery.relabel.containers.output
+	forward_to    = [loki.write.central.receiver]
+	relabel_rules = discovery.relabel.containers.rules
+	labels        = { job = "docker" }
+}
+
+// --- 2. Caddy host access logs ---------------------------------------------------
+local.file_match "caddy" {
+	path_targets = [{ __path__ = "/var/log/caddy/*.log", job = "caddy" }]
+}
+
+loki.source.file "caddy" {
+	targets    = local.file_match.caddy.targets
+	forward_to = [loki.write.central.receiver]
+}
+
+// --- 3. App structured JSON logs (errors / access / audit) -----------------------
+// Mounted read-only from the app's `applogs` volume at /applogs (see the agent
+// compose). Lift `level`/`tag`/`phase` out of the JSON so they're queryable.
+local.file_match "app_jsonl" {
+	path_targets = [{ __path__ = "/applogs/**/*.jsonl", job = "app-json" }]
+}
+
+loki.source.file "app_jsonl" {
+	targets    = local.file_match.app_jsonl.targets
+	forward_to = [loki.process.app_jsonl.receiver]
+}
+
+loki.process "app_jsonl" {
+	forward_to = [loki.write.central.receiver]
+
+	stage.json {
+		expressions = { level = "level", tag = "tag", phase = "phase", status = "status" }
+	}
+	// A record with no explicit level: an error-folder line is an error, else info.
+	stage.static_labels {
+		values = { source = "app" }
+	}
+	stage.labels {
+		values = { level = "", tag = "", phase = "" }
+	}
+}
+
+// --- Ship to the central Loki over the WireGuard mesh ----------------------------
+loki.write "central" {
+	endpoint {
+		url = sys.env("LOKI_URL")
+	}
+	// Every line from this node is stamped with its node name, so one Grafana
+	// view can slice prod vs beta vs dev.
+	external_labels = { host = sys.env("ALLOY_NODE") }
+}
--- a/alloy/docker-compose.agent.yml
+++ b/alloy/docker-compose.agent.yml
@ -0,0 +1,48 @@
+# The Alloy log-collector agent. Runs on EACH node (prod, beta, LAN dev) — one
+# per node — reading that node's Docker containers, Caddy logs, and the app's
+# JSON logs, and shipping them to the central Loki on beta over the mesh.
+#
+# Per-node config comes from the environment (an .env file next to this, or
+# exported before `up`):
+#   ALLOY_NODE   this node's label:  prod | beta | dev
+#   LOKI_URL     push endpoint. On beta:  http://127.0.0.1:3100/loki/api/v1/push
+#                Elsewhere (over wg0):    http://10.10.0.2:3100/loki/api/v1/push
+#
+# Deploy on a node:
+#   ALLOY_NODE=prod LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push \
+#     docker compose -f docker-compose.agent.yml up -d
+#
+# The app's JSON logs are read from its `applogs` Docker volume, named
+# `<project>_applogs`. prod and beta run the app under the compose project
+# `thermograph`, so the external volume below is `thermograph_applogs`. The LAN
+# dev stack uses the project `thermograph-dev` — on that node, change the two
+# `thermograph_applogs` references below to `thermograph-dev_applogs`.
+
+services:
+  alloy:
+    image: grafana/alloy:v1.9.1
+    command:
+      - run
+      - /etc/alloy/config.alloy
+      - --storage.path=/var/lib/alloy/data
+      - --server.http.listen-addr=0.0.0.0:12345
+    environment:
+      ALLOY_NODE: ${ALLOY_NODE:?set ALLOY_NODE (prod|beta|dev)}
+      LOKI_URL: ${LOKI_URL:?set LOKI_URL}
+    volumes:
+      - ./config.alloy:/etc/alloy/config.alloy:ro
+      # Read-only Docker socket: discover + tail every container's stdout/stderr.
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+      # Caddy's host access logs (runs on the host, not a container).
+      - /var/log/caddy:/var/log/caddy:ro
+      # The app's structured JSON logs, straight off its named volume.
+      - thermograph_applogs:/applogs:ro
+      - alloy_data:/var/lib/alloy/data
+    restart: unless-stopped
+
+volumes:
+  # The app's existing log volume (created by the app's own stack). external:true
+  # means compose references it, never creates or deletes it.
+  thermograph_applogs:
+    external: true
+  alloy_data: {}
--- a/caddy-grafana.conf
+++ b/caddy-grafana.conf
@ -0,0 +1,15 @@
+# Append to beta's /etc/caddy/Caddyfile (beta is the monitoring host), then
+# `systemctl reload caddy`. Grafana publishes to 127.0.0.1:3000; this fronts it
+# with TLS, same pattern as beta's other site blocks. Grafana owns its own login,
+# so the UI is safe to expose publicly — but Loki (the log store, on
+# 10.10.0.2:3100) is NOT proxied here and stays mesh-only.
+#
+# Point grafana.thermograph.org's A record at beta (75.119.132.91) first.
+
+grafana.thermograph.org {
+	encode zstd gzip
+	reverse_proxy 127.0.0.1:3000
+	log {
+		output file /var/log/caddy/grafana.log
+	}
+}
--- a/docker-compose.yml
+++ b/docker-compose.yml
@ -0,0 +1,51 @@
+# The central observability stack: Loki (log store) + Grafana (UI). Runs on ONE
+# node — the monitoring host, `beta` (75.119.132.91 / mesh 10.10.0.2) — because
+# beta is an always-on VPS with a public Caddy already fronting it. Every node's
+# Alloy agent (see alloy/) ships logs here; agents on prod and the desktop reach
+# Loki over the WireGuard mesh, so Loki must listen on the mesh interface.
+#
+# Deploy (on beta):   docker compose up -d
+# Grafana is proxied by beta's Caddy at grafana.thermograph.org (see README);
+# Loki is NOT public — it binds the mesh IP only, reachable to agents over wg0.
+
+services:
+  loki:
+    image: grafana/loki:3.5.1
+    command: -config.file=/etc/loki/config.yml
+    volumes:
+      - ./loki/config.yml:/etc/loki/config.yml:ro
+      - loki_data:/loki
+    ports:
+      # Mesh-only: agents on prod (10.10.0.1) and desktop (10.10.0.3) push here
+      # over wg0. Never published on the public interface.
+      - "10.10.0.2:3100:3100"
+      # Also localhost, so the beta-local Alloy agent and curl checks can reach it.
+      - "127.0.0.1:3100:3100"
+    restart: unless-stopped
+
+  grafana:
+    image: grafana/grafana:11.6.1
+    depends_on: [loki]
+    environment:
+      # Grafana owns its own auth. Admin password is injected from the host env
+      # (set GF_SECURITY_ADMIN_PASSWORD before `up`, or in an .env file — see
+      # .env.example); never bake a credential into the compose file.
+      GF_SECURITY_ADMIN_USER: ${GF_ADMIN_USER:-admin}
+      GF_SECURITY_ADMIN_PASSWORD: ${GF_SECURITY_ADMIN_PASSWORD:?set GF_SECURITY_ADMIN_PASSWORD}
+      GF_USERS_ALLOW_SIGN_UP: "false"
+      GF_ANALYTICS_REPORTING_ENABLED: "false"
+      GF_ANALYTICS_CHECK_FOR_UPDATES: "false"
+      # Served behind Caddy at this external URL (sub-path-safe cookie/redirects).
+      GF_SERVER_ROOT_URL: "https://${GRAFANA_DOMAIN:-grafana.thermograph.org}/"
+    volumes:
+      - ./grafana/provisioning:/etc/grafana/provisioning:ro
+      - ./grafana/dashboards:/var/lib/grafana/dashboards:ro
+      - grafana_data:/var/lib/grafana
+    ports:
+      # Host-local; beta's Caddy reverse-proxies to it. Not public directly.
+      - "127.0.0.1:3000:3000"
+    restart: unless-stopped
+
+volumes:
+  loki_data: {}
+  grafana_data: {}
--- a/grafana/dashboards/thermograph-logs.json
+++ b/grafana/dashboards/thermograph-logs.json
@ -0,0 +1,132 @@
+{
+  "annotations": { "list": [] },
+  "editable": true,
+  "graphTooltip": 1,
+  "title": "Thermograph — Fleet Logs",
+  "uid": "thermograph-logs",
+  "tags": ["thermograph"],
+  "timezone": "",
+  "schemaVersion": 39,
+  "refresh": "30s",
+  "time": { "from": "now-6h", "to": "now" },
+  "templating": {
+    "list": [
+      {
+        "name": "host",
+        "type": "custom",
+        "label": "Node",
+        "query": "prod,beta,dev",
+        "current": { "text": "All", "value": "$__all" },
+        "includeAll": true,
+        "allValue": ".+",
+        "multi": true
+      }
+    ]
+  },
+  "panels": [
+    {
+      "type": "timeseries",
+      "title": "Log volume by service",
+      "description": "Lines/sec from every container, split by compose service (app, db, forgejo…).",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 },
+      "datasource": { "type": "loki", "uid": "loki" },
+      "fieldConfig": { "defaults": { "custom": { "drawStyle": "line", "fillOpacity": 15, "stacking": { "mode": "normal" } } } },
+      "targets": [
+        {
+          "datasource": { "type": "loki", "uid": "loki" },
+          "expr": "sum by (service) (count_over_time({job=\"docker\", host=~\"$host\"} [$__auto]))",
+          "legendFormat": "{{service}}"
+        }
+      ]
+    },
+    {
+      "type": "timeseries",
+      "title": "App errors/sec (structured logs)",
+      "description": "count_over_time of the app's errors-*.jsonl lines, by tag.",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 },
+      "datasource": { "type": "loki", "uid": "loki" },
+      "fieldConfig": { "defaults": { "custom": { "drawStyle": "bars", "fillOpacity": 60 }, "color": { "mode": "palette-classic" } } },
+      "targets": [
+        {
+          "datasource": { "type": "loki", "uid": "loki" },
+          "expr": "sum by (tag) (count_over_time({job=\"app-json\", source=\"app\", host=~\"$host\"} | level=~\"error|warn\" [$__auto]))",
+          "legendFormat": "{{tag}}"
+        }
+      ]
+    },
+    {
+      "type": "stat",
+      "title": "Upstream rate-limits (429) — last 1h",
+      "description": "Open-Meteo / upstream 429s pulled from the app's error logs.",
+      "gridPos": { "h": 4, "w": 6, "x": 0, "y": 8 },
+      "datasource": { "type": "loki", "uid": "loki" },
+      "fieldConfig": { "defaults": { "color": { "mode": "thresholds" }, "thresholds": { "steps": [ { "color": "green", "value": null }, { "color": "yellow", "value": 1 }, { "color": "red", "value": 50 } ] } } },
+      "options": { "reduceOptions": { "calcs": ["sum"] }, "colorMode": "background" },
+      "targets": [
+        {
+          "datasource": { "type": "loki", "uid": "loki" },
+          "expr": "sum (count_over_time({job=\"app-json\", host=~\"$host\"} | status=\"429\" [1h]))",
+          "instant": true
+        }
+      ]
+    },
+    {
+      "type": "stat",
+      "title": "Caddy 5xx — last 1h",
+      "description": "Server errors seen by the reverse proxy across all sites.",
+      "gridPos": { "h": 4, "w": 6, "x": 6, "y": 8 },
+      "datasource": { "type": "loki", "uid": "loki" },
+      "fieldConfig": { "defaults": { "color": { "mode": "thresholds" }, "thresholds": { "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 1 } ] } } },
+      "options": { "reduceOptions": { "calcs": ["sum"] }, "colorMode": "background" },
+      "targets": [
+        {
+          "datasource": { "type": "loki", "uid": "loki" },
+          "expr": "sum (count_over_time({job=\"caddy\", host=~\"$host\"} | json | status>=500 [1h]))",
+          "instant": true
+        }
+      ]
+    },
+    {
+      "type": "logs",
+      "title": "Subscription notifier",
+      "description": "Everything the app logs mentioning the notifier — its liveness at a glance.",
+      "gridPos": { "h": 4, "w": 12, "x": 12, "y": 8 },
+      "datasource": { "type": "loki", "uid": "loki" },
+      "options": { "showTime": true, "wrapLogMessage": true, "dedupStrategy": "exact", "sortOrder": "Descending" },
+      "targets": [
+        {
+          "datasource": { "type": "loki", "uid": "loki" },
+          "expr": "{job=\"docker\", service=\"app\", host=~\"$host\"} |~ \"(?i)notifier\""
+        }
+      ]
+    },
+    {
+      "type": "logs",
+      "title": "Recent app errors",
+      "description": "The app's errors-*.jsonl, newest first.",
+      "gridPos": { "h": 9, "w": 12, "x": 0, "y": 12 },
+      "datasource": { "type": "loki", "uid": "loki" },
+      "options": { "showTime": true, "wrapLogMessage": true, "sortOrder": "Descending", "enableLogDetails": true },
+      "targets": [
+        {
+          "datasource": { "type": "loki", "uid": "loki" },
+          "expr": "{job=\"app-json\", host=~\"$host\"} | level=~\"error|warn\""
+        }
+      ]
+    },
+    {
+      "type": "logs",
+      "title": "Live tail — all containers",
+      "description": "Raw stdout/stderr from every container on the selected node(s).",
+      "gridPos": { "h": 9, "w": 12, "x": 12, "y": 12 },
+      "datasource": { "type": "loki", "uid": "loki" },
+      "options": { "showTime": true, "wrapLogMessage": false, "sortOrder": "Descending", "enableLogDetails": true },
+      "targets": [
+        {
+          "datasource": { "type": "loki", "uid": "loki" },
+          "expr": "{job=\"docker\", host=~\"$host\"}"
+        }
+      ]
+    }
+  ]
+}
--- a/grafana/provisioning/dashboards/provider.yml
+++ b/grafana/provisioning/dashboards/provider.yml
@ -0,0 +1,16 @@
+# Load any dashboard JSON dropped in /var/lib/grafana/dashboards (mounted from
+# ./grafana/dashboards) at startup, so the fleet overview is version-controlled
+# in this repo rather than hand-built in the UI.
+apiVersion: 1
+
+providers:
+  - name: thermograph
+    orgId: 1
+    folder: Thermograph
+    type: file
+    disableDeletion: false
+    editable: true
+    updateIntervalSeconds: 30
+    options:
+      path: /var/lib/grafana/dashboards
+      foldersFromFilesStructure: false
--- a/grafana/provisioning/datasources/loki.yml
+++ b/grafana/provisioning/datasources/loki.yml
@ -0,0 +1,13 @@
+# Auto-provision the Loki datasource so Grafana comes up already wired to it —
+# no click-ops. Loki runs in the same compose stack, reachable by service name.
+apiVersion: 1
+
+datasources:
+  - name: Loki
+    type: loki
+    access: proxy
+    url: http://loki:3100
+    uid: loki                 # stable UID so the provisioned dashboards can bind to it
+    isDefault: true
+    jsonData:
+      maxLines: 2000
--- a/loki/config.yml
+++ b/loki/config.yml
@ -0,0 +1,51 @@
+# Loki — the single log store for the whole Thermograph fleet (prod, beta, LAN
+# dev). Every node's Alloy agent pushes here over the WireGuard mesh; Grafana
+# reads from here. Single-binary, filesystem-backed: no object storage, no
+# clustering — right for a three-node hobby fleet, and everything lives on one
+# volume you can back up or blow away.
+auth_enabled: false
+
+server:
+  http_listen_port: 3100
+  grpc_listen_port: 9096
+  log_level: warn
+
+common:
+  instance_addr: 127.0.0.1
+  path_prefix: /loki
+  storage:
+    filesystem:
+      chunks_directory: /loki/chunks
+      rules_directory: /loki/rules
+  replication_factor: 1
+  ring:
+    kvstore:
+      store: inmemory
+
+schema_config:
+  configs:
+    - from: 2024-01-01
+      store: tsdb
+      object_store: filesystem
+      schema: v13
+      index:
+        prefix: index_
+        period: 24h
+
+limits_config:
+  # A hobby fleet's volume is tiny; keep 30 days and cap ingestion generously.
+  retention_period: 720h
+  reject_old_samples: true
+  reject_old_samples_max_age: 168h
+  max_query_series: 5000
+  allow_structured_metadata: true
+  volume_enabled: true
+
+compactor:
+  working_directory: /loki/compactor
+  retention_enabled: true
+  delete_request_store: filesystem
+
+# No analytics phone-home from a private box.
+analytics:
+  reporting_enabled: false