42 lines
2.1 KiB
Markdown
42 lines
2.1 KiB
Markdown
|
# thermograph-observability — agent instructions
|
||
|
|
|
||
|
|
The **logging/metrics stack** for the Thermograph fleet: Loki + Grafana on beta,
|
||
|
|
with a Grafana Alloy agent on every node (prod, beta, dev) shipping container and
|
||
|
|
app logs over the WireGuard mesh. Grafana is fronted by beta's Caddy at
|
||
|
|
`dashboard.thermograph.org` (Google SSO, pre-provisioned users only).
|
||
|
|
|
||
|
|
This is **operational infra config, not application code** — there is no build.
|
||
|
|
It deploys by hand: `docker compose up -d` on beta for Loki+Grafana, and the
|
||
|
|
Alloy agent (`alloy/docker-compose.agent.yml`) on each node. See the README for
|
||
|
|
the full per-node procedure.
|
||
|
|
|
||
|
|
## Layout
|
||
|
|
|
||
|
|
- `docker-compose.yml` — the Loki + Grafana stack (runs on beta).
|
||
|
|
- `loki/config.yml` — Loki config (mesh-only, filesystem storage).
|
||
|
|
- `grafana/provisioning/` — datasource (Loki) + dashboard provider, auto-loaded
|
||
|
|
at startup. `grafana/dashboards/*.json` — the dashboards themselves.
|
||
|
|
- `alloy/config.alloy` + `alloy/docker-compose.agent.yml` — the per-node log
|
||
|
|
shipper. The node name comes from `ALLOY_NODE` (set per host).
|
||
|
|
- `caddy-grafana.conf` — the Caddy vhost for `dashboard.thermograph.org` (lives
|
||
|
|
in beta's Caddy config, kept here for reference).
|
||
|
|
- `.env.example` — copy to `.env` on beta before `docker compose up`. `.env` is
|
||
|
|
gitignored; never commit real OAuth secrets or the admin password.
|
||
|
|
|
||
|
|
## Conventions
|
||
|
|
|
||
|
|
- **Every artifact that ships to a node is CI-validated** (`.forgejo/workflows/validate.yml`):
|
||
|
|
both compose files parse, all dashboard JSON is valid, and the Loki/provisioning
|
||
|
|
YAML parses. Keep new dashboards as valid JSON and new config as valid YAML or
|
||
|
|
CI fails. The Alloy config is not yet CI-validated (needs the `alloy` binary).
|
||
|
|
- The public hostname is **`dashboard.thermograph.org`** everywhere — not
|
||
|
|
`grafana.thermograph.org`. Match it in any new comment/config.
|
||
|
|
- Secrets (OAuth client id/secret, admin password) live only in the host `.env`,
|
||
|
|
never in the repo.
|
||
|
|
|
||
|
|
## Branching
|
||
|
|
|
||
|
|
Single `main` branch, no protection. Open a PR against `main`; the validator
|
||
|
|
gates it. Deploying the change to beta / the nodes is a separate manual step
|
||
|
|
(this repo has no deploy automation — a known gap).
|