d138f00a20
Some checks failed
Sync infra to hosts / sync-beta (push) Has been skipped
Sync infra to hosts / sync-prod (push) Has been skipped
Sync infra to hosts / sync-dev (push) Failing after 6s
secrets-guard / encrypted (push) Successful in 6s
shell-lint / shellcheck (push) Successful in 13s
Validate observability stack / validate (push) Successful in 17s
PR build (required check) / changes (pull_request) Successful in 6s
secrets-guard / encrypted (pull_request) Successful in 5s
PR build (required check) / build-backend (pull_request) Has been skipped
shell-lint / shellcheck (pull_request) Successful in 6s
PR build (required check) / build-frontend (pull_request) Has been skipped
PR build (required check) / validate-observability (pull_request) Successful in 18s
PR build (required check) / gate (pull_request) Successful in 2s
212 lines
11 KiB
Markdown
212 lines
11 KiB
Markdown
# 9. Observability
|
|
|
|
Fleet-wide log aggregation: **Loki + Grafana on vps1**, fed by a **Grafana
|
|
Alloy** agent on every node, all over the WireGuard mesh. vps2 runs two
|
|
environments (beta, prod) behind that one Alloy agent; vps1 runs Alloy
|
|
alongside the things it's shipping logs *from* (Forgejo, Grafana itself, dev).
|
|
|
|
```
|
|
vps2 (10.10.0.1) vps1 (10.10.0.2) desktop (10.10.0.3)
|
|
prod + beta Loki, Grafana, dev
|
|
┌────────────┐ ┌──────────────────┐ ┌────────────┐
|
|
│ Alloy agent│──wg0──┐ │ Loki ◀── Alloy │ ┌───│ Alloy agent│
|
|
└────────────┘ └──▶│ Grafana (Caddy) │◀─┘ └────────────┘
|
|
docker+caddy+app └──────────────────┘ docker (AI models)
|
|
```
|
|
|
|
- **Loki** — filesystem storage, **30-day retention**, listening on the mesh IP
|
|
`10.10.0.2:3100`. Never public.
|
|
- **Grafana** — the UI, fronted by vps1's Caddy at
|
|
**`dashboard.thermograph.org`** (Google SSO, pre-provisioned accounts only,
|
|
`allow_sign_up=false`, plus a break-glass local admin). Use that hostname
|
|
everywhere; **never** `grafana.thermograph.org`.
|
|
- **Alloy** — one agent per *host*, not per environment: vps2's single agent
|
|
ships logs for both beta and prod, vps1's ships Forgejo/Grafana/dev, and the
|
|
desktop's ships whatever runs there now. Each ships three sources: every
|
|
Docker container's stdout/stderr, Caddy's host access logs, and the app's
|
|
structured JSON logs (`errors`/`access`/`audit` `*.jsonl`, parsed so
|
|
`level`/`tag`/`phase` become labels). Every app log line is tagged
|
|
`host = prod|beta|dev` — that label is the *environment*, not the box, which
|
|
matters on vps2 where one Alloy agent ships lines carrying two different
|
|
`host` values.
|
|
|
|
There is **no build and no deploy automation** for this domain — it ships by
|
|
hand. But `observability-validate.yml` parses every artifact that ships to a
|
|
node, so a malformed dashboard or an unparseable Alloy config fails in CI
|
|
instead of on the host.
|
|
|
|
## The rule that matters most
|
|
|
|
**The repo is the only durable path.** Dashboards and alerting are provisioned
|
|
from `observability/grafana/provisioning/` at startup, so **edits made through
|
|
Grafana's UI or API are overwritten on the next provision**. A change goes
|
|
through a PR. Centralis's `dashboard_write` opens one for you.
|
|
|
|
## Reading the fleet
|
|
|
|
Start wide, then narrow:
|
|
|
|
1. `fleet_status` — HTTP reachability plus what's running on each host.
|
|
2. `logs_overview since="1h"` — where the volume is, and what has gone quiet.
|
|
3. `logs_query service=… host=…` — the actual lines.
|
|
|
|
### ⚠ The prod (and now beta) trap
|
|
|
|
Prod and beta both run Swarm stacks — co-resident on vps2 — and their
|
|
`job="docker"` streams carry **no `service` label**. A raw
|
|
`{host="prod", service="backend"}` query matches nothing and reads as *"no
|
|
logs"* when it actually means *"wrong selector"*. This used to be a prod-only
|
|
trap when beta ran compose; it now applies to both environments on vps2, and
|
|
dev (compose, on vps1) is the one environment this does *not* affect.
|
|
|
|
Use Centralis's `logs_query` `service` argument, which rewrites it to a
|
|
container name-regex with the churning task suffix wildcarded. Prod service
|
|
names are `thermograph_web`, `_worker`, `_frontend`, `_db`, `_autoscaler`,
|
|
`_lake`; beta's are the same names prefixed `beta-` (`beta-web`, `beta-worker`,
|
|
…) with no `_db` or autoscaler equivalents.
|
|
|
|
Related: **prod and beta container names both change on every redeploy** (the
|
|
Swarm task suffix). Never hardcode one; resolve via `docker ps --filter name=`
|
|
at call time.
|
|
|
|
## The dashboard
|
|
|
|
`Thermograph — Fleet Logs` (auto-provisioned), with a `Node` selector: log
|
|
volume by service, app error rate by tag, upstream 429 count, Caddy 5xx count, a
|
|
per-node notifier-liveness indicator, recent app errors, and a live
|
|
all-container tail.
|
|
|
|
## Alerting
|
|
|
|
Provisioned from `grafana/provisioning/alerting/`, mounted read-only into
|
|
Grafana.
|
|
|
|
**Every rule is LogQL** — there is no Prometheus anywhere in the fleet. Loki is
|
|
the only datasource, so every signal is derived from log lines. Thresholds were
|
|
backtested against real Loki data and the working is written out in the comments
|
|
beside each rule; **re-derive before changing a number rather than guessing**.
|
|
|
|
### Alerts go to Discord `#ops-alerts`, never email
|
|
|
|
Three reasons, all learned the hard way:
|
|
|
|
- Grafana's factory default pointed at the literal string
|
|
`<example@email.com>` — a paging path that had never delivered a message to
|
|
anyone.
|
|
- vps1's Grafana relays SMTP through **vps2's** Postfix (prod's mail service).
|
|
Email alerts therefore travel through the box most likely to be on fire, and
|
|
vanish exactly when they matter.
|
|
- Discord is off-estate, works when prod is dead, and reaches a phone.
|
|
|
|
`#ops-alerts` is dedicated. `#dev` / `#uat` / `#prod` are the *product's* alert
|
|
subscription output — real deliveries to real subscribers — so routing ops noise
|
|
there would corrupt the evidence.
|
|
|
|
The webhook URL is a secret and lives only in vps1's gitignored `.env` as
|
|
`DISCORD_ALERT_WEBHOOK_URL`. Grafana refuses to start if it's unset. A literal
|
|
webhook URL committed to the repo is a **hard CI failure**.
|
|
|
|
### The twelve rules
|
|
|
|
| Rule | Fires when | Severity |
|
|
|---|---|---|
|
|
| `ProdEdgeDark` | zero prod Caddy log lines in 15m — site dark, or the pipeline is broken | critical |
|
|
| `ProdHealthzFailing` | any non-200 on `/healthz` in 10m | critical |
|
|
| `ProdHTTP5xxBurst` | ≥10 prod 5xx in 10m | critical |
|
|
| `ProdHTTP5xxElevated` | ≥25 prod 5xx over 6h (the error-budget alarm) | warning |
|
|
| `ProdUnhandledExceptions` | ≥3 app `tag=http.error` in 30m | warning |
|
|
| `NotifierHeartbeatMissingProd` | no `tag=heartbeat` from prod in 20m | critical |
|
|
| `NotifierHeartbeatMissingBeta` | same, on beta | warning |
|
|
| `ProdWebContainerSilent` | `thermograph_web` logged nothing in 30m | critical |
|
|
| `ProdDbContainerSilent` | `thermograph_db` logged nothing in 2h | critical |
|
|
| `ProdFrontendContainerSilent` | `thermograph_frontend` logged nothing in 30m | warning |
|
|
| `ProdWorkerContainerSilent` | `thermograph_worker` logged nothing in 2h | warning |
|
|
| `AlertingWatchdog` | always — one Discord message per day | watchdog |
|
|
|
|
`AlertingWatchdog` is the dead-man's switch and fires permanently by design.
|
|
**Its value is its absence:** nothing can report its own death, so if
|
|
`#ops-alerts` has been silent for more than a day, alerting is broken rather
|
|
than the estate healthy.
|
|
|
|
### The strict CI check, and why
|
|
|
|
A rule whose `condition` names a refId that does not exist is **perfectly valid
|
|
YAML, provisions without complaint, and then never fires** — a paging path that
|
|
looks healthy and reaches nobody. `observability-validate.yml` checks that every
|
|
condition names a real refId and every policy routes to a receiver that exists,
|
|
precisely because nothing else would catch it.
|
|
|
|
The same asymmetry shows up in verification: `GET
|
|
/api/v1/provisioning/contact-points` **redacts** the webhook URL, so you cannot
|
|
confirm interpolation by reading it back. A contact point holding the literal
|
|
string `$DISCORD_ALERT_WEBHOOK_URL` looks completely healthy from the API and
|
|
pages nobody. **The only proof is a message actually arriving in
|
|
`#ops-alerts`** — force one with the receivers-test endpoint (the command is in
|
|
`observability/README.md`).
|
|
|
|
## Deploying an observability change
|
|
|
|
> ⚠️ **Merging is not deploying — and this one has already bitten.**
|
|
> `/opt/observability` on vps1 and vps2 is a checkout of the **archived**
|
|
> `emi/thermograph-observability` repo. It can never `git pull` again; the
|
|
> content now lives here under `observability/`. A previous observability PR
|
|
> merged and had **zero live effect** for exactly this reason.
|
|
|
|
Until someone re-points those checkouts at the monorepo, every change ships by
|
|
hand: back up the live file with a UTC-timestamped suffix, `scp` the new one up,
|
|
`sudo cp` it into place, restart the specific container.
|
|
|
|
Two gotchas in that sequence:
|
|
|
|
- Use `sudo docker restart <container>` on **vps2** — not `docker compose`,
|
|
which there demands a `GF_SECURITY_ADMIN_PASSWORD` it has no `.env` to
|
|
interpolate from. vps1's stack *does* have an `.env`, so compose works there.
|
|
- **`docker restart` will not pick up a new environment variable.** Any change
|
|
that introduces one needs `docker compose up -d grafana` on vps1.
|
|
|
|
Alerting lives on **vps1 only** — vps2 (prod and beta alike) and dev run Alloy
|
|
agents, not Grafana.
|
|
|
|
Full step-by-step, including rollback, is in
|
|
[`observability/README.md`](../../observability/README.md).
|
|
|
|
## Known gaps (as documented in the repo)
|
|
|
|
- **Alloy can silently stop tailing a container after Swarm replaces the task.**
|
|
Observed on prod 2026-07-24: `thermograph_worker` was `1/1`, healthy, writing
|
|
~19 lines per 10 minutes to its own docker log, yet not one line reached Loki
|
|
for 13½ hours. What localises it to Alloy: prod's `tag=heartbeat` lines kept
|
|
arriving throughout, and those come from a *different* Alloy source
|
|
(`loki.source.file` on `/applogs`) than container stdout
|
|
(`loki.source.docker`). One source went deaf, the other didn't. Workaround:
|
|
`sudo docker restart alloy-alloy-1` on the node. **Not root-caused.**
|
|
`ProdWorkerContainerSilent` now catches this class of failure.
|
|
- **Some services are too quiet to alert on.** `thermograph_daemon` (23 lines /
|
|
24h), `thermograph_autoscaler` (1) and `_autoscaler-lake` (2) have a healthy
|
|
floor indistinguishable from dead; `thermograph_lake` is bursty ETL with
|
|
legitimate idle gaps. Giving each an `audit.log_heartbeat()` line would make
|
|
all four alertable with the pattern the notifier already uses. **The single
|
|
highest-value follow-up in this domain.**
|
|
- **Grafana cannot alert on its own death.** `AlertingWatchdog` is a *manual*
|
|
dead-man's switch — it relies on somebody noticing the daily message stopped.
|
|
An external uptime pinger from off-estate would close this properly.
|
|
- **vps1 runs a `docker-compose.override.yml` that is not in this repo**, wiring
|
|
Grafana's SMTP to vps2's Postfix (prod's). It should be mirrored into the
|
|
tracked compose file or deleted; alerting no longer depends on it.
|
|
|
|
## When something breaks
|
|
|
|
1. `fleet_status` — is it up at all?
|
|
2. `logs_query host=prod contains="(?i)error|traceback"` since the incident.
|
|
3. `deployed_version env=prod` — did something ship just before it started?
|
|
4. `forge_prs` / `forge_ci` — what landed recently, and did CI actually pass?
|
|
5. **Write down what you found with `notes_write` *before* fixing it.** The
|
|
reconstruction is worth more than the fix, and it will be gone by tomorrow.
|
|
|
|
You are authorised to fix production roadblocks directly — restart or roll a
|
|
wedged service, clear a full disk, kill a runaway process, bounce a crash-looping
|
|
container. What you must not do is make the fix *durable* on the box: writes to
|
|
code, docs and dashboards go through a PR, and `deploy.sh` hard-resets the
|
|
checkout anyway. The on-box action is stabilisation; the PR is the fix.
|
|
|
|
Next: [Recipes](10-recipes.md).
|