docs: rewrite the agent context layer to match the live system
The four domain CLAUDE.md files still described the pre-monorepo split-repo
topology, and several statements were the exact inverse of current reality:
infra/CLAUDE.md told a reader that emi/thermograph is archived and must not be
pointed at, when that is the live repo; backend/ and frontend/ both claimed
there is no Makefile and no requirements-dev.txt when all four files exist;
frontend/ described itself as one of four sibling repos; observability/ claimed
a single unprotected main branch.
These files are read before every change, so a stale one is a correctness
problem rather than a documentation one. Rewritten against the tree:
- Root CLAUDE.md now owns the cross-cutting truth once — branch model, the
SERVICE + *_IMAGE_TAG deploy contract, image names, which orchestrator each
environment runs, and that everything is a PR. Domain files carry only what
differs and are capped at ~70 lines.
- Records that prod runs Swarm from deploy/stack/thermograph-stack.yml while
beta and LAN dev run compose, routed by /etc/thermograph/deploy-mode.
- Documents the *-deploy-dev.yml workflows as inert rather than leaving a
reader to discover it.
Deletes infra/docker-stack.yml. It defined db/app/worker, matched nothing that
deploys, and was referenced only by prose describing it as a future design
record — while the real 8-service prod stack lives under deploy/stack/. A file
that looks authoritative and affects nothing is the worst case for a reader
asked to change the prod stack.
infra/README.md claimed "compose in production today" and that Swarm was "not
currently live"; corrected, and infra-sync.yml's existence is now recorded
instead of "there is no separate infra deploy trigger". The compose file's
timescale-pin comment now points at how deploy-stack.sh actually resolves the
digest from the running container.
2026-07-25 03:59:44 +00:00
|
|
|
# observability/ — agent instructions
|
2026-07-22 18:47:21 +00:00
|
|
|
|
docs: rewrite the agent context layer to match the live system
The four domain CLAUDE.md files still described the pre-monorepo split-repo
topology, and several statements were the exact inverse of current reality:
infra/CLAUDE.md told a reader that emi/thermograph is archived and must not be
pointed at, when that is the live repo; backend/ and frontend/ both claimed
there is no Makefile and no requirements-dev.txt when all four files exist;
frontend/ described itself as one of four sibling repos; observability/ claimed
a single unprotected main branch.
These files are read before every change, so a stale one is a correctness
problem rather than a documentation one. Rewritten against the tree:
- Root CLAUDE.md now owns the cross-cutting truth once — branch model, the
SERVICE + *_IMAGE_TAG deploy contract, image names, which orchestrator each
environment runs, and that everything is a PR. Domain files carry only what
differs and are capped at ~70 lines.
- Records that prod runs Swarm from deploy/stack/thermograph-stack.yml while
beta and LAN dev run compose, routed by /etc/thermograph/deploy-mode.
- Documents the *-deploy-dev.yml workflows as inert rather than leaving a
reader to discover it.
Deletes infra/docker-stack.yml. It defined db/app/worker, matched nothing that
deploys, and was referenced only by prose describing it as a future design
record — while the real 8-service prod stack lives under deploy/stack/. A file
that looks authoritative and affects nothing is the worst case for a reader
asked to change the prod stack.
infra/README.md claimed "compose in production today" and that Swarm was "not
currently live"; corrected, and infra-sync.yml's existence is now recorded
instead of "there is no separate infra deploy trigger". The compose file's
timescale-pin comment now points at how deploy-stack.sh actually resolves the
digest from the running container.
2026-07-25 03:59:44 +00:00
|
|
|
The **logging stack** for the fleet: Loki + Grafana on beta, with a Grafana Alloy
|
|
|
|
|
agent on every node (prod, beta, dev) shipping container and app logs over the
|
|
|
|
|
WireGuard mesh. Grafana is fronted by beta's Caddy at
|
|
|
|
|
**`dashboard.thermograph.org`** (Google SSO, pre-provisioned users only) — use
|
|
|
|
|
that hostname everywhere, never `grafana.thermograph.org`.
|
2026-07-22 18:47:21 +00:00
|
|
|
|
docs: rewrite the agent context layer to match the live system
The four domain CLAUDE.md files still described the pre-monorepo split-repo
topology, and several statements were the exact inverse of current reality:
infra/CLAUDE.md told a reader that emi/thermograph is archived and must not be
pointed at, when that is the live repo; backend/ and frontend/ both claimed
there is no Makefile and no requirements-dev.txt when all four files exist;
frontend/ described itself as one of four sibling repos; observability/ claimed
a single unprotected main branch.
These files are read before every change, so a stale one is a correctness
problem rather than a documentation one. Rewritten against the tree:
- Root CLAUDE.md now owns the cross-cutting truth once — branch model, the
SERVICE + *_IMAGE_TAG deploy contract, image names, which orchestrator each
environment runs, and that everything is a PR. Domain files carry only what
differs and are capped at ~70 lines.
- Records that prod runs Swarm from deploy/stack/thermograph-stack.yml while
beta and LAN dev run compose, routed by /etc/thermograph/deploy-mode.
- Documents the *-deploy-dev.yml workflows as inert rather than leaving a
reader to discover it.
Deletes infra/docker-stack.yml. It defined db/app/worker, matched nothing that
deploys, and was referenced only by prose describing it as a future design
record — while the real 8-service prod stack lives under deploy/stack/. A file
that looks authoritative and affects nothing is the worst case for a reader
asked to change the prod stack.
infra/README.md claimed "compose in production today" and that Swarm was "not
currently live"; corrected, and infra-sync.yml's existence is now recorded
instead of "there is no separate infra deploy trigger". The compose file's
timescale-pin comment now points at how deploy-stack.sh actually resolves the
digest from the running container.
2026-07-25 03:59:44 +00:00
|
|
|
This is operational config, not application code: there is **no build** and no
|
|
|
|
|
deploy automation. It ships by hand — `docker compose up -d` on beta for
|
|
|
|
|
Loki+Grafana, and `alloy/docker-compose.agent.yml` on each node. Read the root
|
|
|
|
|
`CLAUDE.md` first; the branch model there applies here too.
|
2026-07-22 18:47:21 +00:00
|
|
|
|
|
|
|
|
## Layout
|
|
|
|
|
|
docs: rewrite the agent context layer to match the live system
The four domain CLAUDE.md files still described the pre-monorepo split-repo
topology, and several statements were the exact inverse of current reality:
infra/CLAUDE.md told a reader that emi/thermograph is archived and must not be
pointed at, when that is the live repo; backend/ and frontend/ both claimed
there is no Makefile and no requirements-dev.txt when all four files exist;
frontend/ described itself as one of four sibling repos; observability/ claimed
a single unprotected main branch.
These files are read before every change, so a stale one is a correctness
problem rather than a documentation one. Rewritten against the tree:
- Root CLAUDE.md now owns the cross-cutting truth once — branch model, the
SERVICE + *_IMAGE_TAG deploy contract, image names, which orchestrator each
environment runs, and that everything is a PR. Domain files carry only what
differs and are capped at ~70 lines.
- Records that prod runs Swarm from deploy/stack/thermograph-stack.yml while
beta and LAN dev run compose, routed by /etc/thermograph/deploy-mode.
- Documents the *-deploy-dev.yml workflows as inert rather than leaving a
reader to discover it.
Deletes infra/docker-stack.yml. It defined db/app/worker, matched nothing that
deploys, and was referenced only by prose describing it as a future design
record — while the real 8-service prod stack lives under deploy/stack/. A file
that looks authoritative and affects nothing is the worst case for a reader
asked to change the prod stack.
infra/README.md claimed "compose in production today" and that Swarm was "not
currently live"; corrected, and infra-sync.yml's existence is now recorded
instead of "there is no separate infra deploy trigger". The compose file's
timescale-pin comment now points at how deploy-stack.sh actually resolves the
digest from the running container.
2026-07-25 03:59:44 +00:00
|
|
|
- `docker-compose.yml` — the Loki + Grafana stack (beta).
|
|
|
|
|
- `loki/config.yml` — mesh-only, filesystem storage.
|
|
|
|
|
- `grafana/provisioning/` — datasource + dashboard provider, auto-loaded at
|
|
|
|
|
startup. `grafana/dashboards/*.json` — the dashboards.
|
|
|
|
|
- `grafana/provisioning/alerting/` — alert rules, the Discord contact point, and
|
|
|
|
|
the notification policy.
|
|
|
|
|
- `alloy/config.alloy` + `alloy/docker-compose.agent.yml` — the per-node shipper.
|
|
|
|
|
Node name comes from `ALLOY_NODE`, set per host.
|
|
|
|
|
- `caddy-grafana.conf` — the beta Caddy vhost, kept here for reference.
|
|
|
|
|
- `.env.example` — copy to `.env` on beta. `.env` is gitignored; never commit
|
|
|
|
|
OAuth secrets or the admin password.
|
|
|
|
|
|
|
|
|
|
## Rules
|
|
|
|
|
|
|
|
|
|
- **The repo is the only durable path.** Dashboards and alerting are provisioned
|
|
|
|
|
from this directory at startup; edits made through Grafana's UI or API are
|
|
|
|
|
overwritten on the next provision. Changes go through a PR.
|
observability: add the estate's first alerting; supervise Postfix
The estate had zero alert rules. The only contact point was Grafana's factory
default pointing at the literal string <example@email.com>, and beta's
untracked compose override routed Grafana's SMTP at prod's Postfix — so
alerts, had any existed, would have been delivered by the box most likely to
be on fire. Prod served 86 5xx in 24h including five /healthz failures and
nobody was told.
Alerting (deploys to beta, which is where Grafana runs):
- 12 Loki-based rules. There is no Prometheus in this estate and Loki is the
only datasource, so every rule is log-derived.
- Thresholds come from a 24h backtest that happens to contain a real ~20min
prod outage at 04:20Z. Absolute counts, not ratios: prod runs ~7 req/min, so
one bad request is 1.4% and a ratio alert would scream all night. The 5xx
burst rule fires on the outage's 45 and 22 buckets and on nothing else in
the day; the largest benign bucket all day was 5.
- Routed to a new private #ops-alerts channel, not to any existing channel —
#weather-events, #announcements and #prod are product surfaces that notify
real subscribers.
- AlertingWatchdog is a dead-man's switch; its value is its absence.
- Delivery is proven, not assumed: the rules were provisioned into a throwaway
Grafana against beta's live Loki and a real alert arrived in Discord. This
matters because GET /api/v1/provisioning/contact-points returns [REDACTED]
for the URL — a contact point holding an uninterpolated env var looks
perfectly healthy and pages nobody. The only proof is a message arriving.
CI gains a structural check, because the existing one only proves YAML parses:
an alert rule whose condition names a missing refId is valid YAML, provisions
cleanly, and never fires. It also hard-fails on a literal Discord webhook in
the repo. Verified against all three breakages deliberately introduced.
Postfix supervision:
- The 13h outage was a boot-ordering race, not a Docker renumbering: postfix
started at 08:09:49, wg0 came up at :51, postfix fataled at :52 on a missing
docker_gwbridge address, and dockerd did not finish starting until 08:10:16.
Stock postfix@.service is ordered only After=network-online.target and ships
no Restart=, so one lost race became a permanent outage.
- An ExecStartPre gate now blocks up to 60s until every inet_interfaces
address actually exists, which absorbs the transient case inside a single
start attempt. That makes bounded retry correct: 5 attempts in 600s, then
failed — a genuinely broken config reaches a visible failed state in ~100s
instead of re-fataling every 15s forever.
- A 5-minute watchdog timer retries indefinitely and runs reset-failed, so
"failed" still self-heals. Worst case is ~5 minutes, not 13 hours.
- Wants=, not Requires=: a dockerd failure must not take down the loopback and
mesh listeners that do not depend on Docker at all.
Health checks must read config with `postconf -c`, never postmulti/postqueue/
postfix — those three RESOLVE inet_interfaces and so fatal precisely when an
address is missing, which made the first version of this check report
status=ok bound=0/0. A health check that fails open is worse than none.
DEPLOY.md carries the monitoring contract, including that systemctl is-active
postfix is a known-false signal: postfix.service is a wrapper whose
ExecStart=/bin/true, so it reports active forever while the real postfix@-
instance is failed with zero listeners. Reproduced live.
Known gap, documented not fixed: Alloy ships Docker stdout, Caddy files and
app JSONL, not journald — so no Postfix line reaches Loki and the mail rules
cannot fire until loki.source.journal is added.
Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY
2026-07-24 20:19:19 +00:00
|
|
|
- **Every artifact that ships to a node is CI-validated**
|
docs: rewrite the agent context layer to match the live system
The four domain CLAUDE.md files still described the pre-monorepo split-repo
topology, and several statements were the exact inverse of current reality:
infra/CLAUDE.md told a reader that emi/thermograph is archived and must not be
pointed at, when that is the live repo; backend/ and frontend/ both claimed
there is no Makefile and no requirements-dev.txt when all four files exist;
frontend/ described itself as one of four sibling repos; observability/ claimed
a single unprotected main branch.
These files are read before every change, so a stale one is a correctness
problem rather than a documentation one. Rewritten against the tree:
- Root CLAUDE.md now owns the cross-cutting truth once — branch model, the
SERVICE + *_IMAGE_TAG deploy contract, image names, which orchestrator each
environment runs, and that everything is a PR. Domain files carry only what
differs and are capped at ~70 lines.
- Records that prod runs Swarm from deploy/stack/thermograph-stack.yml while
beta and LAN dev run compose, routed by /etc/thermograph/deploy-mode.
- Documents the *-deploy-dev.yml workflows as inert rather than leaving a
reader to discover it.
Deletes infra/docker-stack.yml. It defined db/app/worker, matched nothing that
deploys, and was referenced only by prose describing it as a future design
record — while the real 8-service prod stack lives under deploy/stack/. A file
that looks authoritative and affects nothing is the worst case for a reader
asked to change the prod stack.
infra/README.md claimed "compose in production today" and that Swarm was "not
currently live"; corrected, and infra-sync.yml's existence is now recorded
instead of "there is no separate infra deploy trigger". The compose file's
timescale-pin comment now points at how deploy-stack.sh actually resolves the
digest from the running container.
2026-07-25 03:59:44 +00:00
|
|
|
(`.forgejo/workflows/observability-validate.yml`, at the repo root): both
|
|
|
|
|
compose files parse, all dashboard JSON is valid, the Loki and provisioning
|
|
|
|
|
YAML parse, and the Alloy config is checked with the pinned `alloy` binary
|
|
|
|
|
(v1.9.1, matching what the fleet runs).
|
|
|
|
|
- **Alerting gets a stricter third step.** Every rule's `condition` must name a
|
|
|
|
|
refId that exists, every policy must route to a receiver that exists, and a
|
|
|
|
|
literal Discord webhook URL in the repo is a hard failure. A rule pointing at a
|
|
|
|
|
missing refId is valid YAML, provisions cleanly, and then never fires — that
|
|
|
|
|
silent no-op is what this check exists to prevent.
|
|
|
|
|
- **Alerts go to Discord `#ops-alerts`, never email.** Beta's Grafana relays SMTP
|
|
|
|
|
through prod's Postfix, so email dies exactly when prod does.
|
|
|
|
|
- **Every rule is LogQL** — there is no Prometheus anywhere in the fleet.
|
|
|
|
|
Thresholds were derived from real Loki data and the working is in the comments
|
|
|
|
|
beside each rule; re-derive before changing a number rather than guessing.
|
|
|
|
|
- Secrets (OAuth client id/secret, admin password) live only in the host `.env`.
|
|
|
|
|
|
|
|
|
|
## Commits & PRs
|
|
|
|
|
|
|
|
|
|
Concise and technical. Never mention AI, assistants or automated authorship.
|
