3 commits

Author	SHA1	Message	Date
Emi Griffith	c98512cfcc	docs: rewrite the agent context layer to match the live system ... The four domain CLAUDE.md files still described the pre-monorepo split-repo topology, and several statements were the exact inverse of current reality: infra/CLAUDE.md told a reader that emi/thermograph is archived and must not be pointed at, when that is the live repo; backend/ and frontend/ both claimed there is no Makefile and no requirements-dev.txt when all four files exist; frontend/ described itself as one of four sibling repos; observability/ claimed a single unprotected main branch. These files are read before every change, so a stale one is a correctness problem rather than a documentation one. Rewritten against the tree: - Root CLAUDE.md now owns the cross-cutting truth once — branch model, the SERVICE + *_IMAGE_TAG deploy contract, image names, which orchestrator each environment runs, and that everything is a PR. Domain files carry only what differs and are capped at ~70 lines. - Records that prod runs Swarm from deploy/stack/thermograph-stack.yml while beta and LAN dev run compose, routed by /etc/thermograph/deploy-mode. - Documents the *-deploy-dev.yml workflows as inert rather than leaving a reader to discover it. Deletes infra/docker-stack.yml. It defined db/app/worker, matched nothing that deploys, and was referenced only by prose describing it as a future design record — while the real 8-service prod stack lives under deploy/stack/. A file that looks authoritative and affects nothing is the worst case for a reader asked to change the prod stack. infra/README.md claimed "compose in production today" and that Swarm was "not currently live"; corrected, and infra-sync.yml's existence is now recorded instead of "there is no separate infra deploy trigger". The compose file's timescale-pin comment now points at how deploy-stack.sh actually resolves the digest from the running container.	2026-07-24 20:59:44 -07:00
Emi Griffith	e4693dce58	observability: add the estate's first alerting; supervise Postfix ... The estate had zero alert rules. The only contact point was Grafana's factory default pointing at the literal string <example@email.com>, and beta's untracked compose override routed Grafana's SMTP at prod's Postfix — so alerts, had any existed, would have been delivered by the box most likely to be on fire. Prod served 86 5xx in 24h including five /healthz failures and nobody was told. Alerting (deploys to beta, which is where Grafana runs): - 12 Loki-based rules. There is no Prometheus in this estate and Loki is the only datasource, so every rule is log-derived. - Thresholds come from a 24h backtest that happens to contain a real ~20min prod outage at 04:20Z. Absolute counts, not ratios: prod runs ~7 req/min, so one bad request is 1.4% and a ratio alert would scream all night. The 5xx burst rule fires on the outage's 45 and 22 buckets and on nothing else in the day; the largest benign bucket all day was 5. - Routed to a new private #ops-alerts channel, not to any existing channel — #weather-events, #announcements and #prod are product surfaces that notify real subscribers. - AlertingWatchdog is a dead-man's switch; its value is its absence. - Delivery is proven, not assumed: the rules were provisioned into a throwaway Grafana against beta's live Loki and a real alert arrived in Discord. This matters because GET /api/v1/provisioning/contact-points returns [REDACTED] for the URL — a contact point holding an uninterpolated env var looks perfectly healthy and pages nobody. The only proof is a message arriving. CI gains a structural check, because the existing one only proves YAML parses: an alert rule whose condition names a missing refId is valid YAML, provisions cleanly, and never fires. It also hard-fails on a literal Discord webhook in the repo. Verified against all three breakages deliberately introduced. Postfix supervision: - The 13h outage was a boot-ordering race, not a Docker renumbering: postfix started at 08:09:49, wg0 came up at :51, postfix fataled at :52 on a missing docker_gwbridge address, and dockerd did not finish starting until 08:10:16. Stock postfix@.service is ordered only After=network-online.target and ships no Restart=, so one lost race became a permanent outage. - An ExecStartPre gate now blocks up to 60s until every inet_interfaces address actually exists, which absorbs the transient case inside a single start attempt. That makes bounded retry correct: 5 attempts in 600s, then failed — a genuinely broken config reaches a visible failed state in ~100s instead of re-fataling every 15s forever. - A 5-minute watchdog timer retries indefinitely and runs reset-failed, so "failed" still self-heals. Worst case is ~5 minutes, not 13 hours. - Wants=, not Requires=: a dockerd failure must not take down the loopback and mesh listeners that do not depend on Docker at all. Health checks must read config with `postconf -c`, never postmulti/postqueue/ postfix — those three RESOLVE inet_interfaces and so fatal precisely when an address is missing, which made the first version of this check report status=ok bound=0/0. A health check that fails open is worse than none. DEPLOY.md carries the monitoring contract, including that systemctl is-active postfix is a known-false signal: postfix.service is a wrapper whose ExecStart=/bin/true, so it reports active forever while the real postfix@- instance is failed with zero listeners. Reproduced live. Known gap, documented not fixed: Alloy ships Docker stdout, Caddy files and app JSONL, not journald — so no Postfix line reaches Loki and the mail rules cannot fire until loki.source.journal is added. Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY	2026-07-24 13:19:19 -07:00
Emi Griffith	f2fd8f6835	Subtree-merge thermograph-observability (origin/main) into observability/ ... git-subtree-dir: observability git-subtree-mainline: ae1d9bb534 git-subtree-split: 19e74af9ca	2026-07-22 22:01:11 -07:00