6 commits

Author	SHA1	Message	Date
Emi Griffith	53e2eb7c84	registry: move the registry host to dev.jinemi.com, MCP to mcp.jinemi.com ... Completes the Forgejo domain migration. ROOT_URL moved to dev.jinemi.com earlier; the registry half was deliberately deferred. Every image name, REGISTRY_HOST default, runner label and --add-host pin now names dev.jinemi.com, so the registry host and the bearer-token realm agree again. Both names address the same Forgejo, so no image needs re-pushing and a rollback to a tag pushed under the old prefix still resolves. git.thermograph.org therefore stays served off the same Caddy site block -- one block, so the /v2/* mesh-only matcher keeps covering both names -- for pre-migration tags and for runners holding it as their registered instance URL. Mesh clients now pin both names in /etc/hosts: the new one as registry host and token realm, the old one for pre-migration tags. runner-vps2/config.yaml carries both --add-host entries for the same reason. Also renames Centralis' endpoint to mcp.jinemi.com in the two places this repo names it; Centralis itself is provisioned outside this repo. Host-side steps this cannot do (documented in deploy/forgejo/README.md, "Host-side steps"): the Forgejo Actions variable REGISTRY_HOST, docker login against the new host, and the /etc/hosts pins.	2026-08-01 16:06:22 -07:00
Emi Griffith	af21d8e477	docs: correct file references and the dev reachability claim ... Audited the five CLAUDE.md files and all twenty-one README.md files against the tree, machine-checking every in-repo path they name and verifying the testable claims against the live hosts. The one that matters is in the root file: dev was documented as reachable on the mesh at 10.10.0.2:8137. It is not, and never was from anywhere but vps1 — infra/docker-compose.yml binds the port to 127.0.0.1, and the address answers from neither vps2 nor vps1 itself. Anyone following it gets a connection refused with nothing to explain it. The rest are stale paths, several from the reunification: * assetlinks.json moved under frontend/static/ in the subtree merge; the TWA README kept the pre-merge path in both places it names it. Following it would put the file where nothing serves it and Android app-link verification would fail silently. * push.py and notify.py now live in backend/notifications/. * INFRA.md and deploy/stack/README have never existed in this repo, in any branch. * the Caddyfile is at deploy/stack/lb/Caddyfile. * three bare relative paths that resolve for a reader but not from the directory the file sits in: units.js is the frontend's, deploy.sh is infra's, entrypoint.sh is the backend's. Also records why mesh clients must pin the ROOT_URL host and not only the image host: the registry's bearer-token realm follows ROOT_URL, so pinning git.thermograph.org alone still sends the token request out the public route, where the /v2/* matcher returns 403 and docker falls back to anonymous. That surfaces as `unauthorized: reqPackageAccess`, indistinguishable from a bad credential. Verified true and left alone: the four-domain layout, both .claude runbooks, the absence of any domain-level .forgejo directory, the pinned compose project name, the deploy contract, prod's eight stack services, beta's five prefixed ones with no db of its own, dev's five, and every documented make target.	2026-08-01 11:49:28 -07:00
Emi Griffith	df409f88b3	registry: move image and repo references to the Jinemi namespace ... Repos moved to the Jinemi org; the container packages did not follow, since Forgejo does not transfer packages with a repo. The deploy path still resolved `emi/thermograph/*` — a user_redirect to admin_emi — while build-push.yml derives its push path from ${github.repository}, now jinemi/thermograph. The next backend or frontend build would have published somewhere no deploy looks. Point the image paths at jinemi/thermograph/* (lowercase: OCI references admit no uppercase, which is why build-push.yml already pipes through tr), the clone URLs at Jinemi/thermograph, and the registry logins at admin_emi — the account that actually owns the tokens, rather than the redirect. The live tags and both ci-runner tags were copied into the Jinemi namespace first, so the switch has something to pull. thermograph-infra, thermograph-observability and the retired */app packages stay under admin_emi; they did not move.	2026-08-01 09:25:02 -07:00
emi	d138f00a20	infra: split the estate into vps1/vps2 — beta joins prod, dev gets a home (#103)	2026-07-26 06:56:38 +00:00
Emi Griffith	e4693dce58	observability: add the estate's first alerting; supervise Postfix ... The estate had zero alert rules. The only contact point was Grafana's factory default pointing at the literal string <example@email.com>, and beta's untracked compose override routed Grafana's SMTP at prod's Postfix — so alerts, had any existed, would have been delivered by the box most likely to be on fire. Prod served 86 5xx in 24h including five /healthz failures and nobody was told. Alerting (deploys to beta, which is where Grafana runs): - 12 Loki-based rules. There is no Prometheus in this estate and Loki is the only datasource, so every rule is log-derived. - Thresholds come from a 24h backtest that happens to contain a real ~20min prod outage at 04:20Z. Absolute counts, not ratios: prod runs ~7 req/min, so one bad request is 1.4% and a ratio alert would scream all night. The 5xx burst rule fires on the outage's 45 and 22 buckets and on nothing else in the day; the largest benign bucket all day was 5. - Routed to a new private #ops-alerts channel, not to any existing channel — #weather-events, #announcements and #prod are product surfaces that notify real subscribers. - AlertingWatchdog is a dead-man's switch; its value is its absence. - Delivery is proven, not assumed: the rules were provisioned into a throwaway Grafana against beta's live Loki and a real alert arrived in Discord. This matters because GET /api/v1/provisioning/contact-points returns [REDACTED] for the URL — a contact point holding an uninterpolated env var looks perfectly healthy and pages nobody. The only proof is a message arriving. CI gains a structural check, because the existing one only proves YAML parses: an alert rule whose condition names a missing refId is valid YAML, provisions cleanly, and never fires. It also hard-fails on a literal Discord webhook in the repo. Verified against all three breakages deliberately introduced. Postfix supervision: - The 13h outage was a boot-ordering race, not a Docker renumbering: postfix started at 08:09:49, wg0 came up at :51, postfix fataled at :52 on a missing docker_gwbridge address, and dockerd did not finish starting until 08:10:16. Stock postfix@.service is ordered only After=network-online.target and ships no Restart=, so one lost race became a permanent outage. - An ExecStartPre gate now blocks up to 60s until every inet_interfaces address actually exists, which absorbs the transient case inside a single start attempt. That makes bounded retry correct: 5 attempts in 600s, then failed — a genuinely broken config reaches a visible failed state in ~100s instead of re-fataling every 15s forever. - A 5-minute watchdog timer retries indefinitely and runs reset-failed, so "failed" still self-heals. Worst case is ~5 minutes, not 13 hours. - Wants=, not Requires=: a dockerd failure must not take down the loopback and mesh listeners that do not depend on Docker at all. Health checks must read config with `postconf -c`, never postmulti/postqueue/ postfix — those three RESOLVE inet_interfaces and so fatal precisely when an address is missing, which made the first version of this check report status=ok bound=0/0. A health check that fails open is worse than none. DEPLOY.md carries the monitoring contract, including that systemctl is-active postfix is a known-false signal: postfix.service is a wrapper whose ExecStart=/bin/true, so it reports active forever while the real postfix@- instance is failed with zero listeners. Reproduced live. Known gap, documented not fixed: Alloy ships Docker stdout, Caddy files and app JSONL, not journald — so no Postfix line reaches Loki and the mail rules cannot fire until loki.source.journal is added. Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY	2026-07-24 13:19:19 -07:00
Emi Griffith	f2fd8f6835	Subtree-merge thermograph-observability (origin/main) into observability/ ... git-subtree-dir: observability git-subtree-mainline: ae1d9bb534 git-subtree-split: 19e74af9ca	2026-07-22 22:01:11 -07:00