Thermograph monorepo: graded-climate API + SSR frontend + infra, domain-specific containerized deploys
9c0f493d68
All checks were successful
PR build (required check) / changes (pull_request) Successful in 6s
secrets-guard / encrypted (pull_request) Successful in 5s
PR build (required check) / build-backend (pull_request) Has been skipped
PR build (required check) / build-frontend (pull_request) Has been skipped
shell-lint / shellcheck (pull_request) Successful in 6s
PR build (required check) / validate-observability (pull_request) Successful in 23s
PR build (required check) / gate (pull_request) Successful in 2s
thermograph-observability (and thermograph-backend/frontend/infra) are archived on Forgejo now that the monorepo cutover has landed, so this domain's work lands here in observability/ instead of a standalone repo PR. Alloy (observability/alloy/config.alloy), deployed from /opt/observability on prod and beta (confirmed via docker inspect mounts on the live hosts; the monorepo's own observability/ copy isn't what's running yet -- that's a separate, not-yet-done cutover step for this domain, flagged below): - Raise loki.source.file's file_watch poll interval (caddy + app_jsonl) from the 250ms/250ms PollingFileWatcher default to 2s/10s, and local.file_match's glob-rescan sync_period from 10s to 1m. - Raise discovery.docker's refresh_interval from 60s to 5m so Swarm task churn (~every 90s) stops restarting tailers on every refresh. - Drop noise/duplicate containers (alloy, autoscaler, thermograph-test_*, thermograph-lb, thermograph_worker) via a relabel rule; the worker's stdout is pure /healthz noise, already superseded by its own access/*.jsonl. - Fix loki.source.docker double-relabeling: it took both the already- relabelled target output AND relabel_rules, running every rule twice per entry and making the new drop rules a no-op on the second pass. Pass raw discovery targets + relabel_rules instead. - Add a loki.process stage for Caddy logs that drops well-known crawler/bot lines (semrushbot, claudebot, ahrefsbot, yandexbot, bytespider, mj12bot, petalbot) -- Caddy's own log_skip needs >= 2.7 and both hosts are older (beta confirmed 2.6.2; prod is actually 2.11.4, a version-skew worth addressing separately), so this is the shipper-side equivalent. Loki (observability/loki/config.yml): 30 low-rate streams were hitting the 30m chunk_idle_period long before chunk_target_size, flushing chunks 1.98% full on average. Raise chunk_idle_period to 2h, max_chunk_age to 12h, set chunk_target_size/chunk_encoding explicitly. Add explicit ingestion/stream limits sized for a three-node hobby fleet -- their absence was the source of the sporadic 429s (Loki was falling back to its stricter multi-tenant defaults). CI (.forgejo/workflows/observability-validate.yml): the Alloy config had no syntax coverage. Download the pinned v1.9.1 alloy binary (matches observability/alloy/docker-compose.agent.yml) straight from its GitHub release and run `alloy validate` (minus the one line covering a known gap where validate doesn't recognize the top-level livedebugging singleton block, unlike alloy run). Caddy (infra/terraform/modules/thermograph-host/templates/Caddyfile.tftpl, the true source for /etc/caddy/Caddyfile -- confirmed live on prod, whose deployed file's own header says "RENDERED BY TERRAFORM"; infra/deploy/Caddyfile is the documented manual-apply fallback, updated in step to avoid drift): - health_interval 5s -> 15s: a quarter of the active-healthcheck polling load for the same restart-safety guarantee. - A `filter` format log encoder deleting request>headers, request>tls, and resp_headers (the full-header/TLS-block serialization was the main driver of ~1,133B/line on prod, ~922B on beta), plus roll_size/roll_keep so the file itself doesn't grow unbounded between logrotate runs. - Strip the query string entirely from logged request URIs via a `regexp` field filter -- Caddy's default logger recorded request.uri *including* the query string, so every `?q=<search text>` sat in Loki next to the client IP for the full 30-day retention. Confirmed empirically (via `caddy validate` and a scratch `caddy run` against beta's real 2.6.2 binary) that keeping request>headers>User-Agent alongside a parent-level header delete isn't achievable in this Caddy version -- deleting the parent drops the whole subtree regardless of a child override, and `log_append` (which would let it be hoisted out first) needs a newer Caddy than beta runs. Traded away rather than block on it. Deferred (host-level, not in any repo, applied directly and separately): /etc/docker/daemon.json (10m/3-file json-file log caps) on both hosts -- config written, but NOT restarting dockerd on either: prod is the fleet's sole Swarm manager with Live Restore disabled, so a daemon restart there would both interrupt Swarm's control plane and actually stop/restart every container on the host (no live-restore to keep them up), a real production outage window, not a blip. Left for a planned maintenance window instead. A 14-day retention cron for the app's applogs JSONL volume is applied on both hosts (/etc/cron.d/thermograph-applogs-retention) -- plain find+delete rather than logrotate, since the app already self-dates one file per day per category and logrotate's rotate-in-place model doesn't fit files it doesn't own the naming of. Grafana itself (docker-compose.yml, grafana/, caddy-grafana.conf, dashboard.thermograph.org's Caddy block) is untouched by any of this. |
||
|---|---|---|
| .forgejo/workflows | ||
| backend | ||
| frontend | ||
| infra | ||
| observability | ||
| CLAUDE.md | ||
| CUTOVER-NOTES.md | ||
| README.md | ||
thermograph
The Thermograph monorepo — the split repos reunified (2026-07-22) with full history via subtree merges, while keeping everything the split was actually for: per-domain images, per-domain deploys, and an async FE/BE contract.
Domains
| Dir | What | CI |
|---|---|---|
backend/ |
FastAPI graded-climate API, accounts, notifications (Discord bot, push, mail), data pipeline | backend-build-push → image emi/thermograph/backend; backend-deploy[-prod|-dev] |
frontend/ |
Public client: static JS/CSS + SSR pages | frontend-* mirrors of the above; image emi/thermograph/frontend |
infra/ |
Compose, deploy scripts, terraform, SOPS secrets vault, ops cron | infra-sync (host checkout + secrets render), secrets-guard, ops-cron |
observability/ |
Loki + Grafana + Alloy stack | observability-validate |
thermograph-docs deliberately stays its own repo (ADRs + runbooks, no
build artifacts, different change cadence).
How CI stays decoupled
Every workflow in .forgejo/workflows/ is path-filtered to its domain: a
push touching only frontend/** builds/deploys nothing else. Images stay
separate (emi/thermograph/backend, emi/thermograph/frontend, each tagged
sha-<12hex>), deploys stay per-service (infra/deploy/deploy.sh SERVICE=backend|frontend|all), and the API version contract
(GET /api/version, PAYLOAD_VER) still lets FE and BE ship out of lockstep.
The one intentionally coupled piece is pr-build.yml: a single always-running
gate required check that builds only the domains a PR touches (a
path-filtered required check would deadlock auto-merge).
Branch model (unchanged from the split era): PRs → dev, main → beta,
release → prod; infra tracked via main on all hosts.
Before pointing anything live at this repo, read CUTOVER-NOTES.md.