190 lines
10 KiB
Markdown
190 lines
10 KiB
Markdown
|
# 3. Repo map
|
||
|
|
|
||
|
|
The monorepo was reunified on **2026-07-22** by subtree-merging four split
|
||
|
|
repos with full history (380 commits). The split's *properties* were kept
|
||
|
|
deliberately: per-domain images, per-domain deploys, and an async FE/BE
|
||
|
|
contract. That's why the layout looks like four projects sharing a root.
|
||
|
|
|
||
|
|
```
|
||
|
|
thermograph/
|
||
|
|
├── CLAUDE.md root agent instructions — branch model, deploy contract
|
||
|
|
├── CUTOVER-NOTES.md what is and isn't live; the reunification record
|
||
|
|
├── README.md domain table + how CI stays decoupled
|
||
|
|
├── .forgejo/workflows/ ALL CI. Nine files. Never add workflows elsewhere.
|
||
|
|
├── .claude/ settings.json + enforcement hooks
|
||
|
|
├── backend/ FastAPI API + Go daemon → emi/thermograph/backend
|
||
|
|
├── frontend/ Go SSR + static assets → emi/thermograph/frontend
|
||
|
|
├── infra/ compose, Swarm, deploy, secrets, terraform, ops
|
||
|
|
├── observability/ Loki + Grafana + Alloy
|
||
|
|
└── docs/onboarding/ you are here
|
||
|
|
```
|
||
|
|
|
||
|
|
> **CI lives only in root `.forgejo/workflows/`**, path-filtered per domain.
|
||
|
|
> Forgejo only reads root workflows — a copy under a domain's own `.forgejo/`
|
||
|
|
> is inert, and becomes a trap when someone edits it expecting an effect.
|
||
|
|
|
||
|
|
## `backend/` — the API service
|
||
|
|
|
||
|
|
Python 3.12 / FastAPI. Owns climate data, grading, the API, accounts,
|
||
|
|
notifications, and the SSR-content JSON API the frontend renders from. Serves
|
||
|
|
**no** HTML/CSS/JS.
|
||
|
|
|
||
|
|
```
|
||
|
|
app.py one-line re-export shim for web/app.py — keeps `app:app`
|
||
|
|
stable for systemd/CI/entrypoint. Don't change the name.
|
||
|
|
paths.py every filesystem location, resolved from the repo root.
|
||
|
|
NEVER reintroduce __file__-relative paths in a module.
|
||
|
|
web/app.py (1022) the real FastAPI app: routes, CORS, ETag, middleware,
|
||
|
|
lifespan, role gating, the frontend proxy fallback
|
||
|
|
api/
|
||
|
|
payloads.py (342) pure inputs → response dict builders. One definition per
|
||
|
|
payload shape; PAYLOAD_VER lives here.
|
||
|
|
content_payloads.py the SSR content API's builders (JSON-safe, no Markup)
|
||
|
|
content_routes.py route wiring for /content/* + the same ETag pattern
|
||
|
|
internal_routes.py /internal/* — the Go daemon's control surface. Fails
|
||
|
|
closed: no token configured ⇒ the whole surface 404s.
|
||
|
|
homepage.py (367) the precomputed "unusual right now" feed
|
||
|
|
sitemap.py
|
||
|
|
data/
|
||
|
|
climate.py (1248) fetch + cache the daily record. The source ladder lives
|
||
|
|
here. The single biggest file in the repo.
|
||
|
|
grading.py (532) day-of-year climatology, percentiles, TEMP/RAIN_BANDS
|
||
|
|
grid.py (92) lat/lon → stable ~4 sq mi cell id
|
||
|
|
store.py (329) derived-payload store (SQLite/Postgres). Pure accelerator.
|
||
|
|
climate_store.py(353) raw climate record on TimescaleDB hypertables
|
||
|
|
era5lake.py (173) the ERA5 object-storage lake: layout, grid math, clients
|
||
|
|
scoring.py (293) climate-shift scoring (recent years vs full baseline)
|
||
|
|
places.py (355) typo-tolerant local place index for /suggest
|
||
|
|
cities.py, city_events.py, meteostat.py
|
||
|
|
core/
|
||
|
|
metrics.py (805) since-start traffic counters, multi-worker via SQLite
|
||
|
|
audit.py (167) JSONL audit / errors / access / activity / heartbeat
|
||
|
|
singleton.py (129) leader election: flock (per host) or PG advisory lock
|
||
|
|
(per cluster). Read this before adding any timer.
|
||
|
|
accounts/ fastapi-users: models, schemas, db, users, api_accounts
|
||
|
|
notifications/
|
||
|
|
notify.py (542) the subscription evaluation engine (background thread)
|
||
|
|
push.py, mailer.py, digest.py, discord.py, discord_link.py,
|
||
|
|
discord_interactions.py
|
||
|
|
daemon/ the Go daemon — Discord gateway + cron timers
|
||
|
|
alembic/ Postgres migrations (run by the entrypoint on boot)
|
||
|
|
deploy/entrypoint.sh container entrypoint: alembic migrate, then serve
|
||
|
|
lake_app.py (204) the prod-only `lake` service (DuckDB over the bucket)
|
||
|
|
tests/ the hermetic pytest suite
|
||
|
|
cities.json, cities_flavor.json bundled reference data (generated)
|
||
|
|
```
|
||
|
|
|
||
|
|
Standalone scripts at the root are tooling, not runtime: `warm_cities.py`,
|
||
|
|
`indexnow.py`, `seed_era5.py`, `gen_era5_lake.py`, `gen_cities.py`,
|
||
|
|
`gen_flavor.py`, `drift_check.py`, `migrate*.py`.
|
||
|
|
|
||
|
|
Deep dive: [04-backend.md](04-backend.md).
|
||
|
|
|
||
|
|
## `frontend/` — SSR + static assets
|
||
|
|
|
||
|
|
```
|
||
|
|
server/ ★ THE LIVE SERVICE (Go 1.26, module thermograph/frontend)
|
||
|
|
main.go config + mux + static + graceful shutdown
|
||
|
|
internal/config/ every env var, with the same names/defaults as before
|
||
|
|
internal/contentapi/ backend /content/* client: TTL cache, bounded LRU,
|
||
|
|
per-key single-flight, origin forwarding
|
||
|
|
internal/content/ page handlers, funcmap, SEO
|
||
|
|
internal/contentdata/ glossary.yaml + pages.yaml loader (fail-loud)
|
||
|
|
internal/format/ unit-aware °C/°F formatting + band names
|
||
|
|
internal/handlers/ routing, SPA shells, static
|
||
|
|
internal/render/ html/template over embed.FS + ETag helpers
|
||
|
|
internal/render/templates/*.tmpl the embedded page templates
|
||
|
|
static/ every static asset — app.js, cache.js, shared.js,
|
||
|
|
account.js, calendar/day/score/compare .js + .html
|
||
|
|
shells, style.css (all design tokens), icons, sw.js
|
||
|
|
content/ glossary.yaml, pages.yaml — structured SSR copy
|
||
|
|
templates/*.html.j2 Jinja templates — used by the PYTHON service only
|
||
|
|
tests/ Python tiers: unit (hermetic, fixtures) + integration
|
||
|
|
tests/fixtures/*.json golden payloads — ALSO consumed by the Go tests
|
||
|
|
tools/shoot.py the `make shots` screenshot sweep
|
||
|
|
DESIGN.md the visual system: tokens, components, breakpoints
|
||
|
|
app.py, content.py, ⚠ the superseded Python implementation — not deployed
|
||
|
|
api_client.py, (see traps)
|
||
|
|
format.py, paths.py,
|
||
|
|
content_loader.py
|
||
|
|
```
|
||
|
|
|
||
|
|
Deep dive: [05-frontend.md](05-frontend.md).
|
||
|
|
|
||
|
|
## `infra/` — how and where images run
|
||
|
|
|
||
|
|
```
|
||
|
|
docker-compose.yml the compose stack (beta + LAN dev):
|
||
|
|
db, backend, lake, daemon, frontend.
|
||
|
|
`name: thermograph` is PINNED — see below.
|
||
|
|
docker-compose.dev.yml LAN overlay: uncapped, backend on 0.0.0.0:8137
|
||
|
|
docker-compose.openmeteo.yml self-hosted Open-Meteo overlay (prod-only)
|
||
|
|
Makefile compose orchestration only
|
||
|
|
deploy/
|
||
|
|
deploy.sh ★ the single entry point for beta and prod
|
||
|
|
render-secrets.sh renders /etc/thermograph.env from the SOPS vault
|
||
|
|
secrets/ the vault: common.yaml, prod/beta/dev.yaml,
|
||
|
|
centralis.prod.yaml, example.yaml
|
||
|
|
stack/ the SWARM path (live on prod):
|
||
|
|
thermograph-stack.yml, deploy-stack.sh,
|
||
|
|
autoscale.sh, the LB
|
||
|
|
swarm/, forgejo/ mesh + Forgejo/CI-runner provisioning
|
||
|
|
Caddyfile the reverse proxy (path-splits FE vs BE)
|
||
|
|
provision-*.sh host bootstrap: agent access, dev LAN, mail, secrets
|
||
|
|
db/init/ TimescaleDB init + tuning
|
||
|
|
migrations/ hand-run SQL migrations
|
||
|
|
ops/
|
||
|
|
dbq.sh read-only psql into any environment's db container
|
||
|
|
iceberg.sh duckdb over the ERA5 Iceberg lake
|
||
|
|
terraform/ host provisioning — NO STATE IS PERSISTED
|
||
|
|
ACCESS.md, DEPLOY.md, DEPLOY-DEV.md (partly stale — see traps)
|
||
|
|
```
|
||
|
|
|
||
|
|
**Why the project name is pinned.** Compose creates volumes
|
||
|
|
`thermograph_pgdata` / `_appdata` / `_applogs`, and the Swarm stack declares
|
||
|
|
those exact names as `external: true` at the same mount paths. Running compose
|
||
|
|
from `infra/` without `name: thermograph` derives project `infra`, which makes a
|
||
|
|
whole new stack with empty volumes next to the running one. `deploy-dev.sh`
|
||
|
|
exports `COMPOSE_PROJECT_NAME=thermograph-dev` (env wins over the file key) to
|
||
|
|
keep LAN dev separate on purpose. Keep both halves.
|
||
|
|
|
||
|
|
Deep dive: [08-infra-secrets.md](08-infra-secrets.md).
|
||
|
|
|
||
|
|
## `observability/` — the logging stack
|
||
|
|
|
||
|
|
```
|
||
|
|
docker-compose.yml Loki + Grafana (runs on beta)
|
||
|
|
loki/config.yml mesh-only, filesystem storage, 30-day retention
|
||
|
|
grafana/provisioning/ datasource + dashboard provider (auto-loaded)
|
||
|
|
grafana/provisioning/alerting/ rules, the Discord contact point, the policy
|
||
|
|
grafana/dashboards/*.json the fleet-logs dashboard
|
||
|
|
alloy/config.alloy the per-node shipper
|
||
|
|
alloy/docker-compose.agent.yml runs Alloy on a node (ALLOY_NODE per host)
|
||
|
|
caddy-grafana.conf beta's Grafana vhost, kept for reference
|
||
|
|
```
|
||
|
|
|
||
|
|
No build, no deploy automation — it ships by hand. **The repo is the only
|
||
|
|
durable path**: Grafana is provisioned from this directory at startup, so UI
|
||
|
|
edits are overwritten. Grafana is at **`dashboard.thermograph.org`** (Google
|
||
|
|
SSO, pre-provisioned users only) — never `grafana.thermograph.org`.
|
||
|
|
|
||
|
|
Deep dive: [09-observability.md](09-observability.md).
|
||
|
|
|
||
|
|
## The nine CI workflows
|
||
|
|
|
||
|
|
| Workflow | Trigger | Does |
|
||
|
|
|---|---|---|
|
||
|
|
| `pr-build.yml` | PR → `dev`/`main` | The `gate` required check. Diffs the PR, builds only touched domains. Deliberately **not** path-filtered. |
|
||
|
|
| `build.yml` | `workflow_call` | Build one domain's image; run the backend suite **inside** the built image. |
|
||
|
|
| `build-push.yml` | push to `dev`/`main`/`release` touching an app domain, or a `v*.*.*` tag | Builds + pushes `sha-<12hex>` images. A version tag builds **both**. |
|
||
|
|
| `deploy.yml` | push to `main`/`release` touching an app domain | Branch selects environment; matrix covers services; SSHes and runs `deploy.sh`. |
|
||
|
|
| `infra-sync.yml` | push to `main` touching `infra/**` | Fast-forwards each host's checkout and re-renders secrets. Rolls **no** service. |
|
||
|
|
| `observability-validate.yml` | push touching `observability/**`, or `workflow_call` | Parses every artifact; validates Alloy config with the pinned binary; strict alerting checks. |
|
||
|
|
| `ops-cron.yml` | daily 03:00 UTC | **THE prod backup** (pg_dump) + IndexNow. Uses `PROD_SSH_*`. |
|
||
|
|
| `secrets-guard.yml` | every PR and push | Fails if any `infra/deploy/secrets/*.yaml` isn't SOPS-encrypted. |
|
||
|
|
| `shell-lint.yml` | every PR and push | shellcheck (pinned v0.11.0 + sha256) over every `*.sh`. |
|
||
|
|
|
||
|
|
Deep dive: [07-ci-and-release.md](07-ci-and-release.md).
|
||
|
|
|
||
|
|
Next: [Backend deep dive](04-backend.md).
|