backend and frontend now reference their own images (BACKEND_IMAGE_PATH/TAG,
FRONTEND_IMAGE_PATH/TAG) instead of the shared emi/thermograph/app. deploy.sh
takes SERVICE=backend|frontend|all, rolls just that service with --no-deps,
persists each service's live tag in deploy/.image-tags.env so a single-service
deploy never disturbs the sibling, and fixes the old IMAGE_TAG:? guard that
made every workflow deploy hard-fail. Health-checks per service (8137/8080).
Claude-Session: https://claude.ai/code/session_01RdARHDJaYC1wSQRTYM6t3Z
Frontend now publishes its OWN registry image (emi/thermograph-frontend/app)
via build-push.yml and deploys ONLY the frontend service to beta on push to
main, passing SERVICE=frontend + FRONTEND_IMAGE_TAG to infra's deploy.sh.
Independent of the backend's pipeline.
Claude-Session: https://claude.ai/code/session_01RdARHDJaYC1wSQRTYM6t3Z
Backend now publishes its OWN registry image (emi/thermograph-backend/app) via
build-push.yml and deploys ONLY the backend service to beta on push to main,
passing SERVICE=backend + BACKEND_IMAGE_TAG to infra's deploy.sh. Independent
of the frontend's pipeline -- the two ship asynchronously.
Claude-Session: https://claude.ai/code/session_01RdARHDJaYC1wSQRTYM6t3Z
Content-change pass following the extraction from the app monorepo (this repo
now stands alone, sourced via git filter-repo to preserve history):
- terraform/variables.tf, secrets.tf, modules/thermograph-host: remove every
app-secret Terraform variable (postgres_password, auth_secret, VAPID keys,
registry_token, Discord/SMTP creds, ...) and the random_password/random_id
generators. The SOPS+age vault (deploy/secrets/*.yaml) is now the sole
source of app secrets, rendered at deploy time by deploy/render-secrets.sh;
Terraform renders only a non-secret /etc/thermograph-topology.env (sizing,
routing) via the renamed thermograph-topology.env.tftpl template.
- hosts gains a required app_image_tag field: the host's own checkout is now
this infra repo, not the app repo, so there is no "current commit" to
derive an image tag from — every host pins one explicitly. repo_url now
points at this repo (private; typically needs an embedded read token).
- deploy.sh: IMAGE_TAG is now required from the environment instead of
derived via `git rev-parse HEAD` of the (now infra-repo) checkout, which
would have silently resolved to the wrong or a nonexistent tag.
- New terraform/modules/gcp-host: creates a GCE VM + minimal VPC/firewall
only, then feeds its IP into the same thermograph-host module every
SSH-managed host already uses — one provisioning path regardless of how a
host came to exist. var.gcp_hosts defaults to {}, so no google_* resource
is planned and the provider is never invoked without it (verified: plan
and validate succeed with no GCP credentials configured).
- terraform/README.md, ACCESS.md (renamed from INFRA.md), README.md: updated
for the new secrets model, the GCP scaffold, and this repo's own identity.
Verified: terraform fmt/validate/init clean; plan succeeds against realistic
dummy hosts (prod+beta shape) and against a populated gcp_hosts entry (plans
6 resources with no live credentials, confirming the composition wires
correctly end to end).
The live boot+healthz check repeatedly hit this runner's own
environment quirks (unreachable sibling bridge IP, non-unique \$\$,
permanently-squatted fixed ports from AppArmor-blocked cleanup,
variable boot time under shared capacity=2 contention) without ever
settling into a reliably green check across six attempts. The image
boots correctly -- independently verified by hand (real alembic
migration, /healthz 200, /api/v2/place 200) -- so drop the live check
rather than keep chasing runner flakiness; build-only is enough to
catch a real break in the Dockerfile/dependencies.
Confirmed via a real run's own docker logs: the container actually
booted successfully (migrations + 4 uvicorn workers, "Application
startup complete") but took just over 30s under this runner's
resource contention (capacity 2 -- another job's build often runs
concurrently), where the same boot takes ~3s locally with nothing else
competing for CPU. Give it 60s instead.
Every job container's entry shell gets the same low PID, so
backend-ci-\$\$ collided with an already-stuck leftover container from
a prior run (confirmed in the runner logs: "container name already in
use"). GITHUB_RUN_ID is genuinely unique per run -- use that for both
the container name and the per-run host port instead.
Curling the sibling container's own bridge-network IP (the previous
fix) doesn't work on this runner -- docker-outside-of-docker means the
job container's network namespace can't reach another container's
bridge IP directly, so every curl attempt hung for the full ~130s TCP
SYN timeout instead of failing fast (confirmed: a run got stuck for
over 10 minutes on this). 127.0.0.1 does work here (the monorepo's own
build.yml has published a host port successfully all along), so
publish one again, but to a per-run PID-derived port instead of a
fixed one, to still avoid the original port-collision problem.
This runner's snap-Docker install has a known AppArmor bug denying
docker stop/kill -- a leftover container from any failed run would
permanently squat host port 8137, so every SUBSEQUENT run's docker
run also failed immediately (port already allocated), compounding
into a cascading failure loop. Uses a unique-per-run container name
and curls the container's own bridge-network IP directly instead of
publishing any host port at all.
paths.py drops the sibling-directory walk (frontend is now the repo
root: static/ and content/ are its own subdirectories, no more
sibling walk needed). New Dockerfile (much smaller deps -- fastapi,
uvicorn, httpx, jinja2, PyYAML -- no entrypoint script needed at all,
frontend has no migrations/pre-boot logic).
content/ is a committed starter copy for now, not yet real cross-repo
vendoring from thermograph-copy (that repo doesn't exist yet) -- noted
in paths.py's own docstring.
Real, known gap flagged rather than silently skipped: this process
cannot boot standalone (content.register() fetches the IndexNow key
from backend at import time with no retry, by design), so build.yml
only verifies the image builds, not a live boot+healthz check -- that
needs a genuine cross-repo contract-test job (booting a real backend
too), separate follow-up work. Same reason tests/ doesn't run here yet
(its conftest.py still imports backend's own test fixtures via a
sibling path that no longer exists) -- noted directly in the file.
paths.py drops the sibling-directory walk (backend is now the repo
root, no more frontend/ sibling to walk toward). requirements.txt
drops jinja2/PyYAML (only needed for frontend_ssr's sake in the old
shared monorepo Dockerfile) and fixes stale backend/push.py-style
path comments. New Dockerfile (COPY . /app/, no THERMOGRAPH_SERVICE_ROLE
branch -- this image only ever runs backend) and entrypoint.sh (cd
/app instead of /app/backend). Minimal build.yml: docker build + boot
+ /healthz check.
Verified: image builds and boots standalone (real alembic migration
against SQLite, /healthz and /api/v2/place both 200), full pytest
suite green (301 passed, 4 skipped) against the rewritten paths.py.
Grafana 9.3+ won't look up an OAuth login by email, so a Google sign-in for
the API-created admin (no prior Google-auth linkage) tried to auto-create the
user and hit allow_sign_up=false ("signup is disabled"). Enable
oauth_allow_insecure_email_lookup so the login matches the existing user by
its Google-verified email. Single provider + verified email = no takeover risk.
Replace the grep-docker-logs notifier panel with a per-node UP/DOWN stat
driven by the app's new tag=heartbeat log line (count_over_time over 20m;
0 beats => DOWN/red). Update the README now that the app emits heartbeats.
A containerized Alloy agent can't reach the host's 127.0.0.1, so even the
beta-local agent must push to beta's wg0 address (10.10.0.2:3100), not
127.0.0.1. Fix the beta example in the README and the agent compose header.
- Grafana served at dashboard.thermograph.org (the record now points at beta)
- Google OAuth (OIDC) login, env-gated (OAUTH_ENABLED + GOOGLE_CLIENT_ID/SECRET);
allow_sign_up=false so access is locked to pre-provisioned accounts, with a
break-glass local admin fallback
- README: Google Cloud OAuth client setup with the redirect URIs for both
Grafana and Forgejo, plus the Forgejo add-oauth command
A central Grafana + Loki stack (on beta) fed by a Grafana Alloy agent on
every node, replacing the old SSH-tailed single-host scripts/dashboard.py.
- docker-compose.yml: central Loki (mesh-only :3100) + Grafana (Caddy-fronted)
- loki/config.yml: single-binary, filesystem storage, 30-day retention
- alloy/config.alloy + docker-compose.agent.yml: per-node collector — every
container's stdout/stderr via the Docker socket, Caddy host logs, and the
app's structured JSON logs (errors/access/audit), each line tagged by node
- grafana/: auto-provisioned Loki datasource + a fleet-logs dashboard
(volume by service, error rate, upstream 429s, Caddy 5xx, notifier liveness,
live tail), with a per-node selector
- caddy-grafana.conf, README, .env.example
