A central Grafana + Loki stack (on beta) fed by a Grafana Alloy agent on
every node, replacing the old SSH-tailed single-host scripts/dashboard.py.
- docker-compose.yml: central Loki (mesh-only :3100) + Grafana (Caddy-fronted)
- loki/config.yml: single-binary, filesystem storage, 30-day retention
- alloy/config.alloy + docker-compose.agent.yml: per-node collector — every
container's stdout/stderr via the Docker socket, Caddy host logs, and the
app's structured JSON logs (errors/access/audit), each line tagged by node
- grafana/: auto-provisioned Loki datasource + a fleet-logs dashboard
(volume by service, error rate, upstream 429s, Caddy 5xx, notifier liveness,
live tail), with a per-node selector
- caddy-grafana.conf, README, .env.example
beta (Forgejo's pinned node) is also today's live thermograph.org host,
with Caddy already bound to ports 80/443. A second reverse proxy
binding those same ports would collide with it. Drop Traefik entirely:
forgejo's web port now publishes to 127.0.0.1:3080 only, and a new
Caddy site block (deploy/forgejo/caddy-git.conf) reverse-proxies
git.thermograph.org to it, reusing Caddy's existing automatic-HTTPS
instead of a second ACME flow.
That same Caddy block resolves the registry-exposure question (hazard
#15 in the hop1 runbook) as its first listed option: /v2/* (the
registry API) is blocked to everything outside the WireGuard mesh
CIDR, while the git/web UI stays public. README documents the
/etc/hosts override mesh clients need so registry traffic actually
routes over the tunnel rather than the public IP.
Bring the Swarm+Forgejo layer in line with the canonical topology in
docs/runbooks/implementation-handoff.md: three nodes (prod, beta, and
the desktop LAN dev machine) instead of two, with the Forgejo Actions
runner registered on the desktop as a plain systemd service rather than
a Swarm-scheduled Docker-in-Docker sidecar.
- setup-wireguard.sh: full N-peer mesh instead of point-to-point
- docker-stack.yml: drop the runner/runner-dind services, volumes, and
runner-token secret; only forgejo_db_password remains
- register-lan-runner.sh: register under both docker and
thermograph-lan labels, since one runner now covers both job types
- init-swarm.sh, join-swarm.sh, swarm/README.md, forgejo/README.md,
INFRA.md: updated node lists, order of operations, and access-state
table for three nodes
- deploy.yml: renamed to "Deploy to beta VPS" and reconciled the
main-vs-release branch question against terraform.tfvars.example
(this workflow already targets beta; a release-triggered prod
deploy doesn't exist yet and isn't invented here)
warm_cities.py and indexnow.py are one-shot scripts run manually at deploy
today - nothing keeps the curated city set topped up, or pings IndexNow,
between deploys.
Add notifications/scheduler.py: an in-process APScheduler instance driving
two recurring jobs (city warming, daily; IndexNow --if-changed, every 6h by
default), gated by the same leader election as the notifier - only the
process that already won leadership starts it, so the jobs run in exactly
one place under multi-host Swarm rather than once per replica. Neither job
runs immediately on start: warm_cities already runs at deploy time, so an
immediate duplicate on every worker boot/restart would be wasted work.
Extract indexnow.submit_if_changed() from the CLI's --if-changed branch so
the scheduler and the command line share the exact same skip-when-unchanged
logic instead of two copies drifting apart. The CLI's plain (non-flag) path
is unchanged.
APScheduler over a Postgres-native queue: exactly one process ever runs
this, no fan-out/backpressure/dead-letter need, no new infrastructure.
warm_cities.py and indexnow.py are one-shot scripts run manually at deploy
today - nothing keeps the curated city set topped up, or pings IndexNow,
between deploys.
Add notifications/scheduler.py: an in-process APScheduler instance driving
two recurring jobs (city warming, daily; IndexNow --if-changed, every 6h by
default), gated by the same leader election as the notifier - only the
process that already won leadership starts it, so the jobs run in exactly
one place under multi-host Swarm rather than once per replica. Neither job
runs immediately on start: warm_cities already runs at deploy time, so an
immediate duplicate on every worker boot/restart would be wasted work.
Extract indexnow.submit_if_changed() from the CLI's --if-changed branch so
the scheduler and the command line share the exact same skip-when-unchanged
logic instead of two copies drifting apart. The CLI's plain (non-flag) path
is unchanged.
APScheduler over a Postgres-native queue: exactly one process ever runs
this, no fan-out/backpressure/dead-letter need, no new infrastructure.
* Add a Postgres advisory-lock leader election for multi-host deploys
The subscription notifier elects one leader via a host-local flock
(THERMOGRAPH_SINGLETON_LOCK) so multiple uvicorn workers on one host don't
each run it. Under multi-host Swarm that guard is insufficient: each host
would independently elect its own leader, multiplying Open-Meteo quota use
N-fold again.
Add claim_pg(key) alongside the existing claim(lock_path): a cluster-wide
Postgres advisory lock, visible to every host talking to the same database.
The holding connection is dedicated and kept for the process lifetime
(advisory locks are session-scoped); a dead connection is dropped and
re-election retried on the next call.
claim_leader() dispatches between the two mechanisms from env:
THERMOGRAPH_SINGLETON_PG (+ Postgres) -> claim_pg; else
THERMOGRAPH_SINGLETON_LOCK -> claim; else always leader, unchanged. Wired in
at web/app.py in place of the direct claim() call. Off by default, so
today's single-host behavior is unaffected.
* Split web/worker duties with THERMOGRAPH_ROLE
Background work (the subscription notifier) is welded to the same process
that serves requests, so scaling the web tier to N replicas would also scale
notifier instances unless something restricts it further than leader
election alone.
Add THERMOGRAPH_ROLE (web|worker|all, default all - unchanged single-process
behavior). Every replica runs the same image; ROLE only gates whether a
process is allowed to own the notifier at all, layered on top of the
existing leader election: web replicas never start it even if they'd win
leader election, worker replicas start it if they win. The decision is
pulled into _should_run_notifier() so it's unit-testable without booting the
full app (DB init, places index, neighbor warmer).
Add a minimal /healthz liveness route (no DB/upstream I/O, not under BASE)
so a worker replica - which serves no real traffic - still has something
Swarm can health-check.
* Retrigger CI (no prior check run was ever recorded for this PR)
* Add a Postgres advisory-lock leader election for multi-host deploys
The subscription notifier elects one leader via a host-local flock
(THERMOGRAPH_SINGLETON_LOCK) so multiple uvicorn workers on one host don't
each run it. Under multi-host Swarm that guard is insufficient: each host
would independently elect its own leader, multiplying Open-Meteo quota use
N-fold again.
Add claim_pg(key) alongside the existing claim(lock_path): a cluster-wide
Postgres advisory lock, visible to every host talking to the same database.
The holding connection is dedicated and kept for the process lifetime
(advisory locks are session-scoped); a dead connection is dropped and
re-election retried on the next call.
claim_leader() dispatches between the two mechanisms from env:
THERMOGRAPH_SINGLETON_PG (+ Postgres) -> claim_pg; else
THERMOGRAPH_SINGLETON_LOCK -> claim; else always leader, unchanged. Wired in
at web/app.py in place of the direct claim() call. Off by default, so
today's single-host behavior is unaffected.
* Split web/worker duties with THERMOGRAPH_ROLE
Background work (the subscription notifier) is welded to the same process
that serves requests, so scaling the web tier to N replicas would also scale
notifier instances unless something restricts it further than leader
election alone.
Add THERMOGRAPH_ROLE (web|worker|all, default all - unchanged single-process
behavior). Every replica runs the same image; ROLE only gates whether a
process is allowed to own the notifier at all, layered on top of the
existing leader election: web replicas never start it even if they'd win
leader election, worker replicas start it if they win. The decision is
pulled into _should_run_notifier() so it's unit-testable without booting the
full app (DB init, places index, neighbor warmer).
Add a minimal /healthz liveness route (no DB/upstream I/O, not under BASE)
so a worker replica - which serves no real traffic - still has something
Swarm can health-check.
* Retrigger CI (no prior check run was ever recorded for this PR)
The weather-terms glossary (9 entries) and four static pages' SEO title/
description were hardcoded directly in web/content.py - a 1019-line module
that also owns all SSR rendering logic - mixed in with code that changes on
a completely different cadence and for different reasons.
Add content/glossary.yaml and content/pages.yaml (a new repo-root content/
tree, a sibling of backend/ and frontend/ paths.py resolves the same way -
the seed of a future thermograph-copy repo per the architecture decision
doc's own §4) plus web/content_loader.py: a small loader that validates each
file's shape at load time (required fields present and non-empty, no
duplicate glossary slugs) and fails loudly on a malformed edit rather than
rendering a blank glossary card or an empty <title>. content.py's GLOSSARY
dict and the about/privacy/hub/glossary_index page_title/description
literals now come from the loader.
Scope: only content that is genuinely pure static data with no embedded
template logic. cities_flavor.json (Wikipedia extracts, already its own
generated file) and the homepage's title/description (embedded in
home.html.j2 as Jinja block overrides, a heavily-tested product-critical
template) are deliberately left as they are - a future pass, not required
for this one. UI microcopy bound to frontend logic stays in frontend/,
per the doc's own line between content and frontend.
Verified: content/glossary.yaml generated programmatically from the live
GLOSSARY dict (not hand-transcribed) and round-tripped byte-for-byte
identical against it; content/pages.yaml's four entries checked field-by-
field against the original hardcoded strings. Full backend suite green (362
passed, 4 skipped) with zero existing test changes needed beyond one
assertion made escaping-aware (a pre-existing Jinja double-escape quirk on
the one title containing "&", intentionally preserved not fixed). Built
and booted the real Docker image: content/ present at /app/content, and
curled /glossary, /glossary/percentine, /about, /privacy from inside the
running container - all four render with the exact expected title text.
The weather-terms glossary (9 entries) and four static pages' SEO title/
description were hardcoded directly in web/content.py - a 1019-line module
that also owns all SSR rendering logic - mixed in with code that changes on
a completely different cadence and for different reasons.
Add content/glossary.yaml and content/pages.yaml (a new repo-root content/
tree, a sibling of backend/ and frontend/ paths.py resolves the same way -
the seed of a future thermograph-copy repo per the architecture decision
doc's own §4) plus web/content_loader.py: a small loader that validates each
file's shape at load time (required fields present and non-empty, no
duplicate glossary slugs) and fails loudly on a malformed edit rather than
rendering a blank glossary card or an empty <title>. content.py's GLOSSARY
dict and the about/privacy/hub/glossary_index page_title/description
literals now come from the loader.
Scope: only content that is genuinely pure static data with no embedded
template logic. cities_flavor.json (Wikipedia extracts, already its own
generated file) and the homepage's title/description (embedded in
home.html.j2 as Jinja block overrides, a heavily-tested product-critical
template) are deliberately left as they are - a future pass, not required
for this one. UI microcopy bound to frontend logic stays in frontend/,
per the doc's own line between content and frontend.
Verified: content/glossary.yaml generated programmatically from the live
GLOSSARY dict (not hand-transcribed) and round-tripped byte-for-byte
identical against it; content/pages.yaml's four entries checked field-by-
field against the original hardcoded strings. Full backend suite green (362
passed, 4 skipped) with zero existing test changes needed beyond one
assertion made escaping-aware (a pre-existing Jinja double-escape quirk on
the one title containing "&", intentionally preserved not fixed). Built
and booted the real Docker image: content/ present at /app/content, and
curled /glossary, /glossary/percentine, /about, /privacy from inside the
running container - all four render with the exact expected title text.
Terraform generates the secrets that have no external meaning
(POSTGRES_PASSWORD, AUTH_SECRET, METRICS_TOKEN, INDEXNOW_KEY) via the random
provider instead of requiring the operator to hand-generate and paste each
into terraform.tfvars. Each is pinned with a static keepers value (secrets.tf)
so apply never regenerates a value already in use - the exact incident class
this guards against: every session invalidated, the app<->DB password
mismatched. Rotation is now a deliberate keepers edit, never a side effect.
postgres_password/auth_secret move from required inputs to optional (default
"") - explicit var wins when supplied (seeding an EXISTING live secret during
a migration onto Terraform, hop-1 cutover runbook Stage 0), else Terraform
generates and owns it. metrics_token/indexnow_key are new: neither existed in
Terraform before, both previously left for the app's own fallback generation.
VAPID deliberately stays a required, non-generated input - an EC keypair
where regeneration breaks every existing push subscription outright, unlike
an opaque token.
Sizing tiers: a locals.sizes t-shirt map (nano/small/medium/large ->
{workers, app_cpus, db_cpus, db_memory}), toward the target Proxmox
sizing-tier model (architecture doc SS6) ahead of actually provisioning VMs -
Proxmox itself stays deferred; today a tier just sizes container caps on the
existing SSH-managed hosts. A host can reference one by name (hosts.<name>.
size) or keep hand-picking the four fields, so existing tfvars are
unaffected; prod's example now uses size = "large" (identical numbers),
beta keeps explicit numbers, and a commented uat example demonstrates the
shortcut for a future ephemeral host.
Strengthened terraform/README.md's local-state caveat: more Terraform-
generated secrets landing in tfstate raises the stakes of the existing
never-commit-cleartext-state guidance, not just the sizing.
Verified: terraform validate + fmt clean. A real `terraform plan` against
fake hosts (prod/beta/uat, mixing size="large"/explicit-numbers/size="nano")
resolved every sizing correctly (prod 8/8/4/16g, beta 4/4/2/8g, uat
1/1/1/1g) and planned exactly one instance of each random_password/random_id
resource. Applied just those four resources (real generation, -target to
avoid touching the fake SSH-only host resources) and re-planned: "No
changes" - confirming the keepers pinning holds. Adding an explicit
postgres_password override afterward left the random_password resource
itself completely untouched (0 replace/destroy), confirming the override
path never disturbs the generated resource.
The Swarm/Forgejo standup (a parallel infra track, see INFRA.md) landed real
.forgejo/workflows/{build,pr-build,deploy,deploy-dev}.yml before this PR
merged, making ci.yml a redundant duplicate of their build.yml (same build
gate, same job). Drop it.
Fix the remaining two files to match the real, already-deployed conventions
those files established rather than the guesses this PR shipped with:
runs-on: [self-hosted, thermograph] -> docker (the actual Docker-in-Docker
Swarm-hosted runner label), and appleboy/ssh-action referenced by full GitHub
URL (confirmed not mirrored on this Forgejo instance's default action
registry, per deploy.yml's own header comment) rather than the short form.
actions/checkout needed no change - already confirmed to resolve unchanged
from Forgejo's default mirror.
build-push.yml (image registry push) and ops-cron.yml (backup + IndexNow)
stay: neither duplicates anything in the real mirror, which faithfully
replicates the OLD git-checkout-and-build-in-place deploy model rather than
the registry-based one, and has no scheduled ops jobs at all.
Records the concrete, live state of Track 1 rather than just the setup
procedure: both boxes' roles/IPs, the ssh invocation, where the private key
lives (~/.ssh/thermograph_agent_ed25519, same convention DEPLOY.md already
uses for the CI deploy key), and how to rotate it. Also notes that neither
box has Docker yet and Terraform hasn't been applied to either, which is why
Track 2 is currently paused.
Checks off the two verification-checklist items that are actually confirmed
now (SSH access, password auth dead) rather than leaving them unchecked.
Three workflows for the self-hosted Forgejo instance (Track A chunk 7 of the
infra-design implementation handoff), mirroring/extending the existing
.github/workflows without touching them - GitHub stays the primary repo and
its own build.yml/ci-cd.yml keep gating auto-merge.
ci.yml mirrors build.yml's build gate (deps, backend tests, frontend JS
syntax check, boot + page/API health check) on a Forgejo runner.
build-push.yml builds the app image and pushes it to Forgejo's built-in
registry, tagged by git SHA (every push) and semver (version tags) - the
registry half of the hop-1 cutover's build-once/deploy-everywhere model.
Runs alongside the existing deploy.yml/deploy.sh (git-checkout-and-build-in-
place) without touching it, per the cutover runbook's Stage C.
ops-cron.yml runs a daily pg_dump backup and an IndexNow --if-changed ping,
both via SSH into the prod host (the same SSH secrets deploy.yml already
uses) using the app's already-running compose stack - no new network
exposure, no separate dependency install. These are ops/infra concerns
(scheduled via Forgejo Actions cron), distinct from the app-domain jobs the
worker's own APScheduler runs.
All three need a runner registered with the `thermograph` label and the
registry's public/mesh exposure resolved (Track B).
Verified: raw YAML syntax valid; actionlint clean except the pre-existing,
already-present-on-.github/workflows "unknown custom runner label" warning
(not something these files introduce - the existing thermograph-lan label
triggers the identical warning). One real finding fixed pre-commit: an
unused shellcheck-flagged loop variable in the boot-check retry loop.
* Split web/worker duties with THERMOGRAPH_ROLE
Background work (the subscription notifier) is welded to the same process
that serves requests, so scaling the web tier to N replicas would also scale
notifier instances unless something restricts it further than leader
election alone.
Add THERMOGRAPH_ROLE (web|worker|all, default all - unchanged single-process
behavior). Every replica runs the same image; ROLE only gates whether a
process is allowed to own the notifier at all, layered on top of the
existing leader election: web replicas never start it even if they'd win
leader election, worker replicas start it if they win. The decision is
pulled into _should_run_notifier() so it's unit-testable without booting the
full app (DB init, places index, neighbor warmer).
Add a minimal /healthz liveness route (no DB/upstream I/O, not under BASE)
so a worker replica - which serves no real traffic - still has something
Swarm can health-check.
* Add the Swarm interim stack file, a pinnable TimescaleDB tag, and a Caddy health-gate
Three changes toward the hop-1 interim cutover, all inert until Track B
stands up the platform:
docker-stack.yml: the Swarm stack file for the interim cutover, distinct
from docker-compose.yml (today's plain-compose deploy, unaffected). Pulls a
pre-built image (IMAGE_TAG) instead of building in place; app/worker publish
no host port (127.0.0.1:8137:8137 has no Swarm equivalent - Swarm's routing
mesh publishes on 0.0.0.0, which would expose the plaintext app un-fronted),
reaching Caddy only over an MTU-lowered overlay network (VXLAN-over-WireGuard
needs a smaller MTU or large payloads silently stall); db is placement-
pinned to a labelled node; app/worker skip inline migrations
(RUN_MIGRATIONS=0) so the runbook's one-shot migrate task is the only thing
that ever runs Alembic; secrets are real Swarm secrets mounted at
/run/secrets, read by the entrypoint shim rather than plain env vars.
TIMESCALEDB_TAG: docker-compose.yml's db image now reads this (default
latest-pg18, today's behavior unchanged), wired through Terraform
(timescaledb_tag, default "latest-pg18") so it can actually be pinned to an
exact minor without hand-editing the host - required before any host of the
stack could replicate with another (a floating tag risks mismatched
extension minors, which blocks a physical replica and risks compressed-
chunk corruption on restore).
Caddy active health-gate: both the Terraform-rendered Caddyfile and the live
deploy/Caddyfile now health-check the app on the same cheap /healthz route
its own Docker HEALTHCHECK uses (now /healthz instead of the SSR homepage,
so it's cheap enough for a tight interval and works identically for a
worker replica, which serves no public traffic at all) - Caddy won't
forward into a container that's still booting or unhealthy.
Verified live: built and booted the real image via docker compose - both
containers report healthy via the new /healthz-based HEALTHCHECK, and GET /
still renders the full SSR homepage unchanged. Both Caddyfiles validated
with the real caddy binary. docker-stack.yml validated with docker compose
config (required-var guards fire with clear messages; secrets correctly
mount at /run/secrets/<name>, matching the entrypoint shim's mapping).
docker-compose.yml validated with and without TIMESCALEDB_TAG set, alongside
the existing openmeteo overlay. terraform validate + fmt clean.
* Split web/worker duties with THERMOGRAPH_ROLE
Background work (the subscription notifier) is welded to the same process
that serves requests, so scaling the web tier to N replicas would also scale
notifier instances unless something restricts it further than leader
election alone.
Add THERMOGRAPH_ROLE (web|worker|all, default all - unchanged single-process
behavior). Every replica runs the same image; ROLE only gates whether a
process is allowed to own the notifier at all, layered on top of the
existing leader election: web replicas never start it even if they'd win
leader election, worker replicas start it if they win. The decision is
pulled into _should_run_notifier() so it's unit-testable without booting the
full app (DB init, places index, neighbor warmer).
Add a minimal /healthz liveness route (no DB/upstream I/O, not under BASE)
so a worker replica - which serves no real traffic - still has something
Swarm can health-check.
* Add the Swarm interim stack file, a pinnable TimescaleDB tag, and a Caddy health-gate
Three changes toward the hop-1 interim cutover, all inert until Track B
stands up the platform:
docker-stack.yml: the Swarm stack file for the interim cutover, distinct
from docker-compose.yml (today's plain-compose deploy, unaffected). Pulls a
pre-built image (IMAGE_TAG) instead of building in place; app/worker publish
no host port (127.0.0.1:8137:8137 has no Swarm equivalent - Swarm's routing
mesh publishes on 0.0.0.0, which would expose the plaintext app un-fronted),
reaching Caddy only over an MTU-lowered overlay network (VXLAN-over-WireGuard
needs a smaller MTU or large payloads silently stall); db is placement-
pinned to a labelled node; app/worker skip inline migrations
(RUN_MIGRATIONS=0) so the runbook's one-shot migrate task is the only thing
that ever runs Alembic; secrets are real Swarm secrets mounted at
/run/secrets, read by the entrypoint shim rather than plain env vars.
TIMESCALEDB_TAG: docker-compose.yml's db image now reads this (default
latest-pg18, today's behavior unchanged), wired through Terraform
(timescaledb_tag, default "latest-pg18") so it can actually be pinned to an
exact minor without hand-editing the host - required before any host of the
stack could replicate with another (a floating tag risks mismatched
extension minors, which blocks a physical replica and risks compressed-
chunk corruption on restore).
Caddy active health-gate: both the Terraform-rendered Caddyfile and the live
deploy/Caddyfile now health-check the app on the same cheap /healthz route
its own Docker HEALTHCHECK uses (now /healthz instead of the SSR homepage,
so it's cheap enough for a tight interval and works identically for a
worker replica, which serves no public traffic at all) - Caddy won't
forward into a container that's still booting or unhealthy.
Verified live: built and booted the real image via docker compose - both
containers report healthy via the new /healthz-based HEALTHCHECK, and GET /
still renders the full SSR homepage unchanged. Both Caddyfiles validated
with the real caddy binary. docker-stack.yml validated with docker compose
config (required-var guards fire with clear messages; secrets correctly
mount at /run/secrets/<name>, matching the entrypoint shim's mapping).
docker-compose.yml validated with and without TIMESCALEDB_TAG set, alongside
the existing openmeteo overlay. terraform validate + fmt clean.
Three additive infrastructure layers on top of the two VPS boxes Terraform
already provisions (prod: new 48 GB/12-core box, thermograph.org; beta: old
VPS, 75.119.132.91). None of this touches backend/, Dockerfile,
docker-compose*.yml, terraform/, or deploy/db/ — that stays owned by the
app-containerization work in flight elsewhere; this is strictly the layer on
top. See INFRA.md for the full runbook and order of operations.
- deploy/provision-agent-access.sh: a dedicated, auditable full-sudo login
(not raw root) for agent-driven ops — passwordless sudo under a distinct
username, sshd hardened to key-only auth, auditd logging every
root-effective command. One line to revoke.
- deploy/swarm/: a 2-node Swarm (prod=manager, beta=worker) joined over a
WireGuard tunnel rather than trusting the public internet with the
overlay data plane, which Docker's own guidance says should never face it
directly. Swarm ports locked to the tunnel interface once joined. This
cluster's only workload is Forgejo — it does not orchestrate the
Terraform-managed app deploys, so nothing here can strand the app's
single-writer database.
- deploy/forgejo/ + .forgejo/workflows/: Forgejo + Traefik + a
Docker-in-Docker-sandboxed runner as a Swarm stack pinned to beta, plus
Forgejo Actions workflows mirroring .github/workflows/*.yml. The custom
auto-merge workflow step is dropped — it existed only to work around
GitHub's paywalled branch protection on private free-tier repos, which
Forgejo has no such tier for; native "auto merge when checks succeed"
replaces it, and as a real git push (unlike GitHub's non-triggering
token-merge) it fires the LAN deploy naturally with no double-trigger
logic needed. appleboy/ssh-action is referenced by full URL (not mirrored
on Forgejo's default action registry); actions/checkout and
actions/setup-python resolve unchanged.
Migration is mirror-first: the GitHub repo import and workflow files land
here, but cutting deploy secrets over and retiring GitHub happens only after
verification (INFRA.md 3d) — GitHub stays live as a fallback throughout.
One flagged, unresolved mismatch: deploy.yml still triggers on `main`, but
terraform.tfvars.example names prod's deploy branch `release`. Left as a
faithful mirror rather than guessed at — reconcile with whoever's driving
Terraform/deploy.
homepage.json lives on the appdata volume today, written by whichever
process's notifier last refreshed it. Under Swarm each web replica has its
own disk, so a file only one replica's notifier ever writes leaves every
other replica reading a stale or missing feed indefinitely.
On Postgres, refresh()/load() go through store.py's existing derived-payload
table instead (a fixed sentinel key, since the feed isn't cell-scoped) - a
single upsert is already atomic, and every replica reads the same row.
store.IS_POSTGRES is False without THERMOGRAPH_DATABASE_URL, so dev/tests
keep the plain file path entirely unchanged; no existing test needed to
change.
Verified live against real Postgres: refresh() persists into the shared
derived table and load() reads the exact same feed back.
Two changes to deploy/entrypoint.sh, both needed before the app can run
under Docker Swarm without re-triggering known incidents:
One-shot migrate mode: `entrypoint.sh migrate` (or THERMOGRAPH_MIGRATE_ONLY=1)
runs the Alembic migration and exits, for a Swarm one-shot task that brings
the schema to head once rather than racing it across every web replica's
boot. RUN_MIGRATIONS=0 skips the inline migrate for a deploy that runs the
one-shot task separately. Default (unset) keeps today's behavior unchanged:
every boot migrates itself before serving.
Secrets shim: the app reads config from os.environ everywhere (an unset
THERMOGRAPH_AUTH_SECRET falls back to a random per-process value; an unset
VAPID key pair is freshly minted) - Swarm `secrets:` mount each secret as a
FILE under /run/secrets/ instead, which the app would never see. Before
alembic/uvicorn, mechanically export each /run/secrets/<name> whose
uppercased name is a THERMOGRAPH_* var, unless already set in the process
env; THERMOGRAPH_DATABASE_URL is built from a postgres_password secret the
same way compose's own interpolation builds it today.
Verified against fake alembic/uvicorn binaries in a throwaway container:
default boot, migrate-only mode (both trigger forms), RUN_MIGRATIONS=0,
secrets populating unset vars, an existing env var beating its secret file,
and no /run/secrets present at all (today's plain compose path, unaffected).
shellcheck clean.
The subscription notifier elects one leader via a host-local flock
(THERMOGRAPH_SINGLETON_LOCK) so multiple uvicorn workers on one host don't
each run it. Under multi-host Swarm that guard is insufficient: each host
would independently elect its own leader, multiplying Open-Meteo quota use
N-fold again.
Add claim_pg(key) alongside the existing claim(lock_path): a cluster-wide
Postgres advisory lock, visible to every host talking to the same database.
The holding connection is dedicated and kept for the process lifetime
(advisory locks are session-scoped); a dead connection is dropped and
re-election retried on the next call.
claim_leader() dispatches between the two mechanisms from env:
THERMOGRAPH_SINGLETON_PG (+ Postgres) -> claim_pg; else
THERMOGRAPH_SINGLETON_LOCK -> claim; else always leader, unchanged. Wired in
at web/app.py in place of the direct claim() call. Off by default, so
today's single-host behavior is unaffected.
The subscription notifier elects one leader via a host-local flock
(THERMOGRAPH_SINGLETON_LOCK) so multiple uvicorn workers on one host don't
each run it. Under multi-host Swarm that guard is insufficient: each host
would independently elect its own leader, multiplying Open-Meteo quota use
N-fold again.
Add claim_pg(key) alongside the existing claim(lock_path): a cluster-wide
Postgres advisory lock, visible to every host talking to the same database.
The holding connection is dedicated and kept for the process lifetime
(advisory locks are session-scoped); a dead connection is dropped and
re-election retried on the next call.
claim_leader() dispatches between the two mechanisms from env:
THERMOGRAPH_SINGLETON_PG (+ Postgres) -> claim_pg; else
THERMOGRAPH_SINGLETON_LOCK -> claim; else always leader, unchanged. Wired in
at web/app.py in place of the direct claim() call. Off by default, so
today's single-host behavior is unaffected.
Replace the per-cell parquet cache with TimescaleDB hypertables as the
production backend for the raw daily climate record, and drop pg_duckdb.
Parquet stays the backend whenever THERMOGRAPH_DATABASE_URL is not a
Postgres URL (dev, tests, offline tooling), the same dialect switch the
accounts DB and derived store already use, so CI stays Postgres-free.
- data/climate_store.py: psycopg + polars bridge over climate_history
(a hypertable), climate_recent, and climate_sync (per-cell freshness).
Reads via pl.read_database, writes via COPY + ON CONFLICT upsert;
fail-soft to a cache miss so a DB hiccup degrades to upstream refetch.
- data/climate.py: route every cache/mtime touchpoint through a backend
dispatch. recent_stamp becomes int(recent_synced_at) on Postgres; the
stale-serve path still avoids bumping it, so derived-payload tokens
invalidate on exactly the same events as before.
- alembic 0002: CREATE EXTENSION timescaledb plus the hypertable schema
(compression policy on year-old chunks), guarded to no-op off Postgres.
- migrate_cache_to_pg.py (make migrate-cache): idempotent backfill of the
parquet cache into the hypertables, preserving file mtimes as sync
timestamps so recent_stamp is unchanged across cutover.
- db image -> stock timescale/timescaledb:latest-pg18; drop the custom
pg_duckdb Dockerfile, the read-only /parquet mount, and the duckdb
tuning GUC. Docs updated for the new backend and cutover.
Co-authored-by: Claude <noreply@anthropic.com>
Replace the per-cell parquet cache with TimescaleDB hypertables as the
production backend for the raw daily climate record, and drop pg_duckdb.
Parquet stays the backend whenever THERMOGRAPH_DATABASE_URL is not a
Postgres URL (dev, tests, offline tooling), the same dialect switch the
accounts DB and derived store already use, so CI stays Postgres-free.
- data/climate_store.py: psycopg + polars bridge over climate_history
(a hypertable), climate_recent, and climate_sync (per-cell freshness).
Reads via pl.read_database, writes via COPY + ON CONFLICT upsert;
fail-soft to a cache miss so a DB hiccup degrades to upstream refetch.
- data/climate.py: route every cache/mtime touchpoint through a backend
dispatch. recent_stamp becomes int(recent_synced_at) on Postgres; the
stale-serve path still avoids bumping it, so derived-payload tokens
invalidate on exactly the same events as before.
- alembic 0002: CREATE EXTENSION timescaledb plus the hypertable schema
(compression policy on year-old chunks), guarded to no-op off Postgres.
- migrate_cache_to_pg.py (make migrate-cache): idempotent backfill of the
parquet cache into the hypertables, preserving file mtimes as sync
timestamps so recent_stamp is unchanged across cutover.
- db image -> stock timescale/timescaledb:latest-pg18; drop the custom
pg_duckdb Dockerfile, the read-only /parquet mount, and the duckdb
tuning GUC. Docs updated for the new backend and cutover.
Co-authored-by: Claude <noreply@anthropic.com>
Two prod-readiness hardening changes:
DB tuning scales with the container budget. Replace the fixed 8 GB
20-tuning.sql with 20-tuning.sh, which derives shared_buffers (25%),
effective_cache_size (75%), work_mem, maintenance_work_mem and
duckdb.max_memory (50%) from the DB_MEMORY the compose db service now passes in.
The ratios reproduce the historical 8 GB tuning exactly and scale linearly, so
the 48 GB prod box (db_memory 16g) gets shared_buffers 4 GB / duckdb 8 GB with
no separate edit. Beta/local (8g default) are unchanged. Docs that told
operators to raise the tuning by hand are updated.
Boot ordering for the self-hosted archive. On an openmeteo host, install a
docker.service drop-in (Wants/After rclone-om.service) so Docker starts after
the object-storage mount is ready on every boot — the restart-policy containers
never bind an empty mount point. rclone-om is Type=notify, so After waits for
the mount to actually be ready. Cleaned up when openmeteo is toggled off.
A self-hosted Open-Meteo instance that isn't backfilled yet answers historical
requests with all-null values (or only its recent sync window), which
_finalize_frame reduces to a near-empty frame. That frame is not None, so the
loader would accept it as source=open-meteo and cache it indefinitely as a
complete record — never falling back to NASA and never self-healing after the
backfill lands. Require at least MIN_ARCHIVE_DAYS before accepting an archive
result; below that, fall through to the NASA backup and don't cache the short
frame. Guards the prod cutover window.
Get the 45-year historical record off the rate-limited public Open-Meteo
archive API by running a private Open-Meteo instance that serves the
era5_seamless blend (0.1° ERA5-Land + 0.25° ERA5 for gusts) from the
compressed .om archive in object storage, mounted on the host with rclone.
- climate.py: make ARCHIVE_URL env-driven (THERMOGRAPH_ARCHIVE_URL) and pin
models=era5_seamless on the archive fetches only, so a self-hosted instance
serves the same 0.1° resolution; forecast path unchanged. The public API's
default is already seamless, so dev/beta (URL unset) behave identically.
- docker-compose.openmeteo.yml: open-meteo-api + two rolling sync workers
(era5_land 0.1°, era5 0.25° for gusts), bind-mounting the object-storage
mount; the overlay points the app at the local instance.
- Makefile: om-up / om-down / om-backfill (one-time full-history backfill).
- Terraform: per-host openmeteo flag layers the overlay, renders OM_DATA_DIR,
and provisions the host rclone systemd mount from the bucket credentials.
- deploy/openmeteo: operator runbook + rclone mount unit template.
Get the 45-year historical record off the rate-limited public Open-Meteo
archive API by running a private Open-Meteo instance that serves the
era5_seamless blend (0.1° ERA5-Land + 0.25° ERA5 for gusts) from the
compressed .om archive in object storage, mounted on the host with rclone.
- climate.py: make ARCHIVE_URL env-driven (THERMOGRAPH_ARCHIVE_URL) and pin
models=era5_seamless on the archive fetches only, so a self-hosted instance
serves the same 0.1° resolution; forecast path unchanged. The public API's
default is already seamless, so dev/beta (URL unset) behave identically.
- docker-compose.openmeteo.yml: open-meteo-api + two rolling sync workers
(era5_land 0.1°, era5 0.25° for gusts), bind-mounting the object-storage
mount; the overlay points the app at the local instance.
- Makefile: om-up / om-down / om-backfill (one-time full-history backfill).
- Terraform: per-host openmeteo flag layers the overlay, renders OM_DATA_DIR,
and provisions the host rclone systemd mount from the bucket credentials.
- deploy/openmeteo: operator runbook + rclone mount unit template.
Terraform config under terraform/ manages the two existing VPS hosts and hands the
app to docker-compose, with local state:
- prod: the new 48GB/12-core VPS (release branch, thermograph.org), sized larger.
- beta: the old VPS 75.119.132.91 (main branch, testing tier), no public domain.
- The LAN dev box stays on deploy/deploy-dev.sh (dev branch) — out of Terraform.
A reusable module (modules/thermograph-host) SSHes each host to install docker/
compose/ufw (+ Caddy when a domain is set), sync the checkout to the host's branch,
render /etc/thermograph.env from Terraform variables (secrets pushed via provisioner
content, never on local disk), `docker compose up -d`, and health-check. Named
volumes are preserved on re-apply, so the Postgres data is never recreated.
Container resources are now env-driven in docker-compose.yml (APP_CPUS/DB_CPUS/
DB_MEMORY/WORKERS) with unchanged defaults, so Terraform can size each host.
deploy/db/init/20-tuning.sql sets the memory budget via ALTER SYSTEM (applied on a
fresh volume, effective after the post-init restart): shared_buffers 2GB,
effective_cache_size 6GB, work_mem 64MB, maintenance_work_mem 512MB, max_wal_size
4GB, plus duckdb.max_memory 4GB for parquet processing.
Compose: the db container gets an 8g memory ceiling + 1g shm_size (parallel-query
shared memory) in prod; the dev overlay resets mem_limit so dev memory is uncapped
too (the ~8 GB budget still comes from the tuning). Verified the settings take
effect and pg_duckdb still loads.
Run Thermograph as a docker-compose stack (app + Postgres 18) and standardize the
data layer on Postgres, while keeping the test suite on SQLite.
- accounts/db.py: DSN-driven engines. On Postgres, a per-worker read-write +
read-only asyncpg pair (the RO engine pins read-only transactions, used by the
pure-GET endpoints) plus a sync psycopg engine for the notifier thread; the
SQLite path is preserved for tests/local (selected when THERMOGRAPH_DATABASE_URL
is unset). models.py: boolean server_default -> sa.false().
- store.py / metrics.py: dialect-flexible — Postgres UNLOGGED tables via psycopg
when configured, else the existing raw-sqlite3 paths byte-for-byte; sync
interfaces and every fail-soft contract preserved.
- Alembic (backend/alembic/) manages the accounts schema; the container entrypoint
runs `alembic upgrade head` before uvicorn (4 workers). migrate_accounts_to_pg.py
copies the accounts data SQLite->PG through the ORM (UUID/bool/JSON coerced),
skips access_token, and resets identity sequences.
- Dockerfile + docker-compose.yml: app image (uvicorn, 4 workers, loopback 8137)
and a Postgres 18 db (2 CPUs) running pg_duckdb (deploy/db/) so the parquet
climate cache is queryable in-DB via read_parquet('/parquet/cache/*.parquet').
- deploy.sh/thermograph.service rewired to manage the compose stack; env example,
Makefile targets (up/down/db-up), and deploy/POSTGRES-MIGRATION.md cutover runbook.
Tests stay on SQLite (dialect fallback) — 323 pass. The full Postgres stack was
verified via docker compose: alembic migrations, register/login, the RO endpoint,
store/metrics round-trips, and the accounts data migration.
Run Thermograph as a docker-compose stack (app + Postgres 18) and standardize the
data layer on Postgres, while keeping the test suite on SQLite.
- accounts/db.py: DSN-driven engines. On Postgres, a per-worker read-write +
read-only asyncpg pair (the RO engine pins read-only transactions, used by the
pure-GET endpoints) plus a sync psycopg engine for the notifier thread; the
SQLite path is preserved for tests/local (selected when THERMOGRAPH_DATABASE_URL
is unset). models.py: boolean server_default -> sa.false().
- store.py / metrics.py: dialect-flexible — Postgres UNLOGGED tables via psycopg
when configured, else the existing raw-sqlite3 paths byte-for-byte; sync
interfaces and every fail-soft contract preserved.
- Alembic (backend/alembic/) manages the accounts schema; the container entrypoint
runs `alembic upgrade head` before uvicorn (4 workers). migrate_accounts_to_pg.py
copies the accounts data SQLite->PG through the ORM (UUID/bool/JSON coerced),
skips access_token, and resets identity sequences.
- Dockerfile + docker-compose.yml: app image (uvicorn, 4 workers, loopback 8137)
and a Postgres 18 db (2 CPUs) running pg_duckdb (deploy/db/) so the parquet
climate cache is queryable in-DB via read_parquet('/parquet/cache/*.parquet').
- deploy.sh/thermograph.service rewired to manage the compose stack; env example,
Makefile targets (up/down/db-up), and deploy/POSTGRES-MIGRATION.md cutover runbook.
Tests stay on SQLite (dialect fallback) — 323 pass. The full Postgres stack was
verified via docker compose: alembic migrations, register/login, the RO endpoint,
store/metrics round-trips, and the accounts data migration.
The deploy scripts pulled code and restarted but never applied the
hand-written SQL migrations in deploy/migrations/, so column additions to
the long-lived accounts DB had to be run by hand.
Add deploy/migrate-db.py, an idempotent runner that resolves the accounts
DB the same way the app does (THERMOGRAPH_ACCOUNTS_DB or
<repo>/data/accounts.sqlite), applies any files not yet recorded in a
schema_migrations table, and:
- backs the DB up before touching it;
- baselines a fresh DB (no file, or no user table) rather than ALTERing
a table create_all() will build with the current schema on next start;
- tolerates an already-present column/index (hand-applied or model-built)
by recording it instead of failing.
Wire it into deploy.sh (prod) and deploy-dev.sh (dev) just before the
service restarts, so new code never queries a column an older DB lacks.
No env or systemd changes: the migration runs as the deploy user, who owns
the data dir.
Claude-Session: https://claude.ai/code/session_01XXxmNFy9cZ6Gh8Y9thZn62
The overall climate-shift score was a weighted roll-up (feels-like and humidity
2x, wind and gusts 0.5x, and so on). Make it an equal average instead: every
metric — and, through each metric's own mean over the 10/25/50/75/90 percentiles,
every percentile — counts the same.
- scoring._overall: plain mean of the present metrics' divergences, mapped to
0-100, replacing the weighted sum. Missing metrics still drop out cleanly.
- Drop the now-unused per-metric weights from the METRICS table and the scored
entries (nothing rendered them; only the roll-up read them).
- views.SCORE_VER s4 -> s5 so cached weighted scores are recomputed.
- score.js: the hero note now reads "Every metric counts equally" instead of
"Temperature, feels-like and humidity are weighted most".
Verified against Seattle: overall mad equals the equal mean of the eight metric
mads (7.9), and the entries no longer carry a weight.
Claude-Session: https://claude.ai/code/session_013dRZmX9D3JEntfMKWMTWZ8
The overall climate-shift score was a weighted roll-up (feels-like and humidity
2x, wind and gusts 0.5x, and so on). Make it an equal average instead: every
metric — and, through each metric's own mean over the 10/25/50/75/90 percentiles,
every percentile — counts the same.
- scoring._overall: plain mean of the present metrics' divergences, mapped to
0-100, replacing the weighted sum. Missing metrics still drop out cleanly.
- Drop the now-unused per-metric weights from the METRICS table and the scored
entries (nothing rendered them; only the roll-up read them).
- views.SCORE_VER s4 -> s5 so cached weighted scores are recomputed.
- score.js: the hero note now reads "Every metric counts equally" instead of
"Temperature, feels-like and humidity are weighted most".
Verified against Seattle: overall mad equals the equal mean of the eight metric
mads (7.9), and the entries no longer carry a weight.
Claude-Session: https://claude.ai/code/session_013dRZmX9D3JEntfMKWMTWZ8
* Centralize filesystem paths in a single module
Add paths.py, which resolves the repo root once and derives the cache,
accounts DB, logs, templates, frontend and bundled-city-data locations
from it. Replace the 13 per-module `dirname(__file__)/..` anchors with
references to it, so a module's location no longer determines where the
app reads its data. Env overrides (accounts DB, VAPID, IndexNow) are
unchanged; every resolved path is byte-identical to before.
Groundwork for moving modules into packages without re-pointing paths.
Claude-Session: https://claude.ai/code/session_01XXxmNFy9cZ6Gh8Y9thZn62
* Split the backend into domain packages
Group the flat backend modules into packages that mirror their concerns:
data/ climate, grading, scoring, grid, places, cities,
city_events, store
web/ app, views, homepage, content, schemas
notifications/ notify, digest, push, mailer, discord,
discord_interactions, discord_link
accounts/ models, users, api_accounts, db
core/ metrics, singleton, audit
Intra-project imports are rewritten to the package-qualified form. The
entry scripts (indexnow, warm_cities, migrate, gen_cities, gen_flavor)
and paths.py stay at the backend/ root, and backend/app.py becomes a
shim re-exporting web.app:app so the launch target stays `app:app` —
run.sh, the systemd units, and CI need no change.
Verified: full suite (318) passes, `uvicorn app:app` boots and serves
the home/SEO/static/API surfaces, and every root script imports clean.
Claude-Session: https://claude.ai/code/session_01XXxmNFy9cZ6Gh8Y9thZn62
Rain intensity gains a Severe tier and Dry becomes strictly no-rain:
- The top half of Very Heavy (95th–99th rain-day percentile) becomes a new
"Severe" tier; Very Heavy keeps the 90–95 band. Eight rain tiers now — Trace /
Light / Brisk / Typical / Heavy / Very Heavy / Severe / Extreme — which refill
the wet-2..wet-9 colour ramp contiguously (no gap), so Heavy/Very Heavy shift
one shade lighter and Severe takes the second-darkest.
- A day is Dry only when it didn't rain at all; any measurable rain, however
slight, is at least Trace. _grade_precip splits on > 0 rather than the 0.01"
threshold (which still governs the separate dry-streak metric).
- The distribution strip drops the range under the Dry column — every dry day is
zero, so a "0–0" span was noise.
_precip_ladder derives its percentile marks from RAIN_BANDS now, so adding or
splitting a tier can't leave a hard-coded list behind (that was the bug the 95th
mark would have hit). The detail-view ladder derives from the band table as before.
Claude-Session: https://claude.ai/code/session_013dRZmX9D3JEntfMKWMTWZ8
Rain intensity gains a Severe tier and Dry becomes strictly no-rain:
- The top half of Very Heavy (95th–99th rain-day percentile) becomes a new
"Severe" tier; Very Heavy keeps the 90–95 band. Eight rain tiers now — Trace /
Light / Brisk / Typical / Heavy / Very Heavy / Severe / Extreme — which refill
the wet-2..wet-9 colour ramp contiguously (no gap), so Heavy/Very Heavy shift
one shade lighter and Severe takes the second-darkest.
- A day is Dry only when it didn't rain at all; any measurable rain, however
slight, is at least Trace. _grade_precip splits on > 0 rather than the 0.01"
threshold (which still governs the separate dry-streak metric).
- The distribution strip drops the range under the Dry column — every dry day is
zero, so a "0–0" span was noise.
_precip_ladder derives its percentile marks from RAIN_BANDS now, so adding or
splitting a tier can't leave a hard-coded list behind (that was the bug the 95th
mark would have hit). The detail-view ladder derives from the band table as before.
Claude-Session: https://claude.ai/code/session_013dRZmX9D3JEntfMKWMTWZ8
Move the flat backend/tests/*.py into domain subfolders so the suite
mirrors the code's concerns:
data/ climate, grading, scoring, grid, places, store
web/ api, content, homepage, views
notifications/ notify, digest, discord (+ dm/interactions/link)
accounts/ api_accounts
core/ metrics, singleton, dashboard
conftest.py stays at the tests/ root, so its sys.path setup and shared
fixtures still apply to every subfolder. The four tests that derive repo
paths from __file__ get their depth bumped one level to match their new
location. Pytest discovers the subfolders recursively; the CI command
(python -m pytest backend/tests) is unchanged.
Claude-Session: https://claude.ai/code/session_01XXxmNFy9cZ6Gh8Y9thZn62
The rain-intensity scale drops its two hyphenated compound labels for single
words and loses a tier, going from eight to seven:
Trace / Light / Brisk / Typical / Heavy / Very Heavy / Extreme
- Light–Mod -> Brisk, Moderate -> Typical (renames only; classes unchanged).
- Mod–Heavy is merged up into Heavy, whose floor drops from the 75th to the 60th
rain-day percentile, so Heavy now spans 60–90.
- The lightest tier (already the merged Very Light) is renamed Trace.
grading.py RAIN_BANDS and the frontend SCALE_RAIN mirror stay in lockstep, and
the detail-view ladder derives from the table so it follows automatically. The
now-unreferenced wet-6 colour token is kept so any day still cached under that
class renders until the derived store recomputes; the chart's percentile fan also
keeps it for a smooth gradient.
Claude-Session: https://claude.ai/code/session_013dRZmX9D3JEntfMKWMTWZ8
The rain-intensity scale drops its two hyphenated compound labels for single
words and loses a tier, going from eight to seven:
Trace / Light / Brisk / Typical / Heavy / Very Heavy / Extreme
- Light–Mod -> Brisk, Moderate -> Typical (renames only; classes unchanged).
- Mod–Heavy is merged up into Heavy, whose floor drops from the 75th to the 60th
rain-day percentile, so Heavy now spans 60–90.
- The lightest tier (already the merged Very Light) is renamed Trace.
grading.py RAIN_BANDS and the frontend SCALE_RAIN mirror stay in lockstep, and
the detail-view ladder derives from the table so it follows automatically. The
now-unreferenced wet-6 colour token is kept so any day still cached under that
class renders until the derived store recomputes; the chart's percentile fan also
keeps it for a smooth gradient.
Claude-Session: https://claude.ai/code/session_013dRZmX9D3JEntfMKWMTWZ8
The account menu offered 'Link Discord' to every signed-in user even on a
server with no Discord OAuth app configured, where it dead-ends (the start route
303-bounces to /alerts). Add GET /api/v2/discord/config reporting whether linking
is enabled, and have the menu render the entry only when it is. On a server
without Discord set up nothing surfaces; setting the OAuth env vars later makes
the entry appear on its own — no code change needed to turn it on.
The account menu offered 'Link Discord' to every signed-in user even on a
server with no Discord OAuth app configured, where it dead-ends (the start route
303-bounces to /alerts). Add GET /api/v2/discord/config reporting whether linking
is enabled, and have the menu render the entry only when it is. On a server
without Discord set up nothing surfaces; setting the OAuth env vars later makes
the entry appear on its own — no code change needed to turn it on.
